Oct 30 19:12:59.801795 ip-10-0-132-82 systemd[1]: kubelet.service: Failed to load environment files: No such file or directory
Oct 30 19:12:59.801808 ip-10-0-132-82 systemd[1]: kubelet.service: Failed to run 'start-pre' task: No such file or directory
Oct 30 19:12:59.801816 ip-10-0-132-82 systemd[1]: kubelet.service: Failed with result 'resources'.
Oct 30 19:12:59.802118 ip-10-0-132-82 systemd[1]: Failed to start Kubernetes Kubelet.
Oct 30 19:13:09.981959 ip-10-0-132-82 systemd[1]: kubelet.service: Failed to schedule restart job: Unit crio.service not found.
Oct 30 19:13:09.981980 ip-10-0-132-82 systemd[1]: kubelet.service: Failed with result 'resources'.
-- Boot d31c3a64bf374f8ca6a4b569507d4ade --
Oct 30 19:15:08.173256 ip-10-0-132-82 systemd[1]: Starting Kubernetes Kubelet...
Oct 30 19:15:08.526094 ip-10-0-132-82 kubenswrapper[2577]: Flag --container-runtime-endpoint has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Oct 30 19:15:08.526094 ip-10-0-132-82 kubenswrapper[2577]: Flag --minimum-container-ttl-duration has been deprecated, Use --eviction-hard or --eviction-soft instead. Will be removed in a future version.
Oct 30 19:15:08.526094 ip-10-0-132-82 kubenswrapper[2577]: Flag --volume-plugin-dir has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Oct 30 19:15:08.526094 ip-10-0-132-82 kubenswrapper[2577]: Flag --system-reserved has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Oct 30 19:15:08.532884 ip-10-0-132-82 kubenswrapper[2577]: W1030 19:15:08.532068    2577 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Oct 30 19:15:08.532884 ip-10-0-132-82 kubenswrapper[2577]: W1030 19:15:08.532455    2577 feature_gate.go:386] unrecognized feature gate: EventTTL
Oct 30 19:15:08.532884 ip-10-0-132-82 kubenswrapper[2577]: W1030 19:15:08.532462    2577 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Oct 30 19:15:08.532884 ip-10-0-132-82 kubenswrapper[2577]: W1030 19:15:08.532466    2577 feature_gate.go:386] unrecognized feature gate: NewOLM
Oct 30 19:15:08.532884 ip-10-0-132-82 kubenswrapper[2577]: W1030 19:15:08.532471    2577 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Oct 30 19:15:08.532884 ip-10-0-132-82 kubenswrapper[2577]: W1030 19:15:08.532475    2577 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Oct 30 19:15:08.532884 ip-10-0-132-82 kubenswrapper[2577]: W1030 19:15:08.532480    2577 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Oct 30 19:15:08.532884 ip-10-0-132-82 kubenswrapper[2577]: W1030 19:15:08.532491    2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Oct 30 19:15:08.532884 ip-10-0-132-82 kubenswrapper[2577]: W1030 19:15:08.532495    2577 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Oct 30 19:15:08.532884 ip-10-0-132-82 kubenswrapper[2577]: W1030 19:15:08.532499    2577 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Oct 30 19:15:08.532884 ip-10-0-132-82 kubenswrapper[2577]: W1030 19:15:08.532503    2577 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Oct 30 19:15:08.532884 ip-10-0-132-82 kubenswrapper[2577]: W1030 19:15:08.532515    2577 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Oct 30 19:15:08.532884 ip-10-0-132-82 kubenswrapper[2577]: W1030 19:15:08.532519    2577 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Oct 30 19:15:08.532884 ip-10-0-132-82 kubenswrapper[2577]: W1030 19:15:08.532524    2577 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Oct 30 19:15:08.532884 ip-10-0-132-82 kubenswrapper[2577]: W1030 19:15:08.532528    2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Oct 30 19:15:08.532884 ip-10-0-132-82 kubenswrapper[2577]: W1030 19:15:08.532532    2577 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Oct 30 19:15:08.532884 ip-10-0-132-82 kubenswrapper[2577]: W1030 19:15:08.532536    2577 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Oct 30 19:15:08.532884 ip-10-0-132-82 kubenswrapper[2577]: W1030 19:15:08.532541    2577 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Oct 30 19:15:08.532884 ip-10-0-132-82 kubenswrapper[2577]: W1030 19:15:08.532545    2577 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Oct 30 19:15:08.533555 ip-10-0-132-82 kubenswrapper[2577]: W1030 19:15:08.532555    2577 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Oct 30 19:15:08.533555 ip-10-0-132-82 kubenswrapper[2577]: W1030 19:15:08.532559    2577 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Oct 30 19:15:08.533555 ip-10-0-132-82 kubenswrapper[2577]: W1030 19:15:08.532564    2577 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Oct 30 19:15:08.533555 ip-10-0-132-82 kubenswrapper[2577]: W1030 19:15:08.532568    2577 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Oct 30 19:15:08.533555 ip-10-0-132-82 kubenswrapper[2577]: W1030 19:15:08.532572    2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Oct 30 19:15:08.533555 ip-10-0-132-82 kubenswrapper[2577]: W1030 19:15:08.532576    2577 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Oct 30 19:15:08.533555 ip-10-0-132-82 kubenswrapper[2577]: W1030 19:15:08.532580    2577 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Oct 30 19:15:08.533555 ip-10-0-132-82 kubenswrapper[2577]: W1030 19:15:08.532584    2577 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Oct 30 19:15:08.533555 ip-10-0-132-82 kubenswrapper[2577]: W1030 19:15:08.532589    2577 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Oct 30 19:15:08.533555 ip-10-0-132-82 kubenswrapper[2577]: W1030 19:15:08.532593    2577 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Oct 30 19:15:08.533555 ip-10-0-132-82 kubenswrapper[2577]: W1030 19:15:08.532597    2577 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Oct 30 19:15:08.533555 ip-10-0-132-82 kubenswrapper[2577]: W1030 19:15:08.532602    2577 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Oct 30 19:15:08.533555 ip-10-0-132-82 kubenswrapper[2577]: W1030 19:15:08.532606    2577 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Oct 30 19:15:08.533555 ip-10-0-132-82 kubenswrapper[2577]: W1030 19:15:08.532626    2577 feature_gate.go:386] unrecognized feature gate: OSStreams
Oct 30 19:15:08.533555 ip-10-0-132-82 kubenswrapper[2577]: W1030 19:15:08.532631    2577 feature_gate.go:386] unrecognized feature gate: DualReplica
Oct 30 19:15:08.533555 ip-10-0-132-82 kubenswrapper[2577]: W1030 19:15:08.532635    2577 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Oct 30 19:15:08.533555 ip-10-0-132-82 kubenswrapper[2577]: W1030 19:15:08.532640    2577 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Oct 30 19:15:08.533555 ip-10-0-132-82 kubenswrapper[2577]: W1030 19:15:08.532644    2577 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Oct 30 19:15:08.533555 ip-10-0-132-82 kubenswrapper[2577]: W1030 19:15:08.532648    2577 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Oct 30 19:15:08.534066 ip-10-0-132-82 kubenswrapper[2577]: W1030 19:15:08.532653    2577 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Oct 30 19:15:08.534066 ip-10-0-132-82 kubenswrapper[2577]: W1030 19:15:08.532657    2577 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Oct 30 19:15:08.534066 ip-10-0-132-82 kubenswrapper[2577]: W1030 19:15:08.532662    2577 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Oct 30 19:15:08.534066 ip-10-0-132-82 kubenswrapper[2577]: W1030 19:15:08.532669    2577 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Oct 30 19:15:08.534066 ip-10-0-132-82 kubenswrapper[2577]: W1030 19:15:08.532673    2577 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Oct 30 19:15:08.534066 ip-10-0-132-82 kubenswrapper[2577]: W1030 19:15:08.532678    2577 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Oct 30 19:15:08.534066 ip-10-0-132-82 kubenswrapper[2577]: W1030 19:15:08.532683    2577 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Oct 30 19:15:08.534066 ip-10-0-132-82 kubenswrapper[2577]: W1030 19:15:08.532692    2577 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Oct 30 19:15:08.534066 ip-10-0-132-82 kubenswrapper[2577]: W1030 19:15:08.532696    2577 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Oct 30 19:15:08.534066 ip-10-0-132-82 kubenswrapper[2577]: W1030 19:15:08.532700    2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Oct 30 19:15:08.534066 ip-10-0-132-82 kubenswrapper[2577]: W1030 19:15:08.532705    2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Oct 30 19:15:08.534066 ip-10-0-132-82 kubenswrapper[2577]: W1030 19:15:08.532711    2577 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Oct 30 19:15:08.534066 ip-10-0-132-82 kubenswrapper[2577]: W1030 19:15:08.532717    2577 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Oct 30 19:15:08.534066 ip-10-0-132-82 kubenswrapper[2577]: W1030 19:15:08.532722    2577 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Oct 30 19:15:08.534066 ip-10-0-132-82 kubenswrapper[2577]: W1030 19:15:08.532727    2577 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Oct 30 19:15:08.534066 ip-10-0-132-82 kubenswrapper[2577]: W1030 19:15:08.532731    2577 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Oct 30 19:15:08.534066 ip-10-0-132-82 kubenswrapper[2577]: W1030 19:15:08.532736    2577 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Oct 30 19:15:08.534066 ip-10-0-132-82 kubenswrapper[2577]: W1030 19:15:08.532740    2577 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Oct 30 19:15:08.534066 ip-10-0-132-82 kubenswrapper[2577]: W1030 19:15:08.532744    2577 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Oct 30 19:15:08.534515 ip-10-0-132-82 kubenswrapper[2577]: W1030 19:15:08.532753    2577 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Oct 30 19:15:08.534515 ip-10-0-132-82 kubenswrapper[2577]: W1030 19:15:08.532758    2577 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Oct 30 19:15:08.534515 ip-10-0-132-82 kubenswrapper[2577]: W1030 19:15:08.532762    2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Oct 30 19:15:08.534515 ip-10-0-132-82 kubenswrapper[2577]: W1030 19:15:08.532766    2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Oct 30 19:15:08.534515 ip-10-0-132-82 kubenswrapper[2577]: W1030 19:15:08.532770    2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Oct 30 19:15:08.534515 ip-10-0-132-82 kubenswrapper[2577]: W1030 19:15:08.532775    2577 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Oct 30 19:15:08.534515 ip-10-0-132-82 kubenswrapper[2577]: W1030 19:15:08.532779    2577 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Oct 30 19:15:08.534515 ip-10-0-132-82 kubenswrapper[2577]: W1030 19:15:08.532783    2577 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Oct 30 19:15:08.534515 ip-10-0-132-82 kubenswrapper[2577]: W1030 19:15:08.532789    2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Oct 30 19:15:08.534515 ip-10-0-132-82 kubenswrapper[2577]: W1030 19:15:08.532793    2577 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Oct 30 19:15:08.534515 ip-10-0-132-82 kubenswrapper[2577]: W1030 19:15:08.532797    2577 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Oct 30 19:15:08.534515 ip-10-0-132-82 kubenswrapper[2577]: W1030 19:15:08.532804    2577 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Oct 30 19:15:08.534515 ip-10-0-132-82 kubenswrapper[2577]: W1030 19:15:08.532813    2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Oct 30 19:15:08.534515 ip-10-0-132-82 kubenswrapper[2577]: W1030 19:15:08.532818    2577 feature_gate.go:386] unrecognized feature gate: EVPN
Oct 30 19:15:08.534515 ip-10-0-132-82 kubenswrapper[2577]: W1030 19:15:08.532822    2577 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Oct 30 19:15:08.534515 ip-10-0-132-82 kubenswrapper[2577]: W1030 19:15:08.532826    2577 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Oct 30 19:15:08.534515 ip-10-0-132-82 kubenswrapper[2577]: W1030 19:15:08.532830    2577 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Oct 30 19:15:08.534515 ip-10-0-132-82 kubenswrapper[2577]: W1030 19:15:08.532834    2577 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Oct 30 19:15:08.534515 ip-10-0-132-82 kubenswrapper[2577]: W1030 19:15:08.532837    2577 feature_gate.go:386] unrecognized feature gate: OVNObservability
Oct 30 19:15:08.534515 ip-10-0-132-82 kubenswrapper[2577]: W1030 19:15:08.532842    2577 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Oct 30 19:15:08.535010 ip-10-0-132-82 kubenswrapper[2577]: W1030 19:15:08.532846    2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Oct 30 19:15:08.535010 ip-10-0-132-82 kubenswrapper[2577]: W1030 19:15:08.532850    2577 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Oct 30 19:15:08.535010 ip-10-0-132-82 kubenswrapper[2577]: W1030 19:15:08.532854    2577 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Oct 30 19:15:08.535010 ip-10-0-132-82 kubenswrapper[2577]: W1030 19:15:08.532859    2577 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Oct 30 19:15:08.535010 ip-10-0-132-82 kubenswrapper[2577]: W1030 19:15:08.532863    2577 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Oct 30 19:15:08.535010 ip-10-0-132-82 kubenswrapper[2577]: W1030 19:15:08.532871    2577 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Oct 30 19:15:08.535010 ip-10-0-132-82 kubenswrapper[2577]: W1030 19:15:08.532875    2577 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Oct 30 19:15:08.535010 ip-10-0-132-82 kubenswrapper[2577]: W1030 19:15:08.532880    2577 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Oct 30 19:15:08.535010 ip-10-0-132-82 kubenswrapper[2577]: W1030 19:15:08.532948    2577 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Oct 30 19:15:08.535010 ip-10-0-132-82 kubenswrapper[2577]: W1030 19:15:08.532986    2577 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Oct 30 19:15:08.535010 ip-10-0-132-82 kubenswrapper[2577]: W1030 19:15:08.532992    2577 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Oct 30 19:15:08.535010 ip-10-0-132-82 kubenswrapper[2577]: W1030 19:15:08.533000    2577 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Oct 30 19:15:08.535010 ip-10-0-132-82 kubenswrapper[2577]: W1030 19:15:08.533141    2577 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Oct 30 19:15:08.535010 ip-10-0-132-82 kubenswrapper[2577]: W1030 19:15:08.533147    2577 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Oct 30 19:15:08.535010 ip-10-0-132-82 kubenswrapper[2577]: W1030 19:15:08.533150    2577 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Oct 30 19:15:08.535010 ip-10-0-132-82 kubenswrapper[2577]: W1030 19:15:08.533153    2577 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Oct 30 19:15:08.535010 ip-10-0-132-82 kubenswrapper[2577]: W1030 19:15:08.533155    2577 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Oct 30 19:15:08.535010 ip-10-0-132-82 kubenswrapper[2577]: W1030 19:15:08.533160    2577 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Oct 30 19:15:08.535010 ip-10-0-132-82 kubenswrapper[2577]: W1030 19:15:08.533163    2577 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Oct 30 19:15:08.535450 ip-10-0-132-82 kubenswrapper[2577]: W1030 19:15:08.533165    2577 feature_gate.go:386] unrecognized feature gate: Example2
Oct 30 19:15:08.535450 ip-10-0-132-82 kubenswrapper[2577]: W1030 19:15:08.533169    2577 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Oct 30 19:15:08.535450 ip-10-0-132-82 kubenswrapper[2577]: W1030 19:15:08.533172    2577 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Oct 30 19:15:08.535450 ip-10-0-132-82 kubenswrapper[2577]: W1030 19:15:08.533174    2577 feature_gate.go:386] unrecognized feature gate: Example
Oct 30 19:15:08.535450 ip-10-0-132-82 kubenswrapper[2577]: W1030 19:15:08.533181    2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Oct 30 19:15:08.535450 ip-10-0-132-82 kubenswrapper[2577]: W1030 19:15:08.533184    2577 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Oct 30 19:15:08.535450 ip-10-0-132-82 kubenswrapper[2577]: W1030 19:15:08.533187    2577 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Oct 30 19:15:08.535450 ip-10-0-132-82 kubenswrapper[2577]: W1030 19:15:08.533189    2577 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Oct 30 19:15:08.535450 ip-10-0-132-82 kubenswrapper[2577]: W1030 19:15:08.533192    2577 feature_gate.go:386] unrecognized feature gate: SignatureStores
Oct 30 19:15:08.535450 ip-10-0-132-82 kubenswrapper[2577]: W1030 19:15:08.533194    2577 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Oct 30 19:15:08.535677 ip-10-0-132-82 kubenswrapper[2577]: W1030 19:15:08.535584    2577 feature_gate.go:386] unrecognized feature gate: OSStreams
Oct 30 19:15:08.535677 ip-10-0-132-82 kubenswrapper[2577]: W1030 19:15:08.535593    2577 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Oct 30 19:15:08.535677 ip-10-0-132-82 kubenswrapper[2577]: W1030 19:15:08.535596    2577 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Oct 30 19:15:08.535677 ip-10-0-132-82 kubenswrapper[2577]: W1030 19:15:08.535599    2577 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Oct 30 19:15:08.535677 ip-10-0-132-82 kubenswrapper[2577]: W1030 19:15:08.535601    2577 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Oct 30 19:15:08.535677 ip-10-0-132-82 kubenswrapper[2577]: W1030 19:15:08.535604    2577 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Oct 30 19:15:08.535677 ip-10-0-132-82 kubenswrapper[2577]: W1030 19:15:08.535607    2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Oct 30 19:15:08.535677 ip-10-0-132-82 kubenswrapper[2577]: W1030 19:15:08.535611    2577 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Oct 30 19:15:08.535677 ip-10-0-132-82 kubenswrapper[2577]: W1030 19:15:08.535614    2577 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Oct 30 19:15:08.535677 ip-10-0-132-82 kubenswrapper[2577]: W1030 19:15:08.535616    2577 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Oct 30 19:15:08.535677 ip-10-0-132-82 kubenswrapper[2577]: W1030 19:15:08.535619    2577 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Oct 30 19:15:08.535677 ip-10-0-132-82 kubenswrapper[2577]: W1030 19:15:08.535621    2577 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Oct 30 19:15:08.535677 ip-10-0-132-82 kubenswrapper[2577]: W1030 19:15:08.535624    2577 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Oct 30 19:15:08.535677 ip-10-0-132-82 kubenswrapper[2577]: W1030 19:15:08.535627    2577 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Oct 30 19:15:08.535677 ip-10-0-132-82 kubenswrapper[2577]: W1030 19:15:08.535630    2577 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Oct 30 19:15:08.535677 ip-10-0-132-82 kubenswrapper[2577]: W1030 19:15:08.535632    2577 feature_gate.go:386] unrecognized feature gate: Example
Oct 30 19:15:08.535677 ip-10-0-132-82 kubenswrapper[2577]: W1030 19:15:08.535635    2577 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Oct 30 19:15:08.535677 ip-10-0-132-82 kubenswrapper[2577]: W1030 19:15:08.535637    2577 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Oct 30 19:15:08.535677 ip-10-0-132-82 kubenswrapper[2577]: W1030 19:15:08.535640    2577 feature_gate.go:386] unrecognized feature gate: DualReplica
Oct 30 19:15:08.535677 ip-10-0-132-82 kubenswrapper[2577]: W1030 19:15:08.535642    2577 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Oct 30 19:15:08.536258 ip-10-0-132-82 kubenswrapper[2577]: W1030 19:15:08.535645    2577 feature_gate.go:386] unrecognized feature gate: OVNObservability
Oct 30 19:15:08.536258 ip-10-0-132-82 kubenswrapper[2577]: W1030 19:15:08.535647    2577 feature_gate.go:386] unrecognized feature gate: SignatureStores
Oct 30 19:15:08.536258 ip-10-0-132-82 kubenswrapper[2577]: W1030 19:15:08.535649    2577 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Oct 30 19:15:08.536258 ip-10-0-132-82 kubenswrapper[2577]: W1030 19:15:08.535652    2577 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Oct 30 19:15:08.536258 ip-10-0-132-82 kubenswrapper[2577]: W1030 19:15:08.535654    2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Oct 30 19:15:08.536258 ip-10-0-132-82 kubenswrapper[2577]: W1030 19:15:08.535657    2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Oct 30 19:15:08.536258 ip-10-0-132-82 kubenswrapper[2577]: W1030 19:15:08.535659    2577 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Oct 30 19:15:08.536258 ip-10-0-132-82 kubenswrapper[2577]: W1030 19:15:08.535662    2577 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Oct 30 19:15:08.536258 ip-10-0-132-82 kubenswrapper[2577]: W1030 19:15:08.535664    2577 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Oct 30 19:15:08.536258 ip-10-0-132-82 kubenswrapper[2577]: W1030 19:15:08.535667    2577 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Oct 30 19:15:08.536258 ip-10-0-132-82 kubenswrapper[2577]: W1030 19:15:08.535669    2577 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Oct 30 19:15:08.536258 ip-10-0-132-82 kubenswrapper[2577]: W1030 19:15:08.535672    2577 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Oct 30 19:15:08.536258 ip-10-0-132-82 kubenswrapper[2577]: W1030 19:15:08.535675    2577 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Oct 30 19:15:08.536258 ip-10-0-132-82 kubenswrapper[2577]: W1030 19:15:08.535678    2577 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Oct 30 19:15:08.536258 ip-10-0-132-82 kubenswrapper[2577]: W1030 19:15:08.535689    2577 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Oct 30 19:15:08.536258 ip-10-0-132-82 kubenswrapper[2577]: W1030 19:15:08.535693    2577 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Oct 30 19:15:08.536258 ip-10-0-132-82 kubenswrapper[2577]: W1030 19:15:08.535696    2577 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Oct 30 19:15:08.536258 ip-10-0-132-82 kubenswrapper[2577]: W1030 19:15:08.535698    2577 feature_gate.go:386] unrecognized feature gate: NewOLM
Oct 30 19:15:08.536258 ip-10-0-132-82 kubenswrapper[2577]: W1030 19:15:08.535701    2577 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Oct 30 19:15:08.536258 ip-10-0-132-82 kubenswrapper[2577]: W1030 19:15:08.535704    2577 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Oct 30 19:15:08.536748 ip-10-0-132-82 kubenswrapper[2577]: W1030 19:15:08.535706    2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Oct 30 19:15:08.536748 ip-10-0-132-82 kubenswrapper[2577]: W1030 19:15:08.535709    2577 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Oct 30 19:15:08.536748 ip-10-0-132-82 kubenswrapper[2577]: W1030 19:15:08.535711    2577 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Oct 30 19:15:08.536748 ip-10-0-132-82 kubenswrapper[2577]: W1030 19:15:08.535714    2577 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Oct 30 19:15:08.536748 ip-10-0-132-82 kubenswrapper[2577]: W1030 19:15:08.535716    2577 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Oct 30 19:15:08.536748 ip-10-0-132-82 kubenswrapper[2577]: W1030 19:15:08.535719    2577 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Oct 30 19:15:08.536748 ip-10-0-132-82 kubenswrapper[2577]: W1030 19:15:08.535721    2577 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Oct 30 19:15:08.536748 ip-10-0-132-82 kubenswrapper[2577]: W1030 19:15:08.535724    2577 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Oct 30 19:15:08.536748 ip-10-0-132-82 kubenswrapper[2577]: W1030 19:15:08.535726    2577 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Oct 30 19:15:08.536748 ip-10-0-132-82 kubenswrapper[2577]: W1030 19:15:08.535728    2577 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Oct 30 19:15:08.536748 ip-10-0-132-82 kubenswrapper[2577]: W1030 19:15:08.535731    2577 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Oct 30 19:15:08.536748 ip-10-0-132-82 kubenswrapper[2577]: W1030 19:15:08.535733    2577 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Oct 30 19:15:08.536748 ip-10-0-132-82 kubenswrapper[2577]: W1030 19:15:08.535736    2577 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Oct 30 19:15:08.536748 ip-10-0-132-82 kubenswrapper[2577]: W1030 19:15:08.535738    2577 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Oct 30 19:15:08.536748 ip-10-0-132-82 kubenswrapper[2577]: W1030 19:15:08.535741    2577 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Oct 30 19:15:08.536748 ip-10-0-132-82 kubenswrapper[2577]: W1030 19:15:08.535743    2577 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Oct 30 19:15:08.536748 ip-10-0-132-82 kubenswrapper[2577]: W1030 19:15:08.535746    2577 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Oct 30 19:15:08.536748 ip-10-0-132-82 kubenswrapper[2577]: W1030 19:15:08.535749    2577 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Oct 30 19:15:08.536748 ip-10-0-132-82 kubenswrapper[2577]: W1030 19:15:08.535751    2577 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Oct 30 19:15:08.537233 ip-10-0-132-82 kubenswrapper[2577]: W1030 19:15:08.535754    2577 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Oct 30 19:15:08.537233 ip-10-0-132-82 kubenswrapper[2577]: W1030 19:15:08.535756    2577 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Oct 30 19:15:08.537233 ip-10-0-132-82 kubenswrapper[2577]: W1030 19:15:08.535759    2577 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Oct 30 19:15:08.537233 ip-10-0-132-82 kubenswrapper[2577]: W1030 19:15:08.535761    2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Oct 30 19:15:08.537233 ip-10-0-132-82 kubenswrapper[2577]: W1030 19:15:08.535764    2577 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Oct 30 19:15:08.537233 ip-10-0-132-82 kubenswrapper[2577]: W1030 19:15:08.535766    2577 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Oct 30 19:15:08.537233 ip-10-0-132-82 kubenswrapper[2577]: W1030 19:15:08.535770    2577 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Oct 30 19:15:08.537233 ip-10-0-132-82 kubenswrapper[2577]: W1030 19:15:08.535773    2577 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Oct 30 19:15:08.537233 ip-10-0-132-82 kubenswrapper[2577]: W1030 19:15:08.535776    2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Oct 30 19:15:08.537233 ip-10-0-132-82 kubenswrapper[2577]: W1030 19:15:08.535778    2577 feature_gate.go:386] unrecognized feature gate: Example2
Oct 30 19:15:08.537233 ip-10-0-132-82 kubenswrapper[2577]: W1030 19:15:08.535781    2577 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Oct 30 19:15:08.537233 ip-10-0-132-82 kubenswrapper[2577]: W1030 19:15:08.535783    2577 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Oct 30 19:15:08.537233 ip-10-0-132-82 kubenswrapper[2577]: W1030 19:15:08.535786    2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Oct 30 19:15:08.537233 ip-10-0-132-82 kubenswrapper[2577]: W1030 19:15:08.535789    2577 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Oct 30 19:15:08.537233 ip-10-0-132-82 kubenswrapper[2577]: W1030 19:15:08.535791    2577 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Oct 30 19:15:08.537233 ip-10-0-132-82 kubenswrapper[2577]: W1030 19:15:08.535794    2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Oct 30 19:15:08.537233 ip-10-0-132-82 kubenswrapper[2577]: W1030 19:15:08.535797    2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Oct 30 19:15:08.537233 ip-10-0-132-82 kubenswrapper[2577]: W1030 19:15:08.535800    2577 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Oct 30 19:15:08.537233 ip-10-0-132-82 kubenswrapper[2577]: W1030 19:15:08.535804    2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Oct 30 19:15:08.537709 ip-10-0-132-82 kubenswrapper[2577]: W1030 19:15:08.535806    2577 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Oct 30 19:15:08.537709 ip-10-0-132-82 kubenswrapper[2577]: W1030 19:15:08.535809    2577 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Oct 30 19:15:08.537709 ip-10-0-132-82 kubenswrapper[2577]: W1030 19:15:08.535811    2577 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Oct 30 19:15:08.537709 ip-10-0-132-82 kubenswrapper[2577]: W1030 19:15:08.535814    2577 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Oct 30 19:15:08.537709 ip-10-0-132-82 kubenswrapper[2577]: W1030 19:15:08.535816    2577 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Oct 30 19:15:08.537709 ip-10-0-132-82 kubenswrapper[2577]: W1030 19:15:08.535819    2577 feature_gate.go:386] unrecognized feature gate: EVPN
Oct 30 19:15:08.537709 ip-10-0-132-82 kubenswrapper[2577]: W1030 19:15:08.535821    2577 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Oct 30 19:15:08.537709 ip-10-0-132-82 kubenswrapper[2577]: W1030 19:15:08.535824    2577 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Oct 30 19:15:08.537709 ip-10-0-132-82 kubenswrapper[2577]: W1030 19:15:08.535827    2577 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Oct 30 19:15:08.537709 ip-10-0-132-82 kubenswrapper[2577]: W1030 19:15:08.535831    2577 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Oct 30 19:15:08.537709 ip-10-0-132-82 kubenswrapper[2577]: W1030 19:15:08.535835    2577 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Oct 30 19:15:08.537709 ip-10-0-132-82 kubenswrapper[2577]: W1030 19:15:08.535837    2577 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Oct 30 19:15:08.537709 ip-10-0-132-82 kubenswrapper[2577]: W1030 19:15:08.535840    2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Oct 30 19:15:08.537709 ip-10-0-132-82 kubenswrapper[2577]: W1030 19:15:08.535843    2577 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Oct 30 19:15:08.537709 ip-10-0-132-82 kubenswrapper[2577]: W1030 19:15:08.535846    2577 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Oct 30 19:15:08.537709 ip-10-0-132-82 kubenswrapper[2577]: W1030 19:15:08.535850    2577 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Oct 30 19:15:08.537709 ip-10-0-132-82 kubenswrapper[2577]: W1030 19:15:08.535854    2577 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Oct 30 19:15:08.537709 ip-10-0-132-82 kubenswrapper[2577]: W1030 19:15:08.535857    2577 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Oct 30 19:15:08.537709 ip-10-0-132-82 kubenswrapper[2577]: W1030 19:15:08.535860    2577 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Oct 30 19:15:08.538161 ip-10-0-132-82 kubenswrapper[2577]: W1030 19:15:08.535863    2577 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Oct 30 19:15:08.538161 ip-10-0-132-82 kubenswrapper[2577]: W1030 19:15:08.535866    2577 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Oct 30 19:15:08.538161 ip-10-0-132-82 kubenswrapper[2577]: W1030 19:15:08.535870    2577 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Oct 30 19:15:08.538161 ip-10-0-132-82 kubenswrapper[2577]: W1030 19:15:08.535872    2577 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Oct 30 19:15:08.538161 ip-10-0-132-82 kubenswrapper[2577]: W1030 19:15:08.535875    2577 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Oct 30 19:15:08.538161 ip-10-0-132-82 kubenswrapper[2577]: W1030 19:15:08.535877    2577 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Oct 30 19:15:08.538161 ip-10-0-132-82 kubenswrapper[2577]: W1030 19:15:08.535880    2577 feature_gate.go:386] unrecognized feature gate: EventTTL
Oct 30 19:15:08.538161 ip-10-0-132-82 kubenswrapper[2577]: W1030 19:15:08.535883    2577 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Oct 30 19:15:08.538161 ip-10-0-132-82 kubenswrapper[2577]: W1030 19:15:08.535885    2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Oct 30 19:15:08.538161 ip-10-0-132-82 kubenswrapper[2577]: W1030 19:15:08.536394    2577 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Oct 30 19:15:08.538161 ip-10-0-132-82 kubenswrapper[2577]: W1030 19:15:08.536400    2577 feature_gate.go:386] unrecognized feature gate: OSStreams
Oct 30 19:15:08.538161 ip-10-0-132-82 kubenswrapper[2577]: W1030 19:15:08.536403    2577 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Oct 30 19:15:08.538161 ip-10-0-132-82 kubenswrapper[2577]: W1030 19:15:08.536406    2577 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Oct 30 19:15:08.538161 ip-10-0-132-82 kubenswrapper[2577]: W1030 19:15:08.536409    2577 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Oct 30 19:15:08.538161 ip-10-0-132-82 kubenswrapper[2577]: W1030 19:15:08.536412    2577 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Oct 30 19:15:08.538161 ip-10-0-132-82 kubenswrapper[2577]: W1030 19:15:08.536414    2577 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Oct 30 19:15:08.538161 ip-10-0-132-82 kubenswrapper[2577]: W1030 19:15:08.536417    2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Oct 30 19:15:08.538161 ip-10-0-132-82 kubenswrapper[2577]: W1030 19:15:08.536420    2577 feature_gate.go:386] unrecognized feature gate: Example2
Oct 30 19:15:08.538161 ip-10-0-132-82 kubenswrapper[2577]: W1030 19:15:08.536422    2577 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Oct 30 19:15:08.538161 ip-10-0-132-82 kubenswrapper[2577]: W1030 19:15:08.536425    2577 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Oct 30 19:15:08.538630 ip-10-0-132-82 kubenswrapper[2577]: W1030 19:15:08.536427    2577 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Oct 30 19:15:08.538630 ip-10-0-132-82 kubenswrapper[2577]: W1030 19:15:08.536430    2577 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Oct 30 19:15:08.538630 ip-10-0-132-82 kubenswrapper[2577]: W1030 19:15:08.536433    2577 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Oct 30 19:15:08.538630 ip-10-0-132-82 kubenswrapper[2577]: W1030 19:15:08.536435    2577 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Oct 30 19:15:08.538630 ip-10-0-132-82 kubenswrapper[2577]: W1030 19:15:08.536438    2577 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Oct 30 19:15:08.538630 ip-10-0-132-82 kubenswrapper[2577]: W1030 19:15:08.536440    2577 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Oct 30 19:15:08.538630 ip-10-0-132-82 kubenswrapper[2577]: W1030 19:15:08.536443    2577 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Oct 30 19:15:08.538630 ip-10-0-132-82 kubenswrapper[2577]: W1030 19:15:08.536445    2577 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Oct 30 19:15:08.538630 ip-10-0-132-82 kubenswrapper[2577]: W1030 19:15:08.536448    2577 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Oct 30 19:15:08.538630 ip-10-0-132-82 kubenswrapper[2577]: W1030 19:15:08.536450    2577 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Oct 30 19:15:08.538630 ip-10-0-132-82 kubenswrapper[2577]: W1030 19:15:08.536453    2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Oct 30 19:15:08.538630 ip-10-0-132-82 kubenswrapper[2577]: W1030 19:15:08.536455    2577 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Oct 30 19:15:08.538630 ip-10-0-132-82 kubenswrapper[2577]: W1030 19:15:08.536459    2577 feature_gate.go:386] unrecognized feature gate: EventTTL
Oct 30 19:15:08.538630 ip-10-0-132-82 kubenswrapper[2577]: W1030 19:15:08.536461    2577 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Oct 30 19:15:08.538630 ip-10-0-132-82 kubenswrapper[2577]: W1030 19:15:08.536464    2577 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Oct 30 19:15:08.538630 ip-10-0-132-82 kubenswrapper[2577]: W1030 19:15:08.536466    2577 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Oct 30 19:15:08.538630 ip-10-0-132-82 kubenswrapper[2577]: W1030 19:15:08.536469    2577 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Oct 30 19:15:08.538630 ip-10-0-132-82 kubenswrapper[2577]: W1030 19:15:08.536472    2577 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Oct 30 19:15:08.538630 ip-10-0-132-82 kubenswrapper[2577]: W1030 19:15:08.536475    2577 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Oct 30 19:15:08.538630 ip-10-0-132-82 kubenswrapper[2577]: W1030 19:15:08.536477    2577 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Oct 30 19:15:08.539144 ip-10-0-132-82 kubenswrapper[2577]: W1030 19:15:08.536480    2577 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Oct 30 19:15:08.539144 ip-10-0-132-82 kubenswrapper[2577]: W1030 19:15:08.536483    2577 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Oct 30 19:15:08.539144 ip-10-0-132-82 kubenswrapper[2577]: W1030 19:15:08.536487    2577 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Oct 30 19:15:08.539144 ip-10-0-132-82 kubenswrapper[2577]: W1030 19:15:08.536491    2577 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Oct 30 19:15:08.539144 ip-10-0-132-82 kubenswrapper[2577]: W1030 19:15:08.536494    2577 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Oct 30 19:15:08.539144 ip-10-0-132-82 kubenswrapper[2577]: W1030 19:15:08.536497    2577 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Oct 30 19:15:08.539144 ip-10-0-132-82 kubenswrapper[2577]: W1030 19:15:08.536499    2577 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Oct 30 19:15:08.539144 ip-10-0-132-82 kubenswrapper[2577]: W1030 19:15:08.536502    2577 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Oct 30 19:15:08.539144 ip-10-0-132-82 kubenswrapper[2577]: W1030 19:15:08.536504    2577 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Oct 30 19:15:08.539144 ip-10-0-132-82 kubenswrapper[2577]: W1030 19:15:08.536507    2577 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Oct 30 19:15:08.539144 ip-10-0-132-82 kubenswrapper[2577]: W1030 19:15:08.536511    2577 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Oct 30 19:15:08.539144 ip-10-0-132-82 kubenswrapper[2577]: W1030 19:15:08.536515    2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Oct 30 19:15:08.539144 ip-10-0-132-82 kubenswrapper[2577]: W1030 19:15:08.536518    2577 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Oct 30 19:15:08.539144 ip-10-0-132-82 kubenswrapper[2577]: W1030 19:15:08.536521    2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Oct 30 19:15:08.539144 ip-10-0-132-82 kubenswrapper[2577]: W1030 19:15:08.536524    2577 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Oct 30 19:15:08.539144 ip-10-0-132-82 kubenswrapper[2577]: W1030 19:15:08.536526    2577 feature_gate.go:386] unrecognized feature gate: OVNObservability
Oct 30 19:15:08.539144 ip-10-0-132-82 kubenswrapper[2577]: W1030 19:15:08.536529    2577 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Oct 30 19:15:08.539144 ip-10-0-132-82 kubenswrapper[2577]: W1030 19:15:08.536531    2577 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Oct 30 19:15:08.539560 ip-10-0-132-82 kubenswrapper[2577]: W1030 19:15:08.536533    2577 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Oct 30 19:15:08.539560 ip-10-0-132-82 kubenswrapper[2577]: W1030 19:15:08.536536    2577 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Oct 30 19:15:08.539560 ip-10-0-132-82 kubenswrapper[2577]: W1030 19:15:08.536539    2577 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Oct 30 19:15:08.539560 ip-10-0-132-82 kubenswrapper[2577]: W1030 19:15:08.536541    2577 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Oct 30 19:15:08.539560 ip-10-0-132-82 kubenswrapper[2577]: W1030 19:15:08.536544    2577 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Oct 30 19:15:08.539560 ip-10-0-132-82 kubenswrapper[2577]: W1030 19:15:08.536546    2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Oct 30 19:15:08.539560 ip-10-0-132-82 kubenswrapper[2577]: W1030 19:15:08.536549    2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Oct 30 19:15:08.539560 ip-10-0-132-82 kubenswrapper[2577]: W1030 19:15:08.536552    2577 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Oct 30 19:15:08.539560 ip-10-0-132-82 kubenswrapper[2577]: W1030 19:15:08.536554    2577 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Oct 30 19:15:08.539560 ip-10-0-132-82 kubenswrapper[2577]: W1030 19:15:08.536557    2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Oct 30 19:15:08.539560 ip-10-0-132-82 kubenswrapper[2577]: W1030 19:15:08.536559    2577 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Oct 30 19:15:08.539560 ip-10-0-132-82 kubenswrapper[2577]: W1030 19:15:08.536562    2577 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Oct 30 19:15:08.539560 ip-10-0-132-82 kubenswrapper[2577]: W1030 19:15:08.536564    2577 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Oct 30 19:15:08.539560 ip-10-0-132-82 kubenswrapper[2577]: W1030 19:15:08.536567    2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Oct 30 19:15:08.539560 ip-10-0-132-82 kubenswrapper[2577]: W1030 19:15:08.536569    2577 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Oct 30 19:15:08.539560 ip-10-0-132-82 kubenswrapper[2577]: W1030 19:15:08.536571    2577 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Oct 30 19:15:08.539560 ip-10-0-132-82 kubenswrapper[2577]: W1030 19:15:08.536574    2577 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Oct 30 19:15:08.539560 ip-10-0-132-82 kubenswrapper[2577]: W1030 19:15:08.536576    2577 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Oct 30 19:15:08.539560 ip-10-0-132-82 kubenswrapper[2577]: W1030 19:15:08.536579    2577 feature_gate.go:386] unrecognized feature gate: EVPN
Oct 30 19:15:08.540038 ip-10-0-132-82 kubenswrapper[2577]: W1030 19:15:08.536583    2577 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Oct 30 19:15:08.540038 ip-10-0-132-82 kubenswrapper[2577]: W1030 19:15:08.536586    2577 feature_gate.go:386] unrecognized feature gate: NewOLM
Oct 30 19:15:08.540038 ip-10-0-132-82 kubenswrapper[2577]: W1030 19:15:08.536588    2577 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Oct 30 19:15:08.540038 ip-10-0-132-82 kubenswrapper[2577]: W1030 19:15:08.536591    2577 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Oct 30 19:15:08.540038 ip-10-0-132-82 kubenswrapper[2577]: W1030 19:15:08.536593    2577 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Oct 30 19:15:08.540038 ip-10-0-132-82 kubenswrapper[2577]: W1030 19:15:08.536596    2577 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Oct 30 19:15:08.540038 ip-10-0-132-82 kubenswrapper[2577]: W1030 19:15:08.536598    2577 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Oct 30 19:15:08.540038 ip-10-0-132-82 kubenswrapper[2577]: W1030 19:15:08.536601    2577 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Oct 30 19:15:08.540038 ip-10-0-132-82 kubenswrapper[2577]: W1030 19:15:08.536603    2577 feature_gate.go:386] unrecognized feature gate: SignatureStores
Oct 30 19:15:08.540038 ip-10-0-132-82 kubenswrapper[2577]: W1030 19:15:08.536605    2577 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Oct 30 19:15:08.540038 ip-10-0-132-82 kubenswrapper[2577]: W1030 19:15:08.536608    2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Oct 30 19:15:08.540038 ip-10-0-132-82 kubenswrapper[2577]: W1030 19:15:08.536610    2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Oct 30 19:15:08.540038 ip-10-0-132-82 kubenswrapper[2577]: W1030 19:15:08.536613    2577 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Oct 30 19:15:08.540038 ip-10-0-132-82 kubenswrapper[2577]: W1030 19:15:08.536615    2577 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Oct 30 19:15:08.540038 ip-10-0-132-82 kubenswrapper[2577]: W1030 19:15:08.536618    2577 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Oct 30 19:15:08.540038 ip-10-0-132-82 kubenswrapper[2577]: W1030 19:15:08.536620    2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Oct 30 19:15:08.540038 ip-10-0-132-82 kubenswrapper[2577]: W1030 19:15:08.536623    2577 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Oct 30 19:15:08.540038 ip-10-0-132-82 kubenswrapper[2577]: W1030 19:15:08.536625    2577 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Oct 30 19:15:08.540038 ip-10-0-132-82 kubenswrapper[2577]: W1030 19:15:08.536628    2577 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Oct 30 19:15:08.540038 ip-10-0-132-82 kubenswrapper[2577]: W1030 19:15:08.536631    2577 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Oct 30 19:15:08.540535 ip-10-0-132-82 kubenswrapper[2577]: W1030 19:15:08.536633    2577 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Oct 30 19:15:08.540535 ip-10-0-132-82 kubenswrapper[2577]: W1030 19:15:08.536636    2577 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Oct 30 19:15:08.540535 ip-10-0-132-82 kubenswrapper[2577]: W1030 19:15:08.536638    2577 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Oct 30 19:15:08.540535 ip-10-0-132-82 kubenswrapper[2577]: W1030 19:15:08.536641    2577 feature_gate.go:386] unrecognized feature gate: DualReplica
Oct 30 19:15:08.540535 ip-10-0-132-82 kubenswrapper[2577]: W1030 19:15:08.536643    2577 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Oct 30 19:15:08.540535 ip-10-0-132-82 kubenswrapper[2577]: W1030 19:15:08.536646    2577 feature_gate.go:386] unrecognized feature gate: Example
Oct 30 19:15:08.540535 ip-10-0-132-82 kubenswrapper[2577]: W1030 19:15:08.536648    2577 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Oct 30 19:15:08.540535 ip-10-0-132-82 kubenswrapper[2577]: W1030 19:15:08.536651    2577 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Oct 30 19:15:08.540535 ip-10-0-132-82 kubenswrapper[2577]: W1030 19:15:08.536653    2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Oct 30 19:15:08.540535 ip-10-0-132-82 kubenswrapper[2577]: W1030 19:15:08.536656    2577 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Oct 30 19:15:08.540535 ip-10-0-132-82 kubenswrapper[2577]: W1030 19:15:08.536658    2577 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Oct 30 19:15:08.540535 ip-10-0-132-82 kubenswrapper[2577]: W1030 19:15:08.536661    2577 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Oct 30 19:15:08.540535 ip-10-0-132-82 kubenswrapper[2577]: W1030 19:15:08.536663    2577 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Oct 30 19:15:08.540535 ip-10-0-132-82 kubenswrapper[2577]: W1030 19:15:08.536666    2577 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Oct 30 19:15:08.540535 ip-10-0-132-82 kubenswrapper[2577]: W1030 19:15:08.536669    2577 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Oct 30 19:15:08.540535 ip-10-0-132-82 kubenswrapper[2577]: W1030 19:15:08.536674    2577 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Oct 30 19:15:08.540535 ip-10-0-132-82 kubenswrapper[2577]: W1030 19:15:08.536677    2577 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Oct 30 19:15:08.540535 ip-10-0-132-82 kubenswrapper[2577]: W1030 19:15:08.536679    2577 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Oct 30 19:15:08.540535 ip-10-0-132-82 kubenswrapper[2577]: I1030 19:15:08.537688    2577 flags.go:64] FLAG: --address="0.0.0.0"
Oct 30 19:15:08.540535 ip-10-0-132-82 kubenswrapper[2577]: I1030 19:15:08.537697    2577 flags.go:64] FLAG: --allowed-unsafe-sysctls="[]"
Oct 30 19:15:08.540535 ip-10-0-132-82 kubenswrapper[2577]: I1030 19:15:08.537705    2577 flags.go:64] FLAG: --anonymous-auth="true"
Oct 30 19:15:08.541037 ip-10-0-132-82 kubenswrapper[2577]: I1030 19:15:08.537710    2577 flags.go:64] FLAG: --application-metrics-count-limit="100"
Oct 30 19:15:08.541037 ip-10-0-132-82 kubenswrapper[2577]: I1030 19:15:08.537715    2577 flags.go:64] FLAG: --authentication-token-webhook="false"
Oct 30 19:15:08.541037 ip-10-0-132-82 kubenswrapper[2577]: I1030 19:15:08.537718    2577 flags.go:64] FLAG: --authentication-token-webhook-cache-ttl="2m0s"
Oct 30 19:15:08.541037 ip-10-0-132-82 kubenswrapper[2577]: I1030 19:15:08.537725    2577 flags.go:64] FLAG: --authorization-mode="AlwaysAllow"
Oct 30 19:15:08.541037 ip-10-0-132-82 kubenswrapper[2577]: I1030 19:15:08.537730    2577 flags.go:64] FLAG: --authorization-webhook-cache-authorized-ttl="5m0s"
Oct 30 19:15:08.541037 ip-10-0-132-82 kubenswrapper[2577]: I1030 19:15:08.537733    2577 flags.go:64] FLAG: --authorization-webhook-cache-unauthorized-ttl="30s"
Oct 30 19:15:08.541037 ip-10-0-132-82 kubenswrapper[2577]: I1030 19:15:08.537736    2577 flags.go:64] FLAG: --boot-id-file="/proc/sys/kernel/random/boot_id"
Oct 30 19:15:08.541037 ip-10-0-132-82 kubenswrapper[2577]: I1030 19:15:08.537739    2577 flags.go:64] FLAG: --bootstrap-kubeconfig="/etc/kubernetes/kubeconfig"
Oct 30 19:15:08.541037 ip-10-0-132-82 kubenswrapper[2577]: I1030 19:15:08.537742    2577 flags.go:64] FLAG: --cert-dir="/var/lib/kubelet/pki"
Oct 30 19:15:08.541037 ip-10-0-132-82 kubenswrapper[2577]: I1030 19:15:08.537745    2577 flags.go:64] FLAG: --cgroup-driver="cgroupfs"
Oct 30 19:15:08.541037 ip-10-0-132-82 kubenswrapper[2577]: I1030 19:15:08.537749    2577 flags.go:64] FLAG: --cgroup-root=""
Oct 30 19:15:08.541037 ip-10-0-132-82 kubenswrapper[2577]: I1030 19:15:08.537752    2577 flags.go:64] FLAG: --cgroups-per-qos="true"
Oct 30 19:15:08.541037 ip-10-0-132-82 kubenswrapper[2577]: I1030 19:15:08.537755    2577 flags.go:64] FLAG: --client-ca-file=""
Oct 30 19:15:08.541037 ip-10-0-132-82 kubenswrapper[2577]: I1030 19:15:08.537757    2577 flags.go:64] FLAG: --cloud-provider="external"
Oct 30 19:15:08.541037 ip-10-0-132-82 kubenswrapper[2577]: I1030 19:15:08.537761    2577 flags.go:64] FLAG: --cluster-dns="[]"
Oct 30 19:15:08.541037 ip-10-0-132-82 kubenswrapper[2577]: I1030 19:15:08.537765    2577 flags.go:64] FLAG: --cluster-domain=""
Oct 30 19:15:08.541037 ip-10-0-132-82 kubenswrapper[2577]: I1030 19:15:08.537767    2577 flags.go:64] FLAG: --config="/etc/kubernetes/kubelet.conf"
Oct 30 19:15:08.541037 ip-10-0-132-82 kubenswrapper[2577]: I1030 19:15:08.537770    2577 flags.go:64] FLAG: --config-dir="/etc/openshift/kubelet.conf.d"
Oct 30 19:15:08.541037 ip-10-0-132-82 kubenswrapper[2577]: I1030 19:15:08.537773    2577 flags.go:64] FLAG: --container-hints="/etc/cadvisor/container_hints.json"
Oct 30 19:15:08.541037 ip-10-0-132-82 kubenswrapper[2577]: I1030 19:15:08.537777    2577 flags.go:64] FLAG: --container-log-max-files="5"
Oct 30 19:15:08.541037 ip-10-0-132-82 kubenswrapper[2577]: I1030 19:15:08.537781    2577 flags.go:64] FLAG: --container-log-max-size="10Mi"
Oct 30 19:15:08.541037 ip-10-0-132-82 kubenswrapper[2577]: I1030 19:15:08.537784    2577 flags.go:64] FLAG: --container-runtime-endpoint="/var/run/crio/crio.sock"
Oct 30 19:15:08.541037 ip-10-0-132-82 kubenswrapper[2577]: I1030 19:15:08.537787    2577 flags.go:64] FLAG: --containerd="/run/containerd/containerd.sock"
Oct 30 19:15:08.541607 ip-10-0-132-82 kubenswrapper[2577]: I1030 19:15:08.537790    2577 flags.go:64] FLAG: --containerd-namespace="k8s.io"
Oct 30 19:15:08.541607 ip-10-0-132-82 kubenswrapper[2577]: I1030 19:15:08.537793    2577 flags.go:64] FLAG: --contention-profiling="false"
Oct 30 19:15:08.541607 ip-10-0-132-82 kubenswrapper[2577]: I1030 19:15:08.537796    2577 flags.go:64] FLAG: --cpu-cfs-quota="true"
Oct 30 19:15:08.541607 ip-10-0-132-82 kubenswrapper[2577]: I1030 19:15:08.537800    2577 flags.go:64] FLAG: --cpu-cfs-quota-period="100ms"
Oct 30 19:15:08.541607 ip-10-0-132-82 kubenswrapper[2577]: I1030 19:15:08.537803    2577 flags.go:64] FLAG: --cpu-manager-policy="none"
Oct 30 19:15:08.541607 ip-10-0-132-82 kubenswrapper[2577]: I1030 19:15:08.537806    2577 flags.go:64] FLAG: --cpu-manager-policy-options=""
Oct 30 19:15:08.541607 ip-10-0-132-82 kubenswrapper[2577]: I1030 19:15:08.537810    2577 flags.go:64] FLAG: --cpu-manager-reconcile-period="10s"
Oct 30 19:15:08.541607 ip-10-0-132-82 kubenswrapper[2577]: I1030 19:15:08.537813    2577 flags.go:64] FLAG: --enable-controller-attach-detach="true"
Oct 30 19:15:08.541607 ip-10-0-132-82 kubenswrapper[2577]: I1030 19:15:08.537816    2577 flags.go:64] FLAG: --enable-debugging-handlers="true"
Oct 30 19:15:08.541607 ip-10-0-132-82 kubenswrapper[2577]: I1030 19:15:08.537819    2577 flags.go:64] FLAG: --enable-load-reader="false"
Oct 30 19:15:08.541607 ip-10-0-132-82 kubenswrapper[2577]: I1030 19:15:08.537822    2577 flags.go:64] FLAG: --enable-server="true"
Oct 30 19:15:08.541607 ip-10-0-132-82 kubenswrapper[2577]: I1030 19:15:08.537825    2577 flags.go:64] FLAG: --enforce-node-allocatable="[pods]"
Oct 30 19:15:08.541607 ip-10-0-132-82 kubenswrapper[2577]: I1030 19:15:08.537829    2577 flags.go:64] FLAG: --event-burst="100"
Oct 30 19:15:08.541607 ip-10-0-132-82 kubenswrapper[2577]: I1030 19:15:08.537832    2577 flags.go:64] FLAG: --event-qps="50"
Oct 30 19:15:08.541607 ip-10-0-132-82 kubenswrapper[2577]: I1030 19:15:08.537835    2577 flags.go:64] FLAG: --event-storage-age-limit="default=0"
Oct 30 19:15:08.541607 ip-10-0-132-82 kubenswrapper[2577]: I1030 19:15:08.537838    2577 flags.go:64] FLAG: --event-storage-event-limit="default=0"
Oct 30 19:15:08.541607 ip-10-0-132-82 kubenswrapper[2577]: I1030 19:15:08.537841    2577 flags.go:64] FLAG: --eviction-hard=""
Oct 30 19:15:08.541607 ip-10-0-132-82 kubenswrapper[2577]: I1030 19:15:08.537845    2577 flags.go:64] FLAG: --eviction-max-pod-grace-period="0"
Oct 30 19:15:08.541607 ip-10-0-132-82 kubenswrapper[2577]: I1030 19:15:08.537847    2577 flags.go:64] FLAG: --eviction-minimum-reclaim=""
Oct 30 19:15:08.541607 ip-10-0-132-82 kubenswrapper[2577]: I1030 19:15:08.537852    2577 flags.go:64] FLAG: --eviction-pressure-transition-period="5m0s"
Oct 30 19:15:08.541607 ip-10-0-132-82 kubenswrapper[2577]: I1030 19:15:08.537855    2577 flags.go:64] FLAG: --eviction-soft=""
Oct 30 19:15:08.541607 ip-10-0-132-82 kubenswrapper[2577]: I1030 19:15:08.537858    2577 flags.go:64] FLAG: --eviction-soft-grace-period=""
Oct 30 19:15:08.541607 ip-10-0-132-82 kubenswrapper[2577]: I1030 19:15:08.537861    2577 flags.go:64] FLAG: --exit-on-lock-contention="false"
Oct 30 19:15:08.541607 ip-10-0-132-82 kubenswrapper[2577]: I1030 19:15:08.537864    2577 flags.go:64] FLAG: --experimental-allocatable-ignore-eviction="false"
Oct 30 19:15:08.541607 ip-10-0-132-82 kubenswrapper[2577]: I1030 19:15:08.537867    2577 flags.go:64] FLAG: --experimental-mounter-path=""
Oct 30 19:15:08.542215 ip-10-0-132-82 kubenswrapper[2577]: I1030 19:15:08.537869    2577 flags.go:64] FLAG: --fail-cgroupv1="true"
Oct 30 19:15:08.542215 ip-10-0-132-82 kubenswrapper[2577]: I1030 19:15:08.537872    2577 flags.go:64] FLAG: --fail-swap-on="true"
Oct 30 19:15:08.542215 ip-10-0-132-82 kubenswrapper[2577]: I1030 19:15:08.537875    2577 flags.go:64] FLAG: --feature-gates=""
Oct 30 19:15:08.542215 ip-10-0-132-82 kubenswrapper[2577]: I1030 19:15:08.537879    2577 flags.go:64] FLAG: --file-check-frequency="20s"
Oct 30 19:15:08.542215 ip-10-0-132-82 kubenswrapper[2577]: I1030 19:15:08.537882    2577 flags.go:64] FLAG: --global-housekeeping-interval="1m0s"
Oct 30 19:15:08.542215 ip-10-0-132-82 kubenswrapper[2577]: I1030 19:15:08.537885    2577 flags.go:64] FLAG: --hairpin-mode="promiscuous-bridge"
Oct 30 19:15:08.542215 ip-10-0-132-82 kubenswrapper[2577]: I1030 19:15:08.537888    2577 flags.go:64] FLAG: --healthz-bind-address="127.0.0.1"
Oct 30 19:15:08.542215 ip-10-0-132-82 kubenswrapper[2577]: I1030 19:15:08.537891    2577 flags.go:64] FLAG: --healthz-port="10248"
Oct 30 19:15:08.542215 ip-10-0-132-82 kubenswrapper[2577]: I1030 19:15:08.537894    2577 flags.go:64] FLAG: --help="false"
Oct 30 19:15:08.542215 ip-10-0-132-82 kubenswrapper[2577]: I1030 19:15:08.537897    2577 flags.go:64] FLAG: --hostname-override="ip-10-0-132-82.us-west-1.compute.internal"
Oct 30 19:15:08.542215 ip-10-0-132-82 kubenswrapper[2577]: I1030 19:15:08.537901    2577 flags.go:64] FLAG: --housekeeping-interval="10s"
Oct 30 19:15:08.542215 ip-10-0-132-82 kubenswrapper[2577]: I1030 19:15:08.537904    2577 flags.go:64] FLAG: --http-check-frequency="20s"
Oct 30 19:15:08.542215 ip-10-0-132-82 kubenswrapper[2577]: I1030 19:15:08.537907    2577 flags.go:64] FLAG: --image-credential-provider-bin-dir="/usr/libexec/kubelet-image-credential-provider-plugins"
Oct 30 19:15:08.542215 ip-10-0-132-82 kubenswrapper[2577]: I1030 19:15:08.537911    2577 flags.go:64] FLAG: --image-credential-provider-config="/etc/kubernetes/credential-providers/ecr-credential-provider.yaml"
Oct 30 19:15:08.542215 ip-10-0-132-82 kubenswrapper[2577]: I1030 19:15:08.537914    2577 flags.go:64] FLAG: --image-gc-high-threshold="85"
Oct 30 19:15:08.542215 ip-10-0-132-82 kubenswrapper[2577]: I1030 19:15:08.537916    2577 flags.go:64] FLAG: --image-gc-low-threshold="80"
Oct 30 19:15:08.542215 ip-10-0-132-82 kubenswrapper[2577]: I1030 19:15:08.537933    2577 flags.go:64] FLAG: --image-service-endpoint=""
Oct 30 19:15:08.542215 ip-10-0-132-82 kubenswrapper[2577]: I1030 19:15:08.537938    2577 flags.go:64] FLAG: --kernel-memcg-notification="false"
Oct 30 19:15:08.542215 ip-10-0-132-82 kubenswrapper[2577]: I1030 19:15:08.537943    2577 flags.go:64] FLAG: --kube-api-burst="100"
Oct 30 19:15:08.542215 ip-10-0-132-82 kubenswrapper[2577]: I1030 19:15:08.537946    2577 flags.go:64] FLAG: --kube-api-content-type="application/vnd.kubernetes.protobuf"
Oct 30 19:15:08.542215 ip-10-0-132-82 kubenswrapper[2577]: I1030 19:15:08.537949    2577 flags.go:64] FLAG: --kube-api-qps="50"
Oct 30 19:15:08.542215 ip-10-0-132-82 kubenswrapper[2577]: I1030 19:15:08.537952    2577 flags.go:64] FLAG: --kube-reserved=""
Oct 30 19:15:08.542215 ip-10-0-132-82 kubenswrapper[2577]: I1030 19:15:08.537954    2577 flags.go:64] FLAG: --kube-reserved-cgroup=""
Oct 30 19:15:08.542215 ip-10-0-132-82 kubenswrapper[2577]: I1030 19:15:08.537957    2577 flags.go:64] FLAG: --kubeconfig="/var/lib/kubelet/kubeconfig"
Oct 30 19:15:08.542774 ip-10-0-132-82 kubenswrapper[2577]: I1030 19:15:08.537960    2577 flags.go:64] FLAG: --kubelet-cgroups=""
Oct 30 19:15:08.542774 ip-10-0-132-82 kubenswrapper[2577]: I1030 19:15:08.537963    2577 flags.go:64] FLAG: --local-storage-capacity-isolation="true"
Oct 30 19:15:08.542774 ip-10-0-132-82 kubenswrapper[2577]: I1030 19:15:08.537967    2577 flags.go:64] FLAG: --lock-file=""
Oct 30 19:15:08.542774 ip-10-0-132-82 kubenswrapper[2577]: I1030 19:15:08.537970    2577 flags.go:64] FLAG: --log-cadvisor-usage="false"
Oct 30 19:15:08.542774 ip-10-0-132-82 kubenswrapper[2577]: I1030 19:15:08.537973    2577 flags.go:64] FLAG: --log-flush-frequency="5s"
Oct 30 19:15:08.542774 ip-10-0-132-82 kubenswrapper[2577]: I1030 19:15:08.537976    2577 flags.go:64] FLAG: --log-json-info-buffer-size="0"
Oct 30 19:15:08.542774 ip-10-0-132-82 kubenswrapper[2577]: I1030 19:15:08.537986    2577 flags.go:64] FLAG: --log-json-split-stream="false"
Oct 30 19:15:08.542774 ip-10-0-132-82 kubenswrapper[2577]: I1030 19:15:08.537989    2577 flags.go:64] FLAG: --log-text-info-buffer-size="0"
Oct 30 19:15:08.542774 ip-10-0-132-82 kubenswrapper[2577]: I1030 19:15:08.537992    2577 flags.go:64] FLAG: --log-text-split-stream="false"
Oct 30 19:15:08.542774 ip-10-0-132-82 kubenswrapper[2577]: I1030 19:15:08.537995    2577 flags.go:64] FLAG: --logging-format="text"
Oct 30 19:15:08.542774 ip-10-0-132-82 kubenswrapper[2577]: I1030 19:15:08.537998    2577 flags.go:64] FLAG: --machine-id-file="/etc/machine-id,/var/lib/dbus/machine-id"
Oct 30 19:15:08.542774 ip-10-0-132-82 kubenswrapper[2577]: I1030 19:15:08.538002    2577 flags.go:64] FLAG: --make-iptables-util-chains="true"
Oct 30 19:15:08.542774 ip-10-0-132-82 kubenswrapper[2577]: I1030 19:15:08.538004    2577 flags.go:64] FLAG: --manifest-url=""
Oct 30 19:15:08.542774 ip-10-0-132-82 kubenswrapper[2577]: I1030 19:15:08.538007    2577 flags.go:64] FLAG: --manifest-url-header=""
Oct 30 19:15:08.542774 ip-10-0-132-82 kubenswrapper[2577]: I1030 19:15:08.538011    2577 flags.go:64] FLAG: --max-housekeeping-interval="15s"
Oct 30 19:15:08.542774 ip-10-0-132-82 kubenswrapper[2577]: I1030 19:15:08.538014    2577 flags.go:64] FLAG: --max-open-files="1000000"
Oct 30 19:15:08.542774 ip-10-0-132-82 kubenswrapper[2577]: I1030 19:15:08.538018    2577 flags.go:64] FLAG: --max-pods="110"
Oct 30 19:15:08.542774 ip-10-0-132-82 kubenswrapper[2577]: I1030 19:15:08.538021    2577 flags.go:64] FLAG: --maximum-dead-containers="-1"
Oct 30 19:15:08.542774 ip-10-0-132-82 kubenswrapper[2577]: I1030 19:15:08.538024    2577 flags.go:64] FLAG: --maximum-dead-containers-per-container="1"
Oct 30 19:15:08.542774 ip-10-0-132-82 kubenswrapper[2577]: I1030 19:15:08.538028    2577 flags.go:64] FLAG: --memory-manager-policy="None"
Oct 30 19:15:08.542774 ip-10-0-132-82 kubenswrapper[2577]: I1030 19:15:08.538031    2577 flags.go:64] FLAG: --minimum-container-ttl-duration="6m0s"
Oct 30 19:15:08.542774 ip-10-0-132-82 kubenswrapper[2577]: I1030 19:15:08.538034    2577 flags.go:64] FLAG: --minimum-image-ttl-duration="2m0s"
Oct 30 19:15:08.542774 ip-10-0-132-82 kubenswrapper[2577]: I1030 19:15:08.538037    2577 flags.go:64] FLAG: --node-ip="0.0.0.0"
Oct 30 19:15:08.542774 ip-10-0-132-82 kubenswrapper[2577]: I1030 19:15:08.538040    2577 flags.go:64] FLAG: --node-labels="node-role.kubernetes.io/worker=,node.openshift.io/os_id=rhel"
Oct 30 19:15:08.542774 ip-10-0-132-82 kubenswrapper[2577]: I1030 19:15:08.538047    2577 flags.go:64] FLAG: --node-status-max-images="50"
Oct 30 19:15:08.543380 ip-10-0-132-82 kubenswrapper[2577]: I1030 19:15:08.538050    2577 flags.go:64] FLAG: --node-status-update-frequency="10s"
Oct 30 19:15:08.543380 ip-10-0-132-82 kubenswrapper[2577]: I1030 19:15:08.538053    2577 flags.go:64] FLAG: --oom-score-adj="-999"
Oct 30 19:15:08.543380 ip-10-0-132-82 kubenswrapper[2577]: I1030 19:15:08.538056    2577 flags.go:64] FLAG: --pod-cidr=""
Oct 30 19:15:08.543380 ip-10-0-132-82 kubenswrapper[2577]: I1030 19:15:08.538059    2577 flags.go:64] FLAG: --pod-manifest-path=""
Oct 30 19:15:08.543380 ip-10-0-132-82 kubenswrapper[2577]: I1030 19:15:08.538062    2577 flags.go:64] FLAG: --pod-max-pids="-1"
Oct 30 19:15:08.543380 ip-10-0-132-82 kubenswrapper[2577]: I1030 19:15:08.538064    2577 flags.go:64] FLAG: --pods-per-core="0"
Oct 30 19:15:08.543380 ip-10-0-132-82 kubenswrapper[2577]: I1030 19:15:08.538067    2577 flags.go:64] FLAG: --port="10250"
Oct 30 19:15:08.543380 ip-10-0-132-82 kubenswrapper[2577]: I1030 19:15:08.538070    2577 flags.go:64] FLAG: --protect-kernel-defaults="false"
Oct 30 19:15:08.543380 ip-10-0-132-82 kubenswrapper[2577]: I1030 19:15:08.538073    2577 flags.go:64] FLAG: --provider-id="aws:///us-west-1a/i-0035bfd3429370984"
Oct 30 19:15:08.543380 ip-10-0-132-82 kubenswrapper[2577]: I1030 19:15:08.538077    2577 flags.go:64] FLAG: --qos-reserved=""
Oct 30 19:15:08.543380 ip-10-0-132-82 kubenswrapper[2577]: I1030 19:15:08.538080    2577 flags.go:64] FLAG: --read-only-port="10255"
Oct 30 19:15:08.543380 ip-10-0-132-82 kubenswrapper[2577]: I1030 19:15:08.538083    2577 flags.go:64] FLAG: --register-node="true"
Oct 30 19:15:08.543380 ip-10-0-132-82 kubenswrapper[2577]: I1030 19:15:08.538086    2577 flags.go:64] FLAG: --register-with-taints=""
Oct 30 19:15:08.543380 ip-10-0-132-82 kubenswrapper[2577]: I1030 19:15:08.538090    2577 flags.go:64] FLAG: --registry-burst="10"
Oct 30 19:15:08.543380 ip-10-0-132-82 kubenswrapper[2577]: I1030 19:15:08.538093    2577 flags.go:64] FLAG: --registry-qps="5"
Oct 30 19:15:08.543380 ip-10-0-132-82 kubenswrapper[2577]: I1030 19:15:08.538095    2577 flags.go:64] FLAG: --reserved-cpus=""
Oct 30 19:15:08.543380 ip-10-0-132-82 kubenswrapper[2577]: I1030 19:15:08.538098    2577 flags.go:64] FLAG: --reserved-memory=""
Oct 30 19:15:08.543380 ip-10-0-132-82 kubenswrapper[2577]: I1030 19:15:08.538102    2577 flags.go:64] FLAG: --resolv-conf="/etc/resolv.conf"
Oct 30 19:15:08.543380 ip-10-0-132-82 kubenswrapper[2577]: I1030 19:15:08.538105    2577 flags.go:64] FLAG: --root-dir="/var/lib/kubelet"
Oct 30 19:15:08.543380 ip-10-0-132-82 kubenswrapper[2577]: I1030 19:15:08.538108    2577 flags.go:64] FLAG: --rotate-certificates="false"
Oct 30 19:15:08.543380 ip-10-0-132-82 kubenswrapper[2577]: I1030 19:15:08.538110    2577 flags.go:64] FLAG: --rotate-server-certificates="false"
Oct 30 19:15:08.543380 ip-10-0-132-82 kubenswrapper[2577]: I1030 19:15:08.538113    2577 flags.go:64] FLAG: --runonce="false"
Oct 30 19:15:08.543380 ip-10-0-132-82 kubenswrapper[2577]: I1030 19:15:08.538116    2577 flags.go:64] FLAG: --runtime-cgroups="/system.slice/crio.service"
Oct 30 19:15:08.543380 ip-10-0-132-82 kubenswrapper[2577]: I1030 19:15:08.538119    2577 flags.go:64] FLAG: --runtime-request-timeout="2m0s"
Oct 30 19:15:08.543380 ip-10-0-132-82 kubenswrapper[2577]: I1030 19:15:08.538122    2577 flags.go:64] FLAG: --seccomp-default="false"
Oct 30 19:15:08.543380 ip-10-0-132-82 kubenswrapper[2577]: I1030 19:15:08.538125    2577 flags.go:64] FLAG: --serialize-image-pulls="true"
Oct 30 19:15:08.544039 ip-10-0-132-82 kubenswrapper[2577]: I1030 19:15:08.538128    2577 flags.go:64] FLAG: --storage-driver-buffer-duration="1m0s"
Oct 30 19:15:08.544039 ip-10-0-132-82 kubenswrapper[2577]: I1030 19:15:08.538132    2577 flags.go:64] FLAG: --storage-driver-db="cadvisor"
Oct 30 19:15:08.544039 ip-10-0-132-82 kubenswrapper[2577]: I1030 19:15:08.538135    2577 flags.go:64] FLAG: --storage-driver-host="localhost:8086"
Oct 30 19:15:08.544039 ip-10-0-132-82 kubenswrapper[2577]: I1030 19:15:08.538138    2577 flags.go:64] FLAG: --storage-driver-password="root"
Oct 30 19:15:08.544039 ip-10-0-132-82 kubenswrapper[2577]: I1030 19:15:08.538141    2577 flags.go:64] FLAG: --storage-driver-secure="false"
Oct 30 19:15:08.544039 ip-10-0-132-82 kubenswrapper[2577]: I1030 19:15:08.538143    2577 flags.go:64] FLAG: --storage-driver-table="stats"
Oct 30 19:15:08.544039 ip-10-0-132-82 kubenswrapper[2577]: I1030 19:15:08.538146    2577 flags.go:64] FLAG: --storage-driver-user="root"
Oct 30 19:15:08.544039 ip-10-0-132-82 kubenswrapper[2577]: I1030 19:15:08.538149    2577 flags.go:64] FLAG: --streaming-connection-idle-timeout="4h0m0s"
Oct 30 19:15:08.544039 ip-10-0-132-82 kubenswrapper[2577]: I1030 19:15:08.538152    2577 flags.go:64] FLAG: --sync-frequency="1m0s"
Oct 30 19:15:08.544039 ip-10-0-132-82 kubenswrapper[2577]: I1030 19:15:08.538155    2577 flags.go:64] FLAG: --system-cgroups=""
Oct 30 19:15:08.544039 ip-10-0-132-82 kubenswrapper[2577]: I1030 19:15:08.538157    2577 flags.go:64] FLAG: --system-reserved="cpu=500m,ephemeral-storage=1Gi,memory=1Gi"
Oct 30 19:15:08.544039 ip-10-0-132-82 kubenswrapper[2577]: I1030 19:15:08.538163    2577 flags.go:64] FLAG: --system-reserved-cgroup=""
Oct 30 19:15:08.544039 ip-10-0-132-82 kubenswrapper[2577]: I1030 19:15:08.538165    2577 flags.go:64] FLAG: --tls-cert-file=""
Oct 30 19:15:08.544039 ip-10-0-132-82 kubenswrapper[2577]: I1030 19:15:08.538169    2577 flags.go:64] FLAG: --tls-cipher-suites="[]"
Oct 30 19:15:08.544039 ip-10-0-132-82 kubenswrapper[2577]: I1030 19:15:08.538173    2577 flags.go:64] FLAG: --tls-min-version=""
Oct 30 19:15:08.544039 ip-10-0-132-82 kubenswrapper[2577]: I1030 19:15:08.538177    2577 flags.go:64] FLAG: --tls-private-key-file=""
Oct 30 19:15:08.544039 ip-10-0-132-82 kubenswrapper[2577]: I1030 19:15:08.538180    2577 flags.go:64] FLAG: --topology-manager-policy="none"
Oct 30 19:15:08.544039 ip-10-0-132-82 kubenswrapper[2577]: I1030 19:15:08.538183    2577 flags.go:64] FLAG: --topology-manager-policy-options=""
Oct 30 19:15:08.544039 ip-10-0-132-82 kubenswrapper[2577]: I1030 19:15:08.538186    2577 flags.go:64] FLAG: --topology-manager-scope="container"
Oct 30 19:15:08.544039 ip-10-0-132-82 kubenswrapper[2577]: I1030 19:15:08.538189    2577 flags.go:64] FLAG: --v="2"
Oct 30 19:15:08.544039 ip-10-0-132-82 kubenswrapper[2577]: I1030 19:15:08.538193    2577 flags.go:64] FLAG: --version="false"
Oct 30 19:15:08.544039 ip-10-0-132-82 kubenswrapper[2577]: I1030 19:15:08.538197    2577 flags.go:64] FLAG: --vmodule=""
Oct 30 19:15:08.544039 ip-10-0-132-82 kubenswrapper[2577]: I1030 19:15:08.538201    2577 flags.go:64] FLAG: --volume-plugin-dir="/etc/kubernetes/kubelet-plugins/volume/exec"
Oct 30 19:15:08.544039 ip-10-0-132-82 kubenswrapper[2577]: I1030 19:15:08.538205    2577 flags.go:64] FLAG: --volume-stats-agg-period="1m0s"
Oct 30 19:15:08.544613 ip-10-0-132-82 kubenswrapper[2577]: W1030 19:15:08.538340    2577 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Oct 30 19:15:08.544613 ip-10-0-132-82 kubenswrapper[2577]: W1030 19:15:08.538344    2577 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Oct 30 19:15:08.544613 ip-10-0-132-82 kubenswrapper[2577]: W1030 19:15:08.538347    2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Oct 30 19:15:08.544613 ip-10-0-132-82 kubenswrapper[2577]: W1030 19:15:08.538350    2577 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Oct 30 19:15:08.544613 ip-10-0-132-82 kubenswrapper[2577]: W1030 19:15:08.538353    2577 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Oct 30 19:15:08.544613 ip-10-0-132-82 kubenswrapper[2577]: W1030 19:15:08.538356    2577 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Oct 30 19:15:08.544613 ip-10-0-132-82 kubenswrapper[2577]: W1030 19:15:08.538358    2577 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Oct 30 19:15:08.544613 ip-10-0-132-82 kubenswrapper[2577]: W1030 19:15:08.538362    2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Oct 30 19:15:08.544613 ip-10-0-132-82 kubenswrapper[2577]: W1030 19:15:08.538365    2577 feature_gate.go:386] unrecognized feature gate: NewOLM
Oct 30 19:15:08.544613 ip-10-0-132-82 kubenswrapper[2577]: W1030 19:15:08.538368    2577 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Oct 30 19:15:08.544613 ip-10-0-132-82 kubenswrapper[2577]: W1030 19:15:08.538371    2577 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Oct 30 19:15:08.544613 ip-10-0-132-82 kubenswrapper[2577]: W1030 19:15:08.538373    2577 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Oct 30 19:15:08.544613 ip-10-0-132-82 kubenswrapper[2577]: W1030 19:15:08.538376    2577 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Oct 30 19:15:08.544613 ip-10-0-132-82 kubenswrapper[2577]: W1030 19:15:08.538378    2577 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Oct 30 19:15:08.544613 ip-10-0-132-82 kubenswrapper[2577]: W1030 19:15:08.538381    2577 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Oct 30 19:15:08.544613 ip-10-0-132-82 kubenswrapper[2577]: W1030 19:15:08.538383    2577 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Oct 30 19:15:08.544613 ip-10-0-132-82 kubenswrapper[2577]: W1030 19:15:08.538386    2577 feature_gate.go:386] unrecognized feature gate: DualReplica
Oct 30 19:15:08.544613 ip-10-0-132-82 kubenswrapper[2577]: W1030 19:15:08.538388    2577 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Oct 30 19:15:08.544613 ip-10-0-132-82 kubenswrapper[2577]: W1030 19:15:08.538391    2577 feature_gate.go:386] unrecognized feature gate: OSStreams
Oct 30 19:15:08.544613 ip-10-0-132-82 kubenswrapper[2577]: W1030 19:15:08.538393    2577 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Oct 30 19:15:08.545095 ip-10-0-132-82 kubenswrapper[2577]: W1030 19:15:08.538396    2577 feature_gate.go:386] unrecognized feature gate: EVPN
Oct 30 19:15:08.545095 ip-10-0-132-82 kubenswrapper[2577]: W1030 19:15:08.538398    2577 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Oct 30 19:15:08.545095 ip-10-0-132-82 kubenswrapper[2577]: W1030 19:15:08.538401    2577 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Oct 30 19:15:08.545095 ip-10-0-132-82 kubenswrapper[2577]: W1030 19:15:08.538404    2577 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Oct 30 19:15:08.545095 ip-10-0-132-82 kubenswrapper[2577]: W1030 19:15:08.538407    2577 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Oct 30 19:15:08.545095 ip-10-0-132-82 kubenswrapper[2577]: W1030 19:15:08.538409    2577 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Oct 30 19:15:08.545095 ip-10-0-132-82 kubenswrapper[2577]: W1030 19:15:08.538412    2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Oct 30 19:15:08.545095 ip-10-0-132-82 kubenswrapper[2577]: W1030 19:15:08.538414    2577 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Oct 30 19:15:08.545095 ip-10-0-132-82 kubenswrapper[2577]: W1030 19:15:08.538417    2577 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Oct 30 19:15:08.545095 ip-10-0-132-82 kubenswrapper[2577]: W1030 19:15:08.538420    2577 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Oct 30 19:15:08.545095 ip-10-0-132-82 kubenswrapper[2577]: W1030 19:15:08.538422    2577 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Oct 30 19:15:08.545095 ip-10-0-132-82 kubenswrapper[2577]: W1030 19:15:08.538425    2577 feature_gate.go:386] unrecognized feature gate: OVNObservability
Oct 30 19:15:08.545095 ip-10-0-132-82 kubenswrapper[2577]: W1030 19:15:08.538427    2577 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Oct 30 19:15:08.545095 ip-10-0-132-82 kubenswrapper[2577]: W1030 19:15:08.538430    2577 feature_gate.go:386] unrecognized feature gate: EventTTL
Oct 30 19:15:08.545095 ip-10-0-132-82 kubenswrapper[2577]: W1030 19:15:08.538432    2577 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Oct 30 19:15:08.545095 ip-10-0-132-82 kubenswrapper[2577]: W1030 19:15:08.538434    2577 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Oct 30 19:15:08.545095 ip-10-0-132-82 kubenswrapper[2577]: W1030 19:15:08.538437    2577 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Oct 30 19:15:08.545095 ip-10-0-132-82 kubenswrapper[2577]: W1030 19:15:08.538439    2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Oct 30 19:15:08.545095 ip-10-0-132-82 kubenswrapper[2577]: W1030 19:15:08.538442    2577 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Oct 30 19:15:08.545095 ip-10-0-132-82 kubenswrapper[2577]: W1030 19:15:08.538446    2577 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Oct 30 19:15:08.545564 ip-10-0-132-82 kubenswrapper[2577]: W1030 19:15:08.538449    2577 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Oct 30 19:15:08.545564 ip-10-0-132-82 kubenswrapper[2577]: W1030 19:15:08.538451    2577 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Oct 30 19:15:08.545564 ip-10-0-132-82 kubenswrapper[2577]: W1030 19:15:08.538454    2577 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Oct 30 19:15:08.545564 ip-10-0-132-82 kubenswrapper[2577]: W1030 19:15:08.538458    2577 feature_gate.go:386] unrecognized feature gate: Example2
Oct 30 19:15:08.545564 ip-10-0-132-82 kubenswrapper[2577]: W1030 19:15:08.538460    2577 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Oct 30 19:15:08.545564 ip-10-0-132-82 kubenswrapper[2577]: W1030 19:15:08.538463    2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Oct 30 19:15:08.545564 ip-10-0-132-82 kubenswrapper[2577]: W1030 19:15:08.538465    2577 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Oct 30 19:15:08.545564 ip-10-0-132-82 kubenswrapper[2577]: W1030 19:15:08.538468    2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Oct 30 19:15:08.545564 ip-10-0-132-82 kubenswrapper[2577]: W1030 19:15:08.538470    2577 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Oct 30 19:15:08.545564 ip-10-0-132-82 kubenswrapper[2577]: W1030 19:15:08.538473    2577 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Oct 30 19:15:08.545564 ip-10-0-132-82 kubenswrapper[2577]: W1030 19:15:08.538475    2577 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Oct 30 19:15:08.545564 ip-10-0-132-82 kubenswrapper[2577]: W1030 19:15:08.538478    2577 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Oct 30 19:15:08.545564 ip-10-0-132-82 kubenswrapper[2577]: W1030 19:15:08.538480    2577 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Oct 30 19:15:08.545564 ip-10-0-132-82 kubenswrapper[2577]: W1030 19:15:08.538483    2577 feature_gate.go:386] unrecognized feature gate: SignatureStores
Oct 30 19:15:08.545564 ip-10-0-132-82 kubenswrapper[2577]: W1030 19:15:08.538485    2577 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Oct 30 19:15:08.545564 ip-10-0-132-82 kubenswrapper[2577]: W1030 19:15:08.538489    2577 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Oct 30 19:15:08.545564 ip-10-0-132-82 kubenswrapper[2577]: W1030 19:15:08.538492    2577 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Oct 30 19:15:08.545564 ip-10-0-132-82 kubenswrapper[2577]: W1030 19:15:08.538494    2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Oct 30 19:15:08.545564 ip-10-0-132-82 kubenswrapper[2577]: W1030 19:15:08.538497    2577 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Oct 30 19:15:08.546025 ip-10-0-132-82 kubenswrapper[2577]: W1030 19:15:08.538499    2577 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Oct 30 19:15:08.546025 ip-10-0-132-82 kubenswrapper[2577]: W1030 19:15:08.538502    2577 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Oct 30 19:15:08.546025 ip-10-0-132-82 kubenswrapper[2577]: W1030 19:15:08.538504    2577 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Oct 30 19:15:08.546025 ip-10-0-132-82 kubenswrapper[2577]: W1030 19:15:08.538507    2577 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Oct 30 19:15:08.546025 ip-10-0-132-82 kubenswrapper[2577]: W1030 19:15:08.538509    2577 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Oct 30 19:15:08.546025 ip-10-0-132-82 kubenswrapper[2577]: W1030 19:15:08.538512    2577 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Oct 30 19:15:08.546025 ip-10-0-132-82 kubenswrapper[2577]: W1030 19:15:08.538514    2577 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Oct 30 19:15:08.546025 ip-10-0-132-82 kubenswrapper[2577]: W1030 19:15:08.538517    2577 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Oct 30 19:15:08.546025 ip-10-0-132-82 kubenswrapper[2577]: W1030 19:15:08.538520    2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Oct 30 19:15:08.546025 ip-10-0-132-82 kubenswrapper[2577]: W1030 19:15:08.538522    2577 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Oct 30 19:15:08.546025 ip-10-0-132-82 kubenswrapper[2577]: W1030 19:15:08.538525    2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Oct 30 19:15:08.546025 ip-10-0-132-82 kubenswrapper[2577]: W1030 19:15:08.538527    2577 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Oct 30 19:15:08.546025 ip-10-0-132-82 kubenswrapper[2577]: W1030 19:15:08.538531    2577 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Oct 30 19:15:08.546025 ip-10-0-132-82 kubenswrapper[2577]: W1030 19:15:08.538534    2577 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Oct 30 19:15:08.546025 ip-10-0-132-82 kubenswrapper[2577]: W1030 19:15:08.538536    2577 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Oct 30 19:15:08.546025 ip-10-0-132-82 kubenswrapper[2577]: W1030 19:15:08.538538    2577 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Oct 30 19:15:08.546025 ip-10-0-132-82 kubenswrapper[2577]: W1030 19:15:08.538541    2577 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Oct 30 19:15:08.546025 ip-10-0-132-82 kubenswrapper[2577]: W1030 19:15:08.538544    2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Oct 30 19:15:08.546025 ip-10-0-132-82 kubenswrapper[2577]: W1030 19:15:08.538547    2577 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Oct 30 19:15:08.546025 ip-10-0-132-82 kubenswrapper[2577]: W1030 19:15:08.538550    2577 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Oct 30 19:15:08.546507 ip-10-0-132-82 kubenswrapper[2577]: W1030 19:15:08.538552    2577 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Oct 30 19:15:08.546507 ip-10-0-132-82 kubenswrapper[2577]: W1030 19:15:08.538554    2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Oct 30 19:15:08.546507 ip-10-0-132-82 kubenswrapper[2577]: W1030 19:15:08.538557    2577 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Oct 30 19:15:08.546507 ip-10-0-132-82 kubenswrapper[2577]: W1030 19:15:08.538559    2577 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Oct 30 19:15:08.546507 ip-10-0-132-82 kubenswrapper[2577]: W1030 19:15:08.538562    2577 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Oct 30 19:15:08.546507 ip-10-0-132-82 kubenswrapper[2577]: W1030 19:15:08.538564    2577 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Oct 30 19:15:08.546507 ip-10-0-132-82 kubenswrapper[2577]: W1030 19:15:08.538568    2577 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Oct 30 19:15:08.546507 ip-10-0-132-82 kubenswrapper[2577]: W1030 19:15:08.538572    2577 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Oct 30 19:15:08.546507 ip-10-0-132-82 kubenswrapper[2577]: W1030 19:15:08.538576    2577 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Oct 30 19:15:08.546507 ip-10-0-132-82 kubenswrapper[2577]: W1030 19:15:08.538579    2577 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Oct 30 19:15:08.546507 ip-10-0-132-82 kubenswrapper[2577]: W1030 19:15:08.538582    2577 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Oct 30 19:15:08.546507 ip-10-0-132-82 kubenswrapper[2577]: W1030 19:15:08.538585    2577 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Oct 30 19:15:08.546507 ip-10-0-132-82 kubenswrapper[2577]: W1030 19:15:08.538587    2577 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Oct 30 19:15:08.546507 ip-10-0-132-82 kubenswrapper[2577]: W1030 19:15:08.538590    2577 feature_gate.go:386] unrecognized feature gate: Example
Oct 30 19:15:08.546507 ip-10-0-132-82 kubenswrapper[2577]: W1030 19:15:08.538592    2577 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Oct 30 19:15:08.546507 ip-10-0-132-82 kubenswrapper[2577]: W1030 19:15:08.538595    2577 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Oct 30 19:15:08.546507 ip-10-0-132-82 kubenswrapper[2577]: W1030 19:15:08.538597    2577 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Oct 30 19:15:08.546507 ip-10-0-132-82 kubenswrapper[2577]: W1030 19:15:08.538600    2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Oct 30 19:15:08.546507 ip-10-0-132-82 kubenswrapper[2577]: W1030 19:15:08.538602    2577 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Oct 30 19:15:08.546963 ip-10-0-132-82 kubenswrapper[2577]: W1030 19:15:08.538605    2577 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Oct 30 19:15:08.546963 ip-10-0-132-82 kubenswrapper[2577]: W1030 19:15:08.538609    2577 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Oct 30 19:15:08.546963 ip-10-0-132-82 kubenswrapper[2577]: W1030 19:15:08.538612    2577 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Oct 30 19:15:08.546963 ip-10-0-132-82 kubenswrapper[2577]: W1030 19:15:08.538615    2577 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Oct 30 19:15:08.546963 ip-10-0-132-82 kubenswrapper[2577]: W1030 19:15:08.538617    2577 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Oct 30 19:15:08.546963 ip-10-0-132-82 kubenswrapper[2577]: W1030 19:15:08.538621    2577 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Oct 30 19:15:08.546963 ip-10-0-132-82 kubenswrapper[2577]: W1030 19:15:08.538624    2577 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Oct 30 19:15:08.546963 ip-10-0-132-82 kubenswrapper[2577]: W1030 19:15:08.538626    2577 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Oct 30 19:15:08.546963 ip-10-0-132-82 kubenswrapper[2577]: I1030 19:15:08.538652    2577 feature_gate.go:465] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false DRAPartitionableDevices:false EventedPLEG:false KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:true MutatingAdmissionPolicy:false RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true UserNamespacesSupport:true]}
Oct 30 19:15:08.546963 ip-10-0-132-82 kubenswrapper[2577]: W1030 19:15:08.538695    2577 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Oct 30 19:15:08.546963 ip-10-0-132-82 kubenswrapper[2577]: W1030 19:15:08.538700    2577 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Oct 30 19:15:08.546963 ip-10-0-132-82 kubenswrapper[2577]: W1030 19:15:08.538704    2577 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Oct 30 19:15:08.546963 ip-10-0-132-82 kubenswrapper[2577]: W1030 19:15:08.538706    2577 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Oct 30 19:15:08.546963 ip-10-0-132-82 kubenswrapper[2577]: W1030 19:15:08.538709    2577 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Oct 30 19:15:08.546963 ip-10-0-132-82 kubenswrapper[2577]: W1030 19:15:08.538712    2577 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Oct 30 19:15:08.547326 ip-10-0-132-82 kubenswrapper[2577]: W1030 19:15:08.538715    2577 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Oct 30 19:15:08.547326 ip-10-0-132-82 kubenswrapper[2577]: W1030 19:15:08.538717    2577 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Oct 30 19:15:08.547326 ip-10-0-132-82 kubenswrapper[2577]: W1030 19:15:08.538720    2577 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Oct 30 19:15:08.547326 ip-10-0-132-82 kubenswrapper[2577]: W1030 19:15:08.538722    2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Oct 30 19:15:08.547326 ip-10-0-132-82 kubenswrapper[2577]: W1030 19:15:08.538725    2577 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Oct 30 19:15:08.547326 ip-10-0-132-82 kubenswrapper[2577]: W1030 19:15:08.538727    2577 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Oct 30 19:15:08.547326 ip-10-0-132-82 kubenswrapper[2577]: W1030 19:15:08.538730    2577 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Oct 30 19:15:08.547326 ip-10-0-132-82 kubenswrapper[2577]: W1030 19:15:08.538733    2577 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Oct 30 19:15:08.547326 ip-10-0-132-82 kubenswrapper[2577]: W1030 19:15:08.538735    2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Oct 30 19:15:08.547326 ip-10-0-132-82 kubenswrapper[2577]: W1030 19:15:08.538738    2577 feature_gate.go:386] unrecognized feature gate: NewOLM
Oct 30 19:15:08.547326 ip-10-0-132-82 kubenswrapper[2577]: W1030 19:15:08.538740    2577 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Oct 30 19:15:08.547326 ip-10-0-132-82 kubenswrapper[2577]: W1030 19:15:08.538743    2577 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Oct 30 19:15:08.547326 ip-10-0-132-82 kubenswrapper[2577]: W1030 19:15:08.538745    2577 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Oct 30 19:15:08.547326 ip-10-0-132-82 kubenswrapper[2577]: W1030 19:15:08.538748    2577 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Oct 30 19:15:08.547326 ip-10-0-132-82 kubenswrapper[2577]: W1030 19:15:08.538750    2577 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Oct 30 19:15:08.547326 ip-10-0-132-82 kubenswrapper[2577]: W1030 19:15:08.538753    2577 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Oct 30 19:15:08.547326 ip-10-0-132-82 kubenswrapper[2577]: W1030 19:15:08.538755    2577 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Oct 30 19:15:08.547326 ip-10-0-132-82 kubenswrapper[2577]: W1030 19:15:08.538757    2577 feature_gate.go:386] unrecognized feature gate: DualReplica
Oct 30 19:15:08.547326 ip-10-0-132-82 kubenswrapper[2577]: W1030 19:15:08.538760    2577 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Oct 30 19:15:08.547326 ip-10-0-132-82 kubenswrapper[2577]: W1030 19:15:08.538763    2577 feature_gate.go:386] unrecognized feature gate: OSStreams
Oct 30 19:15:08.547792 ip-10-0-132-82 kubenswrapper[2577]: W1030 19:15:08.538765    2577 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Oct 30 19:15:08.547792 ip-10-0-132-82 kubenswrapper[2577]: W1030 19:15:08.538767    2577 feature_gate.go:386] unrecognized feature gate: EVPN
Oct 30 19:15:08.547792 ip-10-0-132-82 kubenswrapper[2577]: W1030 19:15:08.538770    2577 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Oct 30 19:15:08.547792 ip-10-0-132-82 kubenswrapper[2577]: W1030 19:15:08.538773    2577 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Oct 30 19:15:08.547792 ip-10-0-132-82 kubenswrapper[2577]: W1030 19:15:08.538776    2577 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Oct 30 19:15:08.547792 ip-10-0-132-82 kubenswrapper[2577]: W1030 19:15:08.538778    2577 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Oct 30 19:15:08.547792 ip-10-0-132-82 kubenswrapper[2577]: W1030 19:15:08.538781    2577 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Oct 30 19:15:08.547792 ip-10-0-132-82 kubenswrapper[2577]: W1030 19:15:08.538784    2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Oct 30 19:15:08.547792 ip-10-0-132-82 kubenswrapper[2577]: W1030 19:15:08.538787    2577 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Oct 30 19:15:08.547792 ip-10-0-132-82 kubenswrapper[2577]: W1030 19:15:08.538790    2577 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Oct 30 19:15:08.547792 ip-10-0-132-82 kubenswrapper[2577]: W1030 19:15:08.538793    2577 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Oct 30 19:15:08.547792 ip-10-0-132-82 kubenswrapper[2577]: W1030 19:15:08.538796    2577 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Oct 30 19:15:08.547792 ip-10-0-132-82 kubenswrapper[2577]: W1030 19:15:08.538798    2577 feature_gate.go:386] unrecognized feature gate: OVNObservability
Oct 30 19:15:08.547792 ip-10-0-132-82 kubenswrapper[2577]: W1030 19:15:08.538801    2577 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Oct 30 19:15:08.547792 ip-10-0-132-82 kubenswrapper[2577]: W1030 19:15:08.538803    2577 feature_gate.go:386] unrecognized feature gate: EventTTL
Oct 30 19:15:08.547792 ip-10-0-132-82 kubenswrapper[2577]: W1030 19:15:08.538806    2577 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Oct 30 19:15:08.547792 ip-10-0-132-82 kubenswrapper[2577]: W1030 19:15:08.538809    2577 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Oct 30 19:15:08.547792 ip-10-0-132-82 kubenswrapper[2577]: W1030 19:15:08.538811    2577 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Oct 30 19:15:08.547792 ip-10-0-132-82 kubenswrapper[2577]: W1030 19:15:08.538814    2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Oct 30 19:15:08.547792 ip-10-0-132-82 kubenswrapper[2577]: W1030 19:15:08.538816    2577 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Oct 30 19:15:08.548279 ip-10-0-132-82 kubenswrapper[2577]: W1030 19:15:08.538819    2577 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Oct 30 19:15:08.548279 ip-10-0-132-82 kubenswrapper[2577]: W1030 19:15:08.538821    2577 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Oct 30 19:15:08.548279 ip-10-0-132-82 kubenswrapper[2577]: W1030 19:15:08.538824    2577 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Oct 30 19:15:08.548279 ip-10-0-132-82 kubenswrapper[2577]: W1030 19:15:08.538826    2577 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Oct 30 19:15:08.548279 ip-10-0-132-82 kubenswrapper[2577]: W1030 19:15:08.538828    2577 feature_gate.go:386] unrecognized feature gate: Example2
Oct 30 19:15:08.548279 ip-10-0-132-82 kubenswrapper[2577]: W1030 19:15:08.538831    2577 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Oct 30 19:15:08.548279 ip-10-0-132-82 kubenswrapper[2577]: W1030 19:15:08.538833    2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Oct 30 19:15:08.548279 ip-10-0-132-82 kubenswrapper[2577]: W1030 19:15:08.538836    2577 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Oct 30 19:15:08.548279 ip-10-0-132-82 kubenswrapper[2577]: W1030 19:15:08.538838    2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Oct 30 19:15:08.548279 ip-10-0-132-82 kubenswrapper[2577]: W1030 19:15:08.538840    2577 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Oct 30 19:15:08.548279 ip-10-0-132-82 kubenswrapper[2577]: W1030 19:15:08.538843    2577 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Oct 30 19:15:08.548279 ip-10-0-132-82 kubenswrapper[2577]: W1030 19:15:08.538845    2577 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Oct 30 19:15:08.548279 ip-10-0-132-82 kubenswrapper[2577]: W1030 19:15:08.538848    2577 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Oct 30 19:15:08.548279 ip-10-0-132-82 kubenswrapper[2577]: W1030 19:15:08.538850    2577 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Oct 30 19:15:08.548279 ip-10-0-132-82 kubenswrapper[2577]: W1030 19:15:08.538853    2577 feature_gate.go:386] unrecognized feature gate: SignatureStores
Oct 30 19:15:08.548279 ip-10-0-132-82 kubenswrapper[2577]: W1030 19:15:08.538856    2577 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Oct 30 19:15:08.548279 ip-10-0-132-82 kubenswrapper[2577]: W1030 19:15:08.538859    2577 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Oct 30 19:15:08.548279 ip-10-0-132-82 kubenswrapper[2577]: W1030 19:15:08.538862    2577 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Oct 30 19:15:08.548279 ip-10-0-132-82 kubenswrapper[2577]: W1030 19:15:08.538864    2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Oct 30 19:15:08.548715 ip-10-0-132-82 kubenswrapper[2577]: W1030 19:15:08.538867    2577 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Oct 30 19:15:08.548715 ip-10-0-132-82 kubenswrapper[2577]: W1030 19:15:08.538870    2577 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Oct 30 19:15:08.548715 ip-10-0-132-82 kubenswrapper[2577]: W1030 19:15:08.538872    2577 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Oct 30 19:15:08.548715 ip-10-0-132-82 kubenswrapper[2577]: W1030 19:15:08.538875    2577 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Oct 30 19:15:08.548715 ip-10-0-132-82 kubenswrapper[2577]: W1030 19:15:08.538878    2577 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Oct 30 19:15:08.548715 ip-10-0-132-82 kubenswrapper[2577]: W1030 19:15:08.538880    2577 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Oct 30 19:15:08.548715 ip-10-0-132-82 kubenswrapper[2577]: W1030 19:15:08.538883    2577 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Oct 30 19:15:08.548715 ip-10-0-132-82 kubenswrapper[2577]: W1030 19:15:08.538885    2577 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Oct 30 19:15:08.548715 ip-10-0-132-82 kubenswrapper[2577]: W1030 19:15:08.538889    2577 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Oct 30 19:15:08.548715 ip-10-0-132-82 kubenswrapper[2577]: W1030 19:15:08.538891    2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Oct 30 19:15:08.548715 ip-10-0-132-82 kubenswrapper[2577]: W1030 19:15:08.538894    2577 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Oct 30 19:15:08.548715 ip-10-0-132-82 kubenswrapper[2577]: W1030 19:15:08.538896    2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Oct 30 19:15:08.548715 ip-10-0-132-82 kubenswrapper[2577]: W1030 19:15:08.538899    2577 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Oct 30 19:15:08.548715 ip-10-0-132-82 kubenswrapper[2577]: W1030 19:15:08.538901    2577 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Oct 30 19:15:08.548715 ip-10-0-132-82 kubenswrapper[2577]: W1030 19:15:08.538904    2577 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Oct 30 19:15:08.548715 ip-10-0-132-82 kubenswrapper[2577]: W1030 19:15:08.538906    2577 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Oct 30 19:15:08.548715 ip-10-0-132-82 kubenswrapper[2577]: W1030 19:15:08.538909    2577 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Oct 30 19:15:08.548715 ip-10-0-132-82 kubenswrapper[2577]: W1030 19:15:08.538911    2577 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Oct 30 19:15:08.548715 ip-10-0-132-82 kubenswrapper[2577]: W1030 19:15:08.538914    2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Oct 30 19:15:08.548715 ip-10-0-132-82 kubenswrapper[2577]: W1030 19:15:08.538916    2577 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Oct 30 19:15:08.549299 ip-10-0-132-82 kubenswrapper[2577]: W1030 19:15:08.538918    2577 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Oct 30 19:15:08.549299 ip-10-0-132-82 kubenswrapper[2577]: W1030 19:15:08.538934    2577 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Oct 30 19:15:08.549299 ip-10-0-132-82 kubenswrapper[2577]: W1030 19:15:08.538937    2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Oct 30 19:15:08.549299 ip-10-0-132-82 kubenswrapper[2577]: W1030 19:15:08.538939    2577 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Oct 30 19:15:08.549299 ip-10-0-132-82 kubenswrapper[2577]: W1030 19:15:08.538942    2577 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Oct 30 19:15:08.549299 ip-10-0-132-82 kubenswrapper[2577]: W1030 19:15:08.538944    2577 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Oct 30 19:15:08.549299 ip-10-0-132-82 kubenswrapper[2577]: W1030 19:15:08.538947    2577 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Oct 30 19:15:08.549299 ip-10-0-132-82 kubenswrapper[2577]: W1030 19:15:08.538951    2577 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Oct 30 19:15:08.549299 ip-10-0-132-82 kubenswrapper[2577]: W1030 19:15:08.538958    2577 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Oct 30 19:15:08.549299 ip-10-0-132-82 kubenswrapper[2577]: W1030 19:15:08.538961    2577 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Oct 30 19:15:08.549299 ip-10-0-132-82 kubenswrapper[2577]: W1030 19:15:08.538964    2577 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Oct 30 19:15:08.549299 ip-10-0-132-82 kubenswrapper[2577]: W1030 19:15:08.538967    2577 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Oct 30 19:15:08.549299 ip-10-0-132-82 kubenswrapper[2577]: W1030 19:15:08.538970    2577 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Oct 30 19:15:08.549299 ip-10-0-132-82 kubenswrapper[2577]: W1030 19:15:08.538972    2577 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Oct 30 19:15:08.549299 ip-10-0-132-82 kubenswrapper[2577]: W1030 19:15:08.538975    2577 feature_gate.go:386] unrecognized feature gate: Example
Oct 30 19:15:08.549299 ip-10-0-132-82 kubenswrapper[2577]: W1030 19:15:08.538977    2577 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Oct 30 19:15:08.549299 ip-10-0-132-82 kubenswrapper[2577]: W1030 19:15:08.538984    2577 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Oct 30 19:15:08.549299 ip-10-0-132-82 kubenswrapper[2577]: W1030 19:15:08.538988    2577 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Oct 30 19:15:08.549299 ip-10-0-132-82 kubenswrapper[2577]: W1030 19:15:08.538990    2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Oct 30 19:15:08.549722 ip-10-0-132-82 kubenswrapper[2577]: W1030 19:15:08.538993    2577 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Oct 30 19:15:08.549722 ip-10-0-132-82 kubenswrapper[2577]: W1030 19:15:08.538995    2577 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Oct 30 19:15:08.549722 ip-10-0-132-82 kubenswrapper[2577]: I1030 19:15:08.545794    2577 server.go:525] "Kubelet version" kubeletVersion="v1.35.9"
Oct 30 19:15:08.549722 ip-10-0-132-82 kubenswrapper[2577]: I1030 19:15:08.545807    2577 server.go:527] "Golang settings" GOGC="" GOMAXPROCS="" GOTRACEBACK=""
Oct 30 19:15:08.549722 ip-10-0-132-82 kubenswrapper[2577]: I1030 19:15:08.546480    2577 watchdog_linux.go:95] "Systemd watchdog is not enabled"
Oct 30 19:15:08.549722 ip-10-0-132-82 kubenswrapper[2577]: I1030 19:15:08.546486    2577 watchdog_linux.go:138] "Systemd watchdog is not enabled or the interval is invalid, so health checking will not be started."
Oct 30 19:15:08.549722 ip-10-0-132-82 kubenswrapper[2577]: W1030 19:15:08.546551    2577 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Oct 30 19:15:08.549722 ip-10-0-132-82 kubenswrapper[2577]: W1030 19:15:08.546556    2577 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Oct 30 19:15:08.549722 ip-10-0-132-82 kubenswrapper[2577]: W1030 19:15:08.546560    2577 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Oct 30 19:15:08.549722 ip-10-0-132-82 kubenswrapper[2577]: W1030 19:15:08.546563    2577 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Oct 30 19:15:08.549722 ip-10-0-132-82 kubenswrapper[2577]: W1030 19:15:08.546566    2577 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Oct 30 19:15:08.549722 ip-10-0-132-82 kubenswrapper[2577]: W1030 19:15:08.546569    2577 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Oct 30 19:15:08.549722 ip-10-0-132-82 kubenswrapper[2577]: W1030 19:15:08.546571    2577 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Oct 30 19:15:08.549722 ip-10-0-132-82 kubenswrapper[2577]: W1030 19:15:08.546574    2577 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Oct 30 19:15:08.549722 ip-10-0-132-82 kubenswrapper[2577]: W1030 19:15:08.546577    2577 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Oct 30 19:15:08.549722 ip-10-0-132-82 kubenswrapper[2577]: W1030 19:15:08.546579    2577 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Oct 30 19:15:08.549722 ip-10-0-132-82 kubenswrapper[2577]: W1030 19:15:08.546582    2577 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Oct 30 19:15:08.549722 ip-10-0-132-82 kubenswrapper[2577]: W1030 19:15:08.546584    2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Oct 30 19:15:08.549722 ip-10-0-132-82 kubenswrapper[2577]: W1030 19:15:08.546587    2577 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Oct 30 19:15:08.550179 ip-10-0-132-82 kubenswrapper[2577]: W1030 19:15:08.546589    2577 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Oct 30 19:15:08.550179 ip-10-0-132-82 kubenswrapper[2577]: W1030 19:15:08.546592    2577 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Oct 30 19:15:08.550179 ip-10-0-132-82 kubenswrapper[2577]: W1030 19:15:08.546594    2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Oct 30 19:15:08.550179 ip-10-0-132-82 kubenswrapper[2577]: W1030 19:15:08.546597    2577 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Oct 30 19:15:08.550179 ip-10-0-132-82 kubenswrapper[2577]: W1030 19:15:08.546600    2577 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Oct 30 19:15:08.550179 ip-10-0-132-82 kubenswrapper[2577]: W1030 19:15:08.546602    2577 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Oct 30 19:15:08.550179 ip-10-0-132-82 kubenswrapper[2577]: W1030 19:15:08.546605    2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Oct 30 19:15:08.550179 ip-10-0-132-82 kubenswrapper[2577]: W1030 19:15:08.546607    2577 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Oct 30 19:15:08.550179 ip-10-0-132-82 kubenswrapper[2577]: W1030 19:15:08.546610    2577 feature_gate.go:386] unrecognized feature gate: DualReplica
Oct 30 19:15:08.550179 ip-10-0-132-82 kubenswrapper[2577]: W1030 19:15:08.546612    2577 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Oct 30 19:15:08.550179 ip-10-0-132-82 kubenswrapper[2577]: W1030 19:15:08.546616    2577 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Oct 30 19:15:08.550179 ip-10-0-132-82 kubenswrapper[2577]: W1030 19:15:08.546618    2577 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Oct 30 19:15:08.550179 ip-10-0-132-82 kubenswrapper[2577]: W1030 19:15:08.546621    2577 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Oct 30 19:15:08.550179 ip-10-0-132-82 kubenswrapper[2577]: W1030 19:15:08.546623    2577 feature_gate.go:386] unrecognized feature gate: OSStreams
Oct 30 19:15:08.550179 ip-10-0-132-82 kubenswrapper[2577]: W1030 19:15:08.546626    2577 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Oct 30 19:15:08.550179 ip-10-0-132-82 kubenswrapper[2577]: W1030 19:15:08.546628    2577 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Oct 30 19:15:08.550179 ip-10-0-132-82 kubenswrapper[2577]: W1030 19:15:08.546631    2577 feature_gate.go:386] unrecognized feature gate: SignatureStores
Oct 30 19:15:08.550179 ip-10-0-132-82 kubenswrapper[2577]: W1030 19:15:08.546634    2577 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Oct 30 19:15:08.550179 ip-10-0-132-82 kubenswrapper[2577]: W1030 19:15:08.546637    2577 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Oct 30 19:15:08.550179 ip-10-0-132-82 kubenswrapper[2577]: W1030 19:15:08.546639    2577 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Oct 30 19:15:08.550635 ip-10-0-132-82 kubenswrapper[2577]: W1030 19:15:08.546642    2577 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Oct 30 19:15:08.550635 ip-10-0-132-82 kubenswrapper[2577]: W1030 19:15:08.546645    2577 feature_gate.go:386] unrecognized feature gate: EVPN
Oct 30 19:15:08.550635 ip-10-0-132-82 kubenswrapper[2577]: W1030 19:15:08.546647    2577 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Oct 30 19:15:08.550635 ip-10-0-132-82 kubenswrapper[2577]: W1030 19:15:08.546650    2577 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Oct 30 19:15:08.550635 ip-10-0-132-82 kubenswrapper[2577]: W1030 19:15:08.546652    2577 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Oct 30 19:15:08.550635 ip-10-0-132-82 kubenswrapper[2577]: W1030 19:15:08.546655    2577 feature_gate.go:386] unrecognized feature gate: NewOLM
Oct 30 19:15:08.550635 ip-10-0-132-82 kubenswrapper[2577]: W1030 19:15:08.546657    2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Oct 30 19:15:08.550635 ip-10-0-132-82 kubenswrapper[2577]: W1030 19:15:08.546660    2577 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Oct 30 19:15:08.550635 ip-10-0-132-82 kubenswrapper[2577]: W1030 19:15:08.546662    2577 feature_gate.go:386] unrecognized feature gate: EventTTL
Oct 30 19:15:08.550635 ip-10-0-132-82 kubenswrapper[2577]: W1030 19:15:08.546665    2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Oct 30 19:15:08.550635 ip-10-0-132-82 kubenswrapper[2577]: W1030 19:15:08.546667    2577 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Oct 30 19:15:08.550635 ip-10-0-132-82 kubenswrapper[2577]: W1030 19:15:08.546670    2577 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Oct 30 19:15:08.550635 ip-10-0-132-82 kubenswrapper[2577]: W1030 19:15:08.546672    2577 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Oct 30 19:15:08.550635 ip-10-0-132-82 kubenswrapper[2577]: W1030 19:15:08.546675    2577 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Oct 30 19:15:08.550635 ip-10-0-132-82 kubenswrapper[2577]: W1030 19:15:08.546679    2577 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Oct 30 19:15:08.550635 ip-10-0-132-82 kubenswrapper[2577]: W1030 19:15:08.546683    2577 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Oct 30 19:15:08.550635 ip-10-0-132-82 kubenswrapper[2577]: W1030 19:15:08.546686    2577 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Oct 30 19:15:08.550635 ip-10-0-132-82 kubenswrapper[2577]: W1030 19:15:08.546688    2577 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Oct 30 19:15:08.550635 ip-10-0-132-82 kubenswrapper[2577]: W1030 19:15:08.546691    2577 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Oct 30 19:15:08.550635 ip-10-0-132-82 kubenswrapper[2577]: W1030 19:15:08.546694    2577 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Oct 30 19:15:08.551096 ip-10-0-132-82 kubenswrapper[2577]: W1030 19:15:08.546696    2577 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Oct 30 19:15:08.551096 ip-10-0-132-82 kubenswrapper[2577]: W1030 19:15:08.546699    2577 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Oct 30 19:15:08.551096 ip-10-0-132-82 kubenswrapper[2577]: W1030 19:15:08.546702    2577 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Oct 30 19:15:08.551096 ip-10-0-132-82 kubenswrapper[2577]: W1030 19:15:08.546705    2577 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Oct 30 19:15:08.551096 ip-10-0-132-82 kubenswrapper[2577]: W1030 19:15:08.546708    2577 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Oct 30 19:15:08.551096 ip-10-0-132-82 kubenswrapper[2577]: W1030 19:15:08.546710    2577 feature_gate.go:386] unrecognized feature gate: Example
Oct 30 19:15:08.551096 ip-10-0-132-82 kubenswrapper[2577]: W1030 19:15:08.546713    2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Oct 30 19:15:08.551096 ip-10-0-132-82 kubenswrapper[2577]: W1030 19:15:08.546715    2577 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Oct 30 19:15:08.551096 ip-10-0-132-82 kubenswrapper[2577]: W1030 19:15:08.546719    2577 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Oct 30 19:15:08.551096 ip-10-0-132-82 kubenswrapper[2577]: W1030 19:15:08.546722    2577 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Oct 30 19:15:08.551096 ip-10-0-132-82 kubenswrapper[2577]: W1030 19:15:08.546724    2577 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Oct 30 19:15:08.551096 ip-10-0-132-82 kubenswrapper[2577]: W1030 19:15:08.546727    2577 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Oct 30 19:15:08.551096 ip-10-0-132-82 kubenswrapper[2577]: W1030 19:15:08.546731    2577 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Oct 30 19:15:08.551096 ip-10-0-132-82 kubenswrapper[2577]: W1030 19:15:08.546734    2577 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Oct 30 19:15:08.551096 ip-10-0-132-82 kubenswrapper[2577]: W1030 19:15:08.546737    2577 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Oct 30 19:15:08.551096 ip-10-0-132-82 kubenswrapper[2577]: W1030 19:15:08.546739    2577 feature_gate.go:386] unrecognized feature gate: OVNObservability
Oct 30 19:15:08.551096 ip-10-0-132-82 kubenswrapper[2577]: W1030 19:15:08.546742    2577 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Oct 30 19:15:08.551096 ip-10-0-132-82 kubenswrapper[2577]: W1030 19:15:08.546745    2577 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Oct 30 19:15:08.551096 ip-10-0-132-82 kubenswrapper[2577]: W1030 19:15:08.546747    2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Oct 30 19:15:08.551530 ip-10-0-132-82 kubenswrapper[2577]: W1030 19:15:08.546750    2577 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Oct 30 19:15:08.551530 ip-10-0-132-82 kubenswrapper[2577]: W1030 19:15:08.546752    2577 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Oct 30 19:15:08.551530 ip-10-0-132-82 kubenswrapper[2577]: W1030 19:15:08.546755    2577 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Oct 30 19:15:08.551530 ip-10-0-132-82 kubenswrapper[2577]: W1030 19:15:08.546757    2577 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Oct 30 19:15:08.551530 ip-10-0-132-82 kubenswrapper[2577]: W1030 19:15:08.546759    2577 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Oct 30 19:15:08.551530 ip-10-0-132-82 kubenswrapper[2577]: W1030 19:15:08.546762    2577 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Oct 30 19:15:08.551530 ip-10-0-132-82 kubenswrapper[2577]: W1030 19:15:08.546764    2577 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Oct 30 19:15:08.551530 ip-10-0-132-82 kubenswrapper[2577]: W1030 19:15:08.546767    2577 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Oct 30 19:15:08.551530 ip-10-0-132-82 kubenswrapper[2577]: W1030 19:15:08.546769    2577 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Oct 30 19:15:08.551530 ip-10-0-132-82 kubenswrapper[2577]: W1030 19:15:08.546771    2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Oct 30 19:15:08.551530 ip-10-0-132-82 kubenswrapper[2577]: W1030 19:15:08.546774    2577 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Oct 30 19:15:08.551530 ip-10-0-132-82 kubenswrapper[2577]: W1030 19:15:08.546777    2577 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Oct 30 19:15:08.551530 ip-10-0-132-82 kubenswrapper[2577]: W1030 19:15:08.546779    2577 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Oct 30 19:15:08.551530 ip-10-0-132-82 kubenswrapper[2577]: W1030 19:15:08.546782    2577 feature_gate.go:386] unrecognized feature gate: Example2
Oct 30 19:15:08.551530 ip-10-0-132-82 kubenswrapper[2577]: W1030 19:15:08.546784    2577 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Oct 30 19:15:08.551530 ip-10-0-132-82 kubenswrapper[2577]: W1030 19:15:08.546786    2577 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Oct 30 19:15:08.551530 ip-10-0-132-82 kubenswrapper[2577]: W1030 19:15:08.546789    2577 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Oct 30 19:15:08.551530 ip-10-0-132-82 kubenswrapper[2577]: W1030 19:15:08.546792    2577 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Oct 30 19:15:08.551530 ip-10-0-132-82 kubenswrapper[2577]: W1030 19:15:08.546794    2577 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Oct 30 19:15:08.551997 ip-10-0-132-82 kubenswrapper[2577]: W1030 19:15:08.546797    2577 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Oct 30 19:15:08.551997 ip-10-0-132-82 kubenswrapper[2577]: W1030 19:15:08.546800    2577 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Oct 30 19:15:08.551997 ip-10-0-132-82 kubenswrapper[2577]: W1030 19:15:08.546803    2577 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Oct 30 19:15:08.551997 ip-10-0-132-82 kubenswrapper[2577]: W1030 19:15:08.546805    2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Oct 30 19:15:08.551997 ip-10-0-132-82 kubenswrapper[2577]: W1030 19:15:08.546808    2577 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Oct 30 19:15:08.551997 ip-10-0-132-82 kubenswrapper[2577]: W1030 19:15:08.546811    2577 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Oct 30 19:15:08.551997 ip-10-0-132-82 kubenswrapper[2577]: W1030 19:15:08.546813    2577 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Oct 30 19:15:08.551997 ip-10-0-132-82 kubenswrapper[2577]: W1030 19:15:08.546816    2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Oct 30 19:15:08.551997 ip-10-0-132-82 kubenswrapper[2577]: W1030 19:15:08.546818    2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Oct 30 19:15:08.551997 ip-10-0-132-82 kubenswrapper[2577]: W1030 19:15:08.546821    2577 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Oct 30 19:15:08.551997 ip-10-0-132-82 kubenswrapper[2577]: W1030 19:15:08.546823    2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Oct 30 19:15:08.551997 ip-10-0-132-82 kubenswrapper[2577]: W1030 19:15:08.546826    2577 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Oct 30 19:15:08.551997 ip-10-0-132-82 kubenswrapper[2577]: W1030 19:15:08.546828    2577 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Oct 30 19:15:08.551997 ip-10-0-132-82 kubenswrapper[2577]: W1030 19:15:08.546831    2577 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Oct 30 19:15:08.551997 ip-10-0-132-82 kubenswrapper[2577]: W1030 19:15:08.546833    2577 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Oct 30 19:15:08.551997 ip-10-0-132-82 kubenswrapper[2577]: I1030 19:15:08.546861    2577 feature_gate.go:465] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false DRAPartitionableDevices:false EventedPLEG:false KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:true MutatingAdmissionPolicy:false RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true UserNamespacesSupport:true]}
Oct 30 19:15:08.552367 ip-10-0-132-82 kubenswrapper[2577]: W1030 19:15:08.547013    2577 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Oct 30 19:15:08.552367 ip-10-0-132-82 kubenswrapper[2577]: W1030 19:15:08.547018    2577 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Oct 30 19:15:08.552367 ip-10-0-132-82 kubenswrapper[2577]: W1030 19:15:08.547021    2577 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Oct 30 19:15:08.552367 ip-10-0-132-82 kubenswrapper[2577]: W1030 19:15:08.547023    2577 feature_gate.go:386] unrecognized feature gate: EventTTL
Oct 30 19:15:08.552367 ip-10-0-132-82 kubenswrapper[2577]: W1030 19:15:08.547026    2577 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Oct 30 19:15:08.552367 ip-10-0-132-82 kubenswrapper[2577]: W1030 19:15:08.547029    2577 feature_gate.go:386] unrecognized feature gate: Example
Oct 30 19:15:08.552367 ip-10-0-132-82 kubenswrapper[2577]: W1030 19:15:08.547031    2577 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Oct 30 19:15:08.552367 ip-10-0-132-82 kubenswrapper[2577]: W1030 19:15:08.547034    2577 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Oct 30 19:15:08.552367 ip-10-0-132-82 kubenswrapper[2577]: W1030 19:15:08.547036    2577 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Oct 30 19:15:08.552367 ip-10-0-132-82 kubenswrapper[2577]: W1030 19:15:08.547039    2577 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Oct 30 19:15:08.552367 ip-10-0-132-82 kubenswrapper[2577]: W1030 19:15:08.547041    2577 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Oct 30 19:15:08.552367 ip-10-0-132-82 kubenswrapper[2577]: W1030 19:15:08.547044    2577 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Oct 30 19:15:08.552367 ip-10-0-132-82 kubenswrapper[2577]: W1030 19:15:08.547046    2577 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Oct 30 19:15:08.552367 ip-10-0-132-82 kubenswrapper[2577]: W1030 19:15:08.547048    2577 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Oct 30 19:15:08.552367 ip-10-0-132-82 kubenswrapper[2577]: W1030 19:15:08.547051    2577 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Oct 30 19:15:08.552367 ip-10-0-132-82 kubenswrapper[2577]: W1030 19:15:08.547054    2577 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Oct 30 19:15:08.552367 ip-10-0-132-82 kubenswrapper[2577]: W1030 19:15:08.547056    2577 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Oct 30 19:15:08.552367 ip-10-0-132-82 kubenswrapper[2577]: W1030 19:15:08.547059    2577 feature_gate.go:386] unrecognized feature gate: DualReplica
Oct 30 19:15:08.552367 ip-10-0-132-82 kubenswrapper[2577]: W1030 19:15:08.547061    2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Oct 30 19:15:08.552367 ip-10-0-132-82 kubenswrapper[2577]: W1030 19:15:08.547064    2577 feature_gate.go:386] unrecognized feature gate: EVPN
Oct 30 19:15:08.552835 ip-10-0-132-82 kubenswrapper[2577]: W1030 19:15:08.547067    2577 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Oct 30 19:15:08.552835 ip-10-0-132-82 kubenswrapper[2577]: W1030 19:15:08.547069    2577 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Oct 30 19:15:08.552835 ip-10-0-132-82 kubenswrapper[2577]: W1030 19:15:08.547071    2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Oct 30 19:15:08.552835 ip-10-0-132-82 kubenswrapper[2577]: W1030 19:15:08.547074    2577 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Oct 30 19:15:08.552835 ip-10-0-132-82 kubenswrapper[2577]: W1030 19:15:08.547076    2577 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Oct 30 19:15:08.552835 ip-10-0-132-82 kubenswrapper[2577]: W1030 19:15:08.547079    2577 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Oct 30 19:15:08.552835 ip-10-0-132-82 kubenswrapper[2577]: W1030 19:15:08.547081    2577 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Oct 30 19:15:08.552835 ip-10-0-132-82 kubenswrapper[2577]: W1030 19:15:08.547083    2577 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Oct 30 19:15:08.552835 ip-10-0-132-82 kubenswrapper[2577]: W1030 19:15:08.547086    2577 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Oct 30 19:15:08.552835 ip-10-0-132-82 kubenswrapper[2577]: W1030 19:15:08.547088    2577 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Oct 30 19:15:08.552835 ip-10-0-132-82 kubenswrapper[2577]: W1030 19:15:08.547091    2577 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Oct 30 19:15:08.552835 ip-10-0-132-82 kubenswrapper[2577]: W1030 19:15:08.547094    2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Oct 30 19:15:08.552835 ip-10-0-132-82 kubenswrapper[2577]: W1030 19:15:08.547096    2577 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Oct 30 19:15:08.552835 ip-10-0-132-82 kubenswrapper[2577]: W1030 19:15:08.547099    2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Oct 30 19:15:08.552835 ip-10-0-132-82 kubenswrapper[2577]: W1030 19:15:08.547101    2577 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Oct 30 19:15:08.552835 ip-10-0-132-82 kubenswrapper[2577]: W1030 19:15:08.547103    2577 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Oct 30 19:15:08.552835 ip-10-0-132-82 kubenswrapper[2577]: W1030 19:15:08.547106    2577 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Oct 30 19:15:08.552835 ip-10-0-132-82 kubenswrapper[2577]: W1030 19:15:08.547109    2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Oct 30 19:15:08.552835 ip-10-0-132-82 kubenswrapper[2577]: W1030 19:15:08.547111    2577 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Oct 30 19:15:08.553283 ip-10-0-132-82 kubenswrapper[2577]: W1030 19:15:08.547114    2577 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Oct 30 19:15:08.553283 ip-10-0-132-82 kubenswrapper[2577]: W1030 19:15:08.547116    2577 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Oct 30 19:15:08.553283 ip-10-0-132-82 kubenswrapper[2577]: W1030 19:15:08.547119    2577 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Oct 30 19:15:08.553283 ip-10-0-132-82 kubenswrapper[2577]: W1030 19:15:08.547121    2577 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Oct 30 19:15:08.553283 ip-10-0-132-82 kubenswrapper[2577]: W1030 19:15:08.547124    2577 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Oct 30 19:15:08.553283 ip-10-0-132-82 kubenswrapper[2577]: W1030 19:15:08.547126    2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Oct 30 19:15:08.553283 ip-10-0-132-82 kubenswrapper[2577]: W1030 19:15:08.547129    2577 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Oct 30 19:15:08.553283 ip-10-0-132-82 kubenswrapper[2577]: W1030 19:15:08.547132    2577 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Oct 30 19:15:08.553283 ip-10-0-132-82 kubenswrapper[2577]: W1030 19:15:08.547134    2577 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Oct 30 19:15:08.553283 ip-10-0-132-82 kubenswrapper[2577]: W1030 19:15:08.547137    2577 feature_gate.go:386] unrecognized feature gate: SignatureStores
Oct 30 19:15:08.553283 ip-10-0-132-82 kubenswrapper[2577]: W1030 19:15:08.547140    2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Oct 30 19:15:08.553283 ip-10-0-132-82 kubenswrapper[2577]: W1030 19:15:08.547142    2577 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Oct 30 19:15:08.553283 ip-10-0-132-82 kubenswrapper[2577]: W1030 19:15:08.547145    2577 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Oct 30 19:15:08.553283 ip-10-0-132-82 kubenswrapper[2577]: W1030 19:15:08.547147    2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Oct 30 19:15:08.553283 ip-10-0-132-82 kubenswrapper[2577]: W1030 19:15:08.547149    2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Oct 30 19:15:08.553283 ip-10-0-132-82 kubenswrapper[2577]: W1030 19:15:08.547152    2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Oct 30 19:15:08.553283 ip-10-0-132-82 kubenswrapper[2577]: W1030 19:15:08.547154    2577 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Oct 30 19:15:08.553283 ip-10-0-132-82 kubenswrapper[2577]: W1030 19:15:08.547156    2577 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Oct 30 19:15:08.553283 ip-10-0-132-82 kubenswrapper[2577]: W1030 19:15:08.547160    2577 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Oct 30 19:15:08.553713 ip-10-0-132-82 kubenswrapper[2577]: W1030 19:15:08.547162    2577 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Oct 30 19:15:08.553713 ip-10-0-132-82 kubenswrapper[2577]: W1030 19:15:08.547164    2577 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Oct 30 19:15:08.553713 ip-10-0-132-82 kubenswrapper[2577]: W1030 19:15:08.547167    2577 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Oct 30 19:15:08.553713 ip-10-0-132-82 kubenswrapper[2577]: W1030 19:15:08.547169    2577 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Oct 30 19:15:08.553713 ip-10-0-132-82 kubenswrapper[2577]: W1030 19:15:08.547172    2577 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Oct 30 19:15:08.553713 ip-10-0-132-82 kubenswrapper[2577]: W1030 19:15:08.547174    2577 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Oct 30 19:15:08.553713 ip-10-0-132-82 kubenswrapper[2577]: W1030 19:15:08.547177    2577 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Oct 30 19:15:08.553713 ip-10-0-132-82 kubenswrapper[2577]: W1030 19:15:08.547179    2577 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Oct 30 19:15:08.553713 ip-10-0-132-82 kubenswrapper[2577]: W1030 19:15:08.547181    2577 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Oct 30 19:15:08.553713 ip-10-0-132-82 kubenswrapper[2577]: W1030 19:15:08.547184    2577 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Oct 30 19:15:08.553713 ip-10-0-132-82 kubenswrapper[2577]: W1030 19:15:08.547186    2577 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Oct 30 19:15:08.553713 ip-10-0-132-82 kubenswrapper[2577]: W1030 19:15:08.547190    2577 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Oct 30 19:15:08.553713 ip-10-0-132-82 kubenswrapper[2577]: W1030 19:15:08.547194    2577 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Oct 30 19:15:08.553713 ip-10-0-132-82 kubenswrapper[2577]: W1030 19:15:08.547196    2577 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Oct 30 19:15:08.553713 ip-10-0-132-82 kubenswrapper[2577]: W1030 19:15:08.547199    2577 feature_gate.go:386] unrecognized feature gate: OSStreams
Oct 30 19:15:08.553713 ip-10-0-132-82 kubenswrapper[2577]: W1030 19:15:08.547203    2577 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Oct 30 19:15:08.553713 ip-10-0-132-82 kubenswrapper[2577]: W1030 19:15:08.547207    2577 feature_gate.go:386] unrecognized feature gate: Example2
Oct 30 19:15:08.553713 ip-10-0-132-82 kubenswrapper[2577]: W1030 19:15:08.547210    2577 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Oct 30 19:15:08.553713 ip-10-0-132-82 kubenswrapper[2577]: W1030 19:15:08.547213    2577 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Oct 30 19:15:08.554181 ip-10-0-132-82 kubenswrapper[2577]: W1030 19:15:08.547216    2577 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Oct 30 19:15:08.554181 ip-10-0-132-82 kubenswrapper[2577]: W1030 19:15:08.547218    2577 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Oct 30 19:15:08.554181 ip-10-0-132-82 kubenswrapper[2577]: W1030 19:15:08.547221    2577 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Oct 30 19:15:08.554181 ip-10-0-132-82 kubenswrapper[2577]: W1030 19:15:08.547223    2577 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Oct 30 19:15:08.554181 ip-10-0-132-82 kubenswrapper[2577]: W1030 19:15:08.547226    2577 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Oct 30 19:15:08.554181 ip-10-0-132-82 kubenswrapper[2577]: W1030 19:15:08.547229    2577 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Oct 30 19:15:08.554181 ip-10-0-132-82 kubenswrapper[2577]: W1030 19:15:08.547231    2577 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Oct 30 19:15:08.554181 ip-10-0-132-82 kubenswrapper[2577]: W1030 19:15:08.547234    2577 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Oct 30 19:15:08.554181 ip-10-0-132-82 kubenswrapper[2577]: W1030 19:15:08.547236    2577 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Oct 30 19:15:08.554181 ip-10-0-132-82 kubenswrapper[2577]: W1030 19:15:08.547238    2577 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Oct 30 19:15:08.554181 ip-10-0-132-82 kubenswrapper[2577]: W1030 19:15:08.547241    2577 feature_gate.go:386] unrecognized feature gate: OVNObservability
Oct 30 19:15:08.554181 ip-10-0-132-82 kubenswrapper[2577]: W1030 19:15:08.547243    2577 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Oct 30 19:15:08.554181 ip-10-0-132-82 kubenswrapper[2577]: W1030 19:15:08.547246    2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Oct 30 19:15:08.554181 ip-10-0-132-82 kubenswrapper[2577]: W1030 19:15:08.547248    2577 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Oct 30 19:15:08.554181 ip-10-0-132-82 kubenswrapper[2577]: W1030 19:15:08.547251    2577 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Oct 30 19:15:08.554181 ip-10-0-132-82 kubenswrapper[2577]: W1030 19:15:08.547253    2577 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Oct 30 19:15:08.554181 ip-10-0-132-82 kubenswrapper[2577]: W1030 19:15:08.547255    2577 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Oct 30 19:15:08.554181 ip-10-0-132-82 kubenswrapper[2577]: W1030 19:15:08.547258    2577 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Oct 30 19:15:08.554181 ip-10-0-132-82 kubenswrapper[2577]: W1030 19:15:08.547260    2577 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Oct 30 19:15:08.554181 ip-10-0-132-82 kubenswrapper[2577]: W1030 19:15:08.547263    2577 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Oct 30 19:15:08.554640 ip-10-0-132-82 kubenswrapper[2577]: W1030 19:15:08.547265    2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Oct 30 19:15:08.554640 ip-10-0-132-82 kubenswrapper[2577]: W1030 19:15:08.547267    2577 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Oct 30 19:15:08.554640 ip-10-0-132-82 kubenswrapper[2577]: W1030 19:15:08.547270    2577 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Oct 30 19:15:08.554640 ip-10-0-132-82 kubenswrapper[2577]: W1030 19:15:08.547273    2577 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Oct 30 19:15:08.554640 ip-10-0-132-82 kubenswrapper[2577]: W1030 19:15:08.547275    2577 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Oct 30 19:15:08.554640 ip-10-0-132-82 kubenswrapper[2577]: W1030 19:15:08.547278    2577 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Oct 30 19:15:08.554640 ip-10-0-132-82 kubenswrapper[2577]: W1030 19:15:08.547280    2577 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Oct 30 19:15:08.554640 ip-10-0-132-82 kubenswrapper[2577]: W1030 19:15:08.547283    2577 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Oct 30 19:15:08.554640 ip-10-0-132-82 kubenswrapper[2577]: W1030 19:15:08.547285    2577 feature_gate.go:386] unrecognized feature gate: NewOLM
Oct 30 19:15:08.554640 ip-10-0-132-82 kubenswrapper[2577]: I1030 19:15:08.547308    2577 feature_gate.go:465] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false DRAPartitionableDevices:false EventedPLEG:false KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:true MutatingAdmissionPolicy:false RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true UserNamespacesSupport:true]}
Oct 30 19:15:08.554640 ip-10-0-132-82 kubenswrapper[2577]: W1030 19:15:08.547342    2577 feature_gate.go:386] unrecognized feature gate: NewOLM
Oct 30 19:15:08.554640 ip-10-0-132-82 kubenswrapper[2577]: W1030 19:15:08.547346    2577 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Oct 30 19:15:08.554640 ip-10-0-132-82 kubenswrapper[2577]: W1030 19:15:08.547349    2577 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Oct 30 19:15:08.554640 ip-10-0-132-82 kubenswrapper[2577]: W1030 19:15:08.547352    2577 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Oct 30 19:15:08.554640 ip-10-0-132-82 kubenswrapper[2577]: W1030 19:15:08.547354    2577 feature_gate.go:386] unrecognized feature gate: EventTTL
Oct 30 19:15:08.554640 ip-10-0-132-82 kubenswrapper[2577]: W1030 19:15:08.547356    2577 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Oct 30 19:15:08.555026 ip-10-0-132-82 kubenswrapper[2577]: W1030 19:15:08.547359    2577 feature_gate.go:386] unrecognized feature gate: Example
Oct 30 19:15:08.555026 ip-10-0-132-82 kubenswrapper[2577]: W1030 19:15:08.547362    2577 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Oct 30 19:15:08.555026 ip-10-0-132-82 kubenswrapper[2577]: W1030 19:15:08.547365    2577 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Oct 30 19:15:08.555026 ip-10-0-132-82 kubenswrapper[2577]: W1030 19:15:08.547368    2577 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Oct 30 19:15:08.555026 ip-10-0-132-82 kubenswrapper[2577]: W1030 19:15:08.547371    2577 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Oct 30 19:15:08.555026 ip-10-0-132-82 kubenswrapper[2577]: W1030 19:15:08.547373    2577 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Oct 30 19:15:08.555026 ip-10-0-132-82 kubenswrapper[2577]: W1030 19:15:08.547376    2577 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Oct 30 19:15:08.555026 ip-10-0-132-82 kubenswrapper[2577]: W1030 19:15:08.547378    2577 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Oct 30 19:15:08.555026 ip-10-0-132-82 kubenswrapper[2577]: W1030 19:15:08.547380    2577 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Oct 30 19:15:08.555026 ip-10-0-132-82 kubenswrapper[2577]: W1030 19:15:08.547383    2577 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Oct 30 19:15:08.555026 ip-10-0-132-82 kubenswrapper[2577]: W1030 19:15:08.547385    2577 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Oct 30 19:15:08.555026 ip-10-0-132-82 kubenswrapper[2577]: W1030 19:15:08.547388    2577 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Oct 30 19:15:08.555026 ip-10-0-132-82 kubenswrapper[2577]: W1030 19:15:08.547390    2577 feature_gate.go:386] unrecognized feature gate: DualReplica
Oct 30 19:15:08.555026 ip-10-0-132-82 kubenswrapper[2577]: W1030 19:15:08.547392    2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Oct 30 19:15:08.555026 ip-10-0-132-82 kubenswrapper[2577]: W1030 19:15:08.547395    2577 feature_gate.go:386] unrecognized feature gate: EVPN
Oct 30 19:15:08.555026 ip-10-0-132-82 kubenswrapper[2577]: W1030 19:15:08.547397    2577 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Oct 30 19:15:08.555026 ip-10-0-132-82 kubenswrapper[2577]: W1030 19:15:08.547400    2577 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Oct 30 19:15:08.555026 ip-10-0-132-82 kubenswrapper[2577]: W1030 19:15:08.547402    2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Oct 30 19:15:08.555026 ip-10-0-132-82 kubenswrapper[2577]: W1030 19:15:08.547404    2577 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Oct 30 19:15:08.555026 ip-10-0-132-82 kubenswrapper[2577]: W1030 19:15:08.547407    2577 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Oct 30 19:15:08.555506 ip-10-0-132-82 kubenswrapper[2577]: W1030 19:15:08.547410    2577 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Oct 30 19:15:08.555506 ip-10-0-132-82 kubenswrapper[2577]: W1030 19:15:08.547412    2577 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Oct 30 19:15:08.555506 ip-10-0-132-82 kubenswrapper[2577]: W1030 19:15:08.547414    2577 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Oct 30 19:15:08.555506 ip-10-0-132-82 kubenswrapper[2577]: W1030 19:15:08.547417    2577 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Oct 30 19:15:08.555506 ip-10-0-132-82 kubenswrapper[2577]: W1030 19:15:08.547419    2577 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Oct 30 19:15:08.555506 ip-10-0-132-82 kubenswrapper[2577]: W1030 19:15:08.547422    2577 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Oct 30 19:15:08.555506 ip-10-0-132-82 kubenswrapper[2577]: W1030 19:15:08.547424    2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Oct 30 19:15:08.555506 ip-10-0-132-82 kubenswrapper[2577]: W1030 19:15:08.547427    2577 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Oct 30 19:15:08.555506 ip-10-0-132-82 kubenswrapper[2577]: W1030 19:15:08.547429    2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Oct 30 19:15:08.555506 ip-10-0-132-82 kubenswrapper[2577]: W1030 19:15:08.547432    2577 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Oct 30 19:15:08.555506 ip-10-0-132-82 kubenswrapper[2577]: W1030 19:15:08.547434    2577 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Oct 30 19:15:08.555506 ip-10-0-132-82 kubenswrapper[2577]: W1030 19:15:08.547436    2577 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Oct 30 19:15:08.555506 ip-10-0-132-82 kubenswrapper[2577]: W1030 19:15:08.547441    2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Oct 30 19:15:08.555506 ip-10-0-132-82 kubenswrapper[2577]: W1030 19:15:08.547444    2577 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Oct 30 19:15:08.555506 ip-10-0-132-82 kubenswrapper[2577]: W1030 19:15:08.547451    2577 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Oct 30 19:15:08.555506 ip-10-0-132-82 kubenswrapper[2577]: W1030 19:15:08.547454    2577 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Oct 30 19:15:08.555506 ip-10-0-132-82 kubenswrapper[2577]: W1030 19:15:08.547457    2577 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Oct 30 19:15:08.555506 ip-10-0-132-82 kubenswrapper[2577]: W1030 19:15:08.547459    2577 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Oct 30 19:15:08.555506 ip-10-0-132-82 kubenswrapper[2577]: W1030 19:15:08.547462    2577 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Oct 30 19:15:08.555506 ip-10-0-132-82 kubenswrapper[2577]: W1030 19:15:08.547464    2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Oct 30 19:15:08.556048 ip-10-0-132-82 kubenswrapper[2577]: W1030 19:15:08.547467    2577 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Oct 30 19:15:08.556048 ip-10-0-132-82 kubenswrapper[2577]: W1030 19:15:08.547469    2577 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Oct 30 19:15:08.556048 ip-10-0-132-82 kubenswrapper[2577]: W1030 19:15:08.547471    2577 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Oct 30 19:15:08.556048 ip-10-0-132-82 kubenswrapper[2577]: W1030 19:15:08.547474    2577 feature_gate.go:386] unrecognized feature gate: SignatureStores
Oct 30 19:15:08.556048 ip-10-0-132-82 kubenswrapper[2577]: W1030 19:15:08.547476    2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Oct 30 19:15:08.556048 ip-10-0-132-82 kubenswrapper[2577]: W1030 19:15:08.547479    2577 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Oct 30 19:15:08.556048 ip-10-0-132-82 kubenswrapper[2577]: W1030 19:15:08.547481    2577 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Oct 30 19:15:08.556048 ip-10-0-132-82 kubenswrapper[2577]: W1030 19:15:08.547483    2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Oct 30 19:15:08.556048 ip-10-0-132-82 kubenswrapper[2577]: W1030 19:15:08.547486    2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Oct 30 19:15:08.556048 ip-10-0-132-82 kubenswrapper[2577]: W1030 19:15:08.547488    2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Oct 30 19:15:08.556048 ip-10-0-132-82 kubenswrapper[2577]: W1030 19:15:08.547491    2577 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Oct 30 19:15:08.556048 ip-10-0-132-82 kubenswrapper[2577]: W1030 19:15:08.547493    2577 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Oct 30 19:15:08.556048 ip-10-0-132-82 kubenswrapper[2577]: W1030 19:15:08.547496    2577 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Oct 30 19:15:08.556048 ip-10-0-132-82 kubenswrapper[2577]: W1030 19:15:08.547498    2577 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Oct 30 19:15:08.556048 ip-10-0-132-82 kubenswrapper[2577]: W1030 19:15:08.547501    2577 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Oct 30 19:15:08.556048 ip-10-0-132-82 kubenswrapper[2577]: W1030 19:15:08.547504    2577 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Oct 30 19:15:08.556048 ip-10-0-132-82 kubenswrapper[2577]: W1030 19:15:08.547507    2577 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Oct 30 19:15:08.556048 ip-10-0-132-82 kubenswrapper[2577]: W1030 19:15:08.547509    2577 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Oct 30 19:15:08.556048 ip-10-0-132-82 kubenswrapper[2577]: W1030 19:15:08.547511    2577 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Oct 30 19:15:08.556519 ip-10-0-132-82 kubenswrapper[2577]: W1030 19:15:08.547514    2577 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Oct 30 19:15:08.556519 ip-10-0-132-82 kubenswrapper[2577]: W1030 19:15:08.547516    2577 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Oct 30 19:15:08.556519 ip-10-0-132-82 kubenswrapper[2577]: W1030 19:15:08.547519    2577 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Oct 30 19:15:08.556519 ip-10-0-132-82 kubenswrapper[2577]: W1030 19:15:08.547521    2577 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Oct 30 19:15:08.556519 ip-10-0-132-82 kubenswrapper[2577]: W1030 19:15:08.547523    2577 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Oct 30 19:15:08.556519 ip-10-0-132-82 kubenswrapper[2577]: W1030 19:15:08.547529    2577 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Oct 30 19:15:08.556519 ip-10-0-132-82 kubenswrapper[2577]: W1030 19:15:08.547533    2577 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Oct 30 19:15:08.556519 ip-10-0-132-82 kubenswrapper[2577]: W1030 19:15:08.547536    2577 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Oct 30 19:15:08.556519 ip-10-0-132-82 kubenswrapper[2577]: W1030 19:15:08.547540    2577 feature_gate.go:386] unrecognized feature gate: OSStreams
Oct 30 19:15:08.556519 ip-10-0-132-82 kubenswrapper[2577]: W1030 19:15:08.547543    2577 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Oct 30 19:15:08.556519 ip-10-0-132-82 kubenswrapper[2577]: W1030 19:15:08.547547    2577 feature_gate.go:386] unrecognized feature gate: Example2
Oct 30 19:15:08.556519 ip-10-0-132-82 kubenswrapper[2577]: W1030 19:15:08.547550    2577 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Oct 30 19:15:08.556519 ip-10-0-132-82 kubenswrapper[2577]: W1030 19:15:08.547553    2577 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Oct 30 19:15:08.556519 ip-10-0-132-82 kubenswrapper[2577]: W1030 19:15:08.547555    2577 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Oct 30 19:15:08.556519 ip-10-0-132-82 kubenswrapper[2577]: W1030 19:15:08.547557    2577 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Oct 30 19:15:08.556519 ip-10-0-132-82 kubenswrapper[2577]: W1030 19:15:08.547560    2577 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Oct 30 19:15:08.556519 ip-10-0-132-82 kubenswrapper[2577]: W1030 19:15:08.547562    2577 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Oct 30 19:15:08.556519 ip-10-0-132-82 kubenswrapper[2577]: W1030 19:15:08.547565    2577 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Oct 30 19:15:08.556519 ip-10-0-132-82 kubenswrapper[2577]: W1030 19:15:08.547567    2577 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Oct 30 19:15:08.557102 ip-10-0-132-82 kubenswrapper[2577]: W1030 19:15:08.547569    2577 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Oct 30 19:15:08.557102 ip-10-0-132-82 kubenswrapper[2577]: W1030 19:15:08.547572    2577 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Oct 30 19:15:08.557102 ip-10-0-132-82 kubenswrapper[2577]: W1030 19:15:08.547574    2577 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Oct 30 19:15:08.557102 ip-10-0-132-82 kubenswrapper[2577]: W1030 19:15:08.547577    2577 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Oct 30 19:15:08.557102 ip-10-0-132-82 kubenswrapper[2577]: W1030 19:15:08.547579    2577 feature_gate.go:386] unrecognized feature gate: OVNObservability
Oct 30 19:15:08.557102 ip-10-0-132-82 kubenswrapper[2577]: W1030 19:15:08.547582    2577 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Oct 30 19:15:08.557102 ip-10-0-132-82 kubenswrapper[2577]: W1030 19:15:08.547584    2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Oct 30 19:15:08.557102 ip-10-0-132-82 kubenswrapper[2577]: W1030 19:15:08.547587    2577 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Oct 30 19:15:08.557102 ip-10-0-132-82 kubenswrapper[2577]: W1030 19:15:08.547589    2577 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Oct 30 19:15:08.557102 ip-10-0-132-82 kubenswrapper[2577]: W1030 19:15:08.547591    2577 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Oct 30 19:15:08.557102 ip-10-0-132-82 kubenswrapper[2577]: W1030 19:15:08.547594    2577 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Oct 30 19:15:08.557102 ip-10-0-132-82 kubenswrapper[2577]: W1030 19:15:08.547596    2577 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Oct 30 19:15:08.557102 ip-10-0-132-82 kubenswrapper[2577]: W1030 19:15:08.547599    2577 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Oct 30 19:15:08.557102 ip-10-0-132-82 kubenswrapper[2577]: W1030 19:15:08.547601    2577 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Oct 30 19:15:08.557102 ip-10-0-132-82 kubenswrapper[2577]: W1030 19:15:08.547603    2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Oct 30 19:15:08.557102 ip-10-0-132-82 kubenswrapper[2577]: W1030 19:15:08.547606    2577 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Oct 30 19:15:08.557102 ip-10-0-132-82 kubenswrapper[2577]: W1030 19:15:08.547608    2577 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Oct 30 19:15:08.557102 ip-10-0-132-82 kubenswrapper[2577]: W1030 19:15:08.547612    2577 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Oct 30 19:15:08.557102 ip-10-0-132-82 kubenswrapper[2577]: W1030 19:15:08.547616    2577 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Oct 30 19:15:08.557651 ip-10-0-132-82 kubenswrapper[2577]: W1030 19:15:08.547619    2577 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Oct 30 19:15:08.557651 ip-10-0-132-82 kubenswrapper[2577]: W1030 19:15:08.547621    2577 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Oct 30 19:15:08.557651 ip-10-0-132-82 kubenswrapper[2577]: W1030 19:15:08.547624    2577 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Oct 30 19:15:08.557651 ip-10-0-132-82 kubenswrapper[2577]: I1030 19:15:08.547674    2577 server.go:957] "Client rotation is on, will bootstrap in background"
Oct 30 19:15:08.557651 ip-10-0-132-82 kubenswrapper[2577]: I1030 19:15:08.550046    2577 bootstrap.go:101] "Use the bootstrap credentials to request a cert, and set kubeconfig to point to the certificate dir"
Oct 30 19:15:08.557651 ip-10-0-132-82 kubenswrapper[2577]: I1030 19:15:08.550964    2577 server.go:1014] "Starting client certificate rotation"
Oct 30 19:15:08.557651 ip-10-0-132-82 kubenswrapper[2577]: I1030 19:15:08.551059    2577 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kube-apiserver-client-kubelet"
Oct 30 19:15:08.557651 ip-10-0-132-82 kubenswrapper[2577]: I1030 19:15:08.551094    2577 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kube-apiserver-client-kubelet"
Oct 30 19:15:08.557651 ip-10-0-132-82 kubenswrapper[2577]: I1030 19:15:08.553083    2577 dynamic_cafile_content.go:123] "Loaded a new CA Bundle and Verifier" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt"
Oct 30 19:15:08.557651 ip-10-0-132-82 kubenswrapper[2577]: I1030 19:15:08.554555    2577 dynamic_cafile_content.go:161] "Starting controller" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt"
Oct 30 19:15:08.565562 ip-10-0-132-82 kubenswrapper[2577]: I1030 19:15:08.565544    2577 log.go:25] "Validated CRI v1 runtime API"
Oct 30 19:15:08.568491 ip-10-0-132-82 kubenswrapper[2577]: I1030 19:15:08.568474    2577 log.go:25] "Validated CRI v1 image API"
Oct 30 19:15:08.569273 ip-10-0-132-82 kubenswrapper[2577]: I1030 19:15:08.569249    2577 server.go:1424] "Using cgroup driver setting received from the CRI runtime" cgroupDriver="systemd"
Oct 30 19:15:08.575713 ip-10-0-132-82 kubenswrapper[2577]: I1030 19:15:08.575678    2577 fs.go:135] Filesystem UUIDs: map[2136bd09-3cc3-4cc1-9474-9c6a9d7fad31:/dev/nvme0n1p3 7B77-95E7:/dev/nvme0n1p2 cb428dfa-9865-415d-9f85-f41e9bc9d06c:/dev/nvme0n1p4]
Oct 30 19:15:08.575713 ip-10-0-132-82 kubenswrapper[2577]: I1030 19:15:08.575707    2577 fs.go:136] Filesystem partitions: map[/dev/nvme0n1p3:{mountpoint:/boot major:259 minor:3 fsType:ext4 blockSize:0} /dev/nvme0n1p4:{mountpoint:/var major:259 minor:4 fsType:xfs blockSize:0} /dev/shm:{mountpoint:/dev/shm major:0 minor:24 fsType:tmpfs blockSize:0} /run:{mountpoint:/run major:0 minor:26 fsType:tmpfs blockSize:0} /tmp:{mountpoint:/tmp major:0 minor:32 fsType:tmpfs blockSize:0} composefs_0-34:{mountpoint:/ major:0 minor:34 fsType:overlay blockSize:0}]
Oct 30 19:15:08.579498 ip-10-0-132-82 kubenswrapper[2577]: I1030 19:15:08.579317    2577 manager.go:217] Machine: {Timestamp:2026-10-30 19:15:08.57885386 +0000 UTC m=+0.308815136 CPUVendorID:GenuineIntel NumCores:8 NumPhysicalCores:4 NumSockets:1 CpuFrequency:3064192 MemoryCapacity:32812007424 SwapCapacity:0 MemoryByType:map[] NVMInfo:{MemoryModeCapacity:0 AppDirectModeCapacity:0 AvgPowerBudget:0} HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] MachineID:ec27e6011dbd50892ffc479345e98622 SystemUUID:ec27e601-1dbd-5089-2ffc-479345e98622 BootID:d31c3a64-bf37-4f8c-a6a4-b569507d4ade Filesystems:[{Device:/dev/shm DeviceMajor:0 DeviceMinor:24 Capacity:16406003712 Type:vfs Inodes:4005372 HasInodes:true} {Device:/run DeviceMajor:0 DeviceMinor:26 Capacity:6562402304 Type:vfs Inodes:819200 HasInodes:true} {Device:composefs_0-34 DeviceMajor:0 DeviceMinor:34 Capacity:6422528 Type:vfs Inodes:18446744073709551615 HasInodes:true} {Device:/dev/nvme0n1p4 DeviceMajor:259 DeviceMinor:4 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/tmp DeviceMajor:0 DeviceMinor:32 Capacity:16406003712 Type:vfs Inodes:1048576 HasInodes:true} {Device:/dev/nvme0n1p3 DeviceMajor:259 DeviceMinor:3 Capacity:366869504 Type:vfs Inodes:98304 HasInodes:true}] DiskMap:map[259:0:{Name:nvme0n1 Major:259 Minor:0 Size:128849018880 Scheduler:none}] NetworkDevices:[{Name:br-ex MacAddress:02:58:ed:71:9c:8b Speed:0 Mtu:9001} {Name:ens5 MacAddress:02:58:ed:71:9c:8b Speed:0 Mtu:9001} {Name:ovs-system MacAddress:5e:c8:09:01:9f:08 Speed:0 Mtu:1500}] Topology:[{Id:0 Memory:32812007424 HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] Cores:[{Id:0 Threads:[0 4] Caches:[{Id:0 Size:32768 Type:Data Level:1} {Id:0 Size:32768 Type:Instruction Level:1} {Id:0 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:1 Threads:[1 5] Caches:[{Id:1 Size:32768 Type:Data Level:1} {Id:1 Size:32768 Type:Instruction Level:1} {Id:1 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:2 Threads:[2 6] Caches:[{Id:2 Size:32768 Type:Data Level:1} {Id:2 Size:32768 Type:Instruction Level:1} {Id:2 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:3 Threads:[3 7] Caches:[{Id:3 Size:32768 Type:Data Level:1} {Id:3 Size:32768 Type:Instruction Level:1} {Id:3 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:}] Caches:[{Id:0 Size:34603008 Type:Unified Level:3}] Distances:[10]}] CloudProvider:Unknown InstanceType:Unknown InstanceID:None}
Oct 30 19:15:08.579498 ip-10-0-132-82 kubenswrapper[2577]: I1030 19:15:08.579494    2577 manager_no_libpfm.go:29] cAdvisor is build without cgo and/or libpfm support. Perf event counters are not available.
Oct 30 19:15:08.579603 ip-10-0-132-82 kubenswrapper[2577]: I1030 19:15:08.579563    2577 manager.go:233] Version: {KernelVersion:5.14.0-687.60.1.el9_8.x86_64 ContainerOsVersion:Red Hat Enterprise Linux CoreOS 9.8.20261023-0 (Plow) DockerVersion: DockerAPIVersion: CadvisorVersion: CadvisorRevision:}
Oct 30 19:15:08.583206 ip-10-0-132-82 kubenswrapper[2577]: I1030 19:15:08.583184    2577 container_manager_linux.go:275] "Container manager verified user specified cgroup-root exists" cgroupRoot=[]
Oct 30 19:15:08.583332 ip-10-0-132-82 kubenswrapper[2577]: I1030 19:15:08.583209    2577 container_manager_linux.go:280] "Creating Container Manager object based on Node Config" nodeConfig={"NodeName":"ip-10-0-132-82.us-west-1.compute.internal","RuntimeCgroupsName":"/system.slice/crio.service","SystemCgroupsName":"/system.slice","KubeletCgroupsName":"","KubeletOOMScoreAdj":-999,"ContainerRuntime":"","CgroupsPerQOS":true,"CgroupRoot":"/","CgroupDriver":"systemd","KubeletRootDir":"/var/lib/kubelet","ProtectKernelDefaults":true,"KubeReservedCgroupName":"","SystemReservedCgroupName":"/system.slice","ReservedSystemCPUs":{},"EnforceNodeAllocatable":{"pods":{},"system-reserved-compressible":{}},"KubeReserved":null,"SystemReserved":{"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"},"HardEvictionThresholds":[{"Signal":"imagefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.15},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"memory.available","Operator":"LessThan","Value":{"Quantity":"100Mi","Percentage":0},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.1},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null}],"QOSReserved":{},"CPUManagerPolicy":"none","CPUManagerPolicyOptions":null,"TopologyManagerScope":"container","CPUManagerReconcilePeriod":10000000000,"MemoryManagerPolicy":"None","MemoryManagerReservedMemory":null,"PodPidsLimit":4096,"EnforceCPULimits":true,"CPUCFSQuotaPeriod":100000000,"TopologyManagerPolicy":"none","TopologyManagerPolicyOptions":null,"CgroupVersion":2}
Oct 30 19:15:08.583376 ip-10-0-132-82 kubenswrapper[2577]: I1030 19:15:08.583334    2577 topology_manager.go:143] "Creating topology manager with none policy"
Oct 30 19:15:08.583376 ip-10-0-132-82 kubenswrapper[2577]: I1030 19:15:08.583342    2577 container_manager_linux.go:311] "Creating device plugin manager"
Oct 30 19:15:08.583376 ip-10-0-132-82 kubenswrapper[2577]: I1030 19:15:08.583365    2577 manager.go:144] "Creating Device Plugin manager" path="/var/lib/kubelet/device-plugins/kubelet.sock"
Oct 30 19:15:08.584323 ip-10-0-132-82 kubenswrapper[2577]: I1030 19:15:08.584313    2577 server.go:74] "Creating device plugin registration server" version="v1beta1" socket="/var/lib/kubelet/device-plugins/kubelet.sock"
Oct 30 19:15:08.585553 ip-10-0-132-82 kubenswrapper[2577]: I1030 19:15:08.585544    2577 container_manager_linux.go:320] "Creating Dynamic Resource Allocation (DRA) manager"
Oct 30 19:15:08.589708 ip-10-0-132-82 kubenswrapper[2577]: I1030 19:15:08.589697    2577 state_mem.go:41] "Initialized" logger="CPUManager state memory"
Oct 30 19:15:08.589822 ip-10-0-132-82 kubenswrapper[2577]: I1030 19:15:08.589813    2577 server.go:1235] "Using root directory" path="/var/lib/kubelet"
Oct 30 19:15:08.590628 ip-10-0-132-82 kubenswrapper[2577]: I1030 19:15:08.590618    2577 kubelet.go:493] "Attempting to sync node with API server"
Oct 30 19:15:08.590666 ip-10-0-132-82 kubenswrapper[2577]: I1030 19:15:08.590632    2577 kubelet.go:394] "Adding static pod path" path="/etc/kubernetes/manifests"
Oct 30 19:15:08.590666 ip-10-0-132-82 kubenswrapper[2577]: I1030 19:15:08.590642    2577 file.go:69] "Watching path" path="/etc/kubernetes/manifests"
Oct 30 19:15:08.590666 ip-10-0-132-82 kubenswrapper[2577]: I1030 19:15:08.590651    2577 kubelet.go:405] "Adding apiserver pod source"
Oct 30 19:15:08.590666 ip-10-0-132-82 kubenswrapper[2577]: I1030 19:15:08.590660    2577 apiserver.go:42] "Waiting for node sync before watching apiserver pods"
Oct 30 19:15:08.591061 ip-10-0-132-82 kubenswrapper[2577]: I1030 19:15:08.591047    2577 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking"
Oct 30 19:15:08.593382 ip-10-0-132-82 kubenswrapper[2577]: I1030 19:15:08.593360    2577 kuberuntime_manager.go:304] "Container runtime initialized" containerRuntime="cri-o" version="1.35.10-3.rhaos4.22.git9383502.el9" apiVersion="v1"
Oct 30 19:15:08.601876 ip-10-0-132-82 kubenswrapper[2577]: I1030 19:15:08.601862    2577 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking"
Oct 30 19:15:08.602790 ip-10-0-132-82 kubenswrapper[2577]: I1030 19:15:08.602777    2577 kubelet.go:963] "Not starting ClusterTrustBundle informer because we are in static kubelet mode or the ClusterTrustBundleProjection featuregate is disabled"
Oct 30 19:15:08.602834 ip-10-0-132-82 kubenswrapper[2577]: I1030 19:15:08.602812    2577 kubelet.go:990] "Not starting PodCertificateRequest manager because we are in static kubelet mode or the PodCertificateProjection feature gate is disabled"
Oct 30 19:15:08.602989 ip-10-0-132-82 kubenswrapper[2577]: I1030 19:15:08.602979    2577 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/portworx-volume"
Oct 30 19:15:08.603023 ip-10-0-132-82 kubenswrapper[2577]: I1030 19:15:08.602995    2577 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/empty-dir"
Oct 30 19:15:08.603023 ip-10-0-132-82 kubenswrapper[2577]: I1030 19:15:08.603001    2577 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/git-repo"
Oct 30 19:15:08.603023 ip-10-0-132-82 kubenswrapper[2577]: I1030 19:15:08.603006    2577 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/host-path"
Oct 30 19:15:08.603023 ip-10-0-132-82 kubenswrapper[2577]: I1030 19:15:08.603013    2577 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/nfs"
Oct 30 19:15:08.603023 ip-10-0-132-82 kubenswrapper[2577]: I1030 19:15:08.603019    2577 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/secret"
Oct 30 19:15:08.603710 ip-10-0-132-82 kubenswrapper[2577]: I1030 19:15:08.603701    2577 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/iscsi"
Oct 30 19:15:08.603739 ip-10-0-132-82 kubenswrapper[2577]: I1030 19:15:08.603711    2577 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/downward-api"
Oct 30 19:15:08.603739 ip-10-0-132-82 kubenswrapper[2577]: I1030 19:15:08.603717    2577 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/fc"
Oct 30 19:15:08.603739 ip-10-0-132-82 kubenswrapper[2577]: I1030 19:15:08.603723    2577 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/configmap"
Oct 30 19:15:08.603739 ip-10-0-132-82 kubenswrapper[2577]: I1030 19:15:08.603732    2577 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/projected"
Oct 30 19:15:08.603739 ip-10-0-132-82 kubenswrapper[2577]: I1030 19:15:08.603739    2577 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/local-volume"
Oct 30 19:15:08.604804 ip-10-0-132-82 kubenswrapper[2577]: I1030 19:15:08.604793    2577 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/csi"
Oct 30 19:15:08.604841 ip-10-0-132-82 kubenswrapper[2577]: I1030 19:15:08.604809    2577 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/image"
Oct 30 19:15:08.608683 ip-10-0-132-82 kubenswrapper[2577]: I1030 19:15:08.608671    2577 server.go:1263] "Started kubelet"
Oct 30 19:15:08.608764 ip-10-0-132-82 kubenswrapper[2577]: I1030 19:15:08.608740    2577 server.go:182] "Starting to listen" address="0.0.0.0" port=10250
Oct 30 19:15:08.608820 ip-10-0-132-82 kubenswrapper[2577]: I1030 19:15:08.608767    2577 ratelimit.go:56] "Setting rate limiting for endpoint" service="podresources" qps=100 burstTokens=10
Oct 30 19:15:08.608864 ip-10-0-132-82 kubenswrapper[2577]: I1030 19:15:08.608845    2577 server_v1.go:49] "podresources" method="list" useActivePods=true
Oct 30 19:15:08.612652 ip-10-0-132-82 kubenswrapper[2577]: I1030 19:15:08.612627    2577 server.go:254] "Starting to serve the podresources API" endpoint="unix:/var/lib/kubelet/pod-resources/kubelet.sock"
Oct 30 19:15:08.612936 ip-10-0-132-82 systemd[1]: Started Kubernetes Kubelet.
Oct 30 19:15:08.615081 ip-10-0-132-82 kubenswrapper[2577]: I1030 19:15:08.615061    2577 server.go:317] "Adding debug handlers to kubelet server"
Oct 30 19:15:08.618837 ip-10-0-132-82 kubenswrapper[2577]: E1030 19:15:08.618813    2577 kubelet.go:1679] "Image garbage collection failed once. Stats initialization may not have completed yet" err="failed to get imageFs info: unable to find data in memory cache"
Oct 30 19:15:08.622360 ip-10-0-132-82 kubenswrapper[2577]: I1030 19:15:08.622341    2577 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kubelet-serving"
Oct 30 19:15:08.623203 ip-10-0-132-82 kubenswrapper[2577]: I1030 19:15:08.623190    2577 fs_resource_analyzer.go:69] "Starting FS ResourceAnalyzer"
Oct 30 19:15:08.623401 ip-10-0-132-82 kubenswrapper[2577]: E1030 19:15:08.623381    2577 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-132-82.us-west-1.compute.internal\" not found"
Oct 30 19:15:08.625124 ip-10-0-132-82 kubenswrapper[2577]: I1030 19:15:08.624830    2577 desired_state_of_world_populator.go:146] "Desired state populator starts to run"
Oct 30 19:15:08.625124 ip-10-0-132-82 kubenswrapper[2577]: I1030 19:15:08.625069    2577 factory.go:55] Registering systemd factory
Oct 30 19:15:08.625124 ip-10-0-132-82 kubenswrapper[2577]: I1030 19:15:08.625082    2577 factory.go:223] Registration of the systemd container factory successfully
Oct 30 19:15:08.625290 ip-10-0-132-82 kubenswrapper[2577]: I1030 19:15:08.625274    2577 volume_manager.go:309] "The desired_state_of_world populator starts"
Oct 30 19:15:08.625290 ip-10-0-132-82 kubenswrapper[2577]: I1030 19:15:08.625290    2577 volume_manager.go:311] "Starting Kubelet Volume Manager"
Oct 30 19:15:08.625417 ip-10-0-132-82 kubenswrapper[2577]: I1030 19:15:08.625403    2577 reconstruct.go:97] "Volume reconstruction finished"
Oct 30 19:15:08.625417 ip-10-0-132-82 kubenswrapper[2577]: I1030 19:15:08.625412    2577 reconciler.go:29] "Reconciler: start to sync state"
Oct 30 19:15:08.625508 ip-10-0-132-82 kubenswrapper[2577]: I1030 19:15:08.625474    2577 factory.go:153] Registering CRI-O factory
Oct 30 19:15:08.625508 ip-10-0-132-82 kubenswrapper[2577]: I1030 19:15:08.625497    2577 factory.go:223] Registration of the crio container factory successfully
Oct 30 19:15:08.625614 ip-10-0-132-82 kubenswrapper[2577]: I1030 19:15:08.625546    2577 factory.go:221] Registration of the containerd container factory failed: unable to create containerd client: containerd: cannot unix dial containerd api service: dial unix /run/containerd/containerd.sock: connect: no such file or directory
Oct 30 19:15:08.625614 ip-10-0-132-82 kubenswrapper[2577]: I1030 19:15:08.625565    2577 factory.go:103] Registering Raw factory
Oct 30 19:15:08.625614 ip-10-0-132-82 kubenswrapper[2577]: I1030 19:15:08.625582    2577 manager.go:1199] Started watching for new ooms in manager
Oct 30 19:15:08.626590 ip-10-0-132-82 kubenswrapper[2577]: I1030 19:15:08.626577    2577 manager.go:319] Starting recovery of all containers
Oct 30 19:15:08.635824 ip-10-0-132-82 kubenswrapper[2577]: I1030 19:15:08.635812    2577 manager.go:324] Recovery completed
Oct 30 19:15:08.637326 ip-10-0-132-82 kubenswrapper[2577]: E1030 19:15:08.637299    2577 watcher.go:152] Failed to watch directory "/sys/fs/cgroup/system.slice/systemd-update-utmp-runlevel.service": inotify_add_watch /sys/fs/cgroup/system.slice/systemd-update-utmp-runlevel.service: no such file or directory
Oct 30 19:15:08.639938 ip-10-0-132-82 kubenswrapper[2577]: I1030 19:15:08.639909    2577 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach"
Oct 30 19:15:08.641421 ip-10-0-132-82 kubenswrapper[2577]: I1030 19:15:08.641396    2577 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-132-82.us-west-1.compute.internal" event="NodeHasSufficientMemory"
Oct 30 19:15:08.641498 ip-10-0-132-82 kubenswrapper[2577]: I1030 19:15:08.641432    2577 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-132-82.us-west-1.compute.internal" event="NodeHasNoDiskPressure"
Oct 30 19:15:08.641498 ip-10-0-132-82 kubenswrapper[2577]: I1030 19:15:08.641443    2577 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-132-82.us-west-1.compute.internal" event="NodeHasSufficientPID"
Oct 30 19:15:08.641898 ip-10-0-132-82 kubenswrapper[2577]: I1030 19:15:08.641885    2577 cpu_manager.go:226] "Starting" policy="none"
Oct 30 19:15:08.641992 ip-10-0-132-82 kubenswrapper[2577]: I1030 19:15:08.641898    2577 cpu_manager.go:227] "Reconciling" reconcilePeriod="10s"
Oct 30 19:15:08.641992 ip-10-0-132-82 kubenswrapper[2577]: I1030 19:15:08.641891    2577 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach"
Oct 30 19:15:08.642047 ip-10-0-132-82 kubenswrapper[2577]: I1030 19:15:08.641944    2577 state_mem.go:41] "Initialized" logger="CPUManager state checkpoint.CPUManager state memory"
Oct 30 19:15:08.642560 ip-10-0-132-82 kubenswrapper[2577]: I1030 19:15:08.642547    2577 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-132-82.us-west-1.compute.internal" event="NodeHasSufficientMemory"
Oct 30 19:15:08.642616 ip-10-0-132-82 kubenswrapper[2577]: I1030 19:15:08.642571    2577 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-132-82.us-west-1.compute.internal" event="NodeHasNoDiskPressure"
Oct 30 19:15:08.642616 ip-10-0-132-82 kubenswrapper[2577]: I1030 19:15:08.642581    2577 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-132-82.us-west-1.compute.internal" event="NodeHasSufficientPID"
Oct 30 19:15:08.644011 ip-10-0-132-82 kubenswrapper[2577]: I1030 19:15:08.643998    2577 policy_none.go:50] "Start"
Oct 30 19:15:08.644060 ip-10-0-132-82 kubenswrapper[2577]: I1030 19:15:08.644021    2577 memory_manager.go:187] "Starting memorymanager" policy="None"
Oct 30 19:15:08.644060 ip-10-0-132-82 kubenswrapper[2577]: I1030 19:15:08.644031    2577 state_mem.go:36] "Initializing new in-memory state store" logger="Memory Manager state checkpoint"
Oct 30 19:15:08.646237 ip-10-0-132-82 kubenswrapper[2577]: I1030 19:15:08.646217    2577 policy_none.go:44] "Start"
Oct 30 19:15:08.707983 ip-10-0-132-82 kubenswrapper[2577]: I1030 19:15:08.687535    2577 node_container_manager_linux.go:130] "Enforcing system reserved compressible on cgroup" cgroupName="/system.slice" limits={"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"}
Oct 30 19:15:08.707983 ip-10-0-132-82 kubenswrapper[2577]: I1030 19:15:08.695460    2577 manager.go:341] "Starting Device Plugin manager"
Oct 30 19:15:08.707983 ip-10-0-132-82 kubenswrapper[2577]: E1030 19:15:08.695495    2577 manager.go:525] "Failed to read data from checkpoint" err="checkpoint is not found" checkpoint="kubelet_internal_checkpoint"
Oct 30 19:15:08.707983 ip-10-0-132-82 kubenswrapper[2577]: I1030 19:15:08.695507    2577 server.go:87] "Starting device plugin registration server"
Oct 30 19:15:08.707983 ip-10-0-132-82 kubenswrapper[2577]: I1030 19:15:08.695755    2577 eviction_manager.go:194] "Eviction manager: starting control loop"
Oct 30 19:15:08.707983 ip-10-0-132-82 kubenswrapper[2577]: I1030 19:15:08.695764    2577 container_log_manager.go:146] "Initializing container log rotate workers" workers=1 monitorPeriod="10s"
Oct 30 19:15:08.707983 ip-10-0-132-82 kubenswrapper[2577]: I1030 19:15:08.695966    2577 plugin_watcher.go:53] "Plugin Watcher Start" path="/var/lib/kubelet/plugins_registry"
Oct 30 19:15:08.707983 ip-10-0-132-82 kubenswrapper[2577]: I1030 19:15:08.696043    2577 plugin_manager.go:119] "The desired_state_of_world populator (plugin watcher) starts"
Oct 30 19:15:08.707983 ip-10-0-132-82 kubenswrapper[2577]: I1030 19:15:08.696050    2577 plugin_manager.go:121] "Starting Kubelet Plugin Manager"
Oct 30 19:15:08.707983 ip-10-0-132-82 kubenswrapper[2577]: E1030 19:15:08.696557    2577 eviction_manager.go:272] "eviction manager: failed to check if we have separate container filesystem. Ignoring." err="non-existent label \"crio-containers\""
Oct 30 19:15:08.707983 ip-10-0-132-82 kubenswrapper[2577]: E1030 19:15:08.696595    2577 eviction_manager.go:297] "Eviction manager: failed to get summary stats" err="failed to get node info: node \"ip-10-0-132-82.us-west-1.compute.internal\" not found"
Oct 30 19:15:08.707983 ip-10-0-132-82 kubenswrapper[2577]: I1030 19:15:08.705563    2577 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv4"
Oct 30 19:15:08.707983 ip-10-0-132-82 kubenswrapper[2577]: I1030 19:15:08.706853    2577 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv6"
Oct 30 19:15:08.707983 ip-10-0-132-82 kubenswrapper[2577]: I1030 19:15:08.706869    2577 status_manager.go:249] "Starting to sync pod status with apiserver"
Oct 30 19:15:08.707983 ip-10-0-132-82 kubenswrapper[2577]: I1030 19:15:08.706887    2577 kubelet.go:2521] "Starting kubelet main sync loop"
Oct 30 19:15:08.707983 ip-10-0-132-82 kubenswrapper[2577]: E1030 19:15:08.706916    2577 kubelet.go:2545] "Skipping pod synchronization" err="PLEG is not healthy: pleg has yet to be successful"
Oct 30 19:15:08.746722 ip-10-0-132-82 kubenswrapper[2577]: I1030 19:15:08.746697    2577 csi_plugin.go:990] Failed to contact API server when waiting for CSINode publishing: csinodes.storage.k8s.io "ip-10-0-132-82.us-west-1.compute.internal" is forbidden: User "system:anonymous" cannot get resource "csinodes" in API group "storage.k8s.io" at the cluster scope
Oct 30 19:15:08.746722 ip-10-0-132-82 kubenswrapper[2577]: E1030 19:15:08.746697    2577 controller.go:201] "Failed to ensure lease exists, will retry" err="leases.coordination.k8s.io \"ip-10-0-132-82.us-west-1.compute.internal\" is forbidden: User \"system:anonymous\" cannot get resource \"leases\" in API group \"coordination.k8s.io\" in the namespace \"kube-node-lease\"" interval="200ms"
Oct 30 19:15:08.747764 ip-10-0-132-82 kubenswrapper[2577]: E1030 19:15:08.746690    2577 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-132-82.us-west-1.compute.internal.18e3649e842b6e4c  default    0 0001-01-01 00:00:00 +0000 UTC <nil> <nil> map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-132-82.us-west-1.compute.internal,UID:,APIVersion:v1,ResourceVersion:,FieldPath:,},Reason:Starting,Message:Starting kubelet.,Source:EventSource{Component:kubelet,Host:ip-10-0-132-82.us-west-1.compute.internal,},FirstTimestamp:2026-10-30 19:15:08.60863854 +0000 UTC m=+0.338599814,LastTimestamp:2026-10-30 19:15:08.60863854 +0000 UTC m=+0.338599814,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-132-82.us-west-1.compute.internal,}"
Oct 30 19:15:08.795985 ip-10-0-132-82 kubenswrapper[2577]: I1030 19:15:08.795908    2577 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach"
Oct 30 19:15:08.797813 ip-10-0-132-82 kubenswrapper[2577]: I1030 19:15:08.797793    2577 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-132-82.us-west-1.compute.internal" event="NodeHasSufficientMemory"
Oct 30 19:15:08.797909 ip-10-0-132-82 kubenswrapper[2577]: I1030 19:15:08.797830    2577 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-132-82.us-west-1.compute.internal" event="NodeHasNoDiskPressure"
Oct 30 19:15:08.797909 ip-10-0-132-82 kubenswrapper[2577]: I1030 19:15:08.797846    2577 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-132-82.us-west-1.compute.internal" event="NodeHasSufficientPID"
Oct 30 19:15:08.797909 ip-10-0-132-82 kubenswrapper[2577]: I1030 19:15:08.797882    2577 kubelet_node_status.go:77] "Attempting to register node" node="ip-10-0-132-82.us-west-1.compute.internal"
Oct 30 19:15:08.799549 ip-10-0-132-82 kubenswrapper[2577]: E1030 19:15:08.799528    2577 reflector.go:204] "Failed to watch" err="failed to list *v1.Service: services is forbidden: User \"system:anonymous\" cannot list resource \"services\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:161" type="*v1.Service"
Oct 30 19:15:08.799671 ip-10-0-132-82 kubenswrapper[2577]: E1030 19:15:08.799536    2577 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-132-82.us-west-1.compute.internal.18e3649e861fa097  default    0 0001-01-01 00:00:00 +0000 UTC <nil> <nil> map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-132-82.us-west-1.compute.internal,UID:,APIVersion:v1,ResourceVersion:,FieldPath:,},Reason:NodeHasSufficientMemory,Message:Node ip-10-0-132-82.us-west-1.compute.internal status is now: NodeHasSufficientMemory,Source:EventSource{Component:kubelet,Host:ip-10-0-132-82.us-west-1.compute.internal,},FirstTimestamp:2026-10-30 19:15:08.641419415 +0000 UTC m=+0.371380693,LastTimestamp:2026-10-30 19:15:08.641419415 +0000 UTC m=+0.371380693,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-132-82.us-west-1.compute.internal,}"
Oct 30 19:15:08.807755 ip-10-0-132-82 kubenswrapper[2577]: I1030 19:15:08.807736    2577 kubelet.go:2608] "SyncLoop ADD" source="file" pods=["openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-132-82.us-west-1.compute.internal","kube-system/kube-apiserver-proxy-ip-10-0-132-82.us-west-1.compute.internal"]
Oct 30 19:15:08.807826 ip-10-0-132-82 kubenswrapper[2577]: I1030 19:15:08.807789    2577 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach"
Oct 30 19:15:08.808652 ip-10-0-132-82 kubenswrapper[2577]: I1030 19:15:08.808635    2577 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-132-82.us-west-1.compute.internal" event="NodeHasSufficientMemory"
Oct 30 19:15:08.808751 ip-10-0-132-82 kubenswrapper[2577]: I1030 19:15:08.808660    2577 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-132-82.us-west-1.compute.internal" event="NodeHasNoDiskPressure"
Oct 30 19:15:08.808751 ip-10-0-132-82 kubenswrapper[2577]: I1030 19:15:08.808669    2577 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-132-82.us-west-1.compute.internal" event="NodeHasSufficientPID"
Oct 30 19:15:08.810842 ip-10-0-132-82 kubenswrapper[2577]: I1030 19:15:08.810830    2577 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach"
Oct 30 19:15:08.811011 ip-10-0-132-82 kubenswrapper[2577]: I1030 19:15:08.810997    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-132-82.us-west-1.compute.internal"
Oct 30 19:15:08.811051 ip-10-0-132-82 kubenswrapper[2577]: I1030 19:15:08.811026    2577 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach"
Oct 30 19:15:08.811581 ip-10-0-132-82 kubenswrapper[2577]: I1030 19:15:08.811563    2577 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-132-82.us-west-1.compute.internal" event="NodeHasSufficientMemory"
Oct 30 19:15:08.811674 ip-10-0-132-82 kubenswrapper[2577]: I1030 19:15:08.811583    2577 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-132-82.us-west-1.compute.internal" event="NodeHasSufficientMemory"
Oct 30 19:15:08.811674 ip-10-0-132-82 kubenswrapper[2577]: I1030 19:15:08.811595    2577 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-132-82.us-west-1.compute.internal" event="NodeHasNoDiskPressure"
Oct 30 19:15:08.811674 ip-10-0-132-82 kubenswrapper[2577]: I1030 19:15:08.811602    2577 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-132-82.us-west-1.compute.internal" event="NodeHasNoDiskPressure"
Oct 30 19:15:08.811674 ip-10-0-132-82 kubenswrapper[2577]: I1030 19:15:08.811609    2577 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-132-82.us-west-1.compute.internal" event="NodeHasSufficientPID"
Oct 30 19:15:08.811674 ip-10-0-132-82 kubenswrapper[2577]: I1030 19:15:08.811613    2577 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-132-82.us-west-1.compute.internal" event="NodeHasSufficientPID"
Oct 30 19:15:08.813704 ip-10-0-132-82 kubenswrapper[2577]: I1030 19:15:08.813688    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-132-82.us-west-1.compute.internal"
Oct 30 19:15:08.813762 ip-10-0-132-82 kubenswrapper[2577]: I1030 19:15:08.813720    2577 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach"
Oct 30 19:15:08.814465 ip-10-0-132-82 kubenswrapper[2577]: I1030 19:15:08.814449    2577 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-132-82.us-west-1.compute.internal" event="NodeHasSufficientMemory"
Oct 30 19:15:08.814531 ip-10-0-132-82 kubenswrapper[2577]: I1030 19:15:08.814483    2577 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-132-82.us-west-1.compute.internal" event="NodeHasNoDiskPressure"
Oct 30 19:15:08.814531 ip-10-0-132-82 kubenswrapper[2577]: I1030 19:15:08.814500    2577 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-132-82.us-west-1.compute.internal" event="NodeHasSufficientPID"
Oct 30 19:15:08.826160 ip-10-0-132-82 kubenswrapper[2577]: I1030 19:15:08.826145    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/16007dbb15fdd01822b4d377fac4e8af-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-132-82.us-west-1.compute.internal\" (UID: \"16007dbb15fdd01822b4d377fac4e8af\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-132-82.us-west-1.compute.internal"
Oct 30 19:15:08.826225 ip-10-0-132-82 kubenswrapper[2577]: I1030 19:15:08.826168    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/16007dbb15fdd01822b4d377fac4e8af-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-132-82.us-west-1.compute.internal\" (UID: \"16007dbb15fdd01822b4d377fac4e8af\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-132-82.us-west-1.compute.internal"
Oct 30 19:15:08.826225 ip-10-0-132-82 kubenswrapper[2577]: I1030 19:15:08.826186    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/f9af2f5b3d8deb871a93a1e3f0011dc6-config\") pod \"kube-apiserver-proxy-ip-10-0-132-82.us-west-1.compute.internal\" (UID: \"f9af2f5b3d8deb871a93a1e3f0011dc6\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-132-82.us-west-1.compute.internal"
Oct 30 19:15:08.845514 ip-10-0-132-82 kubenswrapper[2577]: E1030 19:15:08.845492    2577 kubelet.go:3364] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-132-82.us-west-1.compute.internal\" not found" node="ip-10-0-132-82.us-west-1.compute.internal"
Oct 30 19:15:08.848083 ip-10-0-132-82 kubenswrapper[2577]: E1030 19:15:08.848068    2577 kubelet.go:3364] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-132-82.us-west-1.compute.internal\" not found" node="ip-10-0-132-82.us-west-1.compute.internal"
Oct 30 19:15:08.877084 ip-10-0-132-82 kubenswrapper[2577]: I1030 19:15:08.877066    2577 reflector.go:446] "Caches populated" logger="kubernetes.io/kube-apiserver-client-kubelet" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162"
Oct 30 19:15:08.877191 ip-10-0-132-82 kubenswrapper[2577]: I1030 19:15:08.877144    2577 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-crx6s"
Oct 30 19:15:08.905198 ip-10-0-132-82 kubenswrapper[2577]: E1030 19:15:08.905176    2577 reflector.go:204] "Failed to watch" err="failed to list *v1.Node: nodes \"ip-10-0-132-82.us-west-1.compute.internal\" is forbidden: User \"system:anonymous\" cannot list resource \"nodes\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:161" type="*v1.Node"
Oct 30 19:15:08.905293 ip-10-0-132-82 kubenswrapper[2577]: E1030 19:15:08.905177    2577 reflector.go:204] "Failed to watch" err="failed to list *v1.RuntimeClass: runtimeclasses.node.k8s.io is forbidden: User \"system:anonymous\" cannot list resource \"runtimeclasses\" in API group \"node.k8s.io\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:161" type="*v1.RuntimeClass"
Oct 30 19:15:08.905293 ip-10-0-132-82 kubenswrapper[2577]: E1030 19:15:08.905156    2577 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-132-82.us-west-1.compute.internal.18e3649e861fe64e  default    0 0001-01-01 00:00:00 +0000 UTC <nil> <nil> map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-132-82.us-west-1.compute.internal,UID:,APIVersion:v1,ResourceVersion:,FieldPath:,},Reason:NodeHasNoDiskPressure,Message:Node ip-10-0-132-82.us-west-1.compute.internal status is now: NodeHasNoDiskPressure,Source:EventSource{Component:kubelet,Host:ip-10-0-132-82.us-west-1.compute.internal,},FirstTimestamp:2026-10-30 19:15:08.641437262 +0000 UTC m=+0.371398537,LastTimestamp:2026-10-30 19:15:08.641437262 +0000 UTC m=+0.371398537,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-132-82.us-west-1.compute.internal,}"
Oct 30 19:15:08.905461 ip-10-0-132-82 kubenswrapper[2577]: E1030 19:15:08.905442    2577 reflector.go:204] "Failed to watch" err="failed to list *v1.CSIDriver: csidrivers.storage.k8s.io is forbidden: User \"system:anonymous\" cannot list resource \"csidrivers\" in API group \"storage.k8s.io\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:161" type="*v1.CSIDriver"
Oct 30 19:15:08.926481 ip-10-0-132-82 kubenswrapper[2577]: I1030 19:15:08.926458    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/16007dbb15fdd01822b4d377fac4e8af-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-132-82.us-west-1.compute.internal\" (UID: \"16007dbb15fdd01822b4d377fac4e8af\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-132-82.us-west-1.compute.internal"
Oct 30 19:15:08.926557 ip-10-0-132-82 kubenswrapper[2577]: I1030 19:15:08.926484    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/16007dbb15fdd01822b4d377fac4e8af-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-132-82.us-west-1.compute.internal\" (UID: \"16007dbb15fdd01822b4d377fac4e8af\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-132-82.us-west-1.compute.internal"
Oct 30 19:15:08.926557 ip-10-0-132-82 kubenswrapper[2577]: I1030 19:15:08.926500    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/f9af2f5b3d8deb871a93a1e3f0011dc6-config\") pod \"kube-apiserver-proxy-ip-10-0-132-82.us-west-1.compute.internal\" (UID: \"f9af2f5b3d8deb871a93a1e3f0011dc6\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-132-82.us-west-1.compute.internal"
Oct 30 19:15:08.926557 ip-10-0-132-82 kubenswrapper[2577]: I1030 19:15:08.926547    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/host-path/f9af2f5b3d8deb871a93a1e3f0011dc6-config\") pod \"kube-apiserver-proxy-ip-10-0-132-82.us-west-1.compute.internal\" (UID: \"f9af2f5b3d8deb871a93a1e3f0011dc6\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-132-82.us-west-1.compute.internal"
Oct 30 19:15:08.926652 ip-10-0-132-82 kubenswrapper[2577]: I1030 19:15:08.926567    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/16007dbb15fdd01822b4d377fac4e8af-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-132-82.us-west-1.compute.internal\" (UID: \"16007dbb15fdd01822b4d377fac4e8af\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-132-82.us-west-1.compute.internal"
Oct 30 19:15:08.926652 ip-10-0-132-82 kubenswrapper[2577]: I1030 19:15:08.926567    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/16007dbb15fdd01822b4d377fac4e8af-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-132-82.us-west-1.compute.internal\" (UID: \"16007dbb15fdd01822b4d377fac4e8af\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-132-82.us-west-1.compute.internal"
Oct 30 19:15:08.957504 ip-10-0-132-82 kubenswrapper[2577]: E1030 19:15:08.957482    2577 kubelet_node_status.go:115] "Unable to register node with API server, error getting existing node" err="nodes \"ip-10-0-132-82.us-west-1.compute.internal\" not found" node="ip-10-0-132-82.us-west-1.compute.internal"
Oct 30 19:15:09.052904 ip-10-0-132-82 kubenswrapper[2577]: E1030 19:15:09.052849    2577 nodelease.go:50] "Failed to get node when trying to set owner ref to the node lease" err="nodes \"ip-10-0-132-82.us-west-1.compute.internal\" not found" node="ip-10-0-132-82.us-west-1.compute.internal"
Oct 30 19:15:09.149554 ip-10-0-132-82 kubenswrapper[2577]: I1030 19:15:09.149530    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-132-82.us-west-1.compute.internal"
Oct 30 19:15:09.153449 ip-10-0-132-82 kubenswrapper[2577]: I1030 19:15:09.153435    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-132-82.us-west-1.compute.internal"
Oct 30 19:15:09.157596 ip-10-0-132-82 kubenswrapper[2577]: I1030 19:15:09.157580    2577 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach"
Oct 30 19:15:09.158655 ip-10-0-132-82 kubenswrapper[2577]: I1030 19:15:09.158640    2577 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-132-82.us-west-1.compute.internal" event="NodeHasSufficientMemory"
Oct 30 19:15:09.158733 ip-10-0-132-82 kubenswrapper[2577]: I1030 19:15:09.158667    2577 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-132-82.us-west-1.compute.internal" event="NodeHasNoDiskPressure"
Oct 30 19:15:09.158733 ip-10-0-132-82 kubenswrapper[2577]: I1030 19:15:09.158680    2577 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-132-82.us-west-1.compute.internal" event="NodeHasSufficientPID"
Oct 30 19:15:09.158733 ip-10-0-132-82 kubenswrapper[2577]: I1030 19:15:09.158704    2577 kubelet_node_status.go:77] "Attempting to register node" node="ip-10-0-132-82.us-west-1.compute.internal"
Oct 30 19:15:09.217401 ip-10-0-132-82 kubenswrapper[2577]: I1030 19:15:09.217380    2577 kubelet_node_status.go:80] "Successfully registered node" node="ip-10-0-132-82.us-west-1.compute.internal"
Oct 30 19:15:09.217401 ip-10-0-132-82 kubenswrapper[2577]: E1030 19:15:09.217400    2577 kubelet_node_status.go:524] "Error updating node status, will retry" err="error getting node \"ip-10-0-132-82.us-west-1.compute.internal\": node \"ip-10-0-132-82.us-west-1.compute.internal\" not found"
Oct 30 19:15:09.328007 ip-10-0-132-82 kubenswrapper[2577]: E1030 19:15:09.327956    2577 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-132-82.us-west-1.compute.internal\" not found"
Oct 30 19:15:09.428486 ip-10-0-132-82 kubenswrapper[2577]: E1030 19:15:09.428463    2577 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-132-82.us-west-1.compute.internal\" not found"
Oct 30 19:15:09.528697 ip-10-0-132-82 kubenswrapper[2577]: E1030 19:15:09.528677    2577 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-132-82.us-west-1.compute.internal\" not found"
Oct 30 19:15:09.550823 ip-10-0-132-82 kubenswrapper[2577]: I1030 19:15:09.550805    2577 transport.go:147] "Certificate rotation detected, shutting down client connections to start using new credentials"
Oct 30 19:15:09.551036 ip-10-0-132-82 kubenswrapper[2577]: E1030 19:15:09.550963    2577 event.go:368] "Unable to write event (may retry after sleeping)" err="Patch \"https://a99e23be2ae284760adacf985437c851-69fa068035f08793.elb.us-east-2.amazonaws.com:6443/api/v1/namespaces/default/events/ip-10-0-132-82.us-west-1.compute.internal.18e3649e861fa097\": read tcp 10.0.132.82:37550->18.218.190.50:6443: use of closed network connection" event="&Event{ObjectMeta:{ip-10-0-132-82.us-west-1.compute.internal.18e3649e861fa097  default   4798 0 0001-01-01 00:00:00 +0000 UTC <nil> <nil> map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-132-82.us-west-1.compute.internal,UID:,APIVersion:v1,ResourceVersion:,FieldPath:,},Reason:NodeHasSufficientMemory,Message:Node ip-10-0-132-82.us-west-1.compute.internal status is now: NodeHasSufficientMemory,Source:EventSource{Component:kubelet,Host:ip-10-0-132-82.us-west-1.compute.internal,},FirstTimestamp:2026-10-30 19:15:08 +0000 UTC,LastTimestamp:2026-10-30 19:15:08.808650205 +0000 UTC m=+0.538611480,Count:4,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-132-82.us-west-1.compute.internal,}"
Oct 30 19:15:09.629652 ip-10-0-132-82 kubenswrapper[2577]: E1030 19:15:09.629606    2577 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-132-82.us-west-1.compute.internal\" not found"
Oct 30 19:15:09.730013 ip-10-0-132-82 kubenswrapper[2577]: E1030 19:15:09.729991    2577 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-132-82.us-west-1.compute.internal\" not found"
Oct 30 19:15:09.749773 ip-10-0-132-82 kubenswrapper[2577]: I1030 19:15:09.749755    2577 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach"
Oct 30 19:15:09.750467 ip-10-0-132-82 kubenswrapper[2577]: I1030 19:15:09.750451    2577 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-132-82.us-west-1.compute.internal" event="NodeHasSufficientMemory"
Oct 30 19:15:09.750544 ip-10-0-132-82 kubenswrapper[2577]: I1030 19:15:09.750487    2577 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-132-82.us-west-1.compute.internal" event="NodeHasNoDiskPressure"
Oct 30 19:15:09.750544 ip-10-0-132-82 kubenswrapper[2577]: I1030 19:15:09.750501    2577 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-132-82.us-west-1.compute.internal" event="NodeHasSufficientPID"
Oct 30 19:15:09.830302 ip-10-0-132-82 kubenswrapper[2577]: E1030 19:15:09.830280    2577 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-132-82.us-west-1.compute.internal\" not found"
Oct 30 19:15:09.883090 ip-10-0-132-82 kubenswrapper[2577]: I1030 19:15:09.883026    2577 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kube-apiserver-client-kubelet" expiration="2028-10-29 19:10:10 +0000 UTC" deadline="2028-07-05 07:14:00.34430473 +0000 UTC"
Oct 30 19:15:09.883090 ip-10-0-132-82 kubenswrapper[2577]: I1030 19:15:09.883051    2577 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kube-apiserver-client-kubelet" sleep="14723h58m50.461258165s"
Oct 30 19:15:09.910245 ip-10-0-132-82 kubenswrapper[2577]: I1030 19:15:09.910221    2577 reflector.go:446] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:161"
Oct 30 19:15:09.930815 ip-10-0-132-82 kubenswrapper[2577]: E1030 19:15:09.930792    2577 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-132-82.us-west-1.compute.internal\" not found"
Oct 30 19:15:09.961909 ip-10-0-132-82 kubenswrapper[2577]: I1030 19:15:09.961889    2577 reflector.go:446] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:161"
Oct 30 19:15:10.031346 ip-10-0-132-82 kubenswrapper[2577]: E1030 19:15:10.031326    2577 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-132-82.us-west-1.compute.internal\" not found"
Oct 30 19:15:10.061711 ip-10-0-132-82 kubenswrapper[2577]: I1030 19:15:10.061688    2577 reflector.go:446] "Caches populated" type="*v1.Service" reflector="k8s.io/client-go/informers/factory.go:161"
Oct 30 19:15:10.132189 ip-10-0-132-82 kubenswrapper[2577]: E1030 19:15:10.132160    2577 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-132-82.us-west-1.compute.internal\" not found"
Oct 30 19:15:11.556689 ip-10-0-132-82 kubenswrapper[2577]: E1030 19:15:11.556644    2577 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-132-82.us-west-1.compute.internal\" not found"
Oct 30 19:15:11.600123 ip-10-0-132-82 kubenswrapper[2577]: W1030 19:15:11.600087    2577 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf9af2f5b3d8deb871a93a1e3f0011dc6.slice/crio-3bea07ac8a9e43055344e71268ef13eda8a7b0eb15593518b8076be757e09c3a WatchSource:0}: Error finding container 3bea07ac8a9e43055344e71268ef13eda8a7b0eb15593518b8076be757e09c3a: Status 404 returned error can't find the container with id 3bea07ac8a9e43055344e71268ef13eda8a7b0eb15593518b8076be757e09c3a
Oct 30 19:15:11.601069 ip-10-0-132-82 kubenswrapper[2577]: W1030 19:15:11.601044    2577 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod16007dbb15fdd01822b4d377fac4e8af.slice/crio-b2829769d2e875408df83ae72236364dc323bc32fa4adc204d20ae79308ef283 WatchSource:0}: Error finding container b2829769d2e875408df83ae72236364dc323bc32fa4adc204d20ae79308ef283: Status 404 returned error can't find the container with id b2829769d2e875408df83ae72236364dc323bc32fa4adc204d20ae79308ef283
Oct 30 19:15:11.608226 ip-10-0-132-82 kubenswrapper[2577]: I1030 19:15:11.608207    2577 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Oct 30 19:15:11.657481 ip-10-0-132-82 kubenswrapper[2577]: E1030 19:15:11.657449    2577 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-132-82.us-west-1.compute.internal\" not found"
Oct 30 19:15:11.687917 ip-10-0-132-82 kubenswrapper[2577]: I1030 19:15:11.687902    2577 reflector.go:446] "Caches populated" type="*v1.Node" reflector="k8s.io/client-go/informers/factory.go:161"
Oct 30 19:15:11.748005 ip-10-0-132-82 kubenswrapper[2577]: I1030 19:15:11.747937    2577 kubelet.go:3368] "Creating a mirror pod for static pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-132-82.us-west-1.compute.internal"
Oct 30 19:15:11.811070 ip-10-0-132-82 kubenswrapper[2577]: I1030 19:15:11.811050    2577 warnings.go:107] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]"
Oct 30 19:15:11.812102 ip-10-0-132-82 kubenswrapper[2577]: I1030 19:15:11.812089    2577 kubelet.go:3368] "Creating a mirror pod for static pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-132-82.us-west-1.compute.internal"
Oct 30 19:15:11.870217 ip-10-0-132-82 kubenswrapper[2577]: I1030 19:15:11.870199    2577 warnings.go:107] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]"
Oct 30 19:15:11.915473 ip-10-0-132-82 kubenswrapper[2577]: I1030 19:15:11.915461    2577 apiserver.go:52] "Watching apiserver"
Oct 30 19:15:11.947647 ip-10-0-132-82 kubenswrapper[2577]: I1030 19:15:11.947630    2577 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kubelet-serving"
Oct 30 19:15:12.018487 ip-10-0-132-82 kubenswrapper[2577]: I1030 19:15:12.018439    2577 reflector.go:446] "Caches populated" type="*v1.Pod" reflector="pkg/kubelet/config/apiserver.go:66"
Oct 30 19:15:12.018726 ip-10-0-132-82 kubenswrapper[2577]: I1030 19:15:12.018710    2577 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-132-82.us-west-1.compute.internal","openshift-multus/network-metrics-daemon-822gp","kube-system/global-pull-secret-syncer-76rht","openshift-multus/multus-jvbw5","openshift-network-operator/iptables-alerter-wnn66","openshift-ovn-kubernetes/ovnkube-node-5lwkx","kube-system/kube-apiserver-proxy-ip-10-0-132-82.us-west-1.compute.internal","openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-qgv98","openshift-multus/multus-additional-cni-plugins-4pvgg","openshift-network-diagnostics/network-check-target-c24j9","kube-system/konnectivity-agent-qnm56","openshift-cluster-node-tuning-operator/tuned-2fv9z","openshift-dns/node-resolver-g9xd6","openshift-image-registry/node-ca-kgfvh"]
Oct 30 19:15:12.022349 ip-10-0-132-82 kubenswrapper[2577]: I1030 19:15:12.022335    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-c24j9"
Oct 30 19:15:12.022409 ip-10-0-132-82 kubenswrapper[2577]: E1030 19:15:12.022392    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-c24j9" podUID="4ca01b49-9819-4a90-a851-7553ad568679"
Oct 30 19:15:12.024287 ip-10-0-132-82 kubenswrapper[2577]: I1030 19:15:12.024274    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-g9xd6"
Oct 30 19:15:12.026348 ip-10-0-132-82 kubenswrapper[2577]: I1030 19:15:12.026327    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-76rht"
Oct 30 19:15:12.026437 ip-10-0-132-82 kubenswrapper[2577]: E1030 19:15:12.026390    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-76rht" podUID="c16c0a80-c5ce-4f5d-827a-2e3eb2c3cfb8"
Oct 30 19:15:12.028418 ip-10-0-132-82 kubenswrapper[2577]: I1030 19:15:12.028404    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-jvbw5"
Oct 30 19:15:12.031213 ip-10-0-132-82 kubenswrapper[2577]: I1030 19:15:12.031186    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-wnn66"
Oct 30 19:15:12.033563 ip-10-0-132-82 kubenswrapper[2577]: I1030 19:15:12.033544    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-5lwkx"
Oct 30 19:15:12.035757 ip-10-0-132-82 kubenswrapper[2577]: I1030 19:15:12.035737    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-4pvgg"
Oct 30 19:15:12.037916 ip-10-0-132-82 kubenswrapper[2577]: I1030 19:15:12.037899    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-kgfvh"
Oct 30 19:15:12.040119 ip-10-0-132-82 kubenswrapper[2577]: I1030 19:15:12.040102    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-822gp"
Oct 30 19:15:12.040193 ip-10-0-132-82 kubenswrapper[2577]: E1030 19:15:12.040153    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-822gp" podUID="287ba08f-7b9f-4609-8e35-832c0879c44c"
Oct 30 19:15:12.042328 ip-10-0-132-82 kubenswrapper[2577]: I1030 19:15:12.042312    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-qgv98"
Oct 30 19:15:12.044591 ip-10-0-132-82 kubenswrapper[2577]: I1030 19:15:12.044572    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-qnm56"
Oct 30 19:15:12.046957 ip-10-0-132-82 kubenswrapper[2577]: I1030 19:15:12.046940    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-2fv9z"
Oct 30 19:15:12.049624 ip-10-0-132-82 kubenswrapper[2577]: I1030 19:15:12.049583    2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-132-82.us-west-1.compute.internal" event={"ID":"f9af2f5b3d8deb871a93a1e3f0011dc6","Type":"ContainerStarted","Data":"3bea07ac8a9e43055344e71268ef13eda8a7b0eb15593518b8076be757e09c3a"}
Oct 30 19:15:12.049694 ip-10-0-132-82 kubenswrapper[2577]: I1030 19:15:12.049637    2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-132-82.us-west-1.compute.internal" event={"ID":"16007dbb15fdd01822b4d377fac4e8af","Type":"ContainerStarted","Data":"b2829769d2e875408df83ae72236364dc323bc32fa4adc204d20ae79308ef283"}
Oct 30 19:15:12.049884 ip-10-0-132-82 kubenswrapper[2577]: I1030 19:15:12.049871    2577 desired_state_of_world_populator.go:154] "Finished populating initial desired state of world"
Oct 30 19:15:12.059349 ip-10-0-132-82 kubenswrapper[2577]: I1030 19:15:12.059328    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/39bb8c16-8282-442a-841b-0056baae014d-var-lib-kubelet\") pod \"tuned-2fv9z\" (UID: \"39bb8c16-8282-442a-841b-0056baae014d\") " pod="openshift-cluster-node-tuning-operator/tuned-2fv9z"
Oct 30 19:15:12.059470 ip-10-0-132-82 kubenswrapper[2577]: I1030 19:15:12.059417    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/08ae8ede-9a96-4a84-8ec8-8d364fd476a3-host-kubelet\") pod \"ovnkube-node-5lwkx\" (UID: \"08ae8ede-9a96-4a84-8ec8-8d364fd476a3\") " pod="openshift-ovn-kubernetes/ovnkube-node-5lwkx"
Oct 30 19:15:12.059470 ip-10-0-132-82 kubenswrapper[2577]: I1030 19:15:12.059438    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/08ae8ede-9a96-4a84-8ec8-8d364fd476a3-run-openvswitch\") pod \"ovnkube-node-5lwkx\" (UID: \"08ae8ede-9a96-4a84-8ec8-8d364fd476a3\") " pod="openshift-ovn-kubernetes/ovnkube-node-5lwkx"
Oct 30 19:15:12.059470 ip-10-0-132-82 kubenswrapper[2577]: I1030 19:15:12.059453    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/08ae8ede-9a96-4a84-8ec8-8d364fd476a3-run-ovn\") pod \"ovnkube-node-5lwkx\" (UID: \"08ae8ede-9a96-4a84-8ec8-8d364fd476a3\") " pod="openshift-ovn-kubernetes/ovnkube-node-5lwkx"
Oct 30 19:15:12.059470 ip-10-0-132-82 kubenswrapper[2577]: I1030 19:15:12.059467    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/bab0abf9-46d4-408e-9aa0-afa7279cd9d1-cnibin\") pod \"multus-additional-cni-plugins-4pvgg\" (UID: \"bab0abf9-46d4-408e-9aa0-afa7279cd9d1\") " pod="openshift-multus/multus-additional-cni-plugins-4pvgg"
Oct 30 19:15:12.059609 ip-10-0-132-82 kubenswrapper[2577]: I1030 19:15:12.059499    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/48386105-d79b-4f92-baa0-f330b9658ad0-tmp-dir\") pod \"node-resolver-g9xd6\" (UID: \"48386105-d79b-4f92-baa0-f330b9658ad0\") " pod="openshift-dns/node-resolver-g9xd6"
Oct 30 19:15:12.060771 ip-10-0-132-82 kubenswrapper[2577]: I1030 19:15:12.059921    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/08ae8ede-9a96-4a84-8ec8-8d364fd476a3-var-lib-openvswitch\") pod \"ovnkube-node-5lwkx\" (UID: \"08ae8ede-9a96-4a84-8ec8-8d364fd476a3\") " pod="openshift-ovn-kubernetes/ovnkube-node-5lwkx"
Oct 30 19:15:12.060771 ip-10-0-132-82 kubenswrapper[2577]: I1030 19:15:12.060005    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/dccdaf98-aac7-4756-b93f-22f62a286a1f-cni-binary-copy\") pod \"multus-jvbw5\" (UID: \"dccdaf98-aac7-4756-b93f-22f62a286a1f\") " pod="openshift-multus/multus-jvbw5"
Oct 30 19:15:12.060771 ip-10-0-132-82 kubenswrapper[2577]: I1030 19:15:12.060039    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/dccdaf98-aac7-4756-b93f-22f62a286a1f-host-run-k8s-cni-cncf-io\") pod \"multus-jvbw5\" (UID: \"dccdaf98-aac7-4756-b93f-22f62a286a1f\") " pod="openshift-multus/multus-jvbw5"
Oct 30 19:15:12.060771 ip-10-0-132-82 kubenswrapper[2577]: I1030 19:15:12.060073    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/c3780b96-955e-4728-a2d0-b7ec30d0f6f3-socket-dir\") pod \"aws-ebs-csi-driver-node-qgv98\" (UID: \"c3780b96-955e-4728-a2d0-b7ec30d0f6f3\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-qgv98"
Oct 30 19:15:12.060771 ip-10-0-132-82 kubenswrapper[2577]: I1030 19:15:12.060106    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/c3780b96-955e-4728-a2d0-b7ec30d0f6f3-registration-dir\") pod \"aws-ebs-csi-driver-node-qgv98\" (UID: \"c3780b96-955e-4728-a2d0-b7ec30d0f6f3\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-qgv98"
Oct 30 19:15:12.060771 ip-10-0-132-82 kubenswrapper[2577]: I1030 19:15:12.060166    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/39bb8c16-8282-442a-841b-0056baae014d-run\") pod \"tuned-2fv9z\" (UID: \"39bb8c16-8282-442a-841b-0056baae014d\") " pod="openshift-cluster-node-tuning-operator/tuned-2fv9z"
Oct 30 19:15:12.060771 ip-10-0-132-82 kubenswrapper[2577]: I1030 19:15:12.060216    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/c16c0a80-c5ce-4f5d-827a-2e3eb2c3cfb8-original-pull-secret\") pod \"global-pull-secret-syncer-76rht\" (UID: \"c16c0a80-c5ce-4f5d-827a-2e3eb2c3cfb8\") " pod="kube-system/global-pull-secret-syncer-76rht"
Oct 30 19:15:12.060771 ip-10-0-132-82 kubenswrapper[2577]: I1030 19:15:12.060521    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/08ae8ede-9a96-4a84-8ec8-8d364fd476a3-systemd-units\") pod \"ovnkube-node-5lwkx\" (UID: \"08ae8ede-9a96-4a84-8ec8-8d364fd476a3\") " pod="openshift-ovn-kubernetes/ovnkube-node-5lwkx"
Oct 30 19:15:12.060771 ip-10-0-132-82 kubenswrapper[2577]: I1030 19:15:12.060562    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/dccdaf98-aac7-4756-b93f-22f62a286a1f-multus-cni-dir\") pod \"multus-jvbw5\" (UID: \"dccdaf98-aac7-4756-b93f-22f62a286a1f\") " pod="openshift-multus/multus-jvbw5"
Oct 30 19:15:12.060771 ip-10-0-132-82 kubenswrapper[2577]: I1030 19:15:12.060589    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/dccdaf98-aac7-4756-b93f-22f62a286a1f-host-var-lib-cni-multus\") pod \"multus-jvbw5\" (UID: \"dccdaf98-aac7-4756-b93f-22f62a286a1f\") " pod="openshift-multus/multus-jvbw5"
Oct 30 19:15:12.060771 ip-10-0-132-82 kubenswrapper[2577]: I1030 19:15:12.060611    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/dccdaf98-aac7-4756-b93f-22f62a286a1f-hostroot\") pod \"multus-jvbw5\" (UID: \"dccdaf98-aac7-4756-b93f-22f62a286a1f\") " pod="openshift-multus/multus-jvbw5"
Oct 30 19:15:12.060771 ip-10-0-132-82 kubenswrapper[2577]: I1030 19:15:12.060630    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/39bb8c16-8282-442a-841b-0056baae014d-etc-modprobe-d\") pod \"tuned-2fv9z\" (UID: \"39bb8c16-8282-442a-841b-0056baae014d\") " pod="openshift-cluster-node-tuning-operator/tuned-2fv9z"
Oct 30 19:15:12.062716 ip-10-0-132-82 kubenswrapper[2577]: I1030 19:15:12.061356    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/39bb8c16-8282-442a-841b-0056baae014d-host\") pod \"tuned-2fv9z\" (UID: \"39bb8c16-8282-442a-841b-0056baae014d\") " pod="openshift-cluster-node-tuning-operator/tuned-2fv9z"
Oct 30 19:15:12.062716 ip-10-0-132-82 kubenswrapper[2577]: I1030 19:15:12.061403    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/39bb8c16-8282-442a-841b-0056baae014d-etc-tuned\") pod \"tuned-2fv9z\" (UID: \"39bb8c16-8282-442a-841b-0056baae014d\") " pod="openshift-cluster-node-tuning-operator/tuned-2fv9z"
Oct 30 19:15:12.062716 ip-10-0-132-82 kubenswrapper[2577]: I1030 19:15:12.061456    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/08ae8ede-9a96-4a84-8ec8-8d364fd476a3-host-run-netns\") pod \"ovnkube-node-5lwkx\" (UID: \"08ae8ede-9a96-4a84-8ec8-8d364fd476a3\") " pod="openshift-ovn-kubernetes/ovnkube-node-5lwkx"
Oct 30 19:15:12.062716 ip-10-0-132-82 kubenswrapper[2577]: I1030 19:15:12.061496    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/08ae8ede-9a96-4a84-8ec8-8d364fd476a3-etc-openvswitch\") pod \"ovnkube-node-5lwkx\" (UID: \"08ae8ede-9a96-4a84-8ec8-8d364fd476a3\") " pod="openshift-ovn-kubernetes/ovnkube-node-5lwkx"
Oct 30 19:15:12.062716 ip-10-0-132-82 kubenswrapper[2577]: I1030 19:15:12.061532    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/bab0abf9-46d4-408e-9aa0-afa7279cd9d1-cni-binary-copy\") pod \"multus-additional-cni-plugins-4pvgg\" (UID: \"bab0abf9-46d4-408e-9aa0-afa7279cd9d1\") " pod="openshift-multus/multus-additional-cni-plugins-4pvgg"
Oct 30 19:15:12.062716 ip-10-0-132-82 kubenswrapper[2577]: I1030 19:15:12.061564    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/dccdaf98-aac7-4756-b93f-22f62a286a1f-multus-socket-dir-parent\") pod \"multus-jvbw5\" (UID: \"dccdaf98-aac7-4756-b93f-22f62a286a1f\") " pod="openshift-multus/multus-jvbw5"
Oct 30 19:15:12.062716 ip-10-0-132-82 kubenswrapper[2577]: I1030 19:15:12.061596    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/af36f4d4-f985-4937-a9f7-9884936f3690-agent-certs\") pod \"konnectivity-agent-qnm56\" (UID: \"af36f4d4-f985-4937-a9f7-9884936f3690\") " pod="kube-system/konnectivity-agent-qnm56"
Oct 30 19:15:12.062716 ip-10-0-132-82 kubenswrapper[2577]: I1030 19:15:12.061659    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/08ae8ede-9a96-4a84-8ec8-8d364fd476a3-log-socket\") pod \"ovnkube-node-5lwkx\" (UID: \"08ae8ede-9a96-4a84-8ec8-8d364fd476a3\") " pod="openshift-ovn-kubernetes/ovnkube-node-5lwkx"
Oct 30 19:15:12.062716 ip-10-0-132-82 kubenswrapper[2577]: I1030 19:15:12.061720    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/08ae8ede-9a96-4a84-8ec8-8d364fd476a3-ovnkube-script-lib\") pod \"ovnkube-node-5lwkx\" (UID: \"08ae8ede-9a96-4a84-8ec8-8d364fd476a3\") " pod="openshift-ovn-kubernetes/ovnkube-node-5lwkx"
Oct 30 19:15:12.062716 ip-10-0-132-82 kubenswrapper[2577]: I1030 19:15:12.061760    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/bab0abf9-46d4-408e-9aa0-afa7279cd9d1-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-4pvgg\" (UID: \"bab0abf9-46d4-408e-9aa0-afa7279cd9d1\") " pod="openshift-multus/multus-additional-cni-plugins-4pvgg"
Oct 30 19:15:12.062716 ip-10-0-132-82 kubenswrapper[2577]: I1030 19:15:12.061844    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/bab0abf9-46d4-408e-9aa0-afa7279cd9d1-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-4pvgg\" (UID: \"bab0abf9-46d4-408e-9aa0-afa7279cd9d1\") " pod="openshift-multus/multus-additional-cni-plugins-4pvgg"
Oct 30 19:15:12.062716 ip-10-0-132-82 kubenswrapper[2577]: I1030 19:15:12.061877    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/dccdaf98-aac7-4756-b93f-22f62a286a1f-host-run-netns\") pod \"multus-jvbw5\" (UID: \"dccdaf98-aac7-4756-b93f-22f62a286a1f\") " pod="openshift-multus/multus-jvbw5"
Oct 30 19:15:12.062716 ip-10-0-132-82 kubenswrapper[2577]: I1030 19:15:12.061937    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/dccdaf98-aac7-4756-b93f-22f62a286a1f-host-var-lib-cni-bin\") pod \"multus-jvbw5\" (UID: \"dccdaf98-aac7-4756-b93f-22f62a286a1f\") " pod="openshift-multus/multus-jvbw5"
Oct 30 19:15:12.062716 ip-10-0-132-82 kubenswrapper[2577]: I1030 19:15:12.061976    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/c16c0a80-c5ce-4f5d-827a-2e3eb2c3cfb8-kubelet-config\") pod \"global-pull-secret-syncer-76rht\" (UID: \"c16c0a80-c5ce-4f5d-827a-2e3eb2c3cfb8\") " pod="kube-system/global-pull-secret-syncer-76rht"
Oct 30 19:15:12.062716 ip-10-0-132-82 kubenswrapper[2577]: I1030 19:15:12.062006    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/dccdaf98-aac7-4756-b93f-22f62a286a1f-host-var-lib-kubelet\") pod \"multus-jvbw5\" (UID: \"dccdaf98-aac7-4756-b93f-22f62a286a1f\") " pod="openshift-multus/multus-jvbw5"
Oct 30 19:15:12.062716 ip-10-0-132-82 kubenswrapper[2577]: I1030 19:15:12.062116    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/e0221987-8e5e-40d1-8842-c97993f08c6d-iptables-alerter-script\") pod \"iptables-alerter-wnn66\" (UID: \"e0221987-8e5e-40d1-8842-c97993f08c6d\") " pod="openshift-network-operator/iptables-alerter-wnn66"
Oct 30 19:15:12.063526 ip-10-0-132-82 kubenswrapper[2577]: I1030 19:15:12.062233    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/af36f4d4-f985-4937-a9f7-9884936f3690-konnectivity-ca\") pod \"konnectivity-agent-qnm56\" (UID: \"af36f4d4-f985-4937-a9f7-9884936f3690\") " pod="kube-system/konnectivity-agent-qnm56"
Oct 30 19:15:12.063526 ip-10-0-132-82 kubenswrapper[2577]: I1030 19:15:12.062339    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/08ae8ede-9a96-4a84-8ec8-8d364fd476a3-host-slash\") pod \"ovnkube-node-5lwkx\" (UID: \"08ae8ede-9a96-4a84-8ec8-8d364fd476a3\") " pod="openshift-ovn-kubernetes/ovnkube-node-5lwkx"
Oct 30 19:15:12.063526 ip-10-0-132-82 kubenswrapper[2577]: I1030 19:15:12.062395    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/287ba08f-7b9f-4609-8e35-832c0879c44c-metrics-certs\") pod \"network-metrics-daemon-822gp\" (UID: \"287ba08f-7b9f-4609-8e35-832c0879c44c\") " pod="openshift-multus/network-metrics-daemon-822gp"
Oct 30 19:15:12.063526 ip-10-0-132-82 kubenswrapper[2577]: I1030 19:15:12.062434    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/08ae8ede-9a96-4a84-8ec8-8d364fd476a3-node-log\") pod \"ovnkube-node-5lwkx\" (UID: \"08ae8ede-9a96-4a84-8ec8-8d364fd476a3\") " pod="openshift-ovn-kubernetes/ovnkube-node-5lwkx"
Oct 30 19:15:12.063526 ip-10-0-132-82 kubenswrapper[2577]: I1030 19:15:12.062489    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/dccdaf98-aac7-4756-b93f-22f62a286a1f-host-run-multus-certs\") pod \"multus-jvbw5\" (UID: \"dccdaf98-aac7-4756-b93f-22f62a286a1f\") " pod="openshift-multus/multus-jvbw5"
Oct 30 19:15:12.063526 ip-10-0-132-82 kubenswrapper[2577]: I1030 19:15:12.062525    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/c3780b96-955e-4728-a2d0-b7ec30d0f6f3-sys-fs\") pod \"aws-ebs-csi-driver-node-qgv98\" (UID: \"c3780b96-955e-4728-a2d0-b7ec30d0f6f3\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-qgv98"
Oct 30 19:15:12.063526 ip-10-0-132-82 kubenswrapper[2577]: I1030 19:15:12.062574    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/08ae8ede-9a96-4a84-8ec8-8d364fd476a3-ovn-node-metrics-cert\") pod \"ovnkube-node-5lwkx\" (UID: \"08ae8ede-9a96-4a84-8ec8-8d364fd476a3\") " pod="openshift-ovn-kubernetes/ovnkube-node-5lwkx"
Oct 30 19:15:12.063526 ip-10-0-132-82 kubenswrapper[2577]: I1030 19:15:12.062633    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rvrch\" (UniqueName: \"kubernetes.io/projected/cdb750aa-dd7e-4e2f-8297-9028acc0f559-kube-api-access-rvrch\") pod \"node-ca-kgfvh\" (UID: \"cdb750aa-dd7e-4e2f-8297-9028acc0f559\") " pod="openshift-image-registry/node-ca-kgfvh"
Oct 30 19:15:12.063526 ip-10-0-132-82 kubenswrapper[2577]: I1030 19:15:12.062683    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/e0221987-8e5e-40d1-8842-c97993f08c6d-host-slash\") pod \"iptables-alerter-wnn66\" (UID: \"e0221987-8e5e-40d1-8842-c97993f08c6d\") " pod="openshift-network-operator/iptables-alerter-wnn66"
Oct 30 19:15:12.063526 ip-10-0-132-82 kubenswrapper[2577]: I1030 19:15:12.062732    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fmnd4\" (UniqueName: \"kubernetes.io/projected/e0221987-8e5e-40d1-8842-c97993f08c6d-kube-api-access-fmnd4\") pod \"iptables-alerter-wnn66\" (UID: \"e0221987-8e5e-40d1-8842-c97993f08c6d\") " pod="openshift-network-operator/iptables-alerter-wnn66"
Oct 30 19:15:12.063526 ip-10-0-132-82 kubenswrapper[2577]: I1030 19:15:12.062807    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/c3780b96-955e-4728-a2d0-b7ec30d0f6f3-device-dir\") pod \"aws-ebs-csi-driver-node-qgv98\" (UID: \"c3780b96-955e-4728-a2d0-b7ec30d0f6f3\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-qgv98"
Oct 30 19:15:12.063526 ip-10-0-132-82 kubenswrapper[2577]: I1030 19:15:12.062876    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/08ae8ede-9a96-4a84-8ec8-8d364fd476a3-host-run-ovn-kubernetes\") pod \"ovnkube-node-5lwkx\" (UID: \"08ae8ede-9a96-4a84-8ec8-8d364fd476a3\") " pod="openshift-ovn-kubernetes/ovnkube-node-5lwkx"
Oct 30 19:15:12.063526 ip-10-0-132-82 kubenswrapper[2577]: I1030 19:15:12.062917    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/08ae8ede-9a96-4a84-8ec8-8d364fd476a3-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-5lwkx\" (UID: \"08ae8ede-9a96-4a84-8ec8-8d364fd476a3\") " pod="openshift-ovn-kubernetes/ovnkube-node-5lwkx"
Oct 30 19:15:12.063526 ip-10-0-132-82 kubenswrapper[2577]: I1030 19:15:12.062982    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/cdb750aa-dd7e-4e2f-8297-9028acc0f559-host\") pod \"node-ca-kgfvh\" (UID: \"cdb750aa-dd7e-4e2f-8297-9028acc0f559\") " pod="openshift-image-registry/node-ca-kgfvh"
Oct 30 19:15:12.063526 ip-10-0-132-82 kubenswrapper[2577]: I1030 19:15:12.063022    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/dccdaf98-aac7-4756-b93f-22f62a286a1f-multus-conf-dir\") pod \"multus-jvbw5\" (UID: \"dccdaf98-aac7-4756-b93f-22f62a286a1f\") " pod="openshift-multus/multus-jvbw5"
Oct 30 19:15:12.063526 ip-10-0-132-82 kubenswrapper[2577]: I1030 19:15:12.063055    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/dccdaf98-aac7-4756-b93f-22f62a286a1f-etc-kubernetes\") pod \"multus-jvbw5\" (UID: \"dccdaf98-aac7-4756-b93f-22f62a286a1f\") " pod="openshift-multus/multus-jvbw5"
Oct 30 19:15:12.064024 ip-10-0-132-82 kubenswrapper[2577]: I1030 19:15:12.063089    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-v7kvw\" (UniqueName: \"kubernetes.io/projected/287ba08f-7b9f-4609-8e35-832c0879c44c-kube-api-access-v7kvw\") pod \"network-metrics-daemon-822gp\" (UID: \"287ba08f-7b9f-4609-8e35-832c0879c44c\") " pod="openshift-multus/network-metrics-daemon-822gp"
Oct 30 19:15:12.064024 ip-10-0-132-82 kubenswrapper[2577]: I1030 19:15:12.063154    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/39bb8c16-8282-442a-841b-0056baae014d-etc-sysctl-conf\") pod \"tuned-2fv9z\" (UID: \"39bb8c16-8282-442a-841b-0056baae014d\") " pod="openshift-cluster-node-tuning-operator/tuned-2fv9z"
Oct 30 19:15:12.064024 ip-10-0-132-82 kubenswrapper[2577]: I1030 19:15:12.063224    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/08ae8ede-9a96-4a84-8ec8-8d364fd476a3-host-cni-bin\") pod \"ovnkube-node-5lwkx\" (UID: \"08ae8ede-9a96-4a84-8ec8-8d364fd476a3\") " pod="openshift-ovn-kubernetes/ovnkube-node-5lwkx"
Oct 30 19:15:12.064024 ip-10-0-132-82 kubenswrapper[2577]: I1030 19:15:12.063275    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bnxm6\" (UniqueName: \"kubernetes.io/projected/08ae8ede-9a96-4a84-8ec8-8d364fd476a3-kube-api-access-bnxm6\") pod \"ovnkube-node-5lwkx\" (UID: \"08ae8ede-9a96-4a84-8ec8-8d364fd476a3\") " pod="openshift-ovn-kubernetes/ovnkube-node-5lwkx"
Oct 30 19:15:12.064024 ip-10-0-132-82 kubenswrapper[2577]: I1030 19:15:12.063321    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/cdb750aa-dd7e-4e2f-8297-9028acc0f559-serviceca\") pod \"node-ca-kgfvh\" (UID: \"cdb750aa-dd7e-4e2f-8297-9028acc0f559\") " pod="openshift-image-registry/node-ca-kgfvh"
Oct 30 19:15:12.064024 ip-10-0-132-82 kubenswrapper[2577]: I1030 19:15:12.063348    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/dccdaf98-aac7-4756-b93f-22f62a286a1f-cnibin\") pod \"multus-jvbw5\" (UID: \"dccdaf98-aac7-4756-b93f-22f62a286a1f\") " pod="openshift-multus/multus-jvbw5"
Oct 30 19:15:12.064024 ip-10-0-132-82 kubenswrapper[2577]: I1030 19:15:12.063425    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/dccdaf98-aac7-4756-b93f-22f62a286a1f-multus-daemon-config\") pod \"multus-jvbw5\" (UID: \"dccdaf98-aac7-4756-b93f-22f62a286a1f\") " pod="openshift-multus/multus-jvbw5"
Oct 30 19:15:12.064024 ip-10-0-132-82 kubenswrapper[2577]: I1030 19:15:12.063484    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/39bb8c16-8282-442a-841b-0056baae014d-etc-systemd\") pod \"tuned-2fv9z\" (UID: \"39bb8c16-8282-442a-841b-0056baae014d\") " pod="openshift-cluster-node-tuning-operator/tuned-2fv9z"
Oct 30 19:15:12.064024 ip-10-0-132-82 kubenswrapper[2577]: I1030 19:15:12.063525    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/39bb8c16-8282-442a-841b-0056baae014d-lib-modules\") pod \"tuned-2fv9z\" (UID: \"39bb8c16-8282-442a-841b-0056baae014d\") " pod="openshift-cluster-node-tuning-operator/tuned-2fv9z"
Oct 30 19:15:12.064024 ip-10-0-132-82 kubenswrapper[2577]: I1030 19:15:12.063561    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/c16c0a80-c5ce-4f5d-827a-2e3eb2c3cfb8-dbus\") pod \"global-pull-secret-syncer-76rht\" (UID: \"c16c0a80-c5ce-4f5d-827a-2e3eb2c3cfb8\") " pod="kube-system/global-pull-secret-syncer-76rht"
Oct 30 19:15:12.064024 ip-10-0-132-82 kubenswrapper[2577]: I1030 19:15:12.063617    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/39bb8c16-8282-442a-841b-0056baae014d-etc-sysconfig\") pod \"tuned-2fv9z\" (UID: \"39bb8c16-8282-442a-841b-0056baae014d\") " pod="openshift-cluster-node-tuning-operator/tuned-2fv9z"
Oct 30 19:15:12.064024 ip-10-0-132-82 kubenswrapper[2577]: I1030 19:15:12.063934    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/08ae8ede-9a96-4a84-8ec8-8d364fd476a3-ovnkube-config\") pod \"ovnkube-node-5lwkx\" (UID: \"08ae8ede-9a96-4a84-8ec8-8d364fd476a3\") " pod="openshift-ovn-kubernetes/ovnkube-node-5lwkx"
Oct 30 19:15:12.064024 ip-10-0-132-82 kubenswrapper[2577]: I1030 19:15:12.064005    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4rcdn\" (UniqueName: \"kubernetes.io/projected/bab0abf9-46d4-408e-9aa0-afa7279cd9d1-kube-api-access-4rcdn\") pod \"multus-additional-cni-plugins-4pvgg\" (UID: \"bab0abf9-46d4-408e-9aa0-afa7279cd9d1\") " pod="openshift-multus/multus-additional-cni-plugins-4pvgg"
Oct 30 19:15:12.064362 ip-10-0-132-82 kubenswrapper[2577]: I1030 19:15:12.064048    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/dccdaf98-aac7-4756-b93f-22f62a286a1f-system-cni-dir\") pod \"multus-jvbw5\" (UID: \"dccdaf98-aac7-4756-b93f-22f62a286a1f\") " pod="openshift-multus/multus-jvbw5"
Oct 30 19:15:12.064362 ip-10-0-132-82 kubenswrapper[2577]: I1030 19:15:12.064111    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/c3780b96-955e-4728-a2d0-b7ec30d0f6f3-etc-selinux\") pod \"aws-ebs-csi-driver-node-qgv98\" (UID: \"c3780b96-955e-4728-a2d0-b7ec30d0f6f3\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-qgv98"
Oct 30 19:15:12.064362 ip-10-0-132-82 kubenswrapper[2577]: I1030 19:15:12.064159    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/39bb8c16-8282-442a-841b-0056baae014d-etc-sysctl-d\") pod \"tuned-2fv9z\" (UID: \"39bb8c16-8282-442a-841b-0056baae014d\") " pod="openshift-cluster-node-tuning-operator/tuned-2fv9z"
Oct 30 19:15:12.064362 ip-10-0-132-82 kubenswrapper[2577]: I1030 19:15:12.064193    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/39bb8c16-8282-442a-841b-0056baae014d-sys\") pod \"tuned-2fv9z\" (UID: \"39bb8c16-8282-442a-841b-0056baae014d\") " pod="openshift-cluster-node-tuning-operator/tuned-2fv9z"
Oct 30 19:15:12.064465 ip-10-0-132-82 kubenswrapper[2577]: I1030 19:15:12.064407    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-w8zg9\" (UniqueName: \"kubernetes.io/projected/48386105-d79b-4f92-baa0-f330b9658ad0-kube-api-access-w8zg9\") pod \"node-resolver-g9xd6\" (UID: \"48386105-d79b-4f92-baa0-f330b9658ad0\") " pod="openshift-dns/node-resolver-g9xd6"
Oct 30 19:15:12.064465 ip-10-0-132-82 kubenswrapper[2577]: I1030 19:15:12.064448    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/08ae8ede-9a96-4a84-8ec8-8d364fd476a3-run-systemd\") pod \"ovnkube-node-5lwkx\" (UID: \"08ae8ede-9a96-4a84-8ec8-8d364fd476a3\") " pod="openshift-ovn-kubernetes/ovnkube-node-5lwkx"
Oct 30 19:15:12.064524 ip-10-0-132-82 kubenswrapper[2577]: I1030 19:15:12.064478    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/08ae8ede-9a96-4a84-8ec8-8d364fd476a3-host-cni-netd\") pod \"ovnkube-node-5lwkx\" (UID: \"08ae8ede-9a96-4a84-8ec8-8d364fd476a3\") " pod="openshift-ovn-kubernetes/ovnkube-node-5lwkx"
Oct 30 19:15:12.064524 ip-10-0-132-82 kubenswrapper[2577]: I1030 19:15:12.064497    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/bab0abf9-46d4-408e-9aa0-afa7279cd9d1-tuning-conf-dir\") pod \"multus-additional-cni-plugins-4pvgg\" (UID: \"bab0abf9-46d4-408e-9aa0-afa7279cd9d1\") " pod="openshift-multus/multus-additional-cni-plugins-4pvgg"
Oct 30 19:15:12.064580 ip-10-0-132-82 kubenswrapper[2577]: I1030 19:15:12.064528    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/39bb8c16-8282-442a-841b-0056baae014d-tmp\") pod \"tuned-2fv9z\" (UID: \"39bb8c16-8282-442a-841b-0056baae014d\") " pod="openshift-cluster-node-tuning-operator/tuned-2fv9z"
Oct 30 19:15:12.064580 ip-10-0-132-82 kubenswrapper[2577]: I1030 19:15:12.064551    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/08ae8ede-9a96-4a84-8ec8-8d364fd476a3-env-overrides\") pod \"ovnkube-node-5lwkx\" (UID: \"08ae8ede-9a96-4a84-8ec8-8d364fd476a3\") " pod="openshift-ovn-kubernetes/ovnkube-node-5lwkx"
Oct 30 19:15:12.064635 ip-10-0-132-82 kubenswrapper[2577]: I1030 19:15:12.064601    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gdq6k\" (UniqueName: \"kubernetes.io/projected/39bb8c16-8282-442a-841b-0056baae014d-kube-api-access-gdq6k\") pod \"tuned-2fv9z\" (UID: \"39bb8c16-8282-442a-841b-0056baae014d\") " pod="openshift-cluster-node-tuning-operator/tuned-2fv9z"
Oct 30 19:15:12.064667 ip-10-0-132-82 kubenswrapper[2577]: I1030 19:15:12.064639    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bzw4s\" (UniqueName: \"kubernetes.io/projected/4ca01b49-9819-4a90-a851-7553ad568679-kube-api-access-bzw4s\") pod \"network-check-target-c24j9\" (UID: \"4ca01b49-9819-4a90-a851-7553ad568679\") " pod="openshift-network-diagnostics/network-check-target-c24j9"
Oct 30 19:15:12.064698 ip-10-0-132-82 kubenswrapper[2577]: I1030 19:15:12.064666    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/48386105-d79b-4f92-baa0-f330b9658ad0-hosts-file\") pod \"node-resolver-g9xd6\" (UID: \"48386105-d79b-4f92-baa0-f330b9658ad0\") " pod="openshift-dns/node-resolver-g9xd6"
Oct 30 19:15:12.064698 ip-10-0-132-82 kubenswrapper[2577]: I1030 19:15:12.064691    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/bab0abf9-46d4-408e-9aa0-afa7279cd9d1-system-cni-dir\") pod \"multus-additional-cni-plugins-4pvgg\" (UID: \"bab0abf9-46d4-408e-9aa0-afa7279cd9d1\") " pod="openshift-multus/multus-additional-cni-plugins-4pvgg"
Oct 30 19:15:12.064758 ip-10-0-132-82 kubenswrapper[2577]: I1030 19:15:12.064706    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6bqfr\" (UniqueName: \"kubernetes.io/projected/dccdaf98-aac7-4756-b93f-22f62a286a1f-kube-api-access-6bqfr\") pod \"multus-jvbw5\" (UID: \"dccdaf98-aac7-4756-b93f-22f62a286a1f\") " pod="openshift-multus/multus-jvbw5"
Oct 30 19:15:12.064854 ip-10-0-132-82 kubenswrapper[2577]: I1030 19:15:12.064723    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/c3780b96-955e-4728-a2d0-b7ec30d0f6f3-kubelet-dir\") pod \"aws-ebs-csi-driver-node-qgv98\" (UID: \"c3780b96-955e-4728-a2d0-b7ec30d0f6f3\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-qgv98"
Oct 30 19:15:12.065071 ip-10-0-132-82 kubenswrapper[2577]: I1030 19:15:12.065011    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cdkvt\" (UniqueName: \"kubernetes.io/projected/c3780b96-955e-4728-a2d0-b7ec30d0f6f3-kube-api-access-cdkvt\") pod \"aws-ebs-csi-driver-node-qgv98\" (UID: \"c3780b96-955e-4728-a2d0-b7ec30d0f6f3\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-qgv98"
Oct 30 19:15:12.065163 ip-10-0-132-82 kubenswrapper[2577]: I1030 19:15:12.065102    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/39bb8c16-8282-442a-841b-0056baae014d-etc-kubernetes\") pod \"tuned-2fv9z\" (UID: \"39bb8c16-8282-442a-841b-0056baae014d\") " pod="openshift-cluster-node-tuning-operator/tuned-2fv9z"
Oct 30 19:15:12.078390 ip-10-0-132-82 kubenswrapper[2577]: I1030 19:15:12.078374    2577 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"node-resolver-dockercfg-xzg6c\""
Oct 30 19:15:12.080748 ip-10-0-132-82 kubenswrapper[2577]: I1030 19:15:12.080731    2577 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"openshift-service-ca.crt\""
Oct 30 19:15:12.080829 ip-10-0-132-82 kubenswrapper[2577]: I1030 19:15:12.080764    2577 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"kube-root-ca.crt\""
Oct 30 19:15:12.084731 ip-10-0-132-82 kubenswrapper[2577]: I1030 19:15:12.084718    2577 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"cni-copy-resources\""
Oct 30 19:15:12.152491 ip-10-0-132-82 kubenswrapper[2577]: I1030 19:15:12.152471    2577 reflector.go:446] "Caches populated" logger="kubernetes.io/kubelet-serving" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162"
Oct 30 19:15:12.152567 ip-10-0-132-82 kubenswrapper[2577]: I1030 19:15:12.152504    2577 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kubelet-serving" csr="csr-s9ght"
Oct 30 19:15:12.165303 ip-10-0-132-82 kubenswrapper[2577]: I1030 19:15:12.165285    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/39bb8c16-8282-442a-841b-0056baae014d-run\") pod \"tuned-2fv9z\" (UID: \"39bb8c16-8282-442a-841b-0056baae014d\") " pod="openshift-cluster-node-tuning-operator/tuned-2fv9z"
Oct 30 19:15:12.165381 ip-10-0-132-82 kubenswrapper[2577]: I1030 19:15:12.165308    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/c16c0a80-c5ce-4f5d-827a-2e3eb2c3cfb8-original-pull-secret\") pod \"global-pull-secret-syncer-76rht\" (UID: \"c16c0a80-c5ce-4f5d-827a-2e3eb2c3cfb8\") " pod="kube-system/global-pull-secret-syncer-76rht"
Oct 30 19:15:12.165381 ip-10-0-132-82 kubenswrapper[2577]: I1030 19:15:12.165323    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/08ae8ede-9a96-4a84-8ec8-8d364fd476a3-systemd-units\") pod \"ovnkube-node-5lwkx\" (UID: \"08ae8ede-9a96-4a84-8ec8-8d364fd476a3\") " pod="openshift-ovn-kubernetes/ovnkube-node-5lwkx"
Oct 30 19:15:12.165381 ip-10-0-132-82 kubenswrapper[2577]: I1030 19:15:12.165337    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/dccdaf98-aac7-4756-b93f-22f62a286a1f-multus-cni-dir\") pod \"multus-jvbw5\" (UID: \"dccdaf98-aac7-4756-b93f-22f62a286a1f\") " pod="openshift-multus/multus-jvbw5"
Oct 30 19:15:12.165381 ip-10-0-132-82 kubenswrapper[2577]: I1030 19:15:12.165364    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/dccdaf98-aac7-4756-b93f-22f62a286a1f-host-var-lib-cni-multus\") pod \"multus-jvbw5\" (UID: \"dccdaf98-aac7-4756-b93f-22f62a286a1f\") " pod="openshift-multus/multus-jvbw5"
Oct 30 19:15:12.165381 ip-10-0-132-82 kubenswrapper[2577]: I1030 19:15:12.165378    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/dccdaf98-aac7-4756-b93f-22f62a286a1f-hostroot\") pod \"multus-jvbw5\" (UID: \"dccdaf98-aac7-4756-b93f-22f62a286a1f\") " pod="openshift-multus/multus-jvbw5"
Oct 30 19:15:12.165620 ip-10-0-132-82 kubenswrapper[2577]: I1030 19:15:12.165398    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/39bb8c16-8282-442a-841b-0056baae014d-etc-modprobe-d\") pod \"tuned-2fv9z\" (UID: \"39bb8c16-8282-442a-841b-0056baae014d\") " pod="openshift-cluster-node-tuning-operator/tuned-2fv9z"
Oct 30 19:15:12.165620 ip-10-0-132-82 kubenswrapper[2577]: I1030 19:15:12.165405    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/08ae8ede-9a96-4a84-8ec8-8d364fd476a3-systemd-units\") pod \"ovnkube-node-5lwkx\" (UID: \"08ae8ede-9a96-4a84-8ec8-8d364fd476a3\") " pod="openshift-ovn-kubernetes/ovnkube-node-5lwkx"
Oct 30 19:15:12.165620 ip-10-0-132-82 kubenswrapper[2577]: E1030 19:15:12.165422    2577 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Oct 30 19:15:12.165620 ip-10-0-132-82 kubenswrapper[2577]: I1030 19:15:12.165414    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/39bb8c16-8282-442a-841b-0056baae014d-host\") pod \"tuned-2fv9z\" (UID: \"39bb8c16-8282-442a-841b-0056baae014d\") " pod="openshift-cluster-node-tuning-operator/tuned-2fv9z"
Oct 30 19:15:12.165620 ip-10-0-132-82 kubenswrapper[2577]: I1030 19:15:12.165458    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/dccdaf98-aac7-4756-b93f-22f62a286a1f-host-var-lib-cni-multus\") pod \"multus-jvbw5\" (UID: \"dccdaf98-aac7-4756-b93f-22f62a286a1f\") " pod="openshift-multus/multus-jvbw5"
Oct 30 19:15:12.165620 ip-10-0-132-82 kubenswrapper[2577]: I1030 19:15:12.165473    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/dccdaf98-aac7-4756-b93f-22f62a286a1f-hostroot\") pod \"multus-jvbw5\" (UID: \"dccdaf98-aac7-4756-b93f-22f62a286a1f\") " pod="openshift-multus/multus-jvbw5"
Oct 30 19:15:12.165620 ip-10-0-132-82 kubenswrapper[2577]: I1030 19:15:12.165460    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/39bb8c16-8282-442a-841b-0056baae014d-etc-tuned\") pod \"tuned-2fv9z\" (UID: \"39bb8c16-8282-442a-841b-0056baae014d\") " pod="openshift-cluster-node-tuning-operator/tuned-2fv9z"
Oct 30 19:15:12.165620 ip-10-0-132-82 kubenswrapper[2577]: I1030 19:15:12.165458    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run\" (UniqueName: \"kubernetes.io/host-path/39bb8c16-8282-442a-841b-0056baae014d-run\") pod \"tuned-2fv9z\" (UID: \"39bb8c16-8282-442a-841b-0056baae014d\") " pod="openshift-cluster-node-tuning-operator/tuned-2fv9z"
Oct 30 19:15:12.165620 ip-10-0-132-82 kubenswrapper[2577]: E1030 19:15:12.165497    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/c16c0a80-c5ce-4f5d-827a-2e3eb2c3cfb8-original-pull-secret podName:c16c0a80-c5ce-4f5d-827a-2e3eb2c3cfb8 nodeName:}" failed. No retries permitted until 2026-10-30 19:15:12.665460306 +0000 UTC m=+3.071380735 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/c16c0a80-c5ce-4f5d-827a-2e3eb2c3cfb8-original-pull-secret") pod "global-pull-secret-syncer-76rht" (UID: "c16c0a80-c5ce-4f5d-827a-2e3eb2c3cfb8") : object "kube-system"/"original-pull-secret" not registered
Oct 30 19:15:12.165620 ip-10-0-132-82 kubenswrapper[2577]: I1030 19:15:12.165514    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/39bb8c16-8282-442a-841b-0056baae014d-etc-modprobe-d\") pod \"tuned-2fv9z\" (UID: \"39bb8c16-8282-442a-841b-0056baae014d\") " pod="openshift-cluster-node-tuning-operator/tuned-2fv9z"
Oct 30 19:15:12.165620 ip-10-0-132-82 kubenswrapper[2577]: I1030 19:15:12.165522    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/39bb8c16-8282-442a-841b-0056baae014d-host\") pod \"tuned-2fv9z\" (UID: \"39bb8c16-8282-442a-841b-0056baae014d\") " pod="openshift-cluster-node-tuning-operator/tuned-2fv9z"
Oct 30 19:15:12.165620 ip-10-0-132-82 kubenswrapper[2577]: I1030 19:15:12.165524    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/dccdaf98-aac7-4756-b93f-22f62a286a1f-multus-cni-dir\") pod \"multus-jvbw5\" (UID: \"dccdaf98-aac7-4756-b93f-22f62a286a1f\") " pod="openshift-multus/multus-jvbw5"
Oct 30 19:15:12.165620 ip-10-0-132-82 kubenswrapper[2577]: I1030 19:15:12.165538    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/08ae8ede-9a96-4a84-8ec8-8d364fd476a3-host-run-netns\") pod \"ovnkube-node-5lwkx\" (UID: \"08ae8ede-9a96-4a84-8ec8-8d364fd476a3\") " pod="openshift-ovn-kubernetes/ovnkube-node-5lwkx"
Oct 30 19:15:12.165620 ip-10-0-132-82 kubenswrapper[2577]: I1030 19:15:12.165559    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/08ae8ede-9a96-4a84-8ec8-8d364fd476a3-host-run-netns\") pod \"ovnkube-node-5lwkx\" (UID: \"08ae8ede-9a96-4a84-8ec8-8d364fd476a3\") " pod="openshift-ovn-kubernetes/ovnkube-node-5lwkx"
Oct 30 19:15:12.165620 ip-10-0-132-82 kubenswrapper[2577]: I1030 19:15:12.165567    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/08ae8ede-9a96-4a84-8ec8-8d364fd476a3-etc-openvswitch\") pod \"ovnkube-node-5lwkx\" (UID: \"08ae8ede-9a96-4a84-8ec8-8d364fd476a3\") " pod="openshift-ovn-kubernetes/ovnkube-node-5lwkx"
Oct 30 19:15:12.165620 ip-10-0-132-82 kubenswrapper[2577]: I1030 19:15:12.165593    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/bab0abf9-46d4-408e-9aa0-afa7279cd9d1-cni-binary-copy\") pod \"multus-additional-cni-plugins-4pvgg\" (UID: \"bab0abf9-46d4-408e-9aa0-afa7279cd9d1\") " pod="openshift-multus/multus-additional-cni-plugins-4pvgg"
Oct 30 19:15:12.165620 ip-10-0-132-82 kubenswrapper[2577]: I1030 19:15:12.165617    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/dccdaf98-aac7-4756-b93f-22f62a286a1f-multus-socket-dir-parent\") pod \"multus-jvbw5\" (UID: \"dccdaf98-aac7-4756-b93f-22f62a286a1f\") " pod="openshift-multus/multus-jvbw5"
Oct 30 19:15:12.166260 ip-10-0-132-82 kubenswrapper[2577]: I1030 19:15:12.165619    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/08ae8ede-9a96-4a84-8ec8-8d364fd476a3-etc-openvswitch\") pod \"ovnkube-node-5lwkx\" (UID: \"08ae8ede-9a96-4a84-8ec8-8d364fd476a3\") " pod="openshift-ovn-kubernetes/ovnkube-node-5lwkx"
Oct 30 19:15:12.166260 ip-10-0-132-82 kubenswrapper[2577]: I1030 19:15:12.165639    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/af36f4d4-f985-4937-a9f7-9884936f3690-agent-certs\") pod \"konnectivity-agent-qnm56\" (UID: \"af36f4d4-f985-4937-a9f7-9884936f3690\") " pod="kube-system/konnectivity-agent-qnm56"
Oct 30 19:15:12.166260 ip-10-0-132-82 kubenswrapper[2577]: I1030 19:15:12.165661    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/08ae8ede-9a96-4a84-8ec8-8d364fd476a3-log-socket\") pod \"ovnkube-node-5lwkx\" (UID: \"08ae8ede-9a96-4a84-8ec8-8d364fd476a3\") " pod="openshift-ovn-kubernetes/ovnkube-node-5lwkx"
Oct 30 19:15:12.166260 ip-10-0-132-82 kubenswrapper[2577]: I1030 19:15:12.165679    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/dccdaf98-aac7-4756-b93f-22f62a286a1f-multus-socket-dir-parent\") pod \"multus-jvbw5\" (UID: \"dccdaf98-aac7-4756-b93f-22f62a286a1f\") " pod="openshift-multus/multus-jvbw5"
Oct 30 19:15:12.166260 ip-10-0-132-82 kubenswrapper[2577]: I1030 19:15:12.165684    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/08ae8ede-9a96-4a84-8ec8-8d364fd476a3-ovnkube-script-lib\") pod \"ovnkube-node-5lwkx\" (UID: \"08ae8ede-9a96-4a84-8ec8-8d364fd476a3\") " pod="openshift-ovn-kubernetes/ovnkube-node-5lwkx"
Oct 30 19:15:12.166260 ip-10-0-132-82 kubenswrapper[2577]: I1030 19:15:12.165697    2577 swap_util.go:78] "error creating dir to test if tmpfs noswap is enabled. Assuming not supported" mount path="" error="stat /var/lib/kubelet/plugins/kubernetes.io/empty-dir: no such file or directory"
Oct 30 19:15:12.166260 ip-10-0-132-82 kubenswrapper[2577]: I1030 19:15:12.165725    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/08ae8ede-9a96-4a84-8ec8-8d364fd476a3-log-socket\") pod \"ovnkube-node-5lwkx\" (UID: \"08ae8ede-9a96-4a84-8ec8-8d364fd476a3\") " pod="openshift-ovn-kubernetes/ovnkube-node-5lwkx"
Oct 30 19:15:12.166260 ip-10-0-132-82 kubenswrapper[2577]: I1030 19:15:12.165726    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/bab0abf9-46d4-408e-9aa0-afa7279cd9d1-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-4pvgg\" (UID: \"bab0abf9-46d4-408e-9aa0-afa7279cd9d1\") " pod="openshift-multus/multus-additional-cni-plugins-4pvgg"
Oct 30 19:15:12.166260 ip-10-0-132-82 kubenswrapper[2577]: I1030 19:15:12.165765    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/bab0abf9-46d4-408e-9aa0-afa7279cd9d1-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-4pvgg\" (UID: \"bab0abf9-46d4-408e-9aa0-afa7279cd9d1\") " pod="openshift-multus/multus-additional-cni-plugins-4pvgg"
Oct 30 19:15:12.166260 ip-10-0-132-82 kubenswrapper[2577]: I1030 19:15:12.165786    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/dccdaf98-aac7-4756-b93f-22f62a286a1f-host-run-netns\") pod \"multus-jvbw5\" (UID: \"dccdaf98-aac7-4756-b93f-22f62a286a1f\") " pod="openshift-multus/multus-jvbw5"
Oct 30 19:15:12.166260 ip-10-0-132-82 kubenswrapper[2577]: I1030 19:15:12.165804    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/dccdaf98-aac7-4756-b93f-22f62a286a1f-host-var-lib-cni-bin\") pod \"multus-jvbw5\" (UID: \"dccdaf98-aac7-4756-b93f-22f62a286a1f\") " pod="openshift-multus/multus-jvbw5"
Oct 30 19:15:12.166260 ip-10-0-132-82 kubenswrapper[2577]: I1030 19:15:12.165825    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/c16c0a80-c5ce-4f5d-827a-2e3eb2c3cfb8-kubelet-config\") pod \"global-pull-secret-syncer-76rht\" (UID: \"c16c0a80-c5ce-4f5d-827a-2e3eb2c3cfb8\") " pod="kube-system/global-pull-secret-syncer-76rht"
Oct 30 19:15:12.166260 ip-10-0-132-82 kubenswrapper[2577]: I1030 19:15:12.165846    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/dccdaf98-aac7-4756-b93f-22f62a286a1f-host-var-lib-kubelet\") pod \"multus-jvbw5\" (UID: \"dccdaf98-aac7-4756-b93f-22f62a286a1f\") " pod="openshift-multus/multus-jvbw5"
Oct 30 19:15:12.166260 ip-10-0-132-82 kubenswrapper[2577]: I1030 19:15:12.165859    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/dccdaf98-aac7-4756-b93f-22f62a286a1f-host-run-netns\") pod \"multus-jvbw5\" (UID: \"dccdaf98-aac7-4756-b93f-22f62a286a1f\") " pod="openshift-multus/multus-jvbw5"
Oct 30 19:15:12.166260 ip-10-0-132-82 kubenswrapper[2577]: I1030 19:15:12.165868    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/e0221987-8e5e-40d1-8842-c97993f08c6d-iptables-alerter-script\") pod \"iptables-alerter-wnn66\" (UID: \"e0221987-8e5e-40d1-8842-c97993f08c6d\") " pod="openshift-network-operator/iptables-alerter-wnn66"
Oct 30 19:15:12.166260 ip-10-0-132-82 kubenswrapper[2577]: I1030 19:15:12.165878    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/dccdaf98-aac7-4756-b93f-22f62a286a1f-host-var-lib-cni-bin\") pod \"multus-jvbw5\" (UID: \"dccdaf98-aac7-4756-b93f-22f62a286a1f\") " pod="openshift-multus/multus-jvbw5"
Oct 30 19:15:12.166260 ip-10-0-132-82 kubenswrapper[2577]: I1030 19:15:12.165901    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/af36f4d4-f985-4937-a9f7-9884936f3690-konnectivity-ca\") pod \"konnectivity-agent-qnm56\" (UID: \"af36f4d4-f985-4937-a9f7-9884936f3690\") " pod="kube-system/konnectivity-agent-qnm56"
Oct 30 19:15:12.166260 ip-10-0-132-82 kubenswrapper[2577]: I1030 19:15:12.165926    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/08ae8ede-9a96-4a84-8ec8-8d364fd476a3-host-slash\") pod \"ovnkube-node-5lwkx\" (UID: \"08ae8ede-9a96-4a84-8ec8-8d364fd476a3\") " pod="openshift-ovn-kubernetes/ovnkube-node-5lwkx"
Oct 30 19:15:12.166989 ip-10-0-132-82 kubenswrapper[2577]: I1030 19:15:12.165900    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/c16c0a80-c5ce-4f5d-827a-2e3eb2c3cfb8-kubelet-config\") pod \"global-pull-secret-syncer-76rht\" (UID: \"c16c0a80-c5ce-4f5d-827a-2e3eb2c3cfb8\") " pod="kube-system/global-pull-secret-syncer-76rht"
Oct 30 19:15:12.166989 ip-10-0-132-82 kubenswrapper[2577]: I1030 19:15:12.165956    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/287ba08f-7b9f-4609-8e35-832c0879c44c-metrics-certs\") pod \"network-metrics-daemon-822gp\" (UID: \"287ba08f-7b9f-4609-8e35-832c0879c44c\") " pod="openshift-multus/network-metrics-daemon-822gp"
Oct 30 19:15:12.166989 ip-10-0-132-82 kubenswrapper[2577]: I1030 19:15:12.165990    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/08ae8ede-9a96-4a84-8ec8-8d364fd476a3-host-slash\") pod \"ovnkube-node-5lwkx\" (UID: \"08ae8ede-9a96-4a84-8ec8-8d364fd476a3\") " pod="openshift-ovn-kubernetes/ovnkube-node-5lwkx"
Oct 30 19:15:12.166989 ip-10-0-132-82 kubenswrapper[2577]: I1030 19:15:12.165994    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/dccdaf98-aac7-4756-b93f-22f62a286a1f-host-var-lib-kubelet\") pod \"multus-jvbw5\" (UID: \"dccdaf98-aac7-4756-b93f-22f62a286a1f\") " pod="openshift-multus/multus-jvbw5"
Oct 30 19:15:12.166989 ip-10-0-132-82 kubenswrapper[2577]: I1030 19:15:12.166001    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/08ae8ede-9a96-4a84-8ec8-8d364fd476a3-node-log\") pod \"ovnkube-node-5lwkx\" (UID: \"08ae8ede-9a96-4a84-8ec8-8d364fd476a3\") " pod="openshift-ovn-kubernetes/ovnkube-node-5lwkx"
Oct 30 19:15:12.166989 ip-10-0-132-82 kubenswrapper[2577]: I1030 19:15:12.166026    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/dccdaf98-aac7-4756-b93f-22f62a286a1f-host-run-multus-certs\") pod \"multus-jvbw5\" (UID: \"dccdaf98-aac7-4756-b93f-22f62a286a1f\") " pod="openshift-multus/multus-jvbw5"
Oct 30 19:15:12.166989 ip-10-0-132-82 kubenswrapper[2577]: E1030 19:15:12.166035    2577 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Oct 30 19:15:12.166989 ip-10-0-132-82 kubenswrapper[2577]: I1030 19:15:12.166067    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/c3780b96-955e-4728-a2d0-b7ec30d0f6f3-sys-fs\") pod \"aws-ebs-csi-driver-node-qgv98\" (UID: \"c3780b96-955e-4728-a2d0-b7ec30d0f6f3\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-qgv98"
Oct 30 19:15:12.166989 ip-10-0-132-82 kubenswrapper[2577]: I1030 19:15:12.166076    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/08ae8ede-9a96-4a84-8ec8-8d364fd476a3-node-log\") pod \"ovnkube-node-5lwkx\" (UID: \"08ae8ede-9a96-4a84-8ec8-8d364fd476a3\") " pod="openshift-ovn-kubernetes/ovnkube-node-5lwkx"
Oct 30 19:15:12.166989 ip-10-0-132-82 kubenswrapper[2577]: I1030 19:15:12.166085    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/dccdaf98-aac7-4756-b93f-22f62a286a1f-host-run-multus-certs\") pod \"multus-jvbw5\" (UID: \"dccdaf98-aac7-4756-b93f-22f62a286a1f\") " pod="openshift-multus/multus-jvbw5"
Oct 30 19:15:12.166989 ip-10-0-132-82 kubenswrapper[2577]: E1030 19:15:12.166090    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/287ba08f-7b9f-4609-8e35-832c0879c44c-metrics-certs podName:287ba08f-7b9f-4609-8e35-832c0879c44c nodeName:}" failed. No retries permitted until 2026-10-30 19:15:12.666078236 +0000 UTC m=+3.071998650 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/287ba08f-7b9f-4609-8e35-832c0879c44c-metrics-certs") pod "network-metrics-daemon-822gp" (UID: "287ba08f-7b9f-4609-8e35-832c0879c44c") : object "openshift-multus"/"metrics-daemon-secret" not registered
Oct 30 19:15:12.166989 ip-10-0-132-82 kubenswrapper[2577]: I1030 19:15:12.166098    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/bab0abf9-46d4-408e-9aa0-afa7279cd9d1-cni-binary-copy\") pod \"multus-additional-cni-plugins-4pvgg\" (UID: \"bab0abf9-46d4-408e-9aa0-afa7279cd9d1\") " pod="openshift-multus/multus-additional-cni-plugins-4pvgg"
Oct 30 19:15:12.166989 ip-10-0-132-82 kubenswrapper[2577]: I1030 19:15:12.166102    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/c3780b96-955e-4728-a2d0-b7ec30d0f6f3-sys-fs\") pod \"aws-ebs-csi-driver-node-qgv98\" (UID: \"c3780b96-955e-4728-a2d0-b7ec30d0f6f3\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-qgv98"
Oct 30 19:15:12.166989 ip-10-0-132-82 kubenswrapper[2577]: I1030 19:15:12.166133    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/08ae8ede-9a96-4a84-8ec8-8d364fd476a3-ovn-node-metrics-cert\") pod \"ovnkube-node-5lwkx\" (UID: \"08ae8ede-9a96-4a84-8ec8-8d364fd476a3\") " pod="openshift-ovn-kubernetes/ovnkube-node-5lwkx"
Oct 30 19:15:12.166989 ip-10-0-132-82 kubenswrapper[2577]: I1030 19:15:12.166192    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rvrch\" (UniqueName: \"kubernetes.io/projected/cdb750aa-dd7e-4e2f-8297-9028acc0f559-kube-api-access-rvrch\") pod \"node-ca-kgfvh\" (UID: \"cdb750aa-dd7e-4e2f-8297-9028acc0f559\") " pod="openshift-image-registry/node-ca-kgfvh"
Oct 30 19:15:12.166989 ip-10-0-132-82 kubenswrapper[2577]: I1030 19:15:12.166207    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/e0221987-8e5e-40d1-8842-c97993f08c6d-host-slash\") pod \"iptables-alerter-wnn66\" (UID: \"e0221987-8e5e-40d1-8842-c97993f08c6d\") " pod="openshift-network-operator/iptables-alerter-wnn66"
Oct 30 19:15:12.166989 ip-10-0-132-82 kubenswrapper[2577]: I1030 19:15:12.166221    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fmnd4\" (UniqueName: \"kubernetes.io/projected/e0221987-8e5e-40d1-8842-c97993f08c6d-kube-api-access-fmnd4\") pod \"iptables-alerter-wnn66\" (UID: \"e0221987-8e5e-40d1-8842-c97993f08c6d\") " pod="openshift-network-operator/iptables-alerter-wnn66"
Oct 30 19:15:12.167716 ip-10-0-132-82 kubenswrapper[2577]: I1030 19:15:12.166243    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/c3780b96-955e-4728-a2d0-b7ec30d0f6f3-device-dir\") pod \"aws-ebs-csi-driver-node-qgv98\" (UID: \"c3780b96-955e-4728-a2d0-b7ec30d0f6f3\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-qgv98"
Oct 30 19:15:12.167716 ip-10-0-132-82 kubenswrapper[2577]: I1030 19:15:12.166259    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/08ae8ede-9a96-4a84-8ec8-8d364fd476a3-host-run-ovn-kubernetes\") pod \"ovnkube-node-5lwkx\" (UID: \"08ae8ede-9a96-4a84-8ec8-8d364fd476a3\") " pod="openshift-ovn-kubernetes/ovnkube-node-5lwkx"
Oct 30 19:15:12.167716 ip-10-0-132-82 kubenswrapper[2577]: I1030 19:15:12.166265    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/e0221987-8e5e-40d1-8842-c97993f08c6d-host-slash\") pod \"iptables-alerter-wnn66\" (UID: \"e0221987-8e5e-40d1-8842-c97993f08c6d\") " pod="openshift-network-operator/iptables-alerter-wnn66"
Oct 30 19:15:12.167716 ip-10-0-132-82 kubenswrapper[2577]: I1030 19:15:12.166275    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/08ae8ede-9a96-4a84-8ec8-8d364fd476a3-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-5lwkx\" (UID: \"08ae8ede-9a96-4a84-8ec8-8d364fd476a3\") " pod="openshift-ovn-kubernetes/ovnkube-node-5lwkx"
Oct 30 19:15:12.167716 ip-10-0-132-82 kubenswrapper[2577]: I1030 19:15:12.166327    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/08ae8ede-9a96-4a84-8ec8-8d364fd476a3-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-5lwkx\" (UID: \"08ae8ede-9a96-4a84-8ec8-8d364fd476a3\") " pod="openshift-ovn-kubernetes/ovnkube-node-5lwkx"
Oct 30 19:15:12.167716 ip-10-0-132-82 kubenswrapper[2577]: I1030 19:15:12.166332    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/cdb750aa-dd7e-4e2f-8297-9028acc0f559-host\") pod \"node-ca-kgfvh\" (UID: \"cdb750aa-dd7e-4e2f-8297-9028acc0f559\") " pod="openshift-image-registry/node-ca-kgfvh"
Oct 30 19:15:12.167716 ip-10-0-132-82 kubenswrapper[2577]: I1030 19:15:12.166336    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/08ae8ede-9a96-4a84-8ec8-8d364fd476a3-host-run-ovn-kubernetes\") pod \"ovnkube-node-5lwkx\" (UID: \"08ae8ede-9a96-4a84-8ec8-8d364fd476a3\") " pod="openshift-ovn-kubernetes/ovnkube-node-5lwkx"
Oct 30 19:15:12.167716 ip-10-0-132-82 kubenswrapper[2577]: I1030 19:15:12.166327    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/c3780b96-955e-4728-a2d0-b7ec30d0f6f3-device-dir\") pod \"aws-ebs-csi-driver-node-qgv98\" (UID: \"c3780b96-955e-4728-a2d0-b7ec30d0f6f3\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-qgv98"
Oct 30 19:15:12.167716 ip-10-0-132-82 kubenswrapper[2577]: I1030 19:15:12.166358    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/cdb750aa-dd7e-4e2f-8297-9028acc0f559-host\") pod \"node-ca-kgfvh\" (UID: \"cdb750aa-dd7e-4e2f-8297-9028acc0f559\") " pod="openshift-image-registry/node-ca-kgfvh"
Oct 30 19:15:12.167716 ip-10-0-132-82 kubenswrapper[2577]: I1030 19:15:12.166364    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/dccdaf98-aac7-4756-b93f-22f62a286a1f-multus-conf-dir\") pod \"multus-jvbw5\" (UID: \"dccdaf98-aac7-4756-b93f-22f62a286a1f\") " pod="openshift-multus/multus-jvbw5"
Oct 30 19:15:12.167716 ip-10-0-132-82 kubenswrapper[2577]: I1030 19:15:12.166387    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/dccdaf98-aac7-4756-b93f-22f62a286a1f-etc-kubernetes\") pod \"multus-jvbw5\" (UID: \"dccdaf98-aac7-4756-b93f-22f62a286a1f\") " pod="openshift-multus/multus-jvbw5"
Oct 30 19:15:12.167716 ip-10-0-132-82 kubenswrapper[2577]: I1030 19:15:12.166411    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-v7kvw\" (UniqueName: \"kubernetes.io/projected/287ba08f-7b9f-4609-8e35-832c0879c44c-kube-api-access-v7kvw\") pod \"network-metrics-daemon-822gp\" (UID: \"287ba08f-7b9f-4609-8e35-832c0879c44c\") " pod="openshift-multus/network-metrics-daemon-822gp"
Oct 30 19:15:12.167716 ip-10-0-132-82 kubenswrapper[2577]: I1030 19:15:12.166431    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/dccdaf98-aac7-4756-b93f-22f62a286a1f-multus-conf-dir\") pod \"multus-jvbw5\" (UID: \"dccdaf98-aac7-4756-b93f-22f62a286a1f\") " pod="openshift-multus/multus-jvbw5"
Oct 30 19:15:12.167716 ip-10-0-132-82 kubenswrapper[2577]: I1030 19:15:12.166437    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/dccdaf98-aac7-4756-b93f-22f62a286a1f-etc-kubernetes\") pod \"multus-jvbw5\" (UID: \"dccdaf98-aac7-4756-b93f-22f62a286a1f\") " pod="openshift-multus/multus-jvbw5"
Oct 30 19:15:12.167716 ip-10-0-132-82 kubenswrapper[2577]: I1030 19:15:12.166437    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/39bb8c16-8282-442a-841b-0056baae014d-etc-sysctl-conf\") pod \"tuned-2fv9z\" (UID: \"39bb8c16-8282-442a-841b-0056baae014d\") " pod="openshift-cluster-node-tuning-operator/tuned-2fv9z"
Oct 30 19:15:12.167716 ip-10-0-132-82 kubenswrapper[2577]: I1030 19:15:12.166477    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/08ae8ede-9a96-4a84-8ec8-8d364fd476a3-host-cni-bin\") pod \"ovnkube-node-5lwkx\" (UID: \"08ae8ede-9a96-4a84-8ec8-8d364fd476a3\") " pod="openshift-ovn-kubernetes/ovnkube-node-5lwkx"
Oct 30 19:15:12.167716 ip-10-0-132-82 kubenswrapper[2577]: I1030 19:15:12.166498    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bnxm6\" (UniqueName: \"kubernetes.io/projected/08ae8ede-9a96-4a84-8ec8-8d364fd476a3-kube-api-access-bnxm6\") pod \"ovnkube-node-5lwkx\" (UID: \"08ae8ede-9a96-4a84-8ec8-8d364fd476a3\") " pod="openshift-ovn-kubernetes/ovnkube-node-5lwkx"
Oct 30 19:15:12.168457 ip-10-0-132-82 kubenswrapper[2577]: I1030 19:15:12.166514    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/cdb750aa-dd7e-4e2f-8297-9028acc0f559-serviceca\") pod \"node-ca-kgfvh\" (UID: \"cdb750aa-dd7e-4e2f-8297-9028acc0f559\") " pod="openshift-image-registry/node-ca-kgfvh"
Oct 30 19:15:12.168457 ip-10-0-132-82 kubenswrapper[2577]: I1030 19:15:12.166537    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/dccdaf98-aac7-4756-b93f-22f62a286a1f-cnibin\") pod \"multus-jvbw5\" (UID: \"dccdaf98-aac7-4756-b93f-22f62a286a1f\") " pod="openshift-multus/multus-jvbw5"
Oct 30 19:15:12.168457 ip-10-0-132-82 kubenswrapper[2577]: I1030 19:15:12.166538    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/08ae8ede-9a96-4a84-8ec8-8d364fd476a3-host-cni-bin\") pod \"ovnkube-node-5lwkx\" (UID: \"08ae8ede-9a96-4a84-8ec8-8d364fd476a3\") " pod="openshift-ovn-kubernetes/ovnkube-node-5lwkx"
Oct 30 19:15:12.168457 ip-10-0-132-82 kubenswrapper[2577]: I1030 19:15:12.166561    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/dccdaf98-aac7-4756-b93f-22f62a286a1f-multus-daemon-config\") pod \"multus-jvbw5\" (UID: \"dccdaf98-aac7-4756-b93f-22f62a286a1f\") " pod="openshift-multus/multus-jvbw5"
Oct 30 19:15:12.168457 ip-10-0-132-82 kubenswrapper[2577]: I1030 19:15:12.166589    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/39bb8c16-8282-442a-841b-0056baae014d-etc-systemd\") pod \"tuned-2fv9z\" (UID: \"39bb8c16-8282-442a-841b-0056baae014d\") " pod="openshift-cluster-node-tuning-operator/tuned-2fv9z"
Oct 30 19:15:12.168457 ip-10-0-132-82 kubenswrapper[2577]: I1030 19:15:12.166596    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/dccdaf98-aac7-4756-b93f-22f62a286a1f-cnibin\") pod \"multus-jvbw5\" (UID: \"dccdaf98-aac7-4756-b93f-22f62a286a1f\") " pod="openshift-multus/multus-jvbw5"
Oct 30 19:15:12.168457 ip-10-0-132-82 kubenswrapper[2577]: I1030 19:15:12.166596    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/39bb8c16-8282-442a-841b-0056baae014d-etc-sysctl-conf\") pod \"tuned-2fv9z\" (UID: \"39bb8c16-8282-442a-841b-0056baae014d\") " pod="openshift-cluster-node-tuning-operator/tuned-2fv9z"
Oct 30 19:15:12.168457 ip-10-0-132-82 kubenswrapper[2577]: I1030 19:15:12.166611    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/39bb8c16-8282-442a-841b-0056baae014d-lib-modules\") pod \"tuned-2fv9z\" (UID: \"39bb8c16-8282-442a-841b-0056baae014d\") " pod="openshift-cluster-node-tuning-operator/tuned-2fv9z"
Oct 30 19:15:12.168457 ip-10-0-132-82 kubenswrapper[2577]: I1030 19:15:12.166650    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/39bb8c16-8282-442a-841b-0056baae014d-etc-systemd\") pod \"tuned-2fv9z\" (UID: \"39bb8c16-8282-442a-841b-0056baae014d\") " pod="openshift-cluster-node-tuning-operator/tuned-2fv9z"
Oct 30 19:15:12.168457 ip-10-0-132-82 kubenswrapper[2577]: I1030 19:15:12.166651    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/c16c0a80-c5ce-4f5d-827a-2e3eb2c3cfb8-dbus\") pod \"global-pull-secret-syncer-76rht\" (UID: \"c16c0a80-c5ce-4f5d-827a-2e3eb2c3cfb8\") " pod="kube-system/global-pull-secret-syncer-76rht"
Oct 30 19:15:12.168457 ip-10-0-132-82 kubenswrapper[2577]: I1030 19:15:12.166681    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/39bb8c16-8282-442a-841b-0056baae014d-etc-sysconfig\") pod \"tuned-2fv9z\" (UID: \"39bb8c16-8282-442a-841b-0056baae014d\") " pod="openshift-cluster-node-tuning-operator/tuned-2fv9z"
Oct 30 19:15:12.168457 ip-10-0-132-82 kubenswrapper[2577]: I1030 19:15:12.166700    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/08ae8ede-9a96-4a84-8ec8-8d364fd476a3-ovnkube-config\") pod \"ovnkube-node-5lwkx\" (UID: \"08ae8ede-9a96-4a84-8ec8-8d364fd476a3\") " pod="openshift-ovn-kubernetes/ovnkube-node-5lwkx"
Oct 30 19:15:12.168457 ip-10-0-132-82 kubenswrapper[2577]: I1030 19:15:12.166705    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/c16c0a80-c5ce-4f5d-827a-2e3eb2c3cfb8-dbus\") pod \"global-pull-secret-syncer-76rht\" (UID: \"c16c0a80-c5ce-4f5d-827a-2e3eb2c3cfb8\") " pod="kube-system/global-pull-secret-syncer-76rht"
Oct 30 19:15:12.168457 ip-10-0-132-82 kubenswrapper[2577]: I1030 19:15:12.166720    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4rcdn\" (UniqueName: \"kubernetes.io/projected/bab0abf9-46d4-408e-9aa0-afa7279cd9d1-kube-api-access-4rcdn\") pod \"multus-additional-cni-plugins-4pvgg\" (UID: \"bab0abf9-46d4-408e-9aa0-afa7279cd9d1\") " pod="openshift-multus/multus-additional-cni-plugins-4pvgg"
Oct 30 19:15:12.168457 ip-10-0-132-82 kubenswrapper[2577]: I1030 19:15:12.166743    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/dccdaf98-aac7-4756-b93f-22f62a286a1f-system-cni-dir\") pod \"multus-jvbw5\" (UID: \"dccdaf98-aac7-4756-b93f-22f62a286a1f\") " pod="openshift-multus/multus-jvbw5"
Oct 30 19:15:12.168457 ip-10-0-132-82 kubenswrapper[2577]: I1030 19:15:12.166746    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/39bb8c16-8282-442a-841b-0056baae014d-etc-sysconfig\") pod \"tuned-2fv9z\" (UID: \"39bb8c16-8282-442a-841b-0056baae014d\") " pod="openshift-cluster-node-tuning-operator/tuned-2fv9z"
Oct 30 19:15:12.168457 ip-10-0-132-82 kubenswrapper[2577]: I1030 19:15:12.166725    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/39bb8c16-8282-442a-841b-0056baae014d-lib-modules\") pod \"tuned-2fv9z\" (UID: \"39bb8c16-8282-442a-841b-0056baae014d\") " pod="openshift-cluster-node-tuning-operator/tuned-2fv9z"
Oct 30 19:15:12.168457 ip-10-0-132-82 kubenswrapper[2577]: I1030 19:15:12.166764    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/c3780b96-955e-4728-a2d0-b7ec30d0f6f3-etc-selinux\") pod \"aws-ebs-csi-driver-node-qgv98\" (UID: \"c3780b96-955e-4728-a2d0-b7ec30d0f6f3\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-qgv98"
Oct 30 19:15:12.168927 ip-10-0-132-82 kubenswrapper[2577]: I1030 19:15:12.166776    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/dccdaf98-aac7-4756-b93f-22f62a286a1f-system-cni-dir\") pod \"multus-jvbw5\" (UID: \"dccdaf98-aac7-4756-b93f-22f62a286a1f\") " pod="openshift-multus/multus-jvbw5"
Oct 30 19:15:12.168927 ip-10-0-132-82 kubenswrapper[2577]: I1030 19:15:12.166813    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/39bb8c16-8282-442a-841b-0056baae014d-etc-sysctl-d\") pod \"tuned-2fv9z\" (UID: \"39bb8c16-8282-442a-841b-0056baae014d\") " pod="openshift-cluster-node-tuning-operator/tuned-2fv9z"
Oct 30 19:15:12.168927 ip-10-0-132-82 kubenswrapper[2577]: I1030 19:15:12.166841    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/39bb8c16-8282-442a-841b-0056baae014d-sys\") pod \"tuned-2fv9z\" (UID: \"39bb8c16-8282-442a-841b-0056baae014d\") " pod="openshift-cluster-node-tuning-operator/tuned-2fv9z"
Oct 30 19:15:12.168927 ip-10-0-132-82 kubenswrapper[2577]: I1030 19:15:12.166815    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/c3780b96-955e-4728-a2d0-b7ec30d0f6f3-etc-selinux\") pod \"aws-ebs-csi-driver-node-qgv98\" (UID: \"c3780b96-955e-4728-a2d0-b7ec30d0f6f3\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-qgv98"
Oct 30 19:15:12.168927 ip-10-0-132-82 kubenswrapper[2577]: I1030 19:15:12.166900    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/39bb8c16-8282-442a-841b-0056baae014d-sys\") pod \"tuned-2fv9z\" (UID: \"39bb8c16-8282-442a-841b-0056baae014d\") " pod="openshift-cluster-node-tuning-operator/tuned-2fv9z"
Oct 30 19:15:12.168927 ip-10-0-132-82 kubenswrapper[2577]: I1030 19:15:12.166866    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-w8zg9\" (UniqueName: \"kubernetes.io/projected/48386105-d79b-4f92-baa0-f330b9658ad0-kube-api-access-w8zg9\") pod \"node-resolver-g9xd6\" (UID: \"48386105-d79b-4f92-baa0-f330b9658ad0\") " pod="openshift-dns/node-resolver-g9xd6"
Oct 30 19:15:12.168927 ip-10-0-132-82 kubenswrapper[2577]: I1030 19:15:12.167039    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/08ae8ede-9a96-4a84-8ec8-8d364fd476a3-run-systemd\") pod \"ovnkube-node-5lwkx\" (UID: \"08ae8ede-9a96-4a84-8ec8-8d364fd476a3\") " pod="openshift-ovn-kubernetes/ovnkube-node-5lwkx"
Oct 30 19:15:12.168927 ip-10-0-132-82 kubenswrapper[2577]: I1030 19:15:12.167062    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/08ae8ede-9a96-4a84-8ec8-8d364fd476a3-host-cni-netd\") pod \"ovnkube-node-5lwkx\" (UID: \"08ae8ede-9a96-4a84-8ec8-8d364fd476a3\") " pod="openshift-ovn-kubernetes/ovnkube-node-5lwkx"
Oct 30 19:15:12.168927 ip-10-0-132-82 kubenswrapper[2577]: I1030 19:15:12.167083    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/bab0abf9-46d4-408e-9aa0-afa7279cd9d1-tuning-conf-dir\") pod \"multus-additional-cni-plugins-4pvgg\" (UID: \"bab0abf9-46d4-408e-9aa0-afa7279cd9d1\") " pod="openshift-multus/multus-additional-cni-plugins-4pvgg"
Oct 30 19:15:12.168927 ip-10-0-132-82 kubenswrapper[2577]: I1030 19:15:12.167099    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/39bb8c16-8282-442a-841b-0056baae014d-tmp\") pod \"tuned-2fv9z\" (UID: \"39bb8c16-8282-442a-841b-0056baae014d\") " pod="openshift-cluster-node-tuning-operator/tuned-2fv9z"
Oct 30 19:15:12.168927 ip-10-0-132-82 kubenswrapper[2577]: I1030 19:15:12.167010    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/39bb8c16-8282-442a-841b-0056baae014d-etc-sysctl-d\") pod \"tuned-2fv9z\" (UID: \"39bb8c16-8282-442a-841b-0056baae014d\") " pod="openshift-cluster-node-tuning-operator/tuned-2fv9z"
Oct 30 19:15:12.168927 ip-10-0-132-82 kubenswrapper[2577]: I1030 19:15:12.167115    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/08ae8ede-9a96-4a84-8ec8-8d364fd476a3-env-overrides\") pod \"ovnkube-node-5lwkx\" (UID: \"08ae8ede-9a96-4a84-8ec8-8d364fd476a3\") " pod="openshift-ovn-kubernetes/ovnkube-node-5lwkx"
Oct 30 19:15:12.168927 ip-10-0-132-82 kubenswrapper[2577]: I1030 19:15:12.167115    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/08ae8ede-9a96-4a84-8ec8-8d364fd476a3-run-systemd\") pod \"ovnkube-node-5lwkx\" (UID: \"08ae8ede-9a96-4a84-8ec8-8d364fd476a3\") " pod="openshift-ovn-kubernetes/ovnkube-node-5lwkx"
Oct 30 19:15:12.168927 ip-10-0-132-82 kubenswrapper[2577]: I1030 19:15:12.167153    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/08ae8ede-9a96-4a84-8ec8-8d364fd476a3-host-cni-netd\") pod \"ovnkube-node-5lwkx\" (UID: \"08ae8ede-9a96-4a84-8ec8-8d364fd476a3\") " pod="openshift-ovn-kubernetes/ovnkube-node-5lwkx"
Oct 30 19:15:12.168927 ip-10-0-132-82 kubenswrapper[2577]: I1030 19:15:12.167164    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gdq6k\" (UniqueName: \"kubernetes.io/projected/39bb8c16-8282-442a-841b-0056baae014d-kube-api-access-gdq6k\") pod \"tuned-2fv9z\" (UID: \"39bb8c16-8282-442a-841b-0056baae014d\") " pod="openshift-cluster-node-tuning-operator/tuned-2fv9z"
Oct 30 19:15:12.168927 ip-10-0-132-82 kubenswrapper[2577]: I1030 19:15:12.167195    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bzw4s\" (UniqueName: \"kubernetes.io/projected/4ca01b49-9819-4a90-a851-7553ad568679-kube-api-access-bzw4s\") pod \"network-check-target-c24j9\" (UID: \"4ca01b49-9819-4a90-a851-7553ad568679\") " pod="openshift-network-diagnostics/network-check-target-c24j9"
Oct 30 19:15:12.168927 ip-10-0-132-82 kubenswrapper[2577]: I1030 19:15:12.167222    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/48386105-d79b-4f92-baa0-f330b9658ad0-hosts-file\") pod \"node-resolver-g9xd6\" (UID: \"48386105-d79b-4f92-baa0-f330b9658ad0\") " pod="openshift-dns/node-resolver-g9xd6"
Oct 30 19:15:12.169494 ip-10-0-132-82 kubenswrapper[2577]: I1030 19:15:12.167237    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/bab0abf9-46d4-408e-9aa0-afa7279cd9d1-tuning-conf-dir\") pod \"multus-additional-cni-plugins-4pvgg\" (UID: \"bab0abf9-46d4-408e-9aa0-afa7279cd9d1\") " pod="openshift-multus/multus-additional-cni-plugins-4pvgg"
Oct 30 19:15:12.169494 ip-10-0-132-82 kubenswrapper[2577]: I1030 19:15:12.167248    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/bab0abf9-46d4-408e-9aa0-afa7279cd9d1-system-cni-dir\") pod \"multus-additional-cni-plugins-4pvgg\" (UID: \"bab0abf9-46d4-408e-9aa0-afa7279cd9d1\") " pod="openshift-multus/multus-additional-cni-plugins-4pvgg"
Oct 30 19:15:12.169494 ip-10-0-132-82 kubenswrapper[2577]: I1030 19:15:12.167273    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6bqfr\" (UniqueName: \"kubernetes.io/projected/dccdaf98-aac7-4756-b93f-22f62a286a1f-kube-api-access-6bqfr\") pod \"multus-jvbw5\" (UID: \"dccdaf98-aac7-4756-b93f-22f62a286a1f\") " pod="openshift-multus/multus-jvbw5"
Oct 30 19:15:12.169494 ip-10-0-132-82 kubenswrapper[2577]: I1030 19:15:12.167296    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/c3780b96-955e-4728-a2d0-b7ec30d0f6f3-kubelet-dir\") pod \"aws-ebs-csi-driver-node-qgv98\" (UID: \"c3780b96-955e-4728-a2d0-b7ec30d0f6f3\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-qgv98"
Oct 30 19:15:12.169494 ip-10-0-132-82 kubenswrapper[2577]: I1030 19:15:12.167309    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/48386105-d79b-4f92-baa0-f330b9658ad0-hosts-file\") pod \"node-resolver-g9xd6\" (UID: \"48386105-d79b-4f92-baa0-f330b9658ad0\") " pod="openshift-dns/node-resolver-g9xd6"
Oct 30 19:15:12.169494 ip-10-0-132-82 kubenswrapper[2577]: I1030 19:15:12.167321    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cdkvt\" (UniqueName: \"kubernetes.io/projected/c3780b96-955e-4728-a2d0-b7ec30d0f6f3-kube-api-access-cdkvt\") pod \"aws-ebs-csi-driver-node-qgv98\" (UID: \"c3780b96-955e-4728-a2d0-b7ec30d0f6f3\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-qgv98"
Oct 30 19:15:12.169494 ip-10-0-132-82 kubenswrapper[2577]: I1030 19:15:12.167348    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/bab0abf9-46d4-408e-9aa0-afa7279cd9d1-system-cni-dir\") pod \"multus-additional-cni-plugins-4pvgg\" (UID: \"bab0abf9-46d4-408e-9aa0-afa7279cd9d1\") " pod="openshift-multus/multus-additional-cni-plugins-4pvgg"
Oct 30 19:15:12.169494 ip-10-0-132-82 kubenswrapper[2577]: I1030 19:15:12.167366    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/c3780b96-955e-4728-a2d0-b7ec30d0f6f3-kubelet-dir\") pod \"aws-ebs-csi-driver-node-qgv98\" (UID: \"c3780b96-955e-4728-a2d0-b7ec30d0f6f3\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-qgv98"
Oct 30 19:15:12.169494 ip-10-0-132-82 kubenswrapper[2577]: I1030 19:15:12.167387    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/39bb8c16-8282-442a-841b-0056baae014d-etc-kubernetes\") pod \"tuned-2fv9z\" (UID: \"39bb8c16-8282-442a-841b-0056baae014d\") " pod="openshift-cluster-node-tuning-operator/tuned-2fv9z"
Oct 30 19:15:12.169494 ip-10-0-132-82 kubenswrapper[2577]: I1030 19:15:12.167414    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/39bb8c16-8282-442a-841b-0056baae014d-etc-kubernetes\") pod \"tuned-2fv9z\" (UID: \"39bb8c16-8282-442a-841b-0056baae014d\") " pod="openshift-cluster-node-tuning-operator/tuned-2fv9z"
Oct 30 19:15:12.169494 ip-10-0-132-82 kubenswrapper[2577]: I1030 19:15:12.167414    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/39bb8c16-8282-442a-841b-0056baae014d-var-lib-kubelet\") pod \"tuned-2fv9z\" (UID: \"39bb8c16-8282-442a-841b-0056baae014d\") " pod="openshift-cluster-node-tuning-operator/tuned-2fv9z"
Oct 30 19:15:12.169494 ip-10-0-132-82 kubenswrapper[2577]: I1030 19:15:12.167448    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/08ae8ede-9a96-4a84-8ec8-8d364fd476a3-host-kubelet\") pod \"ovnkube-node-5lwkx\" (UID: \"08ae8ede-9a96-4a84-8ec8-8d364fd476a3\") " pod="openshift-ovn-kubernetes/ovnkube-node-5lwkx"
Oct 30 19:15:12.169494 ip-10-0-132-82 kubenswrapper[2577]: I1030 19:15:12.167446    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/39bb8c16-8282-442a-841b-0056baae014d-var-lib-kubelet\") pod \"tuned-2fv9z\" (UID: \"39bb8c16-8282-442a-841b-0056baae014d\") " pod="openshift-cluster-node-tuning-operator/tuned-2fv9z"
Oct 30 19:15:12.169494 ip-10-0-132-82 kubenswrapper[2577]: I1030 19:15:12.167472    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/08ae8ede-9a96-4a84-8ec8-8d364fd476a3-run-openvswitch\") pod \"ovnkube-node-5lwkx\" (UID: \"08ae8ede-9a96-4a84-8ec8-8d364fd476a3\") " pod="openshift-ovn-kubernetes/ovnkube-node-5lwkx"
Oct 30 19:15:12.169494 ip-10-0-132-82 kubenswrapper[2577]: I1030 19:15:12.167496    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/08ae8ede-9a96-4a84-8ec8-8d364fd476a3-run-openvswitch\") pod \"ovnkube-node-5lwkx\" (UID: \"08ae8ede-9a96-4a84-8ec8-8d364fd476a3\") " pod="openshift-ovn-kubernetes/ovnkube-node-5lwkx"
Oct 30 19:15:12.169494 ip-10-0-132-82 kubenswrapper[2577]: I1030 19:15:12.167512    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/08ae8ede-9a96-4a84-8ec8-8d364fd476a3-run-ovn\") pod \"ovnkube-node-5lwkx\" (UID: \"08ae8ede-9a96-4a84-8ec8-8d364fd476a3\") " pod="openshift-ovn-kubernetes/ovnkube-node-5lwkx"
Oct 30 19:15:12.169494 ip-10-0-132-82 kubenswrapper[2577]: I1030 19:15:12.167537    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/bab0abf9-46d4-408e-9aa0-afa7279cd9d1-cnibin\") pod \"multus-additional-cni-plugins-4pvgg\" (UID: \"bab0abf9-46d4-408e-9aa0-afa7279cd9d1\") " pod="openshift-multus/multus-additional-cni-plugins-4pvgg"
Oct 30 19:15:12.169927 ip-10-0-132-82 kubenswrapper[2577]: I1030 19:15:12.167563    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/48386105-d79b-4f92-baa0-f330b9658ad0-tmp-dir\") pod \"node-resolver-g9xd6\" (UID: \"48386105-d79b-4f92-baa0-f330b9658ad0\") " pod="openshift-dns/node-resolver-g9xd6"
Oct 30 19:15:12.169927 ip-10-0-132-82 kubenswrapper[2577]: I1030 19:15:12.167556    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/08ae8ede-9a96-4a84-8ec8-8d364fd476a3-run-ovn\") pod \"ovnkube-node-5lwkx\" (UID: \"08ae8ede-9a96-4a84-8ec8-8d364fd476a3\") " pod="openshift-ovn-kubernetes/ovnkube-node-5lwkx"
Oct 30 19:15:12.169927 ip-10-0-132-82 kubenswrapper[2577]: I1030 19:15:12.167560    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/bab0abf9-46d4-408e-9aa0-afa7279cd9d1-cnibin\") pod \"multus-additional-cni-plugins-4pvgg\" (UID: \"bab0abf9-46d4-408e-9aa0-afa7279cd9d1\") " pod="openshift-multus/multus-additional-cni-plugins-4pvgg"
Oct 30 19:15:12.169927 ip-10-0-132-82 kubenswrapper[2577]: I1030 19:15:12.167524    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/08ae8ede-9a96-4a84-8ec8-8d364fd476a3-host-kubelet\") pod \"ovnkube-node-5lwkx\" (UID: \"08ae8ede-9a96-4a84-8ec8-8d364fd476a3\") " pod="openshift-ovn-kubernetes/ovnkube-node-5lwkx"
Oct 30 19:15:12.169927 ip-10-0-132-82 kubenswrapper[2577]: I1030 19:15:12.167591    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/08ae8ede-9a96-4a84-8ec8-8d364fd476a3-var-lib-openvswitch\") pod \"ovnkube-node-5lwkx\" (UID: \"08ae8ede-9a96-4a84-8ec8-8d364fd476a3\") " pod="openshift-ovn-kubernetes/ovnkube-node-5lwkx"
Oct 30 19:15:12.169927 ip-10-0-132-82 kubenswrapper[2577]: I1030 19:15:12.167616    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/08ae8ede-9a96-4a84-8ec8-8d364fd476a3-var-lib-openvswitch\") pod \"ovnkube-node-5lwkx\" (UID: \"08ae8ede-9a96-4a84-8ec8-8d364fd476a3\") " pod="openshift-ovn-kubernetes/ovnkube-node-5lwkx"
Oct 30 19:15:12.169927 ip-10-0-132-82 kubenswrapper[2577]: I1030 19:15:12.167620    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/dccdaf98-aac7-4756-b93f-22f62a286a1f-cni-binary-copy\") pod \"multus-jvbw5\" (UID: \"dccdaf98-aac7-4756-b93f-22f62a286a1f\") " pod="openshift-multus/multus-jvbw5"
Oct 30 19:15:12.169927 ip-10-0-132-82 kubenswrapper[2577]: I1030 19:15:12.167644    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/dccdaf98-aac7-4756-b93f-22f62a286a1f-host-run-k8s-cni-cncf-io\") pod \"multus-jvbw5\" (UID: \"dccdaf98-aac7-4756-b93f-22f62a286a1f\") " pod="openshift-multus/multus-jvbw5"
Oct 30 19:15:12.169927 ip-10-0-132-82 kubenswrapper[2577]: I1030 19:15:12.167667    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/c3780b96-955e-4728-a2d0-b7ec30d0f6f3-socket-dir\") pod \"aws-ebs-csi-driver-node-qgv98\" (UID: \"c3780b96-955e-4728-a2d0-b7ec30d0f6f3\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-qgv98"
Oct 30 19:15:12.169927 ip-10-0-132-82 kubenswrapper[2577]: I1030 19:15:12.167689    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/c3780b96-955e-4728-a2d0-b7ec30d0f6f3-registration-dir\") pod \"aws-ebs-csi-driver-node-qgv98\" (UID: \"c3780b96-955e-4728-a2d0-b7ec30d0f6f3\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-qgv98"
Oct 30 19:15:12.169927 ip-10-0-132-82 kubenswrapper[2577]: I1030 19:15:12.167715    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/dccdaf98-aac7-4756-b93f-22f62a286a1f-host-run-k8s-cni-cncf-io\") pod \"multus-jvbw5\" (UID: \"dccdaf98-aac7-4756-b93f-22f62a286a1f\") " pod="openshift-multus/multus-jvbw5"
Oct 30 19:15:12.169927 ip-10-0-132-82 kubenswrapper[2577]: I1030 19:15:12.167804    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/c3780b96-955e-4728-a2d0-b7ec30d0f6f3-socket-dir\") pod \"aws-ebs-csi-driver-node-qgv98\" (UID: \"c3780b96-955e-4728-a2d0-b7ec30d0f6f3\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-qgv98"
Oct 30 19:15:12.169927 ip-10-0-132-82 kubenswrapper[2577]: I1030 19:15:12.167832    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/48386105-d79b-4f92-baa0-f330b9658ad0-tmp-dir\") pod \"node-resolver-g9xd6\" (UID: \"48386105-d79b-4f92-baa0-f330b9658ad0\") " pod="openshift-dns/node-resolver-g9xd6"
Oct 30 19:15:12.169927 ip-10-0-132-82 kubenswrapper[2577]: I1030 19:15:12.167846    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/c3780b96-955e-4728-a2d0-b7ec30d0f6f3-registration-dir\") pod \"aws-ebs-csi-driver-node-qgv98\" (UID: \"c3780b96-955e-4728-a2d0-b7ec30d0f6f3\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-qgv98"
Oct 30 19:15:12.169927 ip-10-0-132-82 kubenswrapper[2577]: I1030 19:15:12.168240    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/dccdaf98-aac7-4756-b93f-22f62a286a1f-cni-binary-copy\") pod \"multus-jvbw5\" (UID: \"dccdaf98-aac7-4756-b93f-22f62a286a1f\") " pod="openshift-multus/multus-jvbw5"
Oct 30 19:15:12.169927 ip-10-0-132-82 kubenswrapper[2577]: I1030 19:15:12.169165    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/39bb8c16-8282-442a-841b-0056baae014d-etc-tuned\") pod \"tuned-2fv9z\" (UID: \"39bb8c16-8282-442a-841b-0056baae014d\") " pod="openshift-cluster-node-tuning-operator/tuned-2fv9z"
Oct 30 19:15:12.169927 ip-10-0-132-82 kubenswrapper[2577]: I1030 19:15:12.169690    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/39bb8c16-8282-442a-841b-0056baae014d-tmp\") pod \"tuned-2fv9z\" (UID: \"39bb8c16-8282-442a-841b-0056baae014d\") " pod="openshift-cluster-node-tuning-operator/tuned-2fv9z"
Oct 30 19:15:12.180245 ip-10-0-132-82 kubenswrapper[2577]: I1030 19:15:12.180228    2577 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"kube-root-ca.crt\""
Oct 30 19:15:12.180318 ip-10-0-132-82 kubenswrapper[2577]: I1030 19:15:12.180276    2577 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"whereabouts-flatfile-config\""
Oct 30 19:15:12.180318 ip-10-0-132-82 kubenswrapper[2577]: I1030 19:15:12.180289    2577 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"openshift-service-ca.crt\""
Oct 30 19:15:12.180408 ip-10-0-132-82 kubenswrapper[2577]: I1030 19:15:12.180331    2577 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"default-cni-sysctl-allowlist\""
Oct 30 19:15:12.180408 ip-10-0-132-82 kubenswrapper[2577]: I1030 19:15:12.180355    2577 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"kube-root-ca.crt\""
Oct 30 19:15:12.180408 ip-10-0-132-82 kubenswrapper[2577]: I1030 19:15:12.180402    2577 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"openshift-service-ca.crt\""
Oct 30 19:15:12.180513 ip-10-0-132-82 kubenswrapper[2577]: I1030 19:15:12.180484    2577 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"image-registry-certificates\""
Oct 30 19:15:12.180565 ip-10-0-132-82 kubenswrapper[2577]: I1030 19:15:12.180526    2577 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"openshift-service-ca.crt\""
Oct 30 19:15:12.180565 ip-10-0-132-82 kubenswrapper[2577]: I1030 19:15:12.180559    2577 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"kube-root-ca.crt\""
Oct 30 19:15:12.180628 ip-10-0-132-82 kubenswrapper[2577]: I1030 19:15:12.180620    2577 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"openshift-service-ca.crt\""
Oct 30 19:15:12.180721 ip-10-0-132-82 kubenswrapper[2577]: I1030 19:15:12.180710    2577 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-config\""
Oct 30 19:15:12.180766 ip-10-0-132-82 kubenswrapper[2577]: I1030 19:15:12.180721    2577 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"konnectivity-ca-bundle\""
Oct 30 19:15:12.180766 ip-10-0-132-82 kubenswrapper[2577]: I1030 19:15:12.180720    2577 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"kube-root-ca.crt\""
Oct 30 19:15:12.181133 ip-10-0-132-82 kubenswrapper[2577]: I1030 19:15:12.181121    2577 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"env-overrides\""
Oct 30 19:15:12.181185 ip-10-0-132-82 kubenswrapper[2577]: I1030 19:15:12.181148    2577 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"openshift-service-ca.crt\""
Oct 30 19:15:12.181185 ip-10-0-132-82 kubenswrapper[2577]: I1030 19:15:12.181148    2577 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"kube-root-ca.crt\""
Oct 30 19:15:12.181254 ip-10-0-132-82 kubenswrapper[2577]: I1030 19:15:12.181192    2577 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-script\""
Oct 30 19:15:12.181254 ip-10-0-132-82 kubenswrapper[2577]: I1030 19:15:12.181209    2577 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"multus-daemon-config\""
Oct 30 19:15:12.181254 ip-10-0-132-82 kubenswrapper[2577]: I1030 19:15:12.181216    2577 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"openshift-service-ca.crt\""
Oct 30 19:15:12.181254 ip-10-0-132-82 kubenswrapper[2577]: I1030 19:15:12.181222    2577 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"kube-root-ca.crt\""
Oct 30 19:15:12.181363 ip-10-0-132-82 kubenswrapper[2577]: I1030 19:15:12.181271    2577 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-script-lib\""
Oct 30 19:15:12.181936 ip-10-0-132-82 kubenswrapper[2577]: I1030 19:15:12.181915    2577 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent\""
Oct 30 19:15:12.187188 ip-10-0-132-82 kubenswrapper[2577]: I1030 19:15:12.187165    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/bab0abf9-46d4-408e-9aa0-afa7279cd9d1-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-4pvgg\" (UID: \"bab0abf9-46d4-408e-9aa0-afa7279cd9d1\") " pod="openshift-multus/multus-additional-cni-plugins-4pvgg"
Oct 30 19:15:12.187275 ip-10-0-132-82 kubenswrapper[2577]: I1030 19:15:12.187199    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/08ae8ede-9a96-4a84-8ec8-8d364fd476a3-ovnkube-script-lib\") pod \"ovnkube-node-5lwkx\" (UID: \"08ae8ede-9a96-4a84-8ec8-8d364fd476a3\") " pod="openshift-ovn-kubernetes/ovnkube-node-5lwkx"
Oct 30 19:15:12.187275 ip-10-0-132-82 kubenswrapper[2577]: I1030 19:15:12.187205    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/bab0abf9-46d4-408e-9aa0-afa7279cd9d1-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-4pvgg\" (UID: \"bab0abf9-46d4-408e-9aa0-afa7279cd9d1\") " pod="openshift-multus/multus-additional-cni-plugins-4pvgg"
Oct 30 19:15:12.187389 ip-10-0-132-82 kubenswrapper[2577]: I1030 19:15:12.187277    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/e0221987-8e5e-40d1-8842-c97993f08c6d-iptables-alerter-script\") pod \"iptables-alerter-wnn66\" (UID: \"e0221987-8e5e-40d1-8842-c97993f08c6d\") " pod="openshift-network-operator/iptables-alerter-wnn66"
Oct 30 19:15:12.187389 ip-10-0-132-82 kubenswrapper[2577]: I1030 19:15:12.187296    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/af36f4d4-f985-4937-a9f7-9884936f3690-konnectivity-ca\") pod \"konnectivity-agent-qnm56\" (UID: \"af36f4d4-f985-4937-a9f7-9884936f3690\") " pod="kube-system/konnectivity-agent-qnm56"
Oct 30 19:15:12.187576 ip-10-0-132-82 kubenswrapper[2577]: I1030 19:15:12.187556    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/08ae8ede-9a96-4a84-8ec8-8d364fd476a3-env-overrides\") pod \"ovnkube-node-5lwkx\" (UID: \"08ae8ede-9a96-4a84-8ec8-8d364fd476a3\") " pod="openshift-ovn-kubernetes/ovnkube-node-5lwkx"
Oct 30 19:15:12.187699 ip-10-0-132-82 kubenswrapper[2577]: I1030 19:15:12.187675    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/08ae8ede-9a96-4a84-8ec8-8d364fd476a3-ovnkube-config\") pod \"ovnkube-node-5lwkx\" (UID: \"08ae8ede-9a96-4a84-8ec8-8d364fd476a3\") " pod="openshift-ovn-kubernetes/ovnkube-node-5lwkx"
Oct 30 19:15:12.187699 ip-10-0-132-82 kubenswrapper[2577]: I1030 19:15:12.187697    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/cdb750aa-dd7e-4e2f-8297-9028acc0f559-serviceca\") pod \"node-ca-kgfvh\" (UID: \"cdb750aa-dd7e-4e2f-8297-9028acc0f559\") " pod="openshift-image-registry/node-ca-kgfvh"
Oct 30 19:15:12.188384 ip-10-0-132-82 kubenswrapper[2577]: I1030 19:15:12.188368    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/dccdaf98-aac7-4756-b93f-22f62a286a1f-multus-daemon-config\") pod \"multus-jvbw5\" (UID: \"dccdaf98-aac7-4756-b93f-22f62a286a1f\") " pod="openshift-multus/multus-jvbw5"
Oct 30 19:15:12.188621 ip-10-0-132-82 kubenswrapper[2577]: I1030 19:15:12.188606    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/af36f4d4-f985-4937-a9f7-9884936f3690-agent-certs\") pod \"konnectivity-agent-qnm56\" (UID: \"af36f4d4-f985-4937-a9f7-9884936f3690\") " pod="kube-system/konnectivity-agent-qnm56"
Oct 30 19:15:12.194042 ip-10-0-132-82 kubenswrapper[2577]: I1030 19:15:12.194026    2577 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-metrics-serving-cert\""
Oct 30 19:15:12.195906 ip-10-0-132-82 kubenswrapper[2577]: I1030 19:15:12.195893    2577 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-node-metrics-cert\""
Oct 30 19:15:12.198058 ip-10-0-132-82 kubenswrapper[2577]: I1030 19:15:12.198040    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/08ae8ede-9a96-4a84-8ec8-8d364fd476a3-ovn-node-metrics-cert\") pod \"ovnkube-node-5lwkx\" (UID: \"08ae8ede-9a96-4a84-8ec8-8d364fd476a3\") " pod="openshift-ovn-kubernetes/ovnkube-node-5lwkx"
Oct 30 19:15:12.199477 ip-10-0-132-82 kubenswrapper[2577]: I1030 19:15:12.199462    2577 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-kubernetes-node-dockercfg-n57d7\""
Oct 30 19:15:12.230098 ip-10-0-132-82 kubenswrapper[2577]: I1030 19:15:12.230074    2577 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"tuned-dockercfg-rhxwm\""
Oct 30 19:15:12.230179 ip-10-0-132-82 kubenswrapper[2577]: I1030 19:15:12.230117    2577 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-sa-dockercfg-csh58\""
Oct 30 19:15:12.230179 ip-10-0-132-82 kubenswrapper[2577]: I1030 19:15:12.230155    2577 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-ancillary-tools-dockercfg-4k9jz\""
Oct 30 19:15:12.230254 ip-10-0-132-82 kubenswrapper[2577]: I1030 19:15:12.230207    2577 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"node-ca-dockercfg-6brtl\""
Oct 30 19:15:12.230254 ip-10-0-132-82 kubenswrapper[2577]: I1030 19:15:12.230226    2577 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent-dockercfg-q7svh\""
Oct 30 19:15:12.230647 ip-10-0-132-82 kubenswrapper[2577]: I1030 19:15:12.230635    2577 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-node-identity-dockercfg-46fzl\""
Oct 30 19:15:12.231439 ip-10-0-132-82 kubenswrapper[2577]: I1030 19:15:12.231128    2577 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-dockercfg-5vtr8\""
Oct 30 19:15:12.231439 ip-10-0-132-82 kubenswrapper[2577]: E1030 19:15:12.231140    2577 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Oct 30 19:15:12.231439 ip-10-0-132-82 kubenswrapper[2577]: E1030 19:15:12.231157    2577 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Oct 30 19:15:12.231439 ip-10-0-132-82 kubenswrapper[2577]: E1030 19:15:12.231169    2577 projected.go:196] Error preparing data for projected volume kube-api-access-bzw4s for pod openshift-network-diagnostics/network-check-target-c24j9: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Oct 30 19:15:12.231439 ip-10-0-132-82 kubenswrapper[2577]: E1030 19:15:12.231249    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/4ca01b49-9819-4a90-a851-7553ad568679-kube-api-access-bzw4s podName:4ca01b49-9819-4a90-a851-7553ad568679 nodeName:}" failed. No retries permitted until 2026-10-30 19:15:12.731228114 +0000 UTC m=+3.137148544 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-api-access-bzw4s" (UniqueName: "kubernetes.io/projected/4ca01b49-9819-4a90-a851-7553ad568679-kube-api-access-bzw4s") pod "network-check-target-c24j9" (UID: "4ca01b49-9819-4a90-a851-7553ad568679") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Oct 30 19:15:12.234055 ip-10-0-132-82 kubenswrapper[2577]: I1030 19:15:12.234031    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fmnd4\" (UniqueName: \"kubernetes.io/projected/e0221987-8e5e-40d1-8842-c97993f08c6d-kube-api-access-fmnd4\") pod \"iptables-alerter-wnn66\" (UID: \"e0221987-8e5e-40d1-8842-c97993f08c6d\") " pod="openshift-network-operator/iptables-alerter-wnn66"
Oct 30 19:15:12.234221 ip-10-0-132-82 kubenswrapper[2577]: I1030 19:15:12.234201    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bnxm6\" (UniqueName: \"kubernetes.io/projected/08ae8ede-9a96-4a84-8ec8-8d364fd476a3-kube-api-access-bnxm6\") pod \"ovnkube-node-5lwkx\" (UID: \"08ae8ede-9a96-4a84-8ec8-8d364fd476a3\") " pod="openshift-ovn-kubernetes/ovnkube-node-5lwkx"
Oct 30 19:15:12.234308 ip-10-0-132-82 kubenswrapper[2577]: I1030 19:15:12.234223    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-cdkvt\" (UniqueName: \"kubernetes.io/projected/c3780b96-955e-4728-a2d0-b7ec30d0f6f3-kube-api-access-cdkvt\") pod \"aws-ebs-csi-driver-node-qgv98\" (UID: \"c3780b96-955e-4728-a2d0-b7ec30d0f6f3\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-qgv98"
Oct 30 19:15:12.234308 ip-10-0-132-82 kubenswrapper[2577]: I1030 19:15:12.234261    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gdq6k\" (UniqueName: \"kubernetes.io/projected/39bb8c16-8282-442a-841b-0056baae014d-kube-api-access-gdq6k\") pod \"tuned-2fv9z\" (UID: \"39bb8c16-8282-442a-841b-0056baae014d\") " pod="openshift-cluster-node-tuning-operator/tuned-2fv9z"
Oct 30 19:15:12.234589 ip-10-0-132-82 kubenswrapper[2577]: I1030 19:15:12.234565    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-v7kvw\" (UniqueName: \"kubernetes.io/projected/287ba08f-7b9f-4609-8e35-832c0879c44c-kube-api-access-v7kvw\") pod \"network-metrics-daemon-822gp\" (UID: \"287ba08f-7b9f-4609-8e35-832c0879c44c\") " pod="openshift-multus/network-metrics-daemon-822gp"
Oct 30 19:15:12.234681 ip-10-0-132-82 kubenswrapper[2577]: I1030 19:15:12.234589    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6bqfr\" (UniqueName: \"kubernetes.io/projected/dccdaf98-aac7-4756-b93f-22f62a286a1f-kube-api-access-6bqfr\") pod \"multus-jvbw5\" (UID: \"dccdaf98-aac7-4756-b93f-22f62a286a1f\") " pod="openshift-multus/multus-jvbw5"
Oct 30 19:15:12.234789 ip-10-0-132-82 kubenswrapper[2577]: I1030 19:15:12.234768    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rvrch\" (UniqueName: \"kubernetes.io/projected/cdb750aa-dd7e-4e2f-8297-9028acc0f559-kube-api-access-rvrch\") pod \"node-ca-kgfvh\" (UID: \"cdb750aa-dd7e-4e2f-8297-9028acc0f559\") " pod="openshift-image-registry/node-ca-kgfvh"
Oct 30 19:15:12.234879 ip-10-0-132-82 kubenswrapper[2577]: I1030 19:15:12.234861    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4rcdn\" (UniqueName: \"kubernetes.io/projected/bab0abf9-46d4-408e-9aa0-afa7279cd9d1-kube-api-access-4rcdn\") pod \"multus-additional-cni-plugins-4pvgg\" (UID: \"bab0abf9-46d4-408e-9aa0-afa7279cd9d1\") " pod="openshift-multus/multus-additional-cni-plugins-4pvgg"
Oct 30 19:15:12.235876 ip-10-0-132-82 kubenswrapper[2577]: I1030 19:15:12.235852    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-w8zg9\" (UniqueName: \"kubernetes.io/projected/48386105-d79b-4f92-baa0-f330b9658ad0-kube-api-access-w8zg9\") pod \"node-resolver-g9xd6\" (UID: \"48386105-d79b-4f92-baa0-f330b9658ad0\") " pod="openshift-dns/node-resolver-g9xd6"
Oct 30 19:15:12.335314 ip-10-0-132-82 kubenswrapper[2577]: I1030 19:15:12.335262    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-g9xd6"
Oct 30 19:15:12.341564 ip-10-0-132-82 kubenswrapper[2577]: I1030 19:15:12.341544    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-jvbw5"
Oct 30 19:15:12.345043 ip-10-0-132-82 kubenswrapper[2577]: W1030 19:15:12.345018    2577 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod48386105_d79b_4f92_baa0_f330b9658ad0.slice/crio-4f8ff29fa81054ed5a37cd86a5ec74649a5651dc5a63c441321ba062767aa578 WatchSource:0}: Error finding container 4f8ff29fa81054ed5a37cd86a5ec74649a5651dc5a63c441321ba062767aa578: Status 404 returned error can't find the container with id 4f8ff29fa81054ed5a37cd86a5ec74649a5651dc5a63c441321ba062767aa578
Oct 30 19:15:12.349691 ip-10-0-132-82 kubenswrapper[2577]: I1030 19:15:12.349675    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-wnn66"
Oct 30 19:15:12.349939 ip-10-0-132-82 kubenswrapper[2577]: W1030 19:15:12.349922    2577 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddccdaf98_aac7_4756_b93f_22f62a286a1f.slice/crio-f4a575fbd0a385888d1b5e4d7a3e43290db13e5198927239e5901160a4449ad2 WatchSource:0}: Error finding container f4a575fbd0a385888d1b5e4d7a3e43290db13e5198927239e5901160a4449ad2: Status 404 returned error can't find the container with id f4a575fbd0a385888d1b5e4d7a3e43290db13e5198927239e5901160a4449ad2
Oct 30 19:15:12.357994 ip-10-0-132-82 kubenswrapper[2577]: W1030 19:15:12.357957    2577 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode0221987_8e5e_40d1_8842_c97993f08c6d.slice/crio-32595290e48470808e0ab2db048e03df52c1e31b97a08ef92d7a4c00da701711 WatchSource:0}: Error finding container 32595290e48470808e0ab2db048e03df52c1e31b97a08ef92d7a4c00da701711: Status 404 returned error can't find the container with id 32595290e48470808e0ab2db048e03df52c1e31b97a08ef92d7a4c00da701711
Oct 30 19:15:12.372889 ip-10-0-132-82 kubenswrapper[2577]: I1030 19:15:12.372873    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-kgfvh"
Oct 30 19:15:12.382734 ip-10-0-132-82 kubenswrapper[2577]: I1030 19:15:12.382714    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-qnm56"
Oct 30 19:15:12.383470 ip-10-0-132-82 kubenswrapper[2577]: W1030 19:15:12.383449    2577 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcdb750aa_dd7e_4e2f_8297_9028acc0f559.slice/crio-929421d442561c4ba4793828037ab0e15ea8bdf99899887708c893190983a48b WatchSource:0}: Error finding container 929421d442561c4ba4793828037ab0e15ea8bdf99899887708c893190983a48b: Status 404 returned error can't find the container with id 929421d442561c4ba4793828037ab0e15ea8bdf99899887708c893190983a48b
Oct 30 19:15:12.384016 ip-10-0-132-82 kubenswrapper[2577]: I1030 19:15:12.384000    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-5lwkx"
Oct 30 19:15:12.388579 ip-10-0-132-82 kubenswrapper[2577]: I1030 19:15:12.388560    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-2fv9z"
Oct 30 19:15:12.392039 ip-10-0-132-82 kubenswrapper[2577]: I1030 19:15:12.392019    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-qgv98"
Oct 30 19:15:12.393030 ip-10-0-132-82 kubenswrapper[2577]: W1030 19:15:12.393011    2577 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podaf36f4d4_f985_4937_a9f7_9884936f3690.slice/crio-0c376387c3362ef564a7120cabcf536b058ed3e285e143402927edceb193291f WatchSource:0}: Error finding container 0c376387c3362ef564a7120cabcf536b058ed3e285e143402927edceb193291f: Status 404 returned error can't find the container with id 0c376387c3362ef564a7120cabcf536b058ed3e285e143402927edceb193291f
Oct 30 19:15:12.394733 ip-10-0-132-82 kubenswrapper[2577]: W1030 19:15:12.394712    2577 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod08ae8ede_9a96_4a84_8ec8_8d364fd476a3.slice/crio-1eeb30806320ba780c1de4b8401be73d6acefab9ba0c36688cdd52db42e2dead WatchSource:0}: Error finding container 1eeb30806320ba780c1de4b8401be73d6acefab9ba0c36688cdd52db42e2dead: Status 404 returned error can't find the container with id 1eeb30806320ba780c1de4b8401be73d6acefab9ba0c36688cdd52db42e2dead
Oct 30 19:15:12.395926 ip-10-0-132-82 kubenswrapper[2577]: I1030 19:15:12.395909    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-4pvgg"
Oct 30 19:15:12.397623 ip-10-0-132-82 kubenswrapper[2577]: W1030 19:15:12.397604    2577 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod39bb8c16_8282_442a_841b_0056baae014d.slice/crio-191d8bc2eed4bb4fe758989cbd1deb001ad9778ca8efe7fa21d6181d5e209fa0 WatchSource:0}: Error finding container 191d8bc2eed4bb4fe758989cbd1deb001ad9778ca8efe7fa21d6181d5e209fa0: Status 404 returned error can't find the container with id 191d8bc2eed4bb4fe758989cbd1deb001ad9778ca8efe7fa21d6181d5e209fa0
Oct 30 19:15:12.408124 ip-10-0-132-82 kubenswrapper[2577]: W1030 19:15:12.408107    2577 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbab0abf9_46d4_408e_9aa0_afa7279cd9d1.slice/crio-5ef42db8ca33f0169c41836d553c454c6d250a87f4246c19237b64f69a39ef34 WatchSource:0}: Error finding container 5ef42db8ca33f0169c41836d553c454c6d250a87f4246c19237b64f69a39ef34: Status 404 returned error can't find the container with id 5ef42db8ca33f0169c41836d553c454c6d250a87f4246c19237b64f69a39ef34
Oct 30 19:15:12.671668 ip-10-0-132-82 kubenswrapper[2577]: I1030 19:15:12.671587    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/c16c0a80-c5ce-4f5d-827a-2e3eb2c3cfb8-original-pull-secret\") pod \"global-pull-secret-syncer-76rht\" (UID: \"c16c0a80-c5ce-4f5d-827a-2e3eb2c3cfb8\") " pod="kube-system/global-pull-secret-syncer-76rht"
Oct 30 19:15:12.671668 ip-10-0-132-82 kubenswrapper[2577]: I1030 19:15:12.671646    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/287ba08f-7b9f-4609-8e35-832c0879c44c-metrics-certs\") pod \"network-metrics-daemon-822gp\" (UID: \"287ba08f-7b9f-4609-8e35-832c0879c44c\") " pod="openshift-multus/network-metrics-daemon-822gp"
Oct 30 19:15:12.671882 ip-10-0-132-82 kubenswrapper[2577]: E1030 19:15:12.671769    2577 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Oct 30 19:15:12.671882 ip-10-0-132-82 kubenswrapper[2577]: E1030 19:15:12.671817    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/287ba08f-7b9f-4609-8e35-832c0879c44c-metrics-certs podName:287ba08f-7b9f-4609-8e35-832c0879c44c nodeName:}" failed. No retries permitted until 2026-10-30 19:15:13.671805132 +0000 UTC m=+4.077725560 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/287ba08f-7b9f-4609-8e35-832c0879c44c-metrics-certs") pod "network-metrics-daemon-822gp" (UID: "287ba08f-7b9f-4609-8e35-832c0879c44c") : object "openshift-multus"/"metrics-daemon-secret" not registered
Oct 30 19:15:12.672012 ip-10-0-132-82 kubenswrapper[2577]: E1030 19:15:12.671945    2577 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Oct 30 19:15:12.672012 ip-10-0-132-82 kubenswrapper[2577]: E1030 19:15:12.671992    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/c16c0a80-c5ce-4f5d-827a-2e3eb2c3cfb8-original-pull-secret podName:c16c0a80-c5ce-4f5d-827a-2e3eb2c3cfb8 nodeName:}" failed. No retries permitted until 2026-10-30 19:15:13.671983915 +0000 UTC m=+4.077904329 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/c16c0a80-c5ce-4f5d-827a-2e3eb2c3cfb8-original-pull-secret") pod "global-pull-secret-syncer-76rht" (UID: "c16c0a80-c5ce-4f5d-827a-2e3eb2c3cfb8") : object "kube-system"/"original-pull-secret" not registered
Oct 30 19:15:12.772824 ip-10-0-132-82 kubenswrapper[2577]: I1030 19:15:12.772793    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bzw4s\" (UniqueName: \"kubernetes.io/projected/4ca01b49-9819-4a90-a851-7553ad568679-kube-api-access-bzw4s\") pod \"network-check-target-c24j9\" (UID: \"4ca01b49-9819-4a90-a851-7553ad568679\") " pod="openshift-network-diagnostics/network-check-target-c24j9"
Oct 30 19:15:12.773013 ip-10-0-132-82 kubenswrapper[2577]: E1030 19:15:12.772905    2577 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Oct 30 19:15:12.773013 ip-10-0-132-82 kubenswrapper[2577]: E1030 19:15:12.772919    2577 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Oct 30 19:15:12.773013 ip-10-0-132-82 kubenswrapper[2577]: E1030 19:15:12.772928    2577 projected.go:196] Error preparing data for projected volume kube-api-access-bzw4s for pod openshift-network-diagnostics/network-check-target-c24j9: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Oct 30 19:15:12.773013 ip-10-0-132-82 kubenswrapper[2577]: E1030 19:15:12.772988    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/4ca01b49-9819-4a90-a851-7553ad568679-kube-api-access-bzw4s podName:4ca01b49-9819-4a90-a851-7553ad568679 nodeName:}" failed. No retries permitted until 2026-10-30 19:15:13.772958134 +0000 UTC m=+4.178878551 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-api-access-bzw4s" (UniqueName: "kubernetes.io/projected/4ca01b49-9819-4a90-a851-7553ad568679-kube-api-access-bzw4s") pod "network-check-target-c24j9" (UID: "4ca01b49-9819-4a90-a851-7553ad568679") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Oct 30 19:15:13.040273 ip-10-0-132-82 kubenswrapper[2577]: I1030 19:15:13.040186    2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-wnn66" event={"ID":"e0221987-8e5e-40d1-8842-c97993f08c6d","Type":"ContainerStarted","Data":"32595290e48470808e0ab2db048e03df52c1e31b97a08ef92d7a4c00da701711"}
Oct 30 19:15:13.041503 ip-10-0-132-82 kubenswrapper[2577]: I1030 19:15:13.041475    2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-g9xd6" event={"ID":"48386105-d79b-4f92-baa0-f330b9658ad0","Type":"ContainerStarted","Data":"4f8ff29fa81054ed5a37cd86a5ec74649a5651dc5a63c441321ba062767aa578"}
Oct 30 19:15:13.042733 ip-10-0-132-82 kubenswrapper[2577]: I1030 19:15:13.042704    2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-2fv9z" event={"ID":"39bb8c16-8282-442a-841b-0056baae014d","Type":"ContainerStarted","Data":"191d8bc2eed4bb4fe758989cbd1deb001ad9778ca8efe7fa21d6181d5e209fa0"}
Oct 30 19:15:13.043978 ip-10-0-132-82 kubenswrapper[2577]: I1030 19:15:13.043936    2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-5lwkx" event={"ID":"08ae8ede-9a96-4a84-8ec8-8d364fd476a3","Type":"ContainerStarted","Data":"1eeb30806320ba780c1de4b8401be73d6acefab9ba0c36688cdd52db42e2dead"}
Oct 30 19:15:13.045465 ip-10-0-132-82 kubenswrapper[2577]: I1030 19:15:13.045448    2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-qnm56" event={"ID":"af36f4d4-f985-4937-a9f7-9884936f3690","Type":"ContainerStarted","Data":"0c376387c3362ef564a7120cabcf536b058ed3e285e143402927edceb193291f"}
Oct 30 19:15:13.046649 ip-10-0-132-82 kubenswrapper[2577]: I1030 19:15:13.046625    2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-jvbw5" event={"ID":"dccdaf98-aac7-4756-b93f-22f62a286a1f","Type":"ContainerStarted","Data":"f4a575fbd0a385888d1b5e4d7a3e43290db13e5198927239e5901160a4449ad2"}
Oct 30 19:15:13.047777 ip-10-0-132-82 kubenswrapper[2577]: I1030 19:15:13.047758    2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-4pvgg" event={"ID":"bab0abf9-46d4-408e-9aa0-afa7279cd9d1","Type":"ContainerStarted","Data":"5ef42db8ca33f0169c41836d553c454c6d250a87f4246c19237b64f69a39ef34"}
Oct 30 19:15:13.049021 ip-10-0-132-82 kubenswrapper[2577]: I1030 19:15:13.048999    2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-qgv98" event={"ID":"c3780b96-955e-4728-a2d0-b7ec30d0f6f3","Type":"ContainerStarted","Data":"71870ffe35cbe4432b34eca4162a3fe61340fcd0c1229a11bee6c775daae713a"}
Oct 30 19:15:13.050048 ip-10-0-132-82 kubenswrapper[2577]: I1030 19:15:13.050029    2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-kgfvh" event={"ID":"cdb750aa-dd7e-4e2f-8297-9028acc0f559","Type":"ContainerStarted","Data":"929421d442561c4ba4793828037ab0e15ea8bdf99899887708c893190983a48b"}
Oct 30 19:15:13.155981 ip-10-0-132-82 kubenswrapper[2577]: I1030 19:15:13.155924    2577 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-10-29 19:10:12 +0000 UTC" deadline="2028-06-22 00:08:46.429734708 +0000 UTC"
Oct 30 19:15:13.155981 ip-10-0-132-82 kubenswrapper[2577]: I1030 19:15:13.155980    2577 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="14404h53m33.273775717s"
Oct 30 19:15:13.679908 ip-10-0-132-82 kubenswrapper[2577]: I1030 19:15:13.679475    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/c16c0a80-c5ce-4f5d-827a-2e3eb2c3cfb8-original-pull-secret\") pod \"global-pull-secret-syncer-76rht\" (UID: \"c16c0a80-c5ce-4f5d-827a-2e3eb2c3cfb8\") " pod="kube-system/global-pull-secret-syncer-76rht"
Oct 30 19:15:13.679908 ip-10-0-132-82 kubenswrapper[2577]: I1030 19:15:13.679535    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/287ba08f-7b9f-4609-8e35-832c0879c44c-metrics-certs\") pod \"network-metrics-daemon-822gp\" (UID: \"287ba08f-7b9f-4609-8e35-832c0879c44c\") " pod="openshift-multus/network-metrics-daemon-822gp"
Oct 30 19:15:13.679908 ip-10-0-132-82 kubenswrapper[2577]: E1030 19:15:13.679709    2577 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Oct 30 19:15:13.679908 ip-10-0-132-82 kubenswrapper[2577]: E1030 19:15:13.679781    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/287ba08f-7b9f-4609-8e35-832c0879c44c-metrics-certs podName:287ba08f-7b9f-4609-8e35-832c0879c44c nodeName:}" failed. No retries permitted until 2026-10-30 19:15:15.679762201 +0000 UTC m=+6.085682621 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/287ba08f-7b9f-4609-8e35-832c0879c44c-metrics-certs") pod "network-metrics-daemon-822gp" (UID: "287ba08f-7b9f-4609-8e35-832c0879c44c") : object "openshift-multus"/"metrics-daemon-secret" not registered
Oct 30 19:15:13.679908 ip-10-0-132-82 kubenswrapper[2577]: E1030 19:15:13.679844    2577 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Oct 30 19:15:13.679908 ip-10-0-132-82 kubenswrapper[2577]: E1030 19:15:13.679876    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/c16c0a80-c5ce-4f5d-827a-2e3eb2c3cfb8-original-pull-secret podName:c16c0a80-c5ce-4f5d-827a-2e3eb2c3cfb8 nodeName:}" failed. No retries permitted until 2026-10-30 19:15:15.679865304 +0000 UTC m=+6.085785721 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/c16c0a80-c5ce-4f5d-827a-2e3eb2c3cfb8-original-pull-secret") pod "global-pull-secret-syncer-76rht" (UID: "c16c0a80-c5ce-4f5d-827a-2e3eb2c3cfb8") : object "kube-system"/"original-pull-secret" not registered
Oct 30 19:15:13.780167 ip-10-0-132-82 kubenswrapper[2577]: I1030 19:15:13.780132    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bzw4s\" (UniqueName: \"kubernetes.io/projected/4ca01b49-9819-4a90-a851-7553ad568679-kube-api-access-bzw4s\") pod \"network-check-target-c24j9\" (UID: \"4ca01b49-9819-4a90-a851-7553ad568679\") " pod="openshift-network-diagnostics/network-check-target-c24j9"
Oct 30 19:15:13.780301 ip-10-0-132-82 kubenswrapper[2577]: E1030 19:15:13.780286    2577 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Oct 30 19:15:13.780334 ip-10-0-132-82 kubenswrapper[2577]: E1030 19:15:13.780304    2577 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Oct 30 19:15:13.780334 ip-10-0-132-82 kubenswrapper[2577]: E1030 19:15:13.780316    2577 projected.go:196] Error preparing data for projected volume kube-api-access-bzw4s for pod openshift-network-diagnostics/network-check-target-c24j9: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Oct 30 19:15:13.780409 ip-10-0-132-82 kubenswrapper[2577]: E1030 19:15:13.780370    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/4ca01b49-9819-4a90-a851-7553ad568679-kube-api-access-bzw4s podName:4ca01b49-9819-4a90-a851-7553ad568679 nodeName:}" failed. No retries permitted until 2026-10-30 19:15:15.780353608 +0000 UTC m=+6.186274027 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-api-access-bzw4s" (UniqueName: "kubernetes.io/projected/4ca01b49-9819-4a90-a851-7553ad568679-kube-api-access-bzw4s") pod "network-check-target-c24j9" (UID: "4ca01b49-9819-4a90-a851-7553ad568679") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Oct 30 19:15:14.034765 ip-10-0-132-82 kubenswrapper[2577]: I1030 19:15:14.034140    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-822gp"
Oct 30 19:15:14.034765 ip-10-0-132-82 kubenswrapper[2577]: E1030 19:15:14.034300    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-822gp" podUID="287ba08f-7b9f-4609-8e35-832c0879c44c"
Oct 30 19:15:14.034765 ip-10-0-132-82 kubenswrapper[2577]: I1030 19:15:14.034408    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-76rht"
Oct 30 19:15:14.034765 ip-10-0-132-82 kubenswrapper[2577]: E1030 19:15:14.034486    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-76rht" podUID="c16c0a80-c5ce-4f5d-827a-2e3eb2c3cfb8"
Oct 30 19:15:14.034765 ip-10-0-132-82 kubenswrapper[2577]: I1030 19:15:14.034571    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-c24j9"
Oct 30 19:15:14.034765 ip-10-0-132-82 kubenswrapper[2577]: E1030 19:15:14.034653    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-c24j9" podUID="4ca01b49-9819-4a90-a851-7553ad568679"
Oct 30 19:15:14.156821 ip-10-0-132-82 kubenswrapper[2577]: I1030 19:15:14.156770    2577 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-10-29 19:10:12 +0000 UTC" deadline="2028-06-12 02:16:23.265424766 +0000 UTC"
Oct 30 19:15:14.156821 ip-10-0-132-82 kubenswrapper[2577]: I1030 19:15:14.156798    2577 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="14167h1m9.10863024s"
Oct 30 19:15:15.700151 ip-10-0-132-82 kubenswrapper[2577]: I1030 19:15:15.699442    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/c16c0a80-c5ce-4f5d-827a-2e3eb2c3cfb8-original-pull-secret\") pod \"global-pull-secret-syncer-76rht\" (UID: \"c16c0a80-c5ce-4f5d-827a-2e3eb2c3cfb8\") " pod="kube-system/global-pull-secret-syncer-76rht"
Oct 30 19:15:15.700151 ip-10-0-132-82 kubenswrapper[2577]: I1030 19:15:15.699701    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/287ba08f-7b9f-4609-8e35-832c0879c44c-metrics-certs\") pod \"network-metrics-daemon-822gp\" (UID: \"287ba08f-7b9f-4609-8e35-832c0879c44c\") " pod="openshift-multus/network-metrics-daemon-822gp"
Oct 30 19:15:15.700151 ip-10-0-132-82 kubenswrapper[2577]: E1030 19:15:15.699850    2577 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Oct 30 19:15:15.700151 ip-10-0-132-82 kubenswrapper[2577]: E1030 19:15:15.699912    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/287ba08f-7b9f-4609-8e35-832c0879c44c-metrics-certs podName:287ba08f-7b9f-4609-8e35-832c0879c44c nodeName:}" failed. No retries permitted until 2026-10-30 19:15:19.69989227 +0000 UTC m=+10.105812699 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/287ba08f-7b9f-4609-8e35-832c0879c44c-metrics-certs") pod "network-metrics-daemon-822gp" (UID: "287ba08f-7b9f-4609-8e35-832c0879c44c") : object "openshift-multus"/"metrics-daemon-secret" not registered
Oct 30 19:15:15.700151 ip-10-0-132-82 kubenswrapper[2577]: E1030 19:15:15.700006    2577 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Oct 30 19:15:15.700151 ip-10-0-132-82 kubenswrapper[2577]: E1030 19:15:15.700040    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/c16c0a80-c5ce-4f5d-827a-2e3eb2c3cfb8-original-pull-secret podName:c16c0a80-c5ce-4f5d-827a-2e3eb2c3cfb8 nodeName:}" failed. No retries permitted until 2026-10-30 19:15:19.700028731 +0000 UTC m=+10.105949159 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/c16c0a80-c5ce-4f5d-827a-2e3eb2c3cfb8-original-pull-secret") pod "global-pull-secret-syncer-76rht" (UID: "c16c0a80-c5ce-4f5d-827a-2e3eb2c3cfb8") : object "kube-system"/"original-pull-secret" not registered
Oct 30 19:15:15.800566 ip-10-0-132-82 kubenswrapper[2577]: I1030 19:15:15.800531    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bzw4s\" (UniqueName: \"kubernetes.io/projected/4ca01b49-9819-4a90-a851-7553ad568679-kube-api-access-bzw4s\") pod \"network-check-target-c24j9\" (UID: \"4ca01b49-9819-4a90-a851-7553ad568679\") " pod="openshift-network-diagnostics/network-check-target-c24j9"
Oct 30 19:15:15.800744 ip-10-0-132-82 kubenswrapper[2577]: E1030 19:15:15.800703    2577 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Oct 30 19:15:15.800744 ip-10-0-132-82 kubenswrapper[2577]: E1030 19:15:15.800724    2577 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Oct 30 19:15:15.800744 ip-10-0-132-82 kubenswrapper[2577]: E1030 19:15:15.800736    2577 projected.go:196] Error preparing data for projected volume kube-api-access-bzw4s for pod openshift-network-diagnostics/network-check-target-c24j9: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Oct 30 19:15:15.800888 ip-10-0-132-82 kubenswrapper[2577]: E1030 19:15:15.800790    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/4ca01b49-9819-4a90-a851-7553ad568679-kube-api-access-bzw4s podName:4ca01b49-9819-4a90-a851-7553ad568679 nodeName:}" failed. No retries permitted until 2026-10-30 19:15:19.80077274 +0000 UTC m=+10.206693181 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-api-access-bzw4s" (UniqueName: "kubernetes.io/projected/4ca01b49-9819-4a90-a851-7553ad568679-kube-api-access-bzw4s") pod "network-check-target-c24j9" (UID: "4ca01b49-9819-4a90-a851-7553ad568679") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Oct 30 19:15:16.033613 ip-10-0-132-82 kubenswrapper[2577]: I1030 19:15:16.033589    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-c24j9"
Oct 30 19:15:16.033735 ip-10-0-132-82 kubenswrapper[2577]: E1030 19:15:16.033697    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-c24j9" podUID="4ca01b49-9819-4a90-a851-7553ad568679"
Oct 30 19:15:16.033796 ip-10-0-132-82 kubenswrapper[2577]: I1030 19:15:16.033766    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-76rht"
Oct 30 19:15:16.033847 ip-10-0-132-82 kubenswrapper[2577]: E1030 19:15:16.033821    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-76rht" podUID="c16c0a80-c5ce-4f5d-827a-2e3eb2c3cfb8"
Oct 30 19:15:16.033898 ip-10-0-132-82 kubenswrapper[2577]: I1030 19:15:16.033868    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-822gp"
Oct 30 19:15:16.033950 ip-10-0-132-82 kubenswrapper[2577]: E1030 19:15:16.033922    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-822gp" podUID="287ba08f-7b9f-4609-8e35-832c0879c44c"
Oct 30 19:15:16.077048 ip-10-0-132-82 kubenswrapper[2577]: I1030 19:15:16.076287    2577 generic.go:356] "Generic (PLEG): container finished" podID="16007dbb15fdd01822b4d377fac4e8af" containerID="c53bce13b8336e42910636a7d114f4c7f7cfb933eb77fede84f6c63870ca7b1c" exitCode=0
Oct 30 19:15:16.077048 ip-10-0-132-82 kubenswrapper[2577]: I1030 19:15:16.076375    2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-132-82.us-west-1.compute.internal" event={"ID":"16007dbb15fdd01822b4d377fac4e8af","Type":"ContainerDied","Data":"c53bce13b8336e42910636a7d114f4c7f7cfb933eb77fede84f6c63870ca7b1c"}
Oct 30 19:15:16.099887 ip-10-0-132-82 kubenswrapper[2577]: I1030 19:15:16.099854    2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-132-82.us-west-1.compute.internal" event={"ID":"f9af2f5b3d8deb871a93a1e3f0011dc6","Type":"ContainerStarted","Data":"a2b2739a883d4be5a965500563e1cc61e2cd3a03a44224b6ec8478014abb02aa"}
Oct 30 19:15:16.307530 ip-10-0-132-82 kubenswrapper[2577]: I1030 19:15:16.307370    2577 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kube-system/kube-apiserver-proxy-ip-10-0-132-82.us-west-1.compute.internal" podStartSLOduration=5.307350752 podStartE2EDuration="5.307350752s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-30 19:15:11 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:15:16.306380383 +0000 UTC m=+6.712300817" watchObservedRunningTime="2026-10-30 19:15:16.307350752 +0000 UTC m=+6.713271189"
Oct 30 19:15:17.112803 ip-10-0-132-82 kubenswrapper[2577]: I1030 19:15:17.112078    2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-132-82.us-west-1.compute.internal" event={"ID":"16007dbb15fdd01822b4d377fac4e8af","Type":"ContainerStarted","Data":"068eb92c94d0b342a135d59cbc174e4b303f7d17a164613a22996eeb2e047965"}
Oct 30 19:15:17.118799 ip-10-0-132-82 kubenswrapper[2577]: I1030 19:15:17.118107    2577 generic.go:356] "Generic (PLEG): container finished" podID="bab0abf9-46d4-408e-9aa0-afa7279cd9d1" containerID="83463f495ed150855f54c4a9c410dc6b8aea0c70c7017e236ab3def9c6213135" exitCode=0
Oct 30 19:15:17.118799 ip-10-0-132-82 kubenswrapper[2577]: I1030 19:15:17.118281    2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-4pvgg" event={"ID":"bab0abf9-46d4-408e-9aa0-afa7279cd9d1","Type":"ContainerDied","Data":"83463f495ed150855f54c4a9c410dc6b8aea0c70c7017e236ab3def9c6213135"}
Oct 30 19:15:17.229752 ip-10-0-132-82 kubenswrapper[2577]: I1030 19:15:17.229654    2577 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-132-82.us-west-1.compute.internal" podStartSLOduration=6.229632765 podStartE2EDuration="6.229632765s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-30 19:15:11 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:15:17.227608394 +0000 UTC m=+7.633528825" watchObservedRunningTime="2026-10-30 19:15:17.229632765 +0000 UTC m=+7.635553202"
Oct 30 19:15:18.032120 ip-10-0-132-82 kubenswrapper[2577]: I1030 19:15:18.032088    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-c24j9"
Oct 30 19:15:18.032287 ip-10-0-132-82 kubenswrapper[2577]: E1030 19:15:18.032210    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-c24j9" podUID="4ca01b49-9819-4a90-a851-7553ad568679"
Oct 30 19:15:18.032287 ip-10-0-132-82 kubenswrapper[2577]: I1030 19:15:18.032282    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-76rht"
Oct 30 19:15:18.032425 ip-10-0-132-82 kubenswrapper[2577]: E1030 19:15:18.032352    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-76rht" podUID="c16c0a80-c5ce-4f5d-827a-2e3eb2c3cfb8"
Oct 30 19:15:18.032498 ip-10-0-132-82 kubenswrapper[2577]: I1030 19:15:18.032481    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-822gp"
Oct 30 19:15:18.032633 ip-10-0-132-82 kubenswrapper[2577]: E1030 19:15:18.032577    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-822gp" podUID="287ba08f-7b9f-4609-8e35-832c0879c44c"
Oct 30 19:15:18.123589 ip-10-0-132-82 kubenswrapper[2577]: I1030 19:15:18.123555    2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-qnm56" event={"ID":"af36f4d4-f985-4937-a9f7-9884936f3690","Type":"ContainerStarted","Data":"1358b8b2f436fe16ee44332fe9d8115c277409a22eb52520989f9092d3345228"}
Oct 30 19:15:18.240461 ip-10-0-132-82 kubenswrapper[2577]: I1030 19:15:18.240087    2577 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kube-system/konnectivity-agent-qnm56" podStartSLOduration=3.648927344 podStartE2EDuration="8.240067682s" totalImagesPullingTime="4.591140338s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:15:10 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:15:18.238615282 +0000 UTC m=+8.644535719" watchObservedRunningTime="2026-10-30 19:15:18.240067682 +0000 UTC m=+8.645988120"
Oct 30 19:15:19.129652 ip-10-0-132-82 kubenswrapper[2577]: I1030 19:15:19.129611    2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-qgv98" event={"ID":"c3780b96-955e-4728-a2d0-b7ec30d0f6f3","Type":"ContainerStarted","Data":"33bccc1c8ed26ed56978d6df32da1c59f4ba307e7543ee353fe4dbb18163a334"}
Oct 30 19:15:19.134665 ip-10-0-132-82 kubenswrapper[2577]: I1030 19:15:19.134635    2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-kgfvh" event={"ID":"cdb750aa-dd7e-4e2f-8297-9028acc0f559","Type":"ContainerStarted","Data":"77da65fa40107a3aaba9c59c347fce9940b2da5eb2d73cb575c2194672eccb1f"}
Oct 30 19:15:19.733555 ip-10-0-132-82 kubenswrapper[2577]: I1030 19:15:19.733494    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/c16c0a80-c5ce-4f5d-827a-2e3eb2c3cfb8-original-pull-secret\") pod \"global-pull-secret-syncer-76rht\" (UID: \"c16c0a80-c5ce-4f5d-827a-2e3eb2c3cfb8\") " pod="kube-system/global-pull-secret-syncer-76rht"
Oct 30 19:15:19.733718 ip-10-0-132-82 kubenswrapper[2577]: I1030 19:15:19.733586    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/287ba08f-7b9f-4609-8e35-832c0879c44c-metrics-certs\") pod \"network-metrics-daemon-822gp\" (UID: \"287ba08f-7b9f-4609-8e35-832c0879c44c\") " pod="openshift-multus/network-metrics-daemon-822gp"
Oct 30 19:15:19.733793 ip-10-0-132-82 kubenswrapper[2577]: E1030 19:15:19.733747    2577 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Oct 30 19:15:19.733847 ip-10-0-132-82 kubenswrapper[2577]: E1030 19:15:19.733816    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/287ba08f-7b9f-4609-8e35-832c0879c44c-metrics-certs podName:287ba08f-7b9f-4609-8e35-832c0879c44c nodeName:}" failed. No retries permitted until 2026-10-30 19:15:27.733798458 +0000 UTC m=+18.139718895 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/287ba08f-7b9f-4609-8e35-832c0879c44c-metrics-certs") pod "network-metrics-daemon-822gp" (UID: "287ba08f-7b9f-4609-8e35-832c0879c44c") : object "openshift-multus"/"metrics-daemon-secret" not registered
Oct 30 19:15:19.734239 ip-10-0-132-82 kubenswrapper[2577]: E1030 19:15:19.734215    2577 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Oct 30 19:15:19.734351 ip-10-0-132-82 kubenswrapper[2577]: E1030 19:15:19.734276    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/c16c0a80-c5ce-4f5d-827a-2e3eb2c3cfb8-original-pull-secret podName:c16c0a80-c5ce-4f5d-827a-2e3eb2c3cfb8 nodeName:}" failed. No retries permitted until 2026-10-30 19:15:27.734260351 +0000 UTC m=+18.140180784 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/c16c0a80-c5ce-4f5d-827a-2e3eb2c3cfb8-original-pull-secret") pod "global-pull-secret-syncer-76rht" (UID: "c16c0a80-c5ce-4f5d-827a-2e3eb2c3cfb8") : object "kube-system"/"original-pull-secret" not registered
Oct 30 19:15:19.834317 ip-10-0-132-82 kubenswrapper[2577]: I1030 19:15:19.834277    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bzw4s\" (UniqueName: \"kubernetes.io/projected/4ca01b49-9819-4a90-a851-7553ad568679-kube-api-access-bzw4s\") pod \"network-check-target-c24j9\" (UID: \"4ca01b49-9819-4a90-a851-7553ad568679\") " pod="openshift-network-diagnostics/network-check-target-c24j9"
Oct 30 19:15:19.834550 ip-10-0-132-82 kubenswrapper[2577]: E1030 19:15:19.834526    2577 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Oct 30 19:15:19.834632 ip-10-0-132-82 kubenswrapper[2577]: E1030 19:15:19.834558    2577 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Oct 30 19:15:19.834632 ip-10-0-132-82 kubenswrapper[2577]: E1030 19:15:19.834576    2577 projected.go:196] Error preparing data for projected volume kube-api-access-bzw4s for pod openshift-network-diagnostics/network-check-target-c24j9: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Oct 30 19:15:19.834747 ip-10-0-132-82 kubenswrapper[2577]: E1030 19:15:19.834654    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/4ca01b49-9819-4a90-a851-7553ad568679-kube-api-access-bzw4s podName:4ca01b49-9819-4a90-a851-7553ad568679 nodeName:}" failed. No retries permitted until 2026-10-30 19:15:27.834624493 +0000 UTC m=+18.240544921 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "kube-api-access-bzw4s" (UniqueName: "kubernetes.io/projected/4ca01b49-9819-4a90-a851-7553ad568679-kube-api-access-bzw4s") pod "network-check-target-c24j9" (UID: "4ca01b49-9819-4a90-a851-7553ad568679") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Oct 30 19:15:20.042940 ip-10-0-132-82 kubenswrapper[2577]: I1030 19:15:20.042868    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-c24j9"
Oct 30 19:15:20.042940 ip-10-0-132-82 kubenswrapper[2577]: I1030 19:15:20.042895    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-822gp"
Oct 30 19:15:20.043169 ip-10-0-132-82 kubenswrapper[2577]: E1030 19:15:20.043064    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-c24j9" podUID="4ca01b49-9819-4a90-a851-7553ad568679"
Oct 30 19:15:20.043627 ip-10-0-132-82 kubenswrapper[2577]: E1030 19:15:20.043599    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-822gp" podUID="287ba08f-7b9f-4609-8e35-832c0879c44c"
Oct 30 19:15:20.043774 ip-10-0-132-82 kubenswrapper[2577]: I1030 19:15:20.043663    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-76rht"
Oct 30 19:15:20.043831 ip-10-0-132-82 kubenswrapper[2577]: E1030 19:15:20.043778    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-76rht" podUID="c16c0a80-c5ce-4f5d-827a-2e3eb2c3cfb8"
Oct 30 19:15:20.580958 ip-10-0-132-82 kubenswrapper[2577]: I1030 19:15:20.580854    2577 plugin_watcher.go:201] "Adding socket path or updating timestamp to desired state cache" path="/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock"
Oct 30 19:15:20.716872 ip-10-0-132-82 kubenswrapper[2577]: I1030 19:15:20.716819    2577 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kube-system/konnectivity-agent-qnm56"
Oct 30 19:15:20.717406 ip-10-0-132-82 kubenswrapper[2577]: I1030 19:15:20.717382    2577 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="kube-system/konnectivity-agent-qnm56"
Oct 30 19:15:20.830638 ip-10-0-132-82 kubenswrapper[2577]: I1030 19:15:20.830578    2577 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-image-registry/node-ca-kgfvh" podStartSLOduration=5.155433426 podStartE2EDuration="10.830560805s" totalImagesPullingTime="5.675127379s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-30 19:15:10 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:15:19.250110912 +0000 UTC m=+9.656031351" watchObservedRunningTime="2026-10-30 19:15:20.830560805 +0000 UTC m=+11.236481240"
Oct 30 19:15:20.981298 ip-10-0-132-82 kubenswrapper[2577]: I1030 19:15:20.981193    2577 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kube-system/konnectivity-agent-qnm56"
Oct 30 19:15:20.981856 ip-10-0-132-82 kubenswrapper[2577]: I1030 19:15:20.981687    2577 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="kube-system/konnectivity-agent-qnm56"
Oct 30 19:15:21.026417 ip-10-0-132-82 kubenswrapper[2577]: I1030 19:15:21.026310    2577 reconciler.go:167] "OperationExecutor.RegisterPlugin started" plugin={"SocketPath":"/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock","Timestamp":"2026-10-30T19:15:20.580869936Z","UUID":"ad087560-83e7-4687-97bf-7d1f77723ef7","Handler":null,"Name":"","Endpoint":""}
Oct 30 19:15:21.027936 ip-10-0-132-82 kubenswrapper[2577]: I1030 19:15:21.027914    2577 csi_plugin.go:106] kubernetes.io/csi: Trying to validate a new CSI Driver with name: ebs.csi.aws.com endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock versions: 1.0.0
Oct 30 19:15:21.028061 ip-10-0-132-82 kubenswrapper[2577]: I1030 19:15:21.027945    2577 csi_plugin.go:119] kubernetes.io/csi: Register new plugin with name: ebs.csi.aws.com at endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock
Oct 30 19:15:21.142248 ip-10-0-132-82 kubenswrapper[2577]: I1030 19:15:21.142216    2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-qgv98" event={"ID":"c3780b96-955e-4728-a2d0-b7ec30d0f6f3","Type":"ContainerStarted","Data":"371c2edfc3d37959de17d7352ed339f68f117e5a98981fcd80927c372e3bb17a"}
Oct 30 19:15:21.144499 ip-10-0-132-82 kubenswrapper[2577]: I1030 19:15:21.144470    2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-g9xd6" event={"ID":"48386105-d79b-4f92-baa0-f330b9658ad0","Type":"ContainerStarted","Data":"b15e3c3d568502ebb289a975049d09105e206e4a41cc9b2318b2a771864cf3e2"}
Oct 30 19:15:21.146375 ip-10-0-132-82 kubenswrapper[2577]: I1030 19:15:21.146346    2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-jvbw5" event={"ID":"dccdaf98-aac7-4756-b93f-22f62a286a1f","Type":"ContainerStarted","Data":"a9e3084b2360eff1bbae46af0e63ece000515967cff03686859523cee0e5dca0"}
Oct 30 19:15:21.260134 ip-10-0-132-82 kubenswrapper[2577]: I1030 19:15:21.260056    2577 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-dns/node-resolver-g9xd6" podStartSLOduration=3.197779705 podStartE2EDuration="11.260038026s" totalImagesPullingTime="8.062258321s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-30 19:15:10 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:15:21.259417244 +0000 UTC m=+11.665337682" watchObservedRunningTime="2026-10-30 19:15:21.260038026 +0000 UTC m=+11.665958462"
Oct 30 19:15:21.376032 ip-10-0-132-82 kubenswrapper[2577]: I1030 19:15:21.375957    2577 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-multus/multus-jvbw5" podStartSLOduration=3.314928534 podStartE2EDuration="11.375945176s" totalImagesPullingTime="8.061016642s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-30 19:15:10 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:15:21.375564877 +0000 UTC m=+11.781485316" watchObservedRunningTime="2026-10-30 19:15:21.375945176 +0000 UTC m=+11.781865612"
Oct 30 19:15:22.031603 ip-10-0-132-82 kubenswrapper[2577]: I1030 19:15:22.031386    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-c24j9"
Oct 30 19:15:22.032505 ip-10-0-132-82 kubenswrapper[2577]: E1030 19:15:22.032480    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-c24j9" podUID="4ca01b49-9819-4a90-a851-7553ad568679"
Oct 30 19:15:22.032505 ip-10-0-132-82 kubenswrapper[2577]: I1030 19:15:22.032370    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-76rht"
Oct 30 19:15:22.032658 ip-10-0-132-82 kubenswrapper[2577]: E1030 19:15:22.032579    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-76rht" podUID="c16c0a80-c5ce-4f5d-827a-2e3eb2c3cfb8"
Oct 30 19:15:22.032658 ip-10-0-132-82 kubenswrapper[2577]: I1030 19:15:22.032329    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-822gp"
Oct 30 19:15:22.032752 ip-10-0-132-82 kubenswrapper[2577]: E1030 19:15:22.032698    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-822gp" podUID="287ba08f-7b9f-4609-8e35-832c0879c44c"
Oct 30 19:15:22.151653 ip-10-0-132-82 kubenswrapper[2577]: I1030 19:15:22.151336    2577 generic.go:356] "Generic (PLEG): container finished" podID="bab0abf9-46d4-408e-9aa0-afa7279cd9d1" containerID="276e24da2fbd424c8f37607af4179597ed72122d871fd1a0b1d86fa4886c553c" exitCode=0
Oct 30 19:15:22.151653 ip-10-0-132-82 kubenswrapper[2577]: I1030 19:15:22.151414    2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-4pvgg" event={"ID":"bab0abf9-46d4-408e-9aa0-afa7279cd9d1","Type":"ContainerDied","Data":"276e24da2fbd424c8f37607af4179597ed72122d871fd1a0b1d86fa4886c553c"}
Oct 30 19:15:22.154981 ip-10-0-132-82 kubenswrapper[2577]: I1030 19:15:22.154940    2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-qgv98" event={"ID":"c3780b96-955e-4728-a2d0-b7ec30d0f6f3","Type":"ContainerStarted","Data":"9e023ded9a95701698edcdf716d6cb83c1a0217afd010cbe4c47730c0f4178d0"}
Oct 30 19:15:22.425972 ip-10-0-132-82 kubenswrapper[2577]: I1030 19:15:22.425916    2577 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-qgv98" podStartSLOduration=3.22932883 podStartE2EDuration="12.425898721s" totalImagesPullingTime="9.196569891s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-30 19:15:10 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:15:22.425181611 +0000 UTC m=+12.831102047" watchObservedRunningTime="2026-10-30 19:15:22.425898721 +0000 UTC m=+12.831819157"
Oct 30 19:15:23.159321 ip-10-0-132-82 kubenswrapper[2577]: I1030 19:15:23.158870    2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-wnn66" event={"ID":"e0221987-8e5e-40d1-8842-c97993f08c6d","Type":"ContainerStarted","Data":"1c041c943b33bfc60439385d2ae82f25b6737e3876af83574d7ba8232a02259f"}
Oct 30 19:15:23.160994 ip-10-0-132-82 kubenswrapper[2577]: I1030 19:15:23.160659    2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-2fv9z" event={"ID":"39bb8c16-8282-442a-841b-0056baae014d","Type":"ContainerStarted","Data":"b4c64ea567f4e2577b435c1da1201da575fb36941932c49fdd34a912437f22e7"}
Oct 30 19:15:23.272907 ip-10-0-132-82 kubenswrapper[2577]: I1030 19:15:23.272815    2577 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-network-operator/iptables-alerter-wnn66" podStartSLOduration=4.999468705 podStartE2EDuration="13.272801865s" totalImagesPullingTime="8.27333316s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-30 19:15:10 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:15:23.272674738 +0000 UTC m=+13.678595174" watchObservedRunningTime="2026-10-30 19:15:23.272801865 +0000 UTC m=+13.678722301"
Oct 30 19:15:23.388074 ip-10-0-132-82 kubenswrapper[2577]: I1030 19:15:23.388028    2577 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-cluster-node-tuning-operator/tuned-2fv9z" podStartSLOduration=2.798257454 podStartE2EDuration="13.388012533s" totalImagesPullingTime="10.589755079s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-30 19:15:10 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:15:23.388002225 +0000 UTC m=+13.793922662" watchObservedRunningTime="2026-10-30 19:15:23.388012533 +0000 UTC m=+13.793932969"
Oct 30 19:15:24.031409 ip-10-0-132-82 kubenswrapper[2577]: I1030 19:15:24.031373    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-822gp"
Oct 30 19:15:24.031555 ip-10-0-132-82 kubenswrapper[2577]: I1030 19:15:24.031386    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-c24j9"
Oct 30 19:15:24.031555 ip-10-0-132-82 kubenswrapper[2577]: E1030 19:15:24.031524    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-822gp" podUID="287ba08f-7b9f-4609-8e35-832c0879c44c"
Oct 30 19:15:24.031648 ip-10-0-132-82 kubenswrapper[2577]: I1030 19:15:24.031388    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-76rht"
Oct 30 19:15:24.031648 ip-10-0-132-82 kubenswrapper[2577]: E1030 19:15:24.031584    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-c24j9" podUID="4ca01b49-9819-4a90-a851-7553ad568679"
Oct 30 19:15:24.031739 ip-10-0-132-82 kubenswrapper[2577]: E1030 19:15:24.031659    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-76rht" podUID="c16c0a80-c5ce-4f5d-827a-2e3eb2c3cfb8"
Oct 30 19:15:24.167658 ip-10-0-132-82 kubenswrapper[2577]: I1030 19:15:24.167624    2577 generic.go:356] "Generic (PLEG): container finished" podID="bab0abf9-46d4-408e-9aa0-afa7279cd9d1" containerID="d14b9cdbefbe3ac90be1fa17d386d32be0c538df487648aef595307389cb423b" exitCode=0
Oct 30 19:15:24.168261 ip-10-0-132-82 kubenswrapper[2577]: I1030 19:15:24.167698    2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-4pvgg" event={"ID":"bab0abf9-46d4-408e-9aa0-afa7279cd9d1","Type":"ContainerDied","Data":"d14b9cdbefbe3ac90be1fa17d386d32be0c538df487648aef595307389cb423b"}
Oct 30 19:15:26.033464 ip-10-0-132-82 kubenswrapper[2577]: I1030 19:15:26.031804    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-76rht"
Oct 30 19:15:26.033464 ip-10-0-132-82 kubenswrapper[2577]: I1030 19:15:26.031874    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-822gp"
Oct 30 19:15:26.033464 ip-10-0-132-82 kubenswrapper[2577]: I1030 19:15:26.031889    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-c24j9"
Oct 30 19:15:26.033464 ip-10-0-132-82 kubenswrapper[2577]: E1030 19:15:26.032022    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-822gp" podUID="287ba08f-7b9f-4609-8e35-832c0879c44c"
Oct 30 19:15:26.033464 ip-10-0-132-82 kubenswrapper[2577]: E1030 19:15:26.032083    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-c24j9" podUID="4ca01b49-9819-4a90-a851-7553ad568679"
Oct 30 19:15:26.033464 ip-10-0-132-82 kubenswrapper[2577]: E1030 19:15:26.032141    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-76rht" podUID="c16c0a80-c5ce-4f5d-827a-2e3eb2c3cfb8"
Oct 30 19:15:26.175548 ip-10-0-132-82 kubenswrapper[2577]: I1030 19:15:26.175514    2577 generic.go:356] "Generic (PLEG): container finished" podID="bab0abf9-46d4-408e-9aa0-afa7279cd9d1" containerID="4109167b89ef6a0b9bac0a265642b2ea08ed42d0f551fefeb2a79fea2663c221" exitCode=0
Oct 30 19:15:26.175726 ip-10-0-132-82 kubenswrapper[2577]: I1030 19:15:26.175565    2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-4pvgg" event={"ID":"bab0abf9-46d4-408e-9aa0-afa7279cd9d1","Type":"ContainerDied","Data":"4109167b89ef6a0b9bac0a265642b2ea08ed42d0f551fefeb2a79fea2663c221"}
Oct 30 19:15:27.808577 ip-10-0-132-82 kubenswrapper[2577]: I1030 19:15:27.808367    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/c16c0a80-c5ce-4f5d-827a-2e3eb2c3cfb8-original-pull-secret\") pod \"global-pull-secret-syncer-76rht\" (UID: \"c16c0a80-c5ce-4f5d-827a-2e3eb2c3cfb8\") " pod="kube-system/global-pull-secret-syncer-76rht"
Oct 30 19:15:27.809110 ip-10-0-132-82 kubenswrapper[2577]: I1030 19:15:27.808616    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/287ba08f-7b9f-4609-8e35-832c0879c44c-metrics-certs\") pod \"network-metrics-daemon-822gp\" (UID: \"287ba08f-7b9f-4609-8e35-832c0879c44c\") " pod="openshift-multus/network-metrics-daemon-822gp"
Oct 30 19:15:27.809110 ip-10-0-132-82 kubenswrapper[2577]: E1030 19:15:27.808528    2577 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Oct 30 19:15:27.809110 ip-10-0-132-82 kubenswrapper[2577]: E1030 19:15:27.808712    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/c16c0a80-c5ce-4f5d-827a-2e3eb2c3cfb8-original-pull-secret podName:c16c0a80-c5ce-4f5d-827a-2e3eb2c3cfb8 nodeName:}" failed. No retries permitted until 2026-10-30 19:15:43.808691468 +0000 UTC m=+34.214611899 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/c16c0a80-c5ce-4f5d-827a-2e3eb2c3cfb8-original-pull-secret") pod "global-pull-secret-syncer-76rht" (UID: "c16c0a80-c5ce-4f5d-827a-2e3eb2c3cfb8") : object "kube-system"/"original-pull-secret" not registered
Oct 30 19:15:27.809110 ip-10-0-132-82 kubenswrapper[2577]: E1030 19:15:27.808757    2577 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Oct 30 19:15:27.809110 ip-10-0-132-82 kubenswrapper[2577]: E1030 19:15:27.808799    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/287ba08f-7b9f-4609-8e35-832c0879c44c-metrics-certs podName:287ba08f-7b9f-4609-8e35-832c0879c44c nodeName:}" failed. No retries permitted until 2026-10-30 19:15:43.808786851 +0000 UTC m=+34.214707294 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/287ba08f-7b9f-4609-8e35-832c0879c44c-metrics-certs") pod "network-metrics-daemon-822gp" (UID: "287ba08f-7b9f-4609-8e35-832c0879c44c") : object "openshift-multus"/"metrics-daemon-secret" not registered
Oct 30 19:15:27.909459 ip-10-0-132-82 kubenswrapper[2577]: I1030 19:15:27.909429    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bzw4s\" (UniqueName: \"kubernetes.io/projected/4ca01b49-9819-4a90-a851-7553ad568679-kube-api-access-bzw4s\") pod \"network-check-target-c24j9\" (UID: \"4ca01b49-9819-4a90-a851-7553ad568679\") " pod="openshift-network-diagnostics/network-check-target-c24j9"
Oct 30 19:15:27.909609 ip-10-0-132-82 kubenswrapper[2577]: E1030 19:15:27.909593    2577 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Oct 30 19:15:27.909661 ip-10-0-132-82 kubenswrapper[2577]: E1030 19:15:27.909616    2577 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Oct 30 19:15:27.909661 ip-10-0-132-82 kubenswrapper[2577]: E1030 19:15:27.909626    2577 projected.go:196] Error preparing data for projected volume kube-api-access-bzw4s for pod openshift-network-diagnostics/network-check-target-c24j9: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Oct 30 19:15:27.909752 ip-10-0-132-82 kubenswrapper[2577]: E1030 19:15:27.909674    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/4ca01b49-9819-4a90-a851-7553ad568679-kube-api-access-bzw4s podName:4ca01b49-9819-4a90-a851-7553ad568679 nodeName:}" failed. No retries permitted until 2026-10-30 19:15:43.909662136 +0000 UTC m=+34.315582550 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "kube-api-access-bzw4s" (UniqueName: "kubernetes.io/projected/4ca01b49-9819-4a90-a851-7553ad568679-kube-api-access-bzw4s") pod "network-check-target-c24j9" (UID: "4ca01b49-9819-4a90-a851-7553ad568679") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Oct 30 19:15:28.032237 ip-10-0-132-82 kubenswrapper[2577]: I1030 19:15:28.032204    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-c24j9"
Oct 30 19:15:28.032237 ip-10-0-132-82 kubenswrapper[2577]: I1030 19:15:28.032233    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-822gp"
Oct 30 19:15:28.032461 ip-10-0-132-82 kubenswrapper[2577]: I1030 19:15:28.032368    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-76rht"
Oct 30 19:15:28.032461 ip-10-0-132-82 kubenswrapper[2577]: E1030 19:15:28.032372    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-c24j9" podUID="4ca01b49-9819-4a90-a851-7553ad568679"
Oct 30 19:15:28.032562 ip-10-0-132-82 kubenswrapper[2577]: E1030 19:15:28.032484    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-822gp" podUID="287ba08f-7b9f-4609-8e35-832c0879c44c"
Oct 30 19:15:28.032616 ip-10-0-132-82 kubenswrapper[2577]: E1030 19:15:28.032578    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-76rht" podUID="c16c0a80-c5ce-4f5d-827a-2e3eb2c3cfb8"
Oct 30 19:15:29.183809 ip-10-0-132-82 kubenswrapper[2577]: I1030 19:15:29.183768    2577 generic.go:356] "Generic (PLEG): container finished" podID="bab0abf9-46d4-408e-9aa0-afa7279cd9d1" containerID="ffd8aa686926a2bdb3d5ed4e9fb9d162a05d20856e7f999b77f538b86c2cad85" exitCode=0
Oct 30 19:15:29.184233 ip-10-0-132-82 kubenswrapper[2577]: I1030 19:15:29.183868    2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-4pvgg" event={"ID":"bab0abf9-46d4-408e-9aa0-afa7279cd9d1","Type":"ContainerDied","Data":"ffd8aa686926a2bdb3d5ed4e9fb9d162a05d20856e7f999b77f538b86c2cad85"}
Oct 30 19:15:29.187396 ip-10-0-132-82 kubenswrapper[2577]: I1030 19:15:29.187371    2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-5lwkx" event={"ID":"08ae8ede-9a96-4a84-8ec8-8d364fd476a3","Type":"ContainerStarted","Data":"109b27fea69361047ab7a334dcb8133945e17ad15b501549528771327812aa9d"}
Oct 30 19:15:29.187483 ip-10-0-132-82 kubenswrapper[2577]: I1030 19:15:29.187455    2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-5lwkx" event={"ID":"08ae8ede-9a96-4a84-8ec8-8d364fd476a3","Type":"ContainerStarted","Data":"d95475a523bcaa11ae3ca23e35163a2e16949ecd0c8822e179cf68b44617b5cf"}
Oct 30 19:15:29.187547 ip-10-0-132-82 kubenswrapper[2577]: I1030 19:15:29.187507    2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-5lwkx" event={"ID":"08ae8ede-9a96-4a84-8ec8-8d364fd476a3","Type":"ContainerStarted","Data":"9a021e374e0191afbf800e567e07ab392178e246b2dea37c3e1a3046448edde4"}
Oct 30 19:15:29.187590 ip-10-0-132-82 kubenswrapper[2577]: I1030 19:15:29.187555    2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-5lwkx" event={"ID":"08ae8ede-9a96-4a84-8ec8-8d364fd476a3","Type":"ContainerStarted","Data":"62202a8e9b3e2303bcd2c5053c69cee34ae474e0b5c1d483cbbeb7b869f74a0b"}
Oct 30 19:15:29.187627 ip-10-0-132-82 kubenswrapper[2577]: I1030 19:15:29.187601    2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-5lwkx" event={"ID":"08ae8ede-9a96-4a84-8ec8-8d364fd476a3","Type":"ContainerStarted","Data":"8e7857ade7272b425822b17539bc630dbeafac54732626b61b5badbf3d23efbd"}
Oct 30 19:15:29.187655 ip-10-0-132-82 kubenswrapper[2577]: I1030 19:15:29.187647    2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-5lwkx" event={"ID":"08ae8ede-9a96-4a84-8ec8-8d364fd476a3","Type":"ContainerStarted","Data":"d1bd4c0ecff49d26f08a6462c8deffbb3fe0447e62f76bb0557ffd90c3824f82"}
Oct 30 19:15:30.032452 ip-10-0-132-82 kubenswrapper[2577]: I1030 19:15:30.032419    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-822gp"
Oct 30 19:15:30.032625 ip-10-0-132-82 kubenswrapper[2577]: E1030 19:15:30.032504    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-822gp" podUID="287ba08f-7b9f-4609-8e35-832c0879c44c"
Oct 30 19:15:30.032625 ip-10-0-132-82 kubenswrapper[2577]: I1030 19:15:30.032546    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-c24j9"
Oct 30 19:15:30.032625 ip-10-0-132-82 kubenswrapper[2577]: E1030 19:15:30.032585    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-c24j9" podUID="4ca01b49-9819-4a90-a851-7553ad568679"
Oct 30 19:15:30.032625 ip-10-0-132-82 kubenswrapper[2577]: I1030 19:15:30.032612    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-76rht"
Oct 30 19:15:30.032766 ip-10-0-132-82 kubenswrapper[2577]: E1030 19:15:30.032655    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-76rht" podUID="c16c0a80-c5ce-4f5d-827a-2e3eb2c3cfb8"
Oct 30 19:15:30.192344 ip-10-0-132-82 kubenswrapper[2577]: I1030 19:15:30.192313    2577 generic.go:356] "Generic (PLEG): container finished" podID="bab0abf9-46d4-408e-9aa0-afa7279cd9d1" containerID="12bddeb50edd8fbac71cdb67cadcebd514f07fdd2a4993c4e59997b923ffb564" exitCode=0
Oct 30 19:15:30.192709 ip-10-0-132-82 kubenswrapper[2577]: I1030 19:15:30.192368    2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-4pvgg" event={"ID":"bab0abf9-46d4-408e-9aa0-afa7279cd9d1","Type":"ContainerDied","Data":"12bddeb50edd8fbac71cdb67cadcebd514f07fdd2a4993c4e59997b923ffb564"}
Oct 30 19:15:31.197208 ip-10-0-132-82 kubenswrapper[2577]: I1030 19:15:31.197180    2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-4pvgg" event={"ID":"bab0abf9-46d4-408e-9aa0-afa7279cd9d1","Type":"ContainerStarted","Data":"22fa04f41d4b3d5d660d0b36406db37afbd18bee60fcb2cfb13cf72b4c96da58"}
Oct 30 19:15:31.313909 ip-10-0-132-82 kubenswrapper[2577]: I1030 19:15:31.313862    2577 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-multus/multus-additional-cni-plugins-4pvgg" podStartSLOduration=8.371003056 podStartE2EDuration="21.313847966s" totalImagesPullingTime="12.94284491s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-30 19:15:10 +0000 UTC" imagePullSessionsCount=5 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:15:31.313384115 +0000 UTC m=+21.719304558" watchObservedRunningTime="2026-10-30 19:15:31.313847966 +0000 UTC m=+21.719768410"
Oct 30 19:15:32.031219 ip-10-0-132-82 kubenswrapper[2577]: I1030 19:15:32.031185    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-822gp"
Oct 30 19:15:32.031219 ip-10-0-132-82 kubenswrapper[2577]: I1030 19:15:32.031206    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-76rht"
Oct 30 19:15:32.031429 ip-10-0-132-82 kubenswrapper[2577]: I1030 19:15:32.031304    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-c24j9"
Oct 30 19:15:32.031429 ip-10-0-132-82 kubenswrapper[2577]: E1030 19:15:32.031313    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-822gp" podUID="287ba08f-7b9f-4609-8e35-832c0879c44c"
Oct 30 19:15:32.031429 ip-10-0-132-82 kubenswrapper[2577]: E1030 19:15:32.031394    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-c24j9" podUID="4ca01b49-9819-4a90-a851-7553ad568679"
Oct 30 19:15:32.031539 ip-10-0-132-82 kubenswrapper[2577]: E1030 19:15:32.031432    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-76rht" podUID="c16c0a80-c5ce-4f5d-827a-2e3eb2c3cfb8"
Oct 30 19:15:32.202937 ip-10-0-132-82 kubenswrapper[2577]: I1030 19:15:32.202911    2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-5lwkx" event={"ID":"08ae8ede-9a96-4a84-8ec8-8d364fd476a3","Type":"ContainerStarted","Data":"b30a08593bfeffbc87189629f36a1c4272e3724be4cb903191007a1b901df771"}
Oct 30 19:15:34.031291 ip-10-0-132-82 kubenswrapper[2577]: I1030 19:15:34.031263    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-822gp"
Oct 30 19:15:34.032267 ip-10-0-132-82 kubenswrapper[2577]: I1030 19:15:34.031259    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-76rht"
Oct 30 19:15:34.032267 ip-10-0-132-82 kubenswrapper[2577]: E1030 19:15:34.031363    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-822gp" podUID="287ba08f-7b9f-4609-8e35-832c0879c44c"
Oct 30 19:15:34.032267 ip-10-0-132-82 kubenswrapper[2577]: I1030 19:15:34.031270    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-c24j9"
Oct 30 19:15:34.032267 ip-10-0-132-82 kubenswrapper[2577]: E1030 19:15:34.031426    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-76rht" podUID="c16c0a80-c5ce-4f5d-827a-2e3eb2c3cfb8"
Oct 30 19:15:34.032267 ip-10-0-132-82 kubenswrapper[2577]: E1030 19:15:34.031518    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-c24j9" podUID="4ca01b49-9819-4a90-a851-7553ad568679"
Oct 30 19:15:34.210379 ip-10-0-132-82 kubenswrapper[2577]: I1030 19:15:34.210352    2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-5lwkx" event={"ID":"08ae8ede-9a96-4a84-8ec8-8d364fd476a3","Type":"ContainerStarted","Data":"c4057f99fa672970b6b8a6d552ba8380026f38066ec1cb4165f2decf646cb0a8"}
Oct 30 19:15:34.210629 ip-10-0-132-82 kubenswrapper[2577]: I1030 19:15:34.210603    2577 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-5lwkx"
Oct 30 19:15:34.210754 ip-10-0-132-82 kubenswrapper[2577]: I1030 19:15:34.210685    2577 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-5lwkx"
Oct 30 19:15:34.225388 ip-10-0-132-82 kubenswrapper[2577]: I1030 19:15:34.225368    2577 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-5lwkx"
Oct 30 19:15:34.481272 ip-10-0-132-82 kubenswrapper[2577]: I1030 19:15:34.481210    2577 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ovn-kubernetes/ovnkube-node-5lwkx" podStartSLOduration=8.076048658 podStartE2EDuration="24.481193118s" totalImagesPullingTime="16.40514446s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-30 19:15:10 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:15:34.480887172 +0000 UTC m=+24.886807610" watchObservedRunningTime="2026-10-30 19:15:34.481193118 +0000 UTC m=+24.887113555"
Oct 30 19:15:35.213005 ip-10-0-132-82 kubenswrapper[2577]: I1030 19:15:35.212974    2577 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-5lwkx"
Oct 30 19:15:35.227940 ip-10-0-132-82 kubenswrapper[2577]: I1030 19:15:35.227920    2577 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-5lwkx"
Oct 30 19:15:35.932406 ip-10-0-132-82 kubenswrapper[2577]: I1030 19:15:35.932369    2577 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-76rht"]
Oct 30 19:15:35.932582 ip-10-0-132-82 kubenswrapper[2577]: I1030 19:15:35.932493    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-76rht"
Oct 30 19:15:35.932649 ip-10-0-132-82 kubenswrapper[2577]: E1030 19:15:35.932581    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-76rht" podUID="c16c0a80-c5ce-4f5d-827a-2e3eb2c3cfb8"
Oct 30 19:15:35.935505 ip-10-0-132-82 kubenswrapper[2577]: I1030 19:15:35.935466    2577 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-c24j9"]
Oct 30 19:15:35.935644 ip-10-0-132-82 kubenswrapper[2577]: I1030 19:15:35.935575    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-c24j9"
Oct 30 19:15:35.935709 ip-10-0-132-82 kubenswrapper[2577]: E1030 19:15:35.935665    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-c24j9" podUID="4ca01b49-9819-4a90-a851-7553ad568679"
Oct 30 19:15:35.936124 ip-10-0-132-82 kubenswrapper[2577]: I1030 19:15:35.936102    2577 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-822gp"]
Oct 30 19:15:35.936250 ip-10-0-132-82 kubenswrapper[2577]: I1030 19:15:35.936191    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-822gp"
Oct 30 19:15:35.936312 ip-10-0-132-82 kubenswrapper[2577]: E1030 19:15:35.936265    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-822gp" podUID="287ba08f-7b9f-4609-8e35-832c0879c44c"
Oct 30 19:15:38.032071 ip-10-0-132-82 kubenswrapper[2577]: I1030 19:15:38.031866    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-822gp"
Oct 30 19:15:38.032502 ip-10-0-132-82 kubenswrapper[2577]: I1030 19:15:38.031886    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-76rht"
Oct 30 19:15:38.032502 ip-10-0-132-82 kubenswrapper[2577]: E1030 19:15:38.032186    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-822gp" podUID="287ba08f-7b9f-4609-8e35-832c0879c44c"
Oct 30 19:15:38.032502 ip-10-0-132-82 kubenswrapper[2577]: I1030 19:15:38.031921    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-c24j9"
Oct 30 19:15:38.032502 ip-10-0-132-82 kubenswrapper[2577]: E1030 19:15:38.032240    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-76rht" podUID="c16c0a80-c5ce-4f5d-827a-2e3eb2c3cfb8"
Oct 30 19:15:38.032502 ip-10-0-132-82 kubenswrapper[2577]: E1030 19:15:38.032298    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-c24j9" podUID="4ca01b49-9819-4a90-a851-7553ad568679"
Oct 30 19:15:40.032158 ip-10-0-132-82 kubenswrapper[2577]: I1030 19:15:40.032115    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-c24j9"
Oct 30 19:15:40.032158 ip-10-0-132-82 kubenswrapper[2577]: I1030 19:15:40.032148    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-822gp"
Oct 30 19:15:40.032158 ip-10-0-132-82 kubenswrapper[2577]: I1030 19:15:40.032127    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-76rht"
Oct 30 19:15:40.032731 ip-10-0-132-82 kubenswrapper[2577]: E1030 19:15:40.032219    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-c24j9" podUID="4ca01b49-9819-4a90-a851-7553ad568679"
Oct 30 19:15:40.032731 ip-10-0-132-82 kubenswrapper[2577]: E1030 19:15:40.032350    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-76rht" podUID="c16c0a80-c5ce-4f5d-827a-2e3eb2c3cfb8"
Oct 30 19:15:40.032731 ip-10-0-132-82 kubenswrapper[2577]: E1030 19:15:40.032451    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-822gp" podUID="287ba08f-7b9f-4609-8e35-832c0879c44c"
Oct 30 19:15:41.335476 ip-10-0-132-82 kubenswrapper[2577]: I1030 19:15:41.335456    2577 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-132-82.us-west-1.compute.internal" event="NodeReady"
Oct 30 19:15:41.335762 ip-10-0-132-82 kubenswrapper[2577]: I1030 19:15:41.335543    2577 kubelet_node_status.go:477] "Fast updating node status as it just became ready"
Oct 30 19:15:41.447978 ip-10-0-132-82 kubenswrapper[2577]: I1030 19:15:41.447935    2577 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-132-82.us-west-1.compute.internal" event="NodeReady"
Oct 30 19:15:41.448207 ip-10-0-132-82 kubenswrapper[2577]: I1030 19:15:41.448191    2577 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kube-system/kas-connection-checker-685895c7c9-tvmks"]
Oct 30 19:15:41.507927 ip-10-0-132-82 kubenswrapper[2577]: I1030 19:15:41.507907    2577 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kube-system/kas-connection-checker-685895c7c9-9h85f"]
Oct 30 19:15:41.508053 ip-10-0-132-82 kubenswrapper[2577]: I1030 19:15:41.508039    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kas-connection-checker-685895c7c9-tvmks"
Oct 30 19:15:41.520569 ip-10-0-132-82 kubenswrapper[2577]: I1030 19:15:41.520552    2577 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kube-system/kas-connection-checker-685895c7c9-p8kjs"]
Oct 30 19:15:41.520661 ip-10-0-132-82 kubenswrapper[2577]: I1030 19:15:41.520648    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kas-connection-checker-685895c7c9-9h85f"
Oct 30 19:15:41.534625 ip-10-0-132-82 kubenswrapper[2577]: I1030 19:15:41.534605    2577 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-image-registry/image-registry-6fcb4848df-8l8cp"]
Oct 30 19:15:41.534737 ip-10-0-132-82 kubenswrapper[2577]: I1030 19:15:41.534723    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kas-connection-checker-685895c7c9-p8kjs"
Oct 30 19:15:41.556809 ip-10-0-132-82 kubenswrapper[2577]: I1030 19:15:41.556786    2577 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-np94b"]
Oct 30 19:15:41.556928 ip-10-0-132-82 kubenswrapper[2577]: I1030 19:15:41.556911    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-6fcb4848df-8l8cp"
Oct 30 19:15:41.569336 ip-10-0-132-82 kubenswrapper[2577]: I1030 19:15:41.569315    2577 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"openshift-service-ca.crt\""
Oct 30 19:15:41.570094 ip-10-0-132-82 kubenswrapper[2577]: I1030 19:15:41.570078    2577 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"kube-root-ca.crt\""
Oct 30 19:15:41.570949 ip-10-0-132-82 kubenswrapper[2577]: I1030 19:15:41.570932    2577 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-insights/insights-operator-7c4986d449-6jdkq"]
Oct 30 19:15:41.571134 ip-10-0-132-82 kubenswrapper[2577]: I1030 19:15:41.571115    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-np94b"
Oct 30 19:15:41.602846 ip-10-0-132-82 kubenswrapper[2577]: I1030 19:15:41.602826    2577 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-cluster-storage-operator/volume-data-source-validator-58c9f5f46d-p42xv"]
Oct 30 19:15:41.602989 ip-10-0-132-82 kubenswrapper[2577]: I1030 19:15:41.602949    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-operator-7c4986d449-6jdkq"
Oct 30 19:15:41.608181 ip-10-0-132-82 kubenswrapper[2577]: I1030 19:15:41.608161    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wmkq8\" (UniqueName: \"kubernetes.io/projected/d260d4d7-b4aa-4f37-92b5-e75b22e13a3b-kube-api-access-wmkq8\") pod \"kas-connection-checker-685895c7c9-tvmks\" (UID: \"d260d4d7-b4aa-4f37-92b5-e75b22e13a3b\") " pod="kube-system/kas-connection-checker-685895c7c9-tvmks"
Oct 30 19:15:41.608275 ip-10-0-132-82 kubenswrapper[2577]: I1030 19:15:41.608263    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-txbm4\" (UniqueName: \"kubernetes.io/projected/87da348d-b1ee-4cd3-806d-0560bff3c6f2-kube-api-access-txbm4\") pod \"kas-connection-checker-685895c7c9-p8kjs\" (UID: \"87da348d-b1ee-4cd3-806d-0560bff3c6f2\") " pod="kube-system/kas-connection-checker-685895c7c9-p8kjs"
Oct 30 19:15:41.608331 ip-10-0-132-82 kubenswrapper[2577]: I1030 19:15:41.608308    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-k62g9\" (UniqueName: \"kubernetes.io/projected/ba2c5571-23a3-46b6-a36d-7e8810a66261-kube-api-access-k62g9\") pod \"kas-connection-checker-685895c7c9-9h85f\" (UID: \"ba2c5571-23a3-46b6-a36d-7e8810a66261\") " pod="kube-system/kas-connection-checker-685895c7c9-9h85f"
Oct 30 19:15:41.611535 ip-10-0-132-82 kubenswrapper[2577]: I1030 19:15:41.611517    2577 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"installation-pull-secrets\""
Oct 30 19:15:41.614651 ip-10-0-132-82 kubenswrapper[2577]: I1030 19:15:41.614634    2577 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-tls\""
Oct 30 19:15:41.617176 ip-10-0-132-82 kubenswrapper[2577]: I1030 19:15:41.617153    2577 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/router-default-776df59447-8nrhh"]
Oct 30 19:15:41.620172 ip-10-0-132-82 kubenswrapper[2577]: I1030 19:15:41.620154    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-storage-operator/volume-data-source-validator-58c9f5f46d-p42xv"
Oct 30 19:15:41.632145 ip-10-0-132-82 kubenswrapper[2577]: I1030 19:15:41.632124    2577 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-scljn"]
Oct 30 19:15:41.632225 ip-10-0-132-82 kubenswrapper[2577]: I1030 19:15:41.632147    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/router-default-776df59447-8nrhh"
Oct 30 19:15:41.646202 ip-10-0-132-82 kubenswrapper[2577]: I1030 19:15:41.646183    2577 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-network-console/networking-console-plugin-86c478bb88-h4twn"]
Oct 30 19:15:41.646314 ip-10-0-132-82 kubenswrapper[2577]: I1030 19:15:41.646297    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-scljn"
Oct 30 19:15:41.664314 ip-10-0-132-82 kubenswrapper[2577]: I1030 19:15:41.664291    2577 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-image-registry/image-registry-d97c68fd4-htsl8"]
Oct 30 19:15:41.664423 ip-10-0-132-82 kubenswrapper[2577]: I1030 19:15:41.664407    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-console/networking-console-plugin-86c478bb88-h4twn"
Oct 30 19:15:41.665808 ip-10-0-132-82 kubenswrapper[2577]: I1030 19:15:41.665788    2577 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-samples-operator\"/\"samples-operator-config\""
Oct 30 19:15:41.665912 ip-10-0-132-82 kubenswrapper[2577]: I1030 19:15:41.665896    2577 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"openshift-service-ca.crt\""
Oct 30 19:15:41.665978 ip-10-0-132-82 kubenswrapper[2577]: I1030 19:15:41.665898    2577 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-root-ca.crt\""
Oct 30 19:15:41.665978 ip-10-0-132-82 kubenswrapper[2577]: I1030 19:15:41.665939    2577 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-samples-operator\"/\"openshift-service-ca.crt\""
Oct 30 19:15:41.665978 ip-10-0-132-82 kubenswrapper[2577]: I1030 19:15:41.665955    2577 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"service-ca-bundle\""
Oct 30 19:15:41.666199 ip-10-0-132-82 kubenswrapper[2577]: I1030 19:15:41.666181    2577 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-samples-operator\"/\"kube-root-ca.crt\""
Oct 30 19:15:41.692382 ip-10-0-132-82 kubenswrapper[2577]: I1030 19:15:41.692356    2577 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-service-ca-operator/service-ca-operator-544d749c8c-nbfv2"]
Oct 30 19:15:41.692480 ip-10-0-132-82 kubenswrapper[2577]: I1030 19:15:41.692465    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-d97c68fd4-htsl8"
Oct 30 19:15:41.705614 ip-10-0-132-82 kubenswrapper[2577]: I1030 19:15:41.705570    2577 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-console-operator/console-operator-7cd56b8dbc-tg5xj"]
Oct 30 19:15:41.705715 ip-10-0-132-82 kubenswrapper[2577]: I1030 19:15:41.705667    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-nbfv2"
Oct 30 19:15:41.708510 ip-10-0-132-82 kubenswrapper[2577]: I1030 19:15:41.708493    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/e3206ddf-35de-4b7c-b979-682f5f666a34-ca-trust-extracted\") pod \"image-registry-6fcb4848df-8l8cp\" (UID: \"e3206ddf-35de-4b7c-b979-682f5f666a34\") " pod="openshift-image-registry/image-registry-6fcb4848df-8l8cp"
Oct 30 19:15:41.708604 ip-10-0-132-82 kubenswrapper[2577]: I1030 19:15:41.708517    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4rmts\" (UniqueName: \"kubernetes.io/projected/2bd84dc4-e96a-43f1-85bf-e1ea4005c193-kube-api-access-4rmts\") pod \"insights-operator-7c4986d449-6jdkq\" (UID: \"2bd84dc4-e96a-43f1-85bf-e1ea4005c193\") " pod="openshift-insights/insights-operator-7c4986d449-6jdkq"
Oct 30 19:15:41.708604 ip-10-0-132-82 kubenswrapper[2577]: I1030 19:15:41.708535    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/87082646-75c3-43b5-864b-3478e9a7373d-metrics-certs\") pod \"router-default-776df59447-8nrhh\" (UID: \"87082646-75c3-43b5-864b-3478e9a7373d\") " pod="openshift-ingress/router-default-776df59447-8nrhh"
Oct 30 19:15:41.708703 ip-10-0-132-82 kubenswrapper[2577]: I1030 19:15:41.708598    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"default-certificate\" (UniqueName: \"kubernetes.io/secret/87082646-75c3-43b5-864b-3478e9a7373d-default-certificate\") pod \"router-default-776df59447-8nrhh\" (UID: \"87082646-75c3-43b5-864b-3478e9a7373d\") " pod="openshift-ingress/router-default-776df59447-8nrhh"
Oct 30 19:15:41.708703 ip-10-0-132-82 kubenswrapper[2577]: I1030 19:15:41.708625    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-txbm4\" (UniqueName: \"kubernetes.io/projected/87da348d-b1ee-4cd3-806d-0560bff3c6f2-kube-api-access-txbm4\") pod \"kas-connection-checker-685895c7c9-p8kjs\" (UID: \"87da348d-b1ee-4cd3-806d-0560bff3c6f2\") " pod="kube-system/kas-connection-checker-685895c7c9-p8kjs"
Oct 30 19:15:41.708703 ip-10-0-132-82 kubenswrapper[2577]: I1030 19:15:41.708658    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/f9a06263-cb99-4004-bffc-581d66ce5ebf-serving-cert\") pod \"kube-storage-version-migrator-operator-7d4bc45dd6-scljn\" (UID: \"f9a06263-cb99-4004-bffc-581d66ce5ebf\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-scljn"
Oct 30 19:15:41.708821 ip-10-0-132-82 kubenswrapper[2577]: I1030 19:15:41.708723    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/87082646-75c3-43b5-864b-3478e9a7373d-service-ca-bundle\") pod \"router-default-776df59447-8nrhh\" (UID: \"87082646-75c3-43b5-864b-3478e9a7373d\") " pod="openshift-ingress/router-default-776df59447-8nrhh"
Oct 30 19:15:41.708821 ip-10-0-132-82 kubenswrapper[2577]: I1030 19:15:41.708761    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/e3206ddf-35de-4b7c-b979-682f5f666a34-bound-sa-token\") pod \"image-registry-6fcb4848df-8l8cp\" (UID: \"e3206ddf-35de-4b7c-b979-682f5f666a34\") " pod="openshift-image-registry/image-registry-6fcb4848df-8l8cp"
Oct 30 19:15:41.708926 ip-10-0-132-82 kubenswrapper[2577]: I1030 19:15:41.708812    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/56274b23-2400-43d4-9686-c928e20d47de-samples-operator-tls\") pod \"cluster-samples-operator-6ddb6c8b94-np94b\" (UID: \"56274b23-2400-43d4-9686-c928e20d47de\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-np94b"
Oct 30 19:15:41.708926 ip-10-0-132-82 kubenswrapper[2577]: I1030 19:15:41.708873    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wmkq8\" (UniqueName: \"kubernetes.io/projected/d260d4d7-b4aa-4f37-92b5-e75b22e13a3b-kube-api-access-wmkq8\") pod \"kas-connection-checker-685895c7c9-tvmks\" (UID: \"d260d4d7-b4aa-4f37-92b5-e75b22e13a3b\") " pod="kube-system/kas-connection-checker-685895c7c9-tvmks"
Oct 30 19:15:41.708926 ip-10-0-132-82 kubenswrapper[2577]: I1030 19:15:41.708894    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/e3206ddf-35de-4b7c-b979-682f5f666a34-installation-pull-secrets\") pod \"image-registry-6fcb4848df-8l8cp\" (UID: \"e3206ddf-35de-4b7c-b979-682f5f666a34\") " pod="openshift-image-registry/image-registry-6fcb4848df-8l8cp"
Oct 30 19:15:41.708926 ip-10-0-132-82 kubenswrapper[2577]: I1030 19:15:41.708909    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7xf2g\" (UniqueName: \"kubernetes.io/projected/4d213db4-73c5-4f50-9930-6d26b880dcf7-kube-api-access-7xf2g\") pod \"volume-data-source-validator-58c9f5f46d-p42xv\" (UID: \"4d213db4-73c5-4f50-9930-6d26b880dcf7\") " pod="openshift-cluster-storage-operator/volume-data-source-validator-58c9f5f46d-p42xv"
Oct 30 19:15:41.709138 ip-10-0-132-82 kubenswrapper[2577]: I1030 19:15:41.708941    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/e3206ddf-35de-4b7c-b979-682f5f666a34-registry-certificates\") pod \"image-registry-6fcb4848df-8l8cp\" (UID: \"e3206ddf-35de-4b7c-b979-682f5f666a34\") " pod="openshift-image-registry/image-registry-6fcb4848df-8l8cp"
Oct 30 19:15:41.709138 ip-10-0-132-82 kubenswrapper[2577]: I1030 19:15:41.708958    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/2bd84dc4-e96a-43f1-85bf-e1ea4005c193-trusted-ca-bundle\") pod \"insights-operator-7c4986d449-6jdkq\" (UID: \"2bd84dc4-e96a-43f1-85bf-e1ea4005c193\") " pod="openshift-insights/insights-operator-7c4986d449-6jdkq"
Oct 30 19:15:41.709138 ip-10-0-132-82 kubenswrapper[2577]: I1030 19:15:41.709033    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8s7g5\" (UniqueName: \"kubernetes.io/projected/87082646-75c3-43b5-864b-3478e9a7373d-kube-api-access-8s7g5\") pod \"router-default-776df59447-8nrhh\" (UID: \"87082646-75c3-43b5-864b-3478e9a7373d\") " pod="openshift-ingress/router-default-776df59447-8nrhh"
Oct 30 19:15:41.709138 ip-10-0-132-82 kubenswrapper[2577]: I1030 19:15:41.709077    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/2bd84dc4-e96a-43f1-85bf-e1ea4005c193-service-ca-bundle\") pod \"insights-operator-7c4986d449-6jdkq\" (UID: \"2bd84dc4-e96a-43f1-85bf-e1ea4005c193\") " pod="openshift-insights/insights-operator-7c4986d449-6jdkq"
Oct 30 19:15:41.709138 ip-10-0-132-82 kubenswrapper[2577]: I1030 19:15:41.709130    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-w9h4f\" (UniqueName: \"kubernetes.io/projected/e3206ddf-35de-4b7c-b979-682f5f666a34-kube-api-access-w9h4f\") pod \"image-registry-6fcb4848df-8l8cp\" (UID: \"e3206ddf-35de-4b7c-b979-682f5f666a34\") " pod="openshift-image-registry/image-registry-6fcb4848df-8l8cp"
Oct 30 19:15:41.709314 ip-10-0-132-82 kubenswrapper[2577]: I1030 19:15:41.709182    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qfw7m\" (UniqueName: \"kubernetes.io/projected/f9a06263-cb99-4004-bffc-581d66ce5ebf-kube-api-access-qfw7m\") pod \"kube-storage-version-migrator-operator-7d4bc45dd6-scljn\" (UID: \"f9a06263-cb99-4004-bffc-581d66ce5ebf\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-scljn"
Oct 30 19:15:41.709314 ip-10-0-132-82 kubenswrapper[2577]: I1030 19:15:41.709198    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"stats-auth\" (UniqueName: \"kubernetes.io/secret/87082646-75c3-43b5-864b-3478e9a7373d-stats-auth\") pod \"router-default-776df59447-8nrhh\" (UID: \"87082646-75c3-43b5-864b-3478e9a7373d\") " pod="openshift-ingress/router-default-776df59447-8nrhh"
Oct 30 19:15:41.709314 ip-10-0-132-82 kubenswrapper[2577]: I1030 19:15:41.709232    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/e3206ddf-35de-4b7c-b979-682f5f666a34-trusted-ca\") pod \"image-registry-6fcb4848df-8l8cp\" (UID: \"e3206ddf-35de-4b7c-b979-682f5f666a34\") " pod="openshift-image-registry/image-registry-6fcb4848df-8l8cp"
Oct 30 19:15:41.709314 ip-10-0-132-82 kubenswrapper[2577]: I1030 19:15:41.709263    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/e3206ddf-35de-4b7c-b979-682f5f666a34-image-registry-private-configuration\") pod \"image-registry-6fcb4848df-8l8cp\" (UID: \"e3206ddf-35de-4b7c-b979-682f5f666a34\") " pod="openshift-image-registry/image-registry-6fcb4848df-8l8cp"
Oct 30 19:15:41.709314 ip-10-0-132-82 kubenswrapper[2577]: I1030 19:15:41.709281    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"samples-operator-config\" (UniqueName: \"kubernetes.io/configmap/56274b23-2400-43d4-9686-c928e20d47de-samples-operator-config\") pod \"cluster-samples-operator-6ddb6c8b94-np94b\" (UID: \"56274b23-2400-43d4-9686-c928e20d47de\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-np94b"
Oct 30 19:15:41.709314 ip-10-0-132-82 kubenswrapper[2577]: I1030 19:15:41.709304    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/2bd84dc4-e96a-43f1-85bf-e1ea4005c193-tmp\") pod \"insights-operator-7c4986d449-6jdkq\" (UID: \"2bd84dc4-e96a-43f1-85bf-e1ea4005c193\") " pod="openshift-insights/insights-operator-7c4986d449-6jdkq"
Oct 30 19:15:41.709542 ip-10-0-132-82 kubenswrapper[2577]: I1030 19:15:41.709336    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/2bd84dc4-e96a-43f1-85bf-e1ea4005c193-serving-cert\") pod \"insights-operator-7c4986d449-6jdkq\" (UID: \"2bd84dc4-e96a-43f1-85bf-e1ea4005c193\") " pod="openshift-insights/insights-operator-7c4986d449-6jdkq"
Oct 30 19:15:41.709542 ip-10-0-132-82 kubenswrapper[2577]: I1030 19:15:41.709422    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"snapshots\" (UniqueName: \"kubernetes.io/empty-dir/2bd84dc4-e96a-43f1-85bf-e1ea4005c193-snapshots\") pod \"insights-operator-7c4986d449-6jdkq\" (UID: \"2bd84dc4-e96a-43f1-85bf-e1ea4005c193\") " pod="openshift-insights/insights-operator-7c4986d449-6jdkq"
Oct 30 19:15:41.709542 ip-10-0-132-82 kubenswrapper[2577]: I1030 19:15:41.709475    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/f9a06263-cb99-4004-bffc-581d66ce5ebf-config\") pod \"kube-storage-version-migrator-operator-7d4bc45dd6-scljn\" (UID: \"f9a06263-cb99-4004-bffc-581d66ce5ebf\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-scljn"
Oct 30 19:15:41.709542 ip-10-0-132-82 kubenswrapper[2577]: I1030 19:15:41.709509    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/e3206ddf-35de-4b7c-b979-682f5f666a34-registry-tls\") pod \"image-registry-6fcb4848df-8l8cp\" (UID: \"e3206ddf-35de-4b7c-b979-682f5f666a34\") " pod="openshift-image-registry/image-registry-6fcb4848df-8l8cp"
Oct 30 19:15:41.709684 ip-10-0-132-82 kubenswrapper[2577]: I1030 19:15:41.709541    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-k62g9\" (UniqueName: \"kubernetes.io/projected/ba2c5571-23a3-46b6-a36d-7e8810a66261-kube-api-access-k62g9\") pod \"kas-connection-checker-685895c7c9-9h85f\" (UID: \"ba2c5571-23a3-46b6-a36d-7e8810a66261\") " pod="kube-system/kas-connection-checker-685895c7c9-9h85f"
Oct 30 19:15:41.709684 ip-10-0-132-82 kubenswrapper[2577]: I1030 19:15:41.709569    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wt5rq\" (UniqueName: \"kubernetes.io/projected/56274b23-2400-43d4-9686-c928e20d47de-kube-api-access-wt5rq\") pod \"cluster-samples-operator-6ddb6c8b94-np94b\" (UID: \"56274b23-2400-43d4-9686-c928e20d47de\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-np94b"
Oct 30 19:15:41.716870 ip-10-0-132-82 kubenswrapper[2577]: I1030 19:15:41.716846    2577 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"openshift-insights-serving-cert\""
Oct 30 19:15:41.717139 ip-10-0-132-82 kubenswrapper[2577]: I1030 19:15:41.717121    2577 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"router-metrics-certs-default\""
Oct 30 19:15:41.717428 ip-10-0-132-82 kubenswrapper[2577]: I1030 19:15:41.717406    2577 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"serving-cert\""
Oct 30 19:15:41.717591 ip-10-0-132-82 kubenswrapper[2577]: I1030 19:15:41.717576    2577 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-samples-operator\"/\"samples-operator-tls\""
Oct 30 19:15:41.717661 ip-10-0-132-82 kubenswrapper[2577]: I1030 19:15:41.717587    2577 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-private-configuration\""
Oct 30 19:15:41.717828 ip-10-0-132-82 kubenswrapper[2577]: I1030 19:15:41.717742    2577 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"default-ingress-cert\""
Oct 30 19:15:41.718196 ip-10-0-132-82 kubenswrapper[2577]: I1030 19:15:41.717958    2577 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"router-stats-default\""
Oct 30 19:15:41.718196 ip-10-0-132-82 kubenswrapper[2577]: I1030 19:15:41.718076    2577 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"registry-dockercfg-2q27v\""
Oct 30 19:15:41.720135 ip-10-0-132-82 kubenswrapper[2577]: I1030 19:15:41.718676    2577 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"trusted-ca-bundle\""
Oct 30 19:15:41.720135 ip-10-0-132-82 kubenswrapper[2577]: I1030 19:15:41.719175    2577 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"trusted-ca\""
Oct 30 19:15:41.730037 ip-10-0-132-82 kubenswrapper[2577]: I1030 19:15:41.730019    2577 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-network-diagnostics/network-check-source-6f78d59f99-2trjq"]
Oct 30 19:15:41.730158 ip-10-0-132-82 kubenswrapper[2577]: I1030 19:15:41.730143    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console-operator/console-operator-7cd56b8dbc-tg5xj"
Oct 30 19:15:41.749594 ip-10-0-132-82 kubenswrapper[2577]: I1030 19:15:41.749578    2577 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-monitoring/cluster-monitoring-operator-884576c57-wkldq"]
Oct 30 19:15:41.749678 ip-10-0-132-82 kubenswrapper[2577]: I1030 19:15:41.749665    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-source-6f78d59f99-2trjq"
Oct 30 19:15:41.764155 ip-10-0-132-82 kubenswrapper[2577]: I1030 19:15:41.764076    2577 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress-canary/ingress-canary-8rwjm"]
Oct 30 19:15:41.764297 ip-10-0-132-82 kubenswrapper[2577]: I1030 19:15:41.764281    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/cluster-monitoring-operator-884576c57-wkldq"
Oct 30 19:15:41.765997 ip-10-0-132-82 kubenswrapper[2577]: I1030 19:15:41.765846    2577 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"openshift-service-ca.crt\""
Oct 30 19:15:41.766089 ip-10-0-132-82 kubenswrapper[2577]: I1030 19:15:41.766052    2577 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca-operator\"/\"service-ca-operator-config\""
Oct 30 19:15:41.766154 ip-10-0-132-82 kubenswrapper[2577]: I1030 19:15:41.766141    2577 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca-operator\"/\"kube-root-ca.crt\""
Oct 30 19:15:41.766301 ip-10-0-132-82 kubenswrapper[2577]: I1030 19:15:41.766191    2577 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"config\""
Oct 30 19:15:41.766405 ip-10-0-132-82 kubenswrapper[2577]: I1030 19:15:41.766389    2577 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"kube-root-ca.crt\""
Oct 30 19:15:41.766405 ip-10-0-132-82 kubenswrapper[2577]: I1030 19:15:41.766402    2577 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-service-ca.crt\""
Oct 30 19:15:41.766514 ip-10-0-132-82 kubenswrapper[2577]: I1030 19:15:41.766420    2577 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"service-ca-bundle\""
Oct 30 19:15:41.766514 ip-10-0-132-82 kubenswrapper[2577]: I1030 19:15:41.766462    2577 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-storage-operator\"/\"openshift-service-ca.crt\""
Oct 30 19:15:41.766514 ip-10-0-132-82 kubenswrapper[2577]: I1030 19:15:41.766466    2577 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-console\"/\"networking-console-plugin\""
Oct 30 19:15:41.766514 ip-10-0-132-82 kubenswrapper[2577]: I1030 19:15:41.766479    2577 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"kube-root-ca.crt\""
Oct 30 19:15:41.766514 ip-10-0-132-82 kubenswrapper[2577]: I1030 19:15:41.766489    2577 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-storage-operator\"/\"kube-root-ca.crt\""
Oct 30 19:15:41.766741 ip-10-0-132-82 kubenswrapper[2577]: I1030 19:15:41.766524    2577 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca-operator\"/\"openshift-service-ca.crt\""
Oct 30 19:15:41.766741 ip-10-0-132-82 kubenswrapper[2577]: I1030 19:15:41.766669    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-txbm4\" (UniqueName: \"kubernetes.io/projected/87da348d-b1ee-4cd3-806d-0560bff3c6f2-kube-api-access-txbm4\") pod \"kas-connection-checker-685895c7c9-p8kjs\" (UID: \"87da348d-b1ee-4cd3-806d-0560bff3c6f2\") " pod="kube-system/kas-connection-checker-685895c7c9-p8kjs"
Oct 30 19:15:41.766741 ip-10-0-132-82 kubenswrapper[2577]: I1030 19:15:41.766692    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wmkq8\" (UniqueName: \"kubernetes.io/projected/d260d4d7-b4aa-4f37-92b5-e75b22e13a3b-kube-api-access-wmkq8\") pod \"kas-connection-checker-685895c7c9-tvmks\" (UID: \"d260d4d7-b4aa-4f37-92b5-e75b22e13a3b\") " pod="kube-system/kas-connection-checker-685895c7c9-tvmks"
Oct 30 19:15:41.766741 ip-10-0-132-82 kubenswrapper[2577]: I1030 19:15:41.766696    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-k62g9\" (UniqueName: \"kubernetes.io/projected/ba2c5571-23a3-46b6-a36d-7e8810a66261-kube-api-access-k62g9\") pod \"kas-connection-checker-685895c7c9-9h85f\" (UID: \"ba2c5571-23a3-46b6-a36d-7e8810a66261\") " pod="kube-system/kas-connection-checker-685895c7c9-9h85f"
Oct 30 19:15:41.788111 ip-10-0-132-82 kubenswrapper[2577]: I1030 19:15:41.788094    2577 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-dns/dns-default-bb9cj"]
Oct 30 19:15:41.788185 ip-10-0-132-82 kubenswrapper[2577]: I1030 19:15:41.788171    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-8rwjm"
Oct 30 19:15:41.809942 ip-10-0-132-82 kubenswrapper[2577]: I1030 19:15:41.809923    2577 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kube-system/kas-connection-checker-685895c7c9-9h85f"]
Oct 30 19:15:41.810040 ip-10-0-132-82 kubenswrapper[2577]: I1030 19:15:41.809958    2577 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kube-system/kas-connection-checker-685895c7c9-tvmks"]
Oct 30 19:15:41.810040 ip-10-0-132-82 kubenswrapper[2577]: I1030 19:15:41.810002    2577 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kube-system/kas-connection-checker-685895c7c9-p8kjs"]
Oct 30 19:15:41.810040 ip-10-0-132-82 kubenswrapper[2577]: I1030 19:15:41.810022    2577 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-cluster-storage-operator/volume-data-source-validator-58c9f5f46d-p42xv"]
Oct 30 19:15:41.810040 ip-10-0-132-82 kubenswrapper[2577]: I1030 19:15:41.810032    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/e3206ddf-35de-4b7c-b979-682f5f666a34-image-registry-private-configuration\") pod \"image-registry-6fcb4848df-8l8cp\" (UID: \"e3206ddf-35de-4b7c-b979-682f5f666a34\") " pod="openshift-image-registry/image-registry-6fcb4848df-8l8cp"
Oct 30 19:15:41.810233 ip-10-0-132-82 kubenswrapper[2577]: I1030 19:15:41.810042    2577 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-np94b"]
Oct 30 19:15:41.810233 ip-10-0-132-82 kubenswrapper[2577]: I1030 19:15:41.810064    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-config\" (UniqueName: \"kubernetes.io/configmap/56274b23-2400-43d4-9686-c928e20d47de-samples-operator-config\") pod \"cluster-samples-operator-6ddb6c8b94-np94b\" (UID: \"56274b23-2400-43d4-9686-c928e20d47de\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-np94b"
Oct 30 19:15:41.810233 ip-10-0-132-82 kubenswrapper[2577]: I1030 19:15:41.810090    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/2bd84dc4-e96a-43f1-85bf-e1ea4005c193-tmp\") pod \"insights-operator-7c4986d449-6jdkq\" (UID: \"2bd84dc4-e96a-43f1-85bf-e1ea4005c193\") " pod="openshift-insights/insights-operator-7c4986d449-6jdkq"
Oct 30 19:15:41.810233 ip-10-0-132-82 kubenswrapper[2577]: I1030 19:15:41.810106    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/2bd84dc4-e96a-43f1-85bf-e1ea4005c193-serving-cert\") pod \"insights-operator-7c4986d449-6jdkq\" (UID: \"2bd84dc4-e96a-43f1-85bf-e1ea4005c193\") " pod="openshift-insights/insights-operator-7c4986d449-6jdkq"
Oct 30 19:15:41.810233 ip-10-0-132-82 kubenswrapper[2577]: I1030 19:15:41.810124    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"snapshots\" (UniqueName: \"kubernetes.io/empty-dir/2bd84dc4-e96a-43f1-85bf-e1ea4005c193-snapshots\") pod \"insights-operator-7c4986d449-6jdkq\" (UID: \"2bd84dc4-e96a-43f1-85bf-e1ea4005c193\") " pod="openshift-insights/insights-operator-7c4986d449-6jdkq"
Oct 30 19:15:41.810233 ip-10-0-132-82 kubenswrapper[2577]: I1030 19:15:41.810151    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/0e179af5-9f52-4941-b4f5-5e3b8789da11-image-registry-private-configuration\") pod \"image-registry-d97c68fd4-htsl8\" (UID: \"0e179af5-9f52-4941-b4f5-5e3b8789da11\") " pod="openshift-image-registry/image-registry-d97c68fd4-htsl8"
Oct 30 19:15:41.810233 ip-10-0-132-82 kubenswrapper[2577]: I1030 19:15:41.810177    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/11db9cec-653f-4aee-8937-0cf5b367efac-trusted-ca\") pod \"console-operator-7cd56b8dbc-tg5xj\" (UID: \"11db9cec-653f-4aee-8937-0cf5b367efac\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-tg5xj"
Oct 30 19:15:41.810233 ip-10-0-132-82 kubenswrapper[2577]: I1030 19:15:41.810040    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-bb9cj"
Oct 30 19:15:41.810233 ip-10-0-132-82 kubenswrapper[2577]: I1030 19:15:41.810204    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/f9a06263-cb99-4004-bffc-581d66ce5ebf-config\") pod \"kube-storage-version-migrator-operator-7d4bc45dd6-scljn\" (UID: \"f9a06263-cb99-4004-bffc-581d66ce5ebf\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-scljn"
Oct 30 19:15:41.810233 ip-10-0-132-82 kubenswrapper[2577]: I1030 19:15:41.810226    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/e3206ddf-35de-4b7c-b979-682f5f666a34-registry-tls\") pod \"image-registry-6fcb4848df-8l8cp\" (UID: \"e3206ddf-35de-4b7c-b979-682f5f666a34\") " pod="openshift-image-registry/image-registry-6fcb4848df-8l8cp"
Oct 30 19:15:41.810950 ip-10-0-132-82 kubenswrapper[2577]: I1030 19:15:41.810254    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wt5rq\" (UniqueName: \"kubernetes.io/projected/56274b23-2400-43d4-9686-c928e20d47de-kube-api-access-wt5rq\") pod \"cluster-samples-operator-6ddb6c8b94-np94b\" (UID: \"56274b23-2400-43d4-9686-c928e20d47de\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-np94b"
Oct 30 19:15:41.810950 ip-10-0-132-82 kubenswrapper[2577]: I1030 19:15:41.810285    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/2c7c7060-def6-4737-8fa9-6889309b7463-nginx-conf\") pod \"networking-console-plugin-86c478bb88-h4twn\" (UID: \"2c7c7060-def6-4737-8fa9-6889309b7463\") " pod="openshift-network-console/networking-console-plugin-86c478bb88-h4twn"
Oct 30 19:15:41.810950 ip-10-0-132-82 kubenswrapper[2577]: I1030 19:15:41.810316    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zj4df\" (UniqueName: \"kubernetes.io/projected/0e179af5-9f52-4941-b4f5-5e3b8789da11-kube-api-access-zj4df\") pod \"image-registry-d97c68fd4-htsl8\" (UID: \"0e179af5-9f52-4941-b4f5-5e3b8789da11\") " pod="openshift-image-registry/image-registry-d97c68fd4-htsl8"
Oct 30 19:15:41.810950 ip-10-0-132-82 kubenswrapper[2577]: I1030 19:15:41.810341    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/0e179af5-9f52-4941-b4f5-5e3b8789da11-trusted-ca\") pod \"image-registry-d97c68fd4-htsl8\" (UID: \"0e179af5-9f52-4941-b4f5-5e3b8789da11\") " pod="openshift-image-registry/image-registry-d97c68fd4-htsl8"
Oct 30 19:15:41.810950 ip-10-0-132-82 kubenswrapper[2577]: I1030 19:15:41.810364    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/11db9cec-653f-4aee-8937-0cf5b367efac-serving-cert\") pod \"console-operator-7cd56b8dbc-tg5xj\" (UID: \"11db9cec-653f-4aee-8937-0cf5b367efac\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-tg5xj"
Oct 30 19:15:41.810950 ip-10-0-132-82 kubenswrapper[2577]: I1030 19:15:41.810390    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/e3206ddf-35de-4b7c-b979-682f5f666a34-ca-trust-extracted\") pod \"image-registry-6fcb4848df-8l8cp\" (UID: \"e3206ddf-35de-4b7c-b979-682f5f666a34\") " pod="openshift-image-registry/image-registry-6fcb4848df-8l8cp"
Oct 30 19:15:41.810950 ip-10-0-132-82 kubenswrapper[2577]: I1030 19:15:41.810423    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4rmts\" (UniqueName: \"kubernetes.io/projected/2bd84dc4-e96a-43f1-85bf-e1ea4005c193-kube-api-access-4rmts\") pod \"insights-operator-7c4986d449-6jdkq\" (UID: \"2bd84dc4-e96a-43f1-85bf-e1ea4005c193\") " pod="openshift-insights/insights-operator-7c4986d449-6jdkq"
Oct 30 19:15:41.810950 ip-10-0-132-82 kubenswrapper[2577]: I1030 19:15:41.810448    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/87082646-75c3-43b5-864b-3478e9a7373d-metrics-certs\") pod \"router-default-776df59447-8nrhh\" (UID: \"87082646-75c3-43b5-864b-3478e9a7373d\") " pod="openshift-ingress/router-default-776df59447-8nrhh"
Oct 30 19:15:41.810950 ip-10-0-132-82 kubenswrapper[2577]: I1030 19:15:41.810485    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"default-certificate\" (UniqueName: \"kubernetes.io/secret/87082646-75c3-43b5-864b-3478e9a7373d-default-certificate\") pod \"router-default-776df59447-8nrhh\" (UID: \"87082646-75c3-43b5-864b-3478e9a7373d\") " pod="openshift-ingress/router-default-776df59447-8nrhh"
Oct 30 19:15:41.810950 ip-10-0-132-82 kubenswrapper[2577]: I1030 19:15:41.810517    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rfxzq\" (UniqueName: \"kubernetes.io/projected/4ba601a6-2a44-4408-8c5b-2c29fd096fed-kube-api-access-rfxzq\") pod \"service-ca-operator-544d749c8c-nbfv2\" (UID: \"4ba601a6-2a44-4408-8c5b-2c29fd096fed\") " pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-nbfv2"
Oct 30 19:15:41.810950 ip-10-0-132-82 kubenswrapper[2577]: I1030 19:15:41.810536    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/2bd84dc4-e96a-43f1-85bf-e1ea4005c193-tmp\") pod \"insights-operator-7c4986d449-6jdkq\" (UID: \"2bd84dc4-e96a-43f1-85bf-e1ea4005c193\") " pod="openshift-insights/insights-operator-7c4986d449-6jdkq"
Oct 30 19:15:41.810950 ip-10-0-132-82 kubenswrapper[2577]: I1030 19:15:41.810547    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/0e179af5-9f52-4941-b4f5-5e3b8789da11-registry-tls\") pod \"image-registry-d97c68fd4-htsl8\" (UID: \"0e179af5-9f52-4941-b4f5-5e3b8789da11\") " pod="openshift-image-registry/image-registry-d97c68fd4-htsl8"
Oct 30 19:15:41.810950 ip-10-0-132-82 kubenswrapper[2577]: I1030 19:15:41.810571    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/4ba601a6-2a44-4408-8c5b-2c29fd096fed-serving-cert\") pod \"service-ca-operator-544d749c8c-nbfv2\" (UID: \"4ba601a6-2a44-4408-8c5b-2c29fd096fed\") " pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-nbfv2"
Oct 30 19:15:41.810950 ip-10-0-132-82 kubenswrapper[2577]: I1030 19:15:41.810070    2577 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/router-default-776df59447-8nrhh"]
Oct 30 19:15:41.810950 ip-10-0-132-82 kubenswrapper[2577]: I1030 19:15:41.810618    2577 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-6fcb4848df-8l8cp"]
Oct 30 19:15:41.810950 ip-10-0-132-82 kubenswrapper[2577]: I1030 19:15:41.810641    2577 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-source-6f78d59f99-2trjq"]
Oct 30 19:15:41.810950 ip-10-0-132-82 kubenswrapper[2577]: I1030 19:15:41.810664    2577 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-scljn"]
Oct 30 19:15:41.810950 ip-10-0-132-82 kubenswrapper[2577]: I1030 19:15:41.810695    2577 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-network-console/networking-console-plugin-86c478bb88-h4twn"]
Oct 30 19:15:41.810950 ip-10-0-132-82 kubenswrapper[2577]: I1030 19:15:41.810714    2577 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-8rwjm"]
Oct 30 19:15:41.810950 ip-10-0-132-82 kubenswrapper[2577]: I1030 19:15:41.810732    2577 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-bb9cj"]
Oct 30 19:15:41.812010 ip-10-0-132-82 kubenswrapper[2577]: I1030 19:15:41.810738    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"snapshots\" (UniqueName: \"kubernetes.io/empty-dir/2bd84dc4-e96a-43f1-85bf-e1ea4005c193-snapshots\") pod \"insights-operator-7c4986d449-6jdkq\" (UID: \"2bd84dc4-e96a-43f1-85bf-e1ea4005c193\") " pod="openshift-insights/insights-operator-7c4986d449-6jdkq"
Oct 30 19:15:41.812010 ip-10-0-132-82 kubenswrapper[2577]: I1030 19:15:41.810766    2577 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-d97c68fd4-htsl8"]
Oct 30 19:15:41.812010 ip-10-0-132-82 kubenswrapper[2577]: I1030 19:15:41.810799    2577 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-service-ca-operator/service-ca-operator-544d749c8c-nbfv2"]
Oct 30 19:15:41.812010 ip-10-0-132-82 kubenswrapper[2577]: I1030 19:15:41.810819    2577 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-console-operator/console-operator-7cd56b8dbc-tg5xj"]
Oct 30 19:15:41.812010 ip-10-0-132-82 kubenswrapper[2577]: I1030 19:15:41.810794    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"samples-operator-config\" (UniqueName: \"kubernetes.io/configmap/56274b23-2400-43d4-9686-c928e20d47de-samples-operator-config\") pod \"cluster-samples-operator-6ddb6c8b94-np94b\" (UID: \"56274b23-2400-43d4-9686-c928e20d47de\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-np94b"
Oct 30 19:15:41.812010 ip-10-0-132-82 kubenswrapper[2577]: I1030 19:15:41.810839    2577 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-operator-7c4986d449-6jdkq"]
Oct 30 19:15:41.812010 ip-10-0-132-82 kubenswrapper[2577]: I1030 19:15:41.810859    2577 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/cluster-monitoring-operator-884576c57-wkldq"]
Oct 30 19:15:41.812010 ip-10-0-132-82 kubenswrapper[2577]: I1030 19:15:41.810815    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/0e179af5-9f52-4941-b4f5-5e3b8789da11-ca-trust-extracted\") pod \"image-registry-d97c68fd4-htsl8\" (UID: \"0e179af5-9f52-4941-b4f5-5e3b8789da11\") " pod="openshift-image-registry/image-registry-d97c68fd4-htsl8"
Oct 30 19:15:41.812010 ip-10-0-132-82 kubenswrapper[2577]: I1030 19:15:41.810918    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/f9a06263-cb99-4004-bffc-581d66ce5ebf-serving-cert\") pod \"kube-storage-version-migrator-operator-7d4bc45dd6-scljn\" (UID: \"f9a06263-cb99-4004-bffc-581d66ce5ebf\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-scljn"
Oct 30 19:15:41.812010 ip-10-0-132-82 kubenswrapper[2577]: E1030 19:15:41.810921    2577 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found
Oct 30 19:15:41.812010 ip-10-0-132-82 kubenswrapper[2577]: I1030 19:15:41.810983    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/e3206ddf-35de-4b7c-b979-682f5f666a34-ca-trust-extracted\") pod \"image-registry-6fcb4848df-8l8cp\" (UID: \"e3206ddf-35de-4b7c-b979-682f5f666a34\") " pod="openshift-image-registry/image-registry-6fcb4848df-8l8cp"
Oct 30 19:15:41.812010 ip-10-0-132-82 kubenswrapper[2577]: E1030 19:15:41.811096    2577 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Oct 30 19:15:41.812010 ip-10-0-132-82 kubenswrapper[2577]: E1030 19:15:41.811112    2577 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-6fcb4848df-8l8cp: secret "image-registry-tls" not found
Oct 30 19:15:41.812010 ip-10-0-132-82 kubenswrapper[2577]: E1030 19:15:41.811137    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/87082646-75c3-43b5-864b-3478e9a7373d-metrics-certs podName:87082646-75c3-43b5-864b-3478e9a7373d nodeName:}" failed. No retries permitted until 2026-10-30 19:15:42.311119448 +0000 UTC m=+32.717039865 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/87082646-75c3-43b5-864b-3478e9a7373d-metrics-certs") pod "router-default-776df59447-8nrhh" (UID: "87082646-75c3-43b5-864b-3478e9a7373d") : secret "router-metrics-certs-default" not found
Oct 30 19:15:41.812010 ip-10-0-132-82 kubenswrapper[2577]: E1030 19:15:41.811159    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/e3206ddf-35de-4b7c-b979-682f5f666a34-registry-tls podName:e3206ddf-35de-4b7c-b979-682f5f666a34 nodeName:}" failed. No retries permitted until 2026-10-30 19:15:42.311149113 +0000 UTC m=+32.717069532 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/e3206ddf-35de-4b7c-b979-682f5f666a34-registry-tls") pod "image-registry-6fcb4848df-8l8cp" (UID: "e3206ddf-35de-4b7c-b979-682f5f666a34") : secret "image-registry-tls" not found
Oct 30 19:15:41.812010 ip-10-0-132-82 kubenswrapper[2577]: I1030 19:15:41.811214    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/87082646-75c3-43b5-864b-3478e9a7373d-service-ca-bundle\") pod \"router-default-776df59447-8nrhh\" (UID: \"87082646-75c3-43b5-864b-3478e9a7373d\") " pod="openshift-ingress/router-default-776df59447-8nrhh"
Oct 30 19:15:41.812010 ip-10-0-132-82 kubenswrapper[2577]: I1030 19:15:41.811256    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5xgwq\" (UniqueName: \"kubernetes.io/projected/11db9cec-653f-4aee-8937-0cf5b367efac-kube-api-access-5xgwq\") pod \"console-operator-7cd56b8dbc-tg5xj\" (UID: \"11db9cec-653f-4aee-8937-0cf5b367efac\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-tg5xj"
Oct 30 19:15:41.812010 ip-10-0-132-82 kubenswrapper[2577]: I1030 19:15:41.811346    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/configmap/f9a06263-cb99-4004-bffc-581d66ce5ebf-config\") pod \"kube-storage-version-migrator-operator-7d4bc45dd6-scljn\" (UID: \"f9a06263-cb99-4004-bffc-581d66ce5ebf\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-scljn"
Oct 30 19:15:41.813031 ip-10-0-132-82 kubenswrapper[2577]: E1030 19:15:41.811365    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/87082646-75c3-43b5-864b-3478e9a7373d-service-ca-bundle podName:87082646-75c3-43b5-864b-3478e9a7373d nodeName:}" failed. No retries permitted until 2026-10-30 19:15:42.311351734 +0000 UTC m=+32.717272165 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/87082646-75c3-43b5-864b-3478e9a7373d-service-ca-bundle") pod "router-default-776df59447-8nrhh" (UID: "87082646-75c3-43b5-864b-3478e9a7373d") : configmap references non-existent config key: service-ca.crt
Oct 30 19:15:41.813031 ip-10-0-132-82 kubenswrapper[2577]: I1030 19:15:41.811413    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/e3206ddf-35de-4b7c-b979-682f5f666a34-bound-sa-token\") pod \"image-registry-6fcb4848df-8l8cp\" (UID: \"e3206ddf-35de-4b7c-b979-682f5f666a34\") " pod="openshift-image-registry/image-registry-6fcb4848df-8l8cp"
Oct 30 19:15:41.813031 ip-10-0-132-82 kubenswrapper[2577]: I1030 19:15:41.811474    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/56274b23-2400-43d4-9686-c928e20d47de-samples-operator-tls\") pod \"cluster-samples-operator-6ddb6c8b94-np94b\" (UID: \"56274b23-2400-43d4-9686-c928e20d47de\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-np94b"
Oct 30 19:15:41.813031 ip-10-0-132-82 kubenswrapper[2577]: I1030 19:15:41.811530    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/2c7c7060-def6-4737-8fa9-6889309b7463-networking-console-plugin-cert\") pod \"networking-console-plugin-86c478bb88-h4twn\" (UID: \"2c7c7060-def6-4737-8fa9-6889309b7463\") " pod="openshift-network-console/networking-console-plugin-86c478bb88-h4twn"
Oct 30 19:15:41.813031 ip-10-0-132-82 kubenswrapper[2577]: E1030 19:15:41.811607    2577 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found
Oct 30 19:15:41.813031 ip-10-0-132-82 kubenswrapper[2577]: E1030 19:15:41.811650    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/56274b23-2400-43d4-9686-c928e20d47de-samples-operator-tls podName:56274b23-2400-43d4-9686-c928e20d47de nodeName:}" failed. No retries permitted until 2026-10-30 19:15:42.311636136 +0000 UTC m=+32.717556552 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/56274b23-2400-43d4-9686-c928e20d47de-samples-operator-tls") pod "cluster-samples-operator-6ddb6c8b94-np94b" (UID: "56274b23-2400-43d4-9686-c928e20d47de") : secret "samples-operator-tls" not found
Oct 30 19:15:41.813031 ip-10-0-132-82 kubenswrapper[2577]: I1030 19:15:41.811738    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/e3206ddf-35de-4b7c-b979-682f5f666a34-installation-pull-secrets\") pod \"image-registry-6fcb4848df-8l8cp\" (UID: \"e3206ddf-35de-4b7c-b979-682f5f666a34\") " pod="openshift-image-registry/image-registry-6fcb4848df-8l8cp"
Oct 30 19:15:41.813031 ip-10-0-132-82 kubenswrapper[2577]: I1030 19:15:41.811908    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7xf2g\" (UniqueName: \"kubernetes.io/projected/4d213db4-73c5-4f50-9930-6d26b880dcf7-kube-api-access-7xf2g\") pod \"volume-data-source-validator-58c9f5f46d-p42xv\" (UID: \"4d213db4-73c5-4f50-9930-6d26b880dcf7\") " pod="openshift-cluster-storage-operator/volume-data-source-validator-58c9f5f46d-p42xv"
Oct 30 19:15:41.813031 ip-10-0-132-82 kubenswrapper[2577]: I1030 19:15:41.811946    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/0e179af5-9f52-4941-b4f5-5e3b8789da11-registry-certificates\") pod \"image-registry-d97c68fd4-htsl8\" (UID: \"0e179af5-9f52-4941-b4f5-5e3b8789da11\") " pod="openshift-image-registry/image-registry-d97c68fd4-htsl8"
Oct 30 19:15:41.813031 ip-10-0-132-82 kubenswrapper[2577]: I1030 19:15:41.812017    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/0e179af5-9f52-4941-b4f5-5e3b8789da11-installation-pull-secrets\") pod \"image-registry-d97c68fd4-htsl8\" (UID: \"0e179af5-9f52-4941-b4f5-5e3b8789da11\") " pod="openshift-image-registry/image-registry-d97c68fd4-htsl8"
Oct 30 19:15:41.813031 ip-10-0-132-82 kubenswrapper[2577]: I1030 19:15:41.812048    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/e3206ddf-35de-4b7c-b979-682f5f666a34-registry-certificates\") pod \"image-registry-6fcb4848df-8l8cp\" (UID: \"e3206ddf-35de-4b7c-b979-682f5f666a34\") " pod="openshift-image-registry/image-registry-6fcb4848df-8l8cp"
Oct 30 19:15:41.813031 ip-10-0-132-82 kubenswrapper[2577]: I1030 19:15:41.812075    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/2bd84dc4-e96a-43f1-85bf-e1ea4005c193-trusted-ca-bundle\") pod \"insights-operator-7c4986d449-6jdkq\" (UID: \"2bd84dc4-e96a-43f1-85bf-e1ea4005c193\") " pod="openshift-insights/insights-operator-7c4986d449-6jdkq"
Oct 30 19:15:41.813031 ip-10-0-132-82 kubenswrapper[2577]: I1030 19:15:41.812100    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8s7g5\" (UniqueName: \"kubernetes.io/projected/87082646-75c3-43b5-864b-3478e9a7373d-kube-api-access-8s7g5\") pod \"router-default-776df59447-8nrhh\" (UID: \"87082646-75c3-43b5-864b-3478e9a7373d\") " pod="openshift-ingress/router-default-776df59447-8nrhh"
Oct 30 19:15:41.813031 ip-10-0-132-82 kubenswrapper[2577]: I1030 19:15:41.812136    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/2bd84dc4-e96a-43f1-85bf-e1ea4005c193-service-ca-bundle\") pod \"insights-operator-7c4986d449-6jdkq\" (UID: \"2bd84dc4-e96a-43f1-85bf-e1ea4005c193\") " pod="openshift-insights/insights-operator-7c4986d449-6jdkq"
Oct 30 19:15:41.813740 ip-10-0-132-82 kubenswrapper[2577]: I1030 19:15:41.812164    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-w9h4f\" (UniqueName: \"kubernetes.io/projected/e3206ddf-35de-4b7c-b979-682f5f666a34-kube-api-access-w9h4f\") pod \"image-registry-6fcb4848df-8l8cp\" (UID: \"e3206ddf-35de-4b7c-b979-682f5f666a34\") " pod="openshift-image-registry/image-registry-6fcb4848df-8l8cp"
Oct 30 19:15:41.813740 ip-10-0-132-82 kubenswrapper[2577]: I1030 19:15:41.812194    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/0e179af5-9f52-4941-b4f5-5e3b8789da11-bound-sa-token\") pod \"image-registry-d97c68fd4-htsl8\" (UID: \"0e179af5-9f52-4941-b4f5-5e3b8789da11\") " pod="openshift-image-registry/image-registry-d97c68fd4-htsl8"
Oct 30 19:15:41.813740 ip-10-0-132-82 kubenswrapper[2577]: I1030 19:15:41.812219    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/4ba601a6-2a44-4408-8c5b-2c29fd096fed-config\") pod \"service-ca-operator-544d749c8c-nbfv2\" (UID: \"4ba601a6-2a44-4408-8c5b-2c29fd096fed\") " pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-nbfv2"
Oct 30 19:15:41.813740 ip-10-0-132-82 kubenswrapper[2577]: I1030 19:15:41.812260    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qfw7m\" (UniqueName: \"kubernetes.io/projected/f9a06263-cb99-4004-bffc-581d66ce5ebf-kube-api-access-qfw7m\") pod \"kube-storage-version-migrator-operator-7d4bc45dd6-scljn\" (UID: \"f9a06263-cb99-4004-bffc-581d66ce5ebf\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-scljn"
Oct 30 19:15:41.813740 ip-10-0-132-82 kubenswrapper[2577]: I1030 19:15:41.812284    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"stats-auth\" (UniqueName: \"kubernetes.io/secret/87082646-75c3-43b5-864b-3478e9a7373d-stats-auth\") pod \"router-default-776df59447-8nrhh\" (UID: \"87082646-75c3-43b5-864b-3478e9a7373d\") " pod="openshift-ingress/router-default-776df59447-8nrhh"
Oct 30 19:15:41.813740 ip-10-0-132-82 kubenswrapper[2577]: I1030 19:15:41.812310    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/e3206ddf-35de-4b7c-b979-682f5f666a34-trusted-ca\") pod \"image-registry-6fcb4848df-8l8cp\" (UID: \"e3206ddf-35de-4b7c-b979-682f5f666a34\") " pod="openshift-image-registry/image-registry-6fcb4848df-8l8cp"
Oct 30 19:15:41.813740 ip-10-0-132-82 kubenswrapper[2577]: I1030 19:15:41.812333    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/11db9cec-653f-4aee-8937-0cf5b367efac-config\") pod \"console-operator-7cd56b8dbc-tg5xj\" (UID: \"11db9cec-653f-4aee-8937-0cf5b367efac\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-tg5xj"
Oct 30 19:15:41.813740 ip-10-0-132-82 kubenswrapper[2577]: I1030 19:15:41.812929    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/e3206ddf-35de-4b7c-b979-682f5f666a34-registry-certificates\") pod \"image-registry-6fcb4848df-8l8cp\" (UID: \"e3206ddf-35de-4b7c-b979-682f5f666a34\") " pod="openshift-image-registry/image-registry-6fcb4848df-8l8cp"
Oct 30 19:15:41.813740 ip-10-0-132-82 kubenswrapper[2577]: I1030 19:15:41.812997    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/2bd84dc4-e96a-43f1-85bf-e1ea4005c193-service-ca-bundle\") pod \"insights-operator-7c4986d449-6jdkq\" (UID: \"2bd84dc4-e96a-43f1-85bf-e1ea4005c193\") " pod="openshift-insights/insights-operator-7c4986d449-6jdkq"
Oct 30 19:15:41.813740 ip-10-0-132-82 kubenswrapper[2577]: I1030 19:15:41.813403    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/e3206ddf-35de-4b7c-b979-682f5f666a34-trusted-ca\") pod \"image-registry-6fcb4848df-8l8cp\" (UID: \"e3206ddf-35de-4b7c-b979-682f5f666a34\") " pod="openshift-image-registry/image-registry-6fcb4848df-8l8cp"
Oct 30 19:15:41.813740 ip-10-0-132-82 kubenswrapper[2577]: I1030 19:15:41.813538    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/2bd84dc4-e96a-43f1-85bf-e1ea4005c193-serving-cert\") pod \"insights-operator-7c4986d449-6jdkq\" (UID: \"2bd84dc4-e96a-43f1-85bf-e1ea4005c193\") " pod="openshift-insights/insights-operator-7c4986d449-6jdkq"
Oct 30 19:15:41.814314 ip-10-0-132-82 kubenswrapper[2577]: I1030 19:15:41.813796    2577 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"network-diagnostics-dockercfg-q7v7l\""
Oct 30 19:15:41.814314 ip-10-0-132-82 kubenswrapper[2577]: I1030 19:15:41.813866    2577 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-console\"/\"networking-console-plugin-cert\""
Oct 30 19:15:41.814314 ip-10-0-132-82 kubenswrapper[2577]: I1030 19:15:41.814023    2577 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-console\"/\"networking-console-plugin-dockercfg-p7m6n\""
Oct 30 19:15:41.814314 ip-10-0-132-82 kubenswrapper[2577]: I1030 19:15:41.814029    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"default-certificate\" (UniqueName: \"kubernetes.io/secret/87082646-75c3-43b5-864b-3478e9a7373d-default-certificate\") pod \"router-default-776df59447-8nrhh\" (UID: \"87082646-75c3-43b5-864b-3478e9a7373d\") " pod="openshift-ingress/router-default-776df59447-8nrhh"
Oct 30 19:15:41.814314 ip-10-0-132-82 kubenswrapper[2577]: I1030 19:15:41.814085    2577 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-service-ca-operator\"/\"serving-cert\""
Oct 30 19:15:41.814314 ip-10-0-132-82 kubenswrapper[2577]: I1030 19:15:41.814139    2577 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console-operator\"/\"serving-cert\""
Oct 30 19:15:41.814314 ip-10-0-132-82 kubenswrapper[2577]: I1030 19:15:41.814190    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/2bd84dc4-e96a-43f1-85bf-e1ea4005c193-trusted-ca-bundle\") pod \"insights-operator-7c4986d449-6jdkq\" (UID: \"2bd84dc4-e96a-43f1-85bf-e1ea4005c193\") " pod="openshift-insights/insights-operator-7c4986d449-6jdkq"
Oct 30 19:15:41.814624 ip-10-0-132-82 kubenswrapper[2577]: I1030 19:15:41.814429    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/e3206ddf-35de-4b7c-b979-682f5f666a34-installation-pull-secrets\") pod \"image-registry-6fcb4848df-8l8cp\" (UID: \"e3206ddf-35de-4b7c-b979-682f5f666a34\") " pod="openshift-image-registry/image-registry-6fcb4848df-8l8cp"
Oct 30 19:15:41.814624 ip-10-0-132-82 kubenswrapper[2577]: I1030 19:15:41.814497    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/e3206ddf-35de-4b7c-b979-682f5f666a34-image-registry-private-configuration\") pod \"image-registry-6fcb4848df-8l8cp\" (UID: \"e3206ddf-35de-4b7c-b979-682f5f666a34\") " pod="openshift-image-registry/image-registry-6fcb4848df-8l8cp"
Oct 30 19:15:41.814808 ip-10-0-132-82 kubenswrapper[2577]: I1030 19:15:41.814793    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/f9a06263-cb99-4004-bffc-581d66ce5ebf-serving-cert\") pod \"kube-storage-version-migrator-operator-7d4bc45dd6-scljn\" (UID: \"f9a06263-cb99-4004-bffc-581d66ce5ebf\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-scljn"
Oct 30 19:15:41.815386 ip-10-0-132-82 kubenswrapper[2577]: I1030 19:15:41.815367    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"stats-auth\" (UniqueName: \"kubernetes.io/secret/87082646-75c3-43b5-864b-3478e9a7373d-stats-auth\") pod \"router-default-776df59447-8nrhh\" (UID: \"87082646-75c3-43b5-864b-3478e9a7373d\") " pod="openshift-ingress/router-default-776df59447-8nrhh"
Oct 30 19:15:41.816008 ip-10-0-132-82 kubenswrapper[2577]: I1030 19:15:41.815990    2577 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"cluster-monitoring-operator-tls\""
Oct 30 19:15:41.829701 ip-10-0-132-82 kubenswrapper[2577]: I1030 19:15:41.829684    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kas-connection-checker-685895c7c9-tvmks"
Oct 30 19:15:41.839858 ip-10-0-132-82 kubenswrapper[2577]: I1030 19:15:41.839843    2577 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"canary-serving-cert\""
Oct 30 19:15:41.839933 ip-10-0-132-82 kubenswrapper[2577]: I1030 19:15:41.839911    2577 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"ingress-canary-dockercfg-qjhd5\""
Oct 30 19:15:41.847546 ip-10-0-132-82 kubenswrapper[2577]: I1030 19:15:41.847532    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kas-connection-checker-685895c7c9-9h85f"
Oct 30 19:15:41.862050 ip-10-0-132-82 kubenswrapper[2577]: I1030 19:15:41.862034    2577 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-default-metrics-tls\""
Oct 30 19:15:41.862212 ip-10-0-132-82 kubenswrapper[2577]: I1030 19:15:41.862201    2577 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-hk6fn\""
Oct 30 19:15:41.865770 ip-10-0-132-82 kubenswrapper[2577]: I1030 19:15:41.865753    2577 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-root-ca.crt\""
Oct 30 19:15:41.865770 ip-10-0-132-82 kubenswrapper[2577]: I1030 19:15:41.865766    2577 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"openshift-service-ca.crt\""
Oct 30 19:15:41.865919 ip-10-0-132-82 kubenswrapper[2577]: I1030 19:15:41.865808    2577 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"openshift-service-ca.crt\""
Oct 30 19:15:41.865919 ip-10-0-132-82 kubenswrapper[2577]: I1030 19:15:41.865841    2577 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\""
Oct 30 19:15:41.865919 ip-10-0-132-82 kubenswrapper[2577]: I1030 19:15:41.865858    2577 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"kube-root-ca.crt\""
Oct 30 19:15:41.865919 ip-10-0-132-82 kubenswrapper[2577]: I1030 19:15:41.865858    2577 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\""
Oct 30 19:15:41.865919 ip-10-0-132-82 kubenswrapper[2577]: I1030 19:15:41.865904    2577 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"openshift-service-ca.crt\""
Oct 30 19:15:41.866166 ip-10-0-132-82 kubenswrapper[2577]: I1030 19:15:41.865941    2577 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"dns-default\""
Oct 30 19:15:41.866166 ip-10-0-132-82 kubenswrapper[2577]: I1030 19:15:41.866105    2577 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"kube-root-ca.crt\""
Oct 30 19:15:41.866237 ip-10-0-132-82 kubenswrapper[2577]: I1030 19:15:41.866154    2577 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"console-operator-config\""
Oct 30 19:15:41.866816 ip-10-0-132-82 kubenswrapper[2577]: I1030 19:15:41.866795    2577 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"telemetry-config\""
Oct 30 19:15:41.867392 ip-10-0-132-82 kubenswrapper[2577]: I1030 19:15:41.867142    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kas-connection-checker-685895c7c9-p8kjs"
Oct 30 19:15:41.867488 ip-10-0-132-82 kubenswrapper[2577]: I1030 19:15:41.867448    2577 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"trusted-ca\""
Oct 30 19:15:41.870676 ip-10-0-132-82 kubenswrapper[2577]: I1030 19:15:41.870410    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/e3206ddf-35de-4b7c-b979-682f5f666a34-bound-sa-token\") pod \"image-registry-6fcb4848df-8l8cp\" (UID: \"e3206ddf-35de-4b7c-b979-682f5f666a34\") " pod="openshift-image-registry/image-registry-6fcb4848df-8l8cp"
Oct 30 19:15:41.870904 ip-10-0-132-82 kubenswrapper[2577]: I1030 19:15:41.870881    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qfw7m\" (UniqueName: \"kubernetes.io/projected/f9a06263-cb99-4004-bffc-581d66ce5ebf-kube-api-access-qfw7m\") pod \"kube-storage-version-migrator-operator-7d4bc45dd6-scljn\" (UID: \"f9a06263-cb99-4004-bffc-581d66ce5ebf\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-scljn"
Oct 30 19:15:41.871433 ip-10-0-132-82 kubenswrapper[2577]: I1030 19:15:41.871340    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4rmts\" (UniqueName: \"kubernetes.io/projected/2bd84dc4-e96a-43f1-85bf-e1ea4005c193-kube-api-access-4rmts\") pod \"insights-operator-7c4986d449-6jdkq\" (UID: \"2bd84dc4-e96a-43f1-85bf-e1ea4005c193\") " pod="openshift-insights/insights-operator-7c4986d449-6jdkq"
Oct 30 19:15:41.871433 ip-10-0-132-82 kubenswrapper[2577]: I1030 19:15:41.871353    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wt5rq\" (UniqueName: \"kubernetes.io/projected/56274b23-2400-43d4-9686-c928e20d47de-kube-api-access-wt5rq\") pod \"cluster-samples-operator-6ddb6c8b94-np94b\" (UID: \"56274b23-2400-43d4-9686-c928e20d47de\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-np94b"
Oct 30 19:15:41.871433 ip-10-0-132-82 kubenswrapper[2577]: I1030 19:15:41.871341    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7xf2g\" (UniqueName: \"kubernetes.io/projected/4d213db4-73c5-4f50-9930-6d26b880dcf7-kube-api-access-7xf2g\") pod \"volume-data-source-validator-58c9f5f46d-p42xv\" (UID: \"4d213db4-73c5-4f50-9930-6d26b880dcf7\") " pod="openshift-cluster-storage-operator/volume-data-source-validator-58c9f5f46d-p42xv"
Oct 30 19:15:41.871433 ip-10-0-132-82 kubenswrapper[2577]: I1030 19:15:41.871381    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-w9h4f\" (UniqueName: \"kubernetes.io/projected/e3206ddf-35de-4b7c-b979-682f5f666a34-kube-api-access-w9h4f\") pod \"image-registry-6fcb4848df-8l8cp\" (UID: \"e3206ddf-35de-4b7c-b979-682f5f666a34\") " pod="openshift-image-registry/image-registry-6fcb4848df-8l8cp"
Oct 30 19:15:41.873366 ip-10-0-132-82 kubenswrapper[2577]: I1030 19:15:41.873345    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8s7g5\" (UniqueName: \"kubernetes.io/projected/87082646-75c3-43b5-864b-3478e9a7373d-kube-api-access-8s7g5\") pod \"router-default-776df59447-8nrhh\" (UID: \"87082646-75c3-43b5-864b-3478e9a7373d\") " pod="openshift-ingress/router-default-776df59447-8nrhh"
Oct 30 19:15:41.912934 ip-10-0-132-82 kubenswrapper[2577]: I1030 19:15:41.912902    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zj4df\" (UniqueName: \"kubernetes.io/projected/0e179af5-9f52-4941-b4f5-5e3b8789da11-kube-api-access-zj4df\") pod \"image-registry-d97c68fd4-htsl8\" (UID: \"0e179af5-9f52-4941-b4f5-5e3b8789da11\") " pod="openshift-image-registry/image-registry-d97c68fd4-htsl8"
Oct 30 19:15:41.913052 ip-10-0-132-82 kubenswrapper[2577]: I1030 19:15:41.912952    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/0e179af5-9f52-4941-b4f5-5e3b8789da11-trusted-ca\") pod \"image-registry-d97c68fd4-htsl8\" (UID: \"0e179af5-9f52-4941-b4f5-5e3b8789da11\") " pod="openshift-image-registry/image-registry-d97c68fd4-htsl8"
Oct 30 19:15:41.913052 ip-10-0-132-82 kubenswrapper[2577]: I1030 19:15:41.913006    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/11db9cec-653f-4aee-8937-0cf5b367efac-serving-cert\") pod \"console-operator-7cd56b8dbc-tg5xj\" (UID: \"11db9cec-653f-4aee-8937-0cf5b367efac\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-tg5xj"
Oct 30 19:15:41.913132 ip-10-0-132-82 kubenswrapper[2577]: I1030 19:15:41.913066    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rfxzq\" (UniqueName: \"kubernetes.io/projected/4ba601a6-2a44-4408-8c5b-2c29fd096fed-kube-api-access-rfxzq\") pod \"service-ca-operator-544d749c8c-nbfv2\" (UID: \"4ba601a6-2a44-4408-8c5b-2c29fd096fed\") " pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-nbfv2"
Oct 30 19:15:41.913132 ip-10-0-132-82 kubenswrapper[2577]: I1030 19:15:41.913097    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/0e179af5-9f52-4941-b4f5-5e3b8789da11-registry-tls\") pod \"image-registry-d97c68fd4-htsl8\" (UID: \"0e179af5-9f52-4941-b4f5-5e3b8789da11\") " pod="openshift-image-registry/image-registry-d97c68fd4-htsl8"
Oct 30 19:15:41.913132 ip-10-0-132-82 kubenswrapper[2577]: I1030 19:15:41.913121    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/4ba601a6-2a44-4408-8c5b-2c29fd096fed-serving-cert\") pod \"service-ca-operator-544d749c8c-nbfv2\" (UID: \"4ba601a6-2a44-4408-8c5b-2c29fd096fed\") " pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-nbfv2"
Oct 30 19:15:41.913232 ip-10-0-132-82 kubenswrapper[2577]: I1030 19:15:41.913146    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/0e179af5-9f52-4941-b4f5-5e3b8789da11-ca-trust-extracted\") pod \"image-registry-d97c68fd4-htsl8\" (UID: \"0e179af5-9f52-4941-b4f5-5e3b8789da11\") " pod="openshift-image-registry/image-registry-d97c68fd4-htsl8"
Oct 30 19:15:41.913232 ip-10-0-132-82 kubenswrapper[2577]: I1030 19:15:41.913174    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-j4wnm\" (UniqueName: \"kubernetes.io/projected/f86c0e38-8144-42fc-b622-25602ada113f-kube-api-access-j4wnm\") pod \"ingress-canary-8rwjm\" (UID: \"f86c0e38-8144-42fc-b622-25602ada113f\") " pod="openshift-ingress-canary/ingress-canary-8rwjm"
Oct 30 19:15:41.913232 ip-10-0-132-82 kubenswrapper[2577]: I1030 19:15:41.913206    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/fade00ba-24a7-41c9-93f7-23e40971874a-config-volume\") pod \"dns-default-bb9cj\" (UID: \"fade00ba-24a7-41c9-93f7-23e40971874a\") " pod="openshift-dns/dns-default-bb9cj"
Oct 30 19:15:41.913324 ip-10-0-132-82 kubenswrapper[2577]: I1030 19:15:41.913243    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/f86c0e38-8144-42fc-b622-25602ada113f-cert\") pod \"ingress-canary-8rwjm\" (UID: \"f86c0e38-8144-42fc-b622-25602ada113f\") " pod="openshift-ingress-canary/ingress-canary-8rwjm"
Oct 30 19:15:41.913324 ip-10-0-132-82 kubenswrapper[2577]: I1030 19:15:41.913269    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-btd8n\" (UniqueName: \"kubernetes.io/projected/33ce6ceb-02a9-4852-9948-f02e2b39e32f-kube-api-access-btd8n\") pod \"network-check-source-6f78d59f99-2trjq\" (UID: \"33ce6ceb-02a9-4852-9948-f02e2b39e32f\") " pod="openshift-network-diagnostics/network-check-source-6f78d59f99-2trjq"
Oct 30 19:15:41.913324 ip-10-0-132-82 kubenswrapper[2577]: I1030 19:15:41.913293    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-q667z\" (UniqueName: \"kubernetes.io/projected/fade00ba-24a7-41c9-93f7-23e40971874a-kube-api-access-q667z\") pod \"dns-default-bb9cj\" (UID: \"fade00ba-24a7-41c9-93f7-23e40971874a\") " pod="openshift-dns/dns-default-bb9cj"
Oct 30 19:15:41.913324 ip-10-0-132-82 kubenswrapper[2577]: I1030 19:15:41.913318    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5xgwq\" (UniqueName: \"kubernetes.io/projected/11db9cec-653f-4aee-8937-0cf5b367efac-kube-api-access-5xgwq\") pod \"console-operator-7cd56b8dbc-tg5xj\" (UID: \"11db9cec-653f-4aee-8937-0cf5b367efac\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-tg5xj"
Oct 30 19:15:41.913443 ip-10-0-132-82 kubenswrapper[2577]: I1030 19:15:41.913366    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/2c7c7060-def6-4737-8fa9-6889309b7463-networking-console-plugin-cert\") pod \"networking-console-plugin-86c478bb88-h4twn\" (UID: \"2c7c7060-def6-4737-8fa9-6889309b7463\") " pod="openshift-network-console/networking-console-plugin-86c478bb88-h4twn"
Oct 30 19:15:41.913443 ip-10-0-132-82 kubenswrapper[2577]: I1030 19:15:41.913402    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/0e179af5-9f52-4941-b4f5-5e3b8789da11-registry-certificates\") pod \"image-registry-d97c68fd4-htsl8\" (UID: \"0e179af5-9f52-4941-b4f5-5e3b8789da11\") " pod="openshift-image-registry/image-registry-d97c68fd4-htsl8"
Oct 30 19:15:41.913443 ip-10-0-132-82 kubenswrapper[2577]: I1030 19:15:41.913425    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/fade00ba-24a7-41c9-93f7-23e40971874a-tmp-dir\") pod \"dns-default-bb9cj\" (UID: \"fade00ba-24a7-41c9-93f7-23e40971874a\") " pod="openshift-dns/dns-default-bb9cj"
Oct 30 19:15:41.913531 ip-10-0-132-82 kubenswrapper[2577]: I1030 19:15:41.913466    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/0e179af5-9f52-4941-b4f5-5e3b8789da11-installation-pull-secrets\") pod \"image-registry-d97c68fd4-htsl8\" (UID: \"0e179af5-9f52-4941-b4f5-5e3b8789da11\") " pod="openshift-image-registry/image-registry-d97c68fd4-htsl8"
Oct 30 19:15:41.914182 ip-10-0-132-82 kubenswrapper[2577]: I1030 19:15:41.914011    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/0e179af5-9f52-4941-b4f5-5e3b8789da11-bound-sa-token\") pod \"image-registry-d97c68fd4-htsl8\" (UID: \"0e179af5-9f52-4941-b4f5-5e3b8789da11\") " pod="openshift-image-registry/image-registry-d97c68fd4-htsl8"
Oct 30 19:15:41.914354 ip-10-0-132-82 kubenswrapper[2577]: I1030 19:15:41.914330    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/4ba601a6-2a44-4408-8c5b-2c29fd096fed-config\") pod \"service-ca-operator-544d749c8c-nbfv2\" (UID: \"4ba601a6-2a44-4408-8c5b-2c29fd096fed\") " pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-nbfv2"
Oct 30 19:15:41.914597 ip-10-0-132-82 kubenswrapper[2577]: I1030 19:15:41.914375    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/e446ca63-5885-4c66-9cb7-da39afbf742c-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-884576c57-wkldq\" (UID: \"e446ca63-5885-4c66-9cb7-da39afbf742c\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-wkldq"
Oct 30 19:15:41.914597 ip-10-0-132-82 kubenswrapper[2577]: I1030 19:15:41.914420    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"telemetry-config\" (UniqueName: \"kubernetes.io/configmap/e446ca63-5885-4c66-9cb7-da39afbf742c-telemetry-config\") pod \"cluster-monitoring-operator-884576c57-wkldq\" (UID: \"e446ca63-5885-4c66-9cb7-da39afbf742c\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-wkldq"
Oct 30 19:15:41.914597 ip-10-0-132-82 kubenswrapper[2577]: I1030 19:15:41.914448    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/11db9cec-653f-4aee-8937-0cf5b367efac-config\") pod \"console-operator-7cd56b8dbc-tg5xj\" (UID: \"11db9cec-653f-4aee-8937-0cf5b367efac\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-tg5xj"
Oct 30 19:15:41.914597 ip-10-0-132-82 kubenswrapper[2577]: I1030 19:15:41.914472    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-j7bln\" (UniqueName: \"kubernetes.io/projected/e446ca63-5885-4c66-9cb7-da39afbf742c-kube-api-access-j7bln\") pod \"cluster-monitoring-operator-884576c57-wkldq\" (UID: \"e446ca63-5885-4c66-9cb7-da39afbf742c\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-wkldq"
Oct 30 19:15:41.914597 ip-10-0-132-82 kubenswrapper[2577]: I1030 19:15:41.914512    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/0e179af5-9f52-4941-b4f5-5e3b8789da11-ca-trust-extracted\") pod \"image-registry-d97c68fd4-htsl8\" (UID: \"0e179af5-9f52-4941-b4f5-5e3b8789da11\") " pod="openshift-image-registry/image-registry-d97c68fd4-htsl8"
Oct 30 19:15:41.914597 ip-10-0-132-82 kubenswrapper[2577]: I1030 19:15:41.914533    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/0e179af5-9f52-4941-b4f5-5e3b8789da11-image-registry-private-configuration\") pod \"image-registry-d97c68fd4-htsl8\" (UID: \"0e179af5-9f52-4941-b4f5-5e3b8789da11\") " pod="openshift-image-registry/image-registry-d97c68fd4-htsl8"
Oct 30 19:15:41.914597 ip-10-0-132-82 kubenswrapper[2577]: E1030 19:15:41.914537    2577 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found
Oct 30 19:15:41.914597 ip-10-0-132-82 kubenswrapper[2577]: E1030 19:15:41.914583    2577 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Oct 30 19:15:41.914597 ip-10-0-132-82 kubenswrapper[2577]: E1030 19:15:41.914597    2577 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-d97c68fd4-htsl8: secret "image-registry-tls" not found
Oct 30 19:15:41.915066 ip-10-0-132-82 kubenswrapper[2577]: E1030 19:15:41.914656    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/0e179af5-9f52-4941-b4f5-5e3b8789da11-registry-tls podName:0e179af5-9f52-4941-b4f5-5e3b8789da11 nodeName:}" failed. No retries permitted until 2026-10-30 19:15:42.41463865 +0000 UTC m=+32.820559064 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/0e179af5-9f52-4941-b4f5-5e3b8789da11-registry-tls") pod "image-registry-d97c68fd4-htsl8" (UID: "0e179af5-9f52-4941-b4f5-5e3b8789da11") : secret "image-registry-tls" not found
Oct 30 19:15:41.915066 ip-10-0-132-82 kubenswrapper[2577]: I1030 19:15:41.914681    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/11db9cec-653f-4aee-8937-0cf5b367efac-trusted-ca\") pod \"console-operator-7cd56b8dbc-tg5xj\" (UID: \"11db9cec-653f-4aee-8937-0cf5b367efac\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-tg5xj"
Oct 30 19:15:41.915066 ip-10-0-132-82 kubenswrapper[2577]: E1030 19:15:41.914737    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/2c7c7060-def6-4737-8fa9-6889309b7463-networking-console-plugin-cert podName:2c7c7060-def6-4737-8fa9-6889309b7463 nodeName:}" failed. No retries permitted until 2026-10-30 19:15:42.414719835 +0000 UTC m=+32.820640266 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/2c7c7060-def6-4737-8fa9-6889309b7463-networking-console-plugin-cert") pod "networking-console-plugin-86c478bb88-h4twn" (UID: "2c7c7060-def6-4737-8fa9-6889309b7463") : secret "networking-console-plugin-cert" not found
Oct 30 19:15:41.915066 ip-10-0-132-82 kubenswrapper[2577]: I1030 19:15:41.914678    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/0e179af5-9f52-4941-b4f5-5e3b8789da11-registry-certificates\") pod \"image-registry-d97c68fd4-htsl8\" (UID: \"0e179af5-9f52-4941-b4f5-5e3b8789da11\") " pod="openshift-image-registry/image-registry-d97c68fd4-htsl8"
Oct 30 19:15:41.915066 ip-10-0-132-82 kubenswrapper[2577]: I1030 19:15:41.914785    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/fade00ba-24a7-41c9-93f7-23e40971874a-metrics-tls\") pod \"dns-default-bb9cj\" (UID: \"fade00ba-24a7-41c9-93f7-23e40971874a\") " pod="openshift-dns/dns-default-bb9cj"
Oct 30 19:15:41.915066 ip-10-0-132-82 kubenswrapper[2577]: I1030 19:15:41.914886    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/2c7c7060-def6-4737-8fa9-6889309b7463-nginx-conf\") pod \"networking-console-plugin-86c478bb88-h4twn\" (UID: \"2c7c7060-def6-4737-8fa9-6889309b7463\") " pod="openshift-network-console/networking-console-plugin-86c478bb88-h4twn"
Oct 30 19:15:41.915066 ip-10-0-132-82 kubenswrapper[2577]: I1030 19:15:41.915000    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/0e179af5-9f52-4941-b4f5-5e3b8789da11-trusted-ca\") pod \"image-registry-d97c68fd4-htsl8\" (UID: \"0e179af5-9f52-4941-b4f5-5e3b8789da11\") " pod="openshift-image-registry/image-registry-d97c68fd4-htsl8"
Oct 30 19:15:41.915430 ip-10-0-132-82 kubenswrapper[2577]: I1030 19:15:41.915280    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/configmap/4ba601a6-2a44-4408-8c5b-2c29fd096fed-config\") pod \"service-ca-operator-544d749c8c-nbfv2\" (UID: \"4ba601a6-2a44-4408-8c5b-2c29fd096fed\") " pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-nbfv2"
Oct 30 19:15:41.915430 ip-10-0-132-82 kubenswrapper[2577]: I1030 19:15:41.915382    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/configmap/11db9cec-653f-4aee-8937-0cf5b367efac-config\") pod \"console-operator-7cd56b8dbc-tg5xj\" (UID: \"11db9cec-653f-4aee-8937-0cf5b367efac\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-tg5xj"
Oct 30 19:15:41.915872 ip-10-0-132-82 kubenswrapper[2577]: I1030 19:15:41.915847    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/2c7c7060-def6-4737-8fa9-6889309b7463-nginx-conf\") pod \"networking-console-plugin-86c478bb88-h4twn\" (UID: \"2c7c7060-def6-4737-8fa9-6889309b7463\") " pod="openshift-network-console/networking-console-plugin-86c478bb88-h4twn"
Oct 30 19:15:41.916431 ip-10-0-132-82 kubenswrapper[2577]: I1030 19:15:41.916373    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/11db9cec-653f-4aee-8937-0cf5b367efac-trusted-ca\") pod \"console-operator-7cd56b8dbc-tg5xj\" (UID: \"11db9cec-653f-4aee-8937-0cf5b367efac\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-tg5xj"
Oct 30 19:15:41.916537 ip-10-0-132-82 kubenswrapper[2577]: I1030 19:15:41.916465    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/4ba601a6-2a44-4408-8c5b-2c29fd096fed-serving-cert\") pod \"service-ca-operator-544d749c8c-nbfv2\" (UID: \"4ba601a6-2a44-4408-8c5b-2c29fd096fed\") " pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-nbfv2"
Oct 30 19:15:41.917168 ip-10-0-132-82 kubenswrapper[2577]: I1030 19:15:41.917148    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/11db9cec-653f-4aee-8937-0cf5b367efac-serving-cert\") pod \"console-operator-7cd56b8dbc-tg5xj\" (UID: \"11db9cec-653f-4aee-8937-0cf5b367efac\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-tg5xj"
Oct 30 19:15:41.917364 ip-10-0-132-82 kubenswrapper[2577]: I1030 19:15:41.917342    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/0e179af5-9f52-4941-b4f5-5e3b8789da11-image-registry-private-configuration\") pod \"image-registry-d97c68fd4-htsl8\" (UID: \"0e179af5-9f52-4941-b4f5-5e3b8789da11\") " pod="openshift-image-registry/image-registry-d97c68fd4-htsl8"
Oct 30 19:15:41.917615 ip-10-0-132-82 kubenswrapper[2577]: I1030 19:15:41.917601    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/0e179af5-9f52-4941-b4f5-5e3b8789da11-installation-pull-secrets\") pod \"image-registry-d97c68fd4-htsl8\" (UID: \"0e179af5-9f52-4941-b4f5-5e3b8789da11\") " pod="openshift-image-registry/image-registry-d97c68fd4-htsl8"
Oct 30 19:15:41.942623 ip-10-0-132-82 kubenswrapper[2577]: I1030 19:15:41.942601    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-operator-7c4986d449-6jdkq"
Oct 30 19:15:41.965159 ip-10-0-132-82 kubenswrapper[2577]: I1030 19:15:41.965128    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-storage-operator/volume-data-source-validator-58c9f5f46d-p42xv"
Oct 30 19:15:41.980686 ip-10-0-132-82 kubenswrapper[2577]: I1030 19:15:41.978462    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rfxzq\" (UniqueName: \"kubernetes.io/projected/4ba601a6-2a44-4408-8c5b-2c29fd096fed-kube-api-access-rfxzq\") pod \"service-ca-operator-544d749c8c-nbfv2\" (UID: \"4ba601a6-2a44-4408-8c5b-2c29fd096fed\") " pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-nbfv2"
Oct 30 19:15:41.983321 ip-10-0-132-82 kubenswrapper[2577]: I1030 19:15:41.983292    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-scljn"
Oct 30 19:15:41.984449 ip-10-0-132-82 kubenswrapper[2577]: I1030 19:15:41.984133    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/0e179af5-9f52-4941-b4f5-5e3b8789da11-bound-sa-token\") pod \"image-registry-d97c68fd4-htsl8\" (UID: \"0e179af5-9f52-4941-b4f5-5e3b8789da11\") " pod="openshift-image-registry/image-registry-d97c68fd4-htsl8"
Oct 30 19:15:41.984449 ip-10-0-132-82 kubenswrapper[2577]: I1030 19:15:41.984199    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zj4df\" (UniqueName: \"kubernetes.io/projected/0e179af5-9f52-4941-b4f5-5e3b8789da11-kube-api-access-zj4df\") pod \"image-registry-d97c68fd4-htsl8\" (UID: \"0e179af5-9f52-4941-b4f5-5e3b8789da11\") " pod="openshift-image-registry/image-registry-d97c68fd4-htsl8"
Oct 30 19:15:41.985508 ip-10-0-132-82 kubenswrapper[2577]: I1030 19:15:41.985481    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5xgwq\" (UniqueName: \"kubernetes.io/projected/11db9cec-653f-4aee-8937-0cf5b367efac-kube-api-access-5xgwq\") pod \"console-operator-7cd56b8dbc-tg5xj\" (UID: \"11db9cec-653f-4aee-8937-0cf5b367efac\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-tg5xj"
Oct 30 19:15:42.017109 ip-10-0-132-82 kubenswrapper[2577]: I1030 19:15:42.016528    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/fade00ba-24a7-41c9-93f7-23e40971874a-config-volume\") pod \"dns-default-bb9cj\" (UID: \"fade00ba-24a7-41c9-93f7-23e40971874a\") " pod="openshift-dns/dns-default-bb9cj"
Oct 30 19:15:42.017109 ip-10-0-132-82 kubenswrapper[2577]: I1030 19:15:42.016597    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/f86c0e38-8144-42fc-b622-25602ada113f-cert\") pod \"ingress-canary-8rwjm\" (UID: \"f86c0e38-8144-42fc-b622-25602ada113f\") " pod="openshift-ingress-canary/ingress-canary-8rwjm"
Oct 30 19:15:42.017109 ip-10-0-132-82 kubenswrapper[2577]: I1030 19:15:42.016624    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-btd8n\" (UniqueName: \"kubernetes.io/projected/33ce6ceb-02a9-4852-9948-f02e2b39e32f-kube-api-access-btd8n\") pod \"network-check-source-6f78d59f99-2trjq\" (UID: \"33ce6ceb-02a9-4852-9948-f02e2b39e32f\") " pod="openshift-network-diagnostics/network-check-source-6f78d59f99-2trjq"
Oct 30 19:15:42.017109 ip-10-0-132-82 kubenswrapper[2577]: I1030 19:15:42.016650    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-q667z\" (UniqueName: \"kubernetes.io/projected/fade00ba-24a7-41c9-93f7-23e40971874a-kube-api-access-q667z\") pod \"dns-default-bb9cj\" (UID: \"fade00ba-24a7-41c9-93f7-23e40971874a\") " pod="openshift-dns/dns-default-bb9cj"
Oct 30 19:15:42.017109 ip-10-0-132-82 kubenswrapper[2577]: I1030 19:15:42.016721    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/fade00ba-24a7-41c9-93f7-23e40971874a-tmp-dir\") pod \"dns-default-bb9cj\" (UID: \"fade00ba-24a7-41c9-93f7-23e40971874a\") " pod="openshift-dns/dns-default-bb9cj"
Oct 30 19:15:42.017109 ip-10-0-132-82 kubenswrapper[2577]: I1030 19:15:42.016777    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/e446ca63-5885-4c66-9cb7-da39afbf742c-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-884576c57-wkldq\" (UID: \"e446ca63-5885-4c66-9cb7-da39afbf742c\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-wkldq"
Oct 30 19:15:42.018823 ip-10-0-132-82 kubenswrapper[2577]: I1030 19:15:42.018795    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/fade00ba-24a7-41c9-93f7-23e40971874a-tmp-dir\") pod \"dns-default-bb9cj\" (UID: \"fade00ba-24a7-41c9-93f7-23e40971874a\") " pod="openshift-dns/dns-default-bb9cj"
Oct 30 19:15:42.019319 ip-10-0-132-82 kubenswrapper[2577]: E1030 19:15:42.019240    2577 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found
Oct 30 19:15:42.019496 ip-10-0-132-82 kubenswrapper[2577]: E1030 19:15:42.019475    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e446ca63-5885-4c66-9cb7-da39afbf742c-cluster-monitoring-operator-tls podName:e446ca63-5885-4c66-9cb7-da39afbf742c nodeName:}" failed. No retries permitted until 2026-10-30 19:15:42.519451044 +0000 UTC m=+32.925371470 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/e446ca63-5885-4c66-9cb7-da39afbf742c-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-884576c57-wkldq" (UID: "e446ca63-5885-4c66-9cb7-da39afbf742c") : secret "cluster-monitoring-operator-tls" not found
Oct 30 19:15:42.019629 ip-10-0-132-82 kubenswrapper[2577]: I1030 19:15:42.019534    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"telemetry-config\" (UniqueName: \"kubernetes.io/configmap/e446ca63-5885-4c66-9cb7-da39afbf742c-telemetry-config\") pod \"cluster-monitoring-operator-884576c57-wkldq\" (UID: \"e446ca63-5885-4c66-9cb7-da39afbf742c\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-wkldq"
Oct 30 19:15:42.019629 ip-10-0-132-82 kubenswrapper[2577]: I1030 19:15:42.019565    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-j7bln\" (UniqueName: \"kubernetes.io/projected/e446ca63-5885-4c66-9cb7-da39afbf742c-kube-api-access-j7bln\") pod \"cluster-monitoring-operator-884576c57-wkldq\" (UID: \"e446ca63-5885-4c66-9cb7-da39afbf742c\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-wkldq"
Oct 30 19:15:42.019629 ip-10-0-132-82 kubenswrapper[2577]: I1030 19:15:42.019621    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/fade00ba-24a7-41c9-93f7-23e40971874a-metrics-tls\") pod \"dns-default-bb9cj\" (UID: \"fade00ba-24a7-41c9-93f7-23e40971874a\") " pod="openshift-dns/dns-default-bb9cj"
Oct 30 19:15:42.019773 ip-10-0-132-82 kubenswrapper[2577]: I1030 19:15:42.019731    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-j4wnm\" (UniqueName: \"kubernetes.io/projected/f86c0e38-8144-42fc-b622-25602ada113f-kube-api-access-j4wnm\") pod \"ingress-canary-8rwjm\" (UID: \"f86c0e38-8144-42fc-b622-25602ada113f\") " pod="openshift-ingress-canary/ingress-canary-8rwjm"
Oct 30 19:15:42.020707 ip-10-0-132-82 kubenswrapper[2577]: I1030 19:15:42.020685    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"telemetry-config\" (UniqueName: \"kubernetes.io/configmap/e446ca63-5885-4c66-9cb7-da39afbf742c-telemetry-config\") pod \"cluster-monitoring-operator-884576c57-wkldq\" (UID: \"e446ca63-5885-4c66-9cb7-da39afbf742c\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-wkldq"
Oct 30 19:15:42.021010 ip-10-0-132-82 kubenswrapper[2577]: E1030 19:15:42.020994    2577 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Oct 30 19:15:42.021073 ip-10-0-132-82 kubenswrapper[2577]: E1030 19:15:42.021042    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/fade00ba-24a7-41c9-93f7-23e40971874a-metrics-tls podName:fade00ba-24a7-41c9-93f7-23e40971874a nodeName:}" failed. No retries permitted until 2026-10-30 19:15:42.521028506 +0000 UTC m=+32.926948935 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/fade00ba-24a7-41c9-93f7-23e40971874a-metrics-tls") pod "dns-default-bb9cj" (UID: "fade00ba-24a7-41c9-93f7-23e40971874a") : secret "dns-default-metrics-tls" not found
Oct 30 19:15:42.021713 ip-10-0-132-82 kubenswrapper[2577]: I1030 19:15:42.021690    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/fade00ba-24a7-41c9-93f7-23e40971874a-config-volume\") pod \"dns-default-bb9cj\" (UID: \"fade00ba-24a7-41c9-93f7-23e40971874a\") " pod="openshift-dns/dns-default-bb9cj"
Oct 30 19:15:42.022359 ip-10-0-132-82 kubenswrapper[2577]: I1030 19:15:42.022331    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/f86c0e38-8144-42fc-b622-25602ada113f-cert\") pod \"ingress-canary-8rwjm\" (UID: \"f86c0e38-8144-42fc-b622-25602ada113f\") " pod="openshift-ingress-canary/ingress-canary-8rwjm"
Oct 30 19:15:42.034638 ip-10-0-132-82 kubenswrapper[2577]: I1030 19:15:42.034609    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-822gp"
Oct 30 19:15:42.035860 ip-10-0-132-82 kubenswrapper[2577]: I1030 19:15:42.035106    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-76rht"
Oct 30 19:15:42.035860 ip-10-0-132-82 kubenswrapper[2577]: I1030 19:15:42.035500    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-c24j9"
Oct 30 19:15:42.060507 ip-10-0-132-82 kubenswrapper[2577]: I1030 19:15:42.060300    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-nbfv2"
Oct 30 19:15:42.077403 ip-10-0-132-82 kubenswrapper[2577]: I1030 19:15:42.077376    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-j4wnm\" (UniqueName: \"kubernetes.io/projected/f86c0e38-8144-42fc-b622-25602ada113f-kube-api-access-j4wnm\") pod \"ingress-canary-8rwjm\" (UID: \"f86c0e38-8144-42fc-b622-25602ada113f\") " pod="openshift-ingress-canary/ingress-canary-8rwjm"
Oct 30 19:15:42.078038 ip-10-0-132-82 kubenswrapper[2577]: I1030 19:15:42.077949    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-q667z\" (UniqueName: \"kubernetes.io/projected/fade00ba-24a7-41c9-93f7-23e40971874a-kube-api-access-q667z\") pod \"dns-default-bb9cj\" (UID: \"fade00ba-24a7-41c9-93f7-23e40971874a\") " pod="openshift-dns/dns-default-bb9cj"
Oct 30 19:15:42.078278 ip-10-0-132-82 kubenswrapper[2577]: I1030 19:15:42.078249    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-btd8n\" (UniqueName: \"kubernetes.io/projected/33ce6ceb-02a9-4852-9948-f02e2b39e32f-kube-api-access-btd8n\") pod \"network-check-source-6f78d59f99-2trjq\" (UID: \"33ce6ceb-02a9-4852-9948-f02e2b39e32f\") " pod="openshift-network-diagnostics/network-check-source-6f78d59f99-2trjq"
Oct 30 19:15:42.078373 ip-10-0-132-82 kubenswrapper[2577]: I1030 19:15:42.078359    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-source-6f78d59f99-2trjq"
Oct 30 19:15:42.079547 ip-10-0-132-82 kubenswrapper[2577]: I1030 19:15:42.079524    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-j7bln\" (UniqueName: \"kubernetes.io/projected/e446ca63-5885-4c66-9cb7-da39afbf742c-kube-api-access-j7bln\") pod \"cluster-monitoring-operator-884576c57-wkldq\" (UID: \"e446ca63-5885-4c66-9cb7-da39afbf742c\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-wkldq"
Oct 30 19:15:42.088388 ip-10-0-132-82 kubenswrapper[2577]: I1030 19:15:42.088311    2577 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\""
Oct 30 19:15:42.088388 ip-10-0-132-82 kubenswrapper[2577]: I1030 19:15:42.088330    2577 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"global-pull-secret-syncer-dockercfg-gt8zl\""
Oct 30 19:15:42.088388 ip-10-0-132-82 kubenswrapper[2577]: I1030 19:15:42.088365    2577 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-jlfgq\""
Oct 30 19:15:42.088647 ip-10-0-132-82 kubenswrapper[2577]: I1030 19:15:42.088615    2577 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\""
Oct 30 19:15:42.088708 ip-10-0-132-82 kubenswrapper[2577]: I1030 19:15:42.088653    2577 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"network-check-target-dockercfg-px7h6\""
Oct 30 19:15:42.102104 ip-10-0-132-82 kubenswrapper[2577]: I1030 19:15:42.102082    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console-operator/console-operator-7cd56b8dbc-tg5xj"
Oct 30 19:15:42.142715 ip-10-0-132-82 kubenswrapper[2577]: I1030 19:15:42.142680    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-8rwjm"
Oct 30 19:15:42.258328 ip-10-0-132-82 kubenswrapper[2577]: I1030 19:15:42.258301    2577 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kube-system/kas-connection-checker-685895c7c9-tvmks"]
Oct 30 19:15:42.261398 ip-10-0-132-82 kubenswrapper[2577]: W1030 19:15:42.261375    2577 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd260d4d7_b4aa_4f37_92b5_e75b22e13a3b.slice/crio-e4dd71d689867fb5f56c774748d9b5a5f2bedffc837d9d7c0ad1723524d414ca WatchSource:0}: Error finding container e4dd71d689867fb5f56c774748d9b5a5f2bedffc837d9d7c0ad1723524d414ca: Status 404 returned error can't find the container with id e4dd71d689867fb5f56c774748d9b5a5f2bedffc837d9d7c0ad1723524d414ca
Oct 30 19:15:42.276093 ip-10-0-132-82 kubenswrapper[2577]: I1030 19:15:42.276031    2577 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kube-system/kas-connection-checker-685895c7c9-p8kjs"]
Oct 30 19:15:42.289770 ip-10-0-132-82 kubenswrapper[2577]: W1030 19:15:42.289744    2577 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod87da348d_b1ee_4cd3_806d_0560bff3c6f2.slice/crio-9a980ae6fdff8b6bb898b3b37c651c25cdbecbdcdeacc75bf47c59da9b0302a0 WatchSource:0}: Error finding container 9a980ae6fdff8b6bb898b3b37c651c25cdbecbdcdeacc75bf47c59da9b0302a0: Status 404 returned error can't find the container with id 9a980ae6fdff8b6bb898b3b37c651c25cdbecbdcdeacc75bf47c59da9b0302a0
Oct 30 19:15:42.322950 ip-10-0-132-82 kubenswrapper[2577]: I1030 19:15:42.322912    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/87082646-75c3-43b5-864b-3478e9a7373d-service-ca-bundle\") pod \"router-default-776df59447-8nrhh\" (UID: \"87082646-75c3-43b5-864b-3478e9a7373d\") " pod="openshift-ingress/router-default-776df59447-8nrhh"
Oct 30 19:15:42.323066 ip-10-0-132-82 kubenswrapper[2577]: I1030 19:15:42.322988    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/56274b23-2400-43d4-9686-c928e20d47de-samples-operator-tls\") pod \"cluster-samples-operator-6ddb6c8b94-np94b\" (UID: \"56274b23-2400-43d4-9686-c928e20d47de\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-np94b"
Oct 30 19:15:42.323155 ip-10-0-132-82 kubenswrapper[2577]: E1030 19:15:42.323096    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/87082646-75c3-43b5-864b-3478e9a7373d-service-ca-bundle podName:87082646-75c3-43b5-864b-3478e9a7373d nodeName:}" failed. No retries permitted until 2026-10-30 19:15:43.323075733 +0000 UTC m=+33.728996164 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/87082646-75c3-43b5-864b-3478e9a7373d-service-ca-bundle") pod "router-default-776df59447-8nrhh" (UID: "87082646-75c3-43b5-864b-3478e9a7373d") : configmap references non-existent config key: service-ca.crt
Oct 30 19:15:42.323155 ip-10-0-132-82 kubenswrapper[2577]: E1030 19:15:42.323147    2577 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found
Oct 30 19:15:42.323262 ip-10-0-132-82 kubenswrapper[2577]: I1030 19:15:42.323158    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/e3206ddf-35de-4b7c-b979-682f5f666a34-registry-tls\") pod \"image-registry-6fcb4848df-8l8cp\" (UID: \"e3206ddf-35de-4b7c-b979-682f5f666a34\") " pod="openshift-image-registry/image-registry-6fcb4848df-8l8cp"
Oct 30 19:15:42.323262 ip-10-0-132-82 kubenswrapper[2577]: E1030 19:15:42.323205    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/56274b23-2400-43d4-9686-c928e20d47de-samples-operator-tls podName:56274b23-2400-43d4-9686-c928e20d47de nodeName:}" failed. No retries permitted until 2026-10-30 19:15:43.323186356 +0000 UTC m=+33.729106791 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/56274b23-2400-43d4-9686-c928e20d47de-samples-operator-tls") pod "cluster-samples-operator-6ddb6c8b94-np94b" (UID: "56274b23-2400-43d4-9686-c928e20d47de") : secret "samples-operator-tls" not found
Oct 30 19:15:42.323262 ip-10-0-132-82 kubenswrapper[2577]: E1030 19:15:42.323253    2577 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Oct 30 19:15:42.323360 ip-10-0-132-82 kubenswrapper[2577]: E1030 19:15:42.323266    2577 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-6fcb4848df-8l8cp: secret "image-registry-tls" not found
Oct 30 19:15:42.323360 ip-10-0-132-82 kubenswrapper[2577]: I1030 19:15:42.323264    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/87082646-75c3-43b5-864b-3478e9a7373d-metrics-certs\") pod \"router-default-776df59447-8nrhh\" (UID: \"87082646-75c3-43b5-864b-3478e9a7373d\") " pod="openshift-ingress/router-default-776df59447-8nrhh"
Oct 30 19:15:42.323360 ip-10-0-132-82 kubenswrapper[2577]: E1030 19:15:42.323305    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/e3206ddf-35de-4b7c-b979-682f5f666a34-registry-tls podName:e3206ddf-35de-4b7c-b979-682f5f666a34 nodeName:}" failed. No retries permitted until 2026-10-30 19:15:43.323292482 +0000 UTC m=+33.729212898 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/e3206ddf-35de-4b7c-b979-682f5f666a34-registry-tls") pod "image-registry-6fcb4848df-8l8cp" (UID: "e3206ddf-35de-4b7c-b979-682f5f666a34") : secret "image-registry-tls" not found
Oct 30 19:15:42.323464 ip-10-0-132-82 kubenswrapper[2577]: E1030 19:15:42.323381    2577 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found
Oct 30 19:15:42.323464 ip-10-0-132-82 kubenswrapper[2577]: E1030 19:15:42.323433    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/87082646-75c3-43b5-864b-3478e9a7373d-metrics-certs podName:87082646-75c3-43b5-864b-3478e9a7373d nodeName:}" failed. No retries permitted until 2026-10-30 19:15:43.323421399 +0000 UTC m=+33.729341847 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/87082646-75c3-43b5-864b-3478e9a7373d-metrics-certs") pod "router-default-776df59447-8nrhh" (UID: "87082646-75c3-43b5-864b-3478e9a7373d") : secret "router-metrics-certs-default" not found
Oct 30 19:15:42.329338 ip-10-0-132-82 kubenswrapper[2577]: I1030 19:15:42.329314    2577 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-operator-7c4986d449-6jdkq"]
Oct 30 19:15:42.339031 ip-10-0-132-82 kubenswrapper[2577]: W1030 19:15:42.338999    2577 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2bd84dc4_e96a_43f1_85bf_e1ea4005c193.slice/crio-9e89eaa98efd97ac57e6419b178d76a3452fdc3097e597ea17f038f98d99832e WatchSource:0}: Error finding container 9e89eaa98efd97ac57e6419b178d76a3452fdc3097e597ea17f038f98d99832e: Status 404 returned error can't find the container with id 9e89eaa98efd97ac57e6419b178d76a3452fdc3097e597ea17f038f98d99832e
Oct 30 19:15:42.362869 ip-10-0-132-82 kubenswrapper[2577]: I1030 19:15:42.362816    2577 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-cluster-storage-operator/volume-data-source-validator-58c9f5f46d-p42xv"]
Oct 30 19:15:42.366075 ip-10-0-132-82 kubenswrapper[2577]: W1030 19:15:42.366054    2577 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4d213db4_73c5_4f50_9930_6d26b880dcf7.slice/crio-238545c7c7675395e4d64454432551c3daf4585d69f1ee4867b334786a60f6f1 WatchSource:0}: Error finding container 238545c7c7675395e4d64454432551c3daf4585d69f1ee4867b334786a60f6f1: Status 404 returned error can't find the container with id 238545c7c7675395e4d64454432551c3daf4585d69f1ee4867b334786a60f6f1
Oct 30 19:15:42.375817 ip-10-0-132-82 kubenswrapper[2577]: I1030 19:15:42.375793    2577 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kube-system/kas-connection-checker-685895c7c9-9h85f"]
Oct 30 19:15:42.377946 ip-10-0-132-82 kubenswrapper[2577]: W1030 19:15:42.377925    2577 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podba2c5571_23a3_46b6_a36d_7e8810a66261.slice/crio-fc2505f6a57fe8bc893233a26b65fea1edffdc09612365750fd0d81c8a31f7d8 WatchSource:0}: Error finding container fc2505f6a57fe8bc893233a26b65fea1edffdc09612365750fd0d81c8a31f7d8: Status 404 returned error can't find the container with id fc2505f6a57fe8bc893233a26b65fea1edffdc09612365750fd0d81c8a31f7d8
Oct 30 19:15:42.384284 ip-10-0-132-82 kubenswrapper[2577]: I1030 19:15:42.384220    2577 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-scljn"]
Oct 30 19:15:42.391415 ip-10-0-132-82 kubenswrapper[2577]: W1030 19:15:42.391391    2577 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf9a06263_cb99_4004_bffc_581d66ce5ebf.slice/crio-f2f64b91fb83a7b6abf1cb091c941acd65de81bf8df0800107508e4e11241e8c WatchSource:0}: Error finding container f2f64b91fb83a7b6abf1cb091c941acd65de81bf8df0800107508e4e11241e8c: Status 404 returned error can't find the container with id f2f64b91fb83a7b6abf1cb091c941acd65de81bf8df0800107508e4e11241e8c
Oct 30 19:15:42.424518 ip-10-0-132-82 kubenswrapper[2577]: I1030 19:15:42.424489    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/0e179af5-9f52-4941-b4f5-5e3b8789da11-registry-tls\") pod \"image-registry-d97c68fd4-htsl8\" (UID: \"0e179af5-9f52-4941-b4f5-5e3b8789da11\") " pod="openshift-image-registry/image-registry-d97c68fd4-htsl8"
Oct 30 19:15:42.424650 ip-10-0-132-82 kubenswrapper[2577]: I1030 19:15:42.424568    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/2c7c7060-def6-4737-8fa9-6889309b7463-networking-console-plugin-cert\") pod \"networking-console-plugin-86c478bb88-h4twn\" (UID: \"2c7c7060-def6-4737-8fa9-6889309b7463\") " pod="openshift-network-console/networking-console-plugin-86c478bb88-h4twn"
Oct 30 19:15:42.424774 ip-10-0-132-82 kubenswrapper[2577]: E1030 19:15:42.424756    2577 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found
Oct 30 19:15:42.424824 ip-10-0-132-82 kubenswrapper[2577]: E1030 19:15:42.424805    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/2c7c7060-def6-4737-8fa9-6889309b7463-networking-console-plugin-cert podName:2c7c7060-def6-4737-8fa9-6889309b7463 nodeName:}" failed. No retries permitted until 2026-10-30 19:15:43.42479276 +0000 UTC m=+33.830713189 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/2c7c7060-def6-4737-8fa9-6889309b7463-networking-console-plugin-cert") pod "networking-console-plugin-86c478bb88-h4twn" (UID: "2c7c7060-def6-4737-8fa9-6889309b7463") : secret "networking-console-plugin-cert" not found
Oct 30 19:15:42.424880 ip-10-0-132-82 kubenswrapper[2577]: E1030 19:15:42.424862    2577 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Oct 30 19:15:42.424880 ip-10-0-132-82 kubenswrapper[2577]: E1030 19:15:42.424871    2577 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-d97c68fd4-htsl8: secret "image-registry-tls" not found
Oct 30 19:15:42.425011 ip-10-0-132-82 kubenswrapper[2577]: E1030 19:15:42.424892    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/0e179af5-9f52-4941-b4f5-5e3b8789da11-registry-tls podName:0e179af5-9f52-4941-b4f5-5e3b8789da11 nodeName:}" failed. No retries permitted until 2026-10-30 19:15:43.424884978 +0000 UTC m=+33.830805405 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/0e179af5-9f52-4941-b4f5-5e3b8789da11-registry-tls") pod "image-registry-d97c68fd4-htsl8" (UID: "0e179af5-9f52-4941-b4f5-5e3b8789da11") : secret "image-registry-tls" not found
Oct 30 19:15:42.439281 ip-10-0-132-82 kubenswrapper[2577]: I1030 19:15:42.439257    2577 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-service-ca-operator/service-ca-operator-544d749c8c-nbfv2"]
Oct 30 19:15:42.442029 ip-10-0-132-82 kubenswrapper[2577]: W1030 19:15:42.442005    2577 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4ba601a6_2a44_4408_8c5b_2c29fd096fed.slice/crio-2edc71ff426362ce04e9fc22239cdc12c104ebc68abb45589d168aa8e283eba0 WatchSource:0}: Error finding container 2edc71ff426362ce04e9fc22239cdc12c104ebc68abb45589d168aa8e283eba0: Status 404 returned error can't find the container with id 2edc71ff426362ce04e9fc22239cdc12c104ebc68abb45589d168aa8e283eba0
Oct 30 19:15:42.467891 ip-10-0-132-82 kubenswrapper[2577]: I1030 19:15:42.467848    2577 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-source-6f78d59f99-2trjq"]
Oct 30 19:15:42.473867 ip-10-0-132-82 kubenswrapper[2577]: W1030 19:15:42.473846    2577 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod33ce6ceb_02a9_4852_9948_f02e2b39e32f.slice/crio-1f9b62b0486573df560be923ea7c2199b7827bf627151bdffb9652a48a0aa8b6 WatchSource:0}: Error finding container 1f9b62b0486573df560be923ea7c2199b7827bf627151bdffb9652a48a0aa8b6: Status 404 returned error can't find the container with id 1f9b62b0486573df560be923ea7c2199b7827bf627151bdffb9652a48a0aa8b6
Oct 30 19:15:42.514288 ip-10-0-132-82 kubenswrapper[2577]: I1030 19:15:42.514259    2577 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-console-operator/console-operator-7cd56b8dbc-tg5xj"]
Oct 30 19:15:42.517140 ip-10-0-132-82 kubenswrapper[2577]: W1030 19:15:42.517107    2577 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod11db9cec_653f_4aee_8937_0cf5b367efac.slice/crio-a203f34f66a5ec4d9297e5f8c1bde08af3007fa46ef54906494feccb340e9a27 WatchSource:0}: Error finding container a203f34f66a5ec4d9297e5f8c1bde08af3007fa46ef54906494feccb340e9a27: Status 404 returned error can't find the container with id a203f34f66a5ec4d9297e5f8c1bde08af3007fa46ef54906494feccb340e9a27
Oct 30 19:15:42.524950 ip-10-0-132-82 kubenswrapper[2577]: I1030 19:15:42.524926    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/e446ca63-5885-4c66-9cb7-da39afbf742c-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-884576c57-wkldq\" (UID: \"e446ca63-5885-4c66-9cb7-da39afbf742c\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-wkldq"
Oct 30 19:15:42.525046 ip-10-0-132-82 kubenswrapper[2577]: I1030 19:15:42.525011    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/fade00ba-24a7-41c9-93f7-23e40971874a-metrics-tls\") pod \"dns-default-bb9cj\" (UID: \"fade00ba-24a7-41c9-93f7-23e40971874a\") " pod="openshift-dns/dns-default-bb9cj"
Oct 30 19:15:42.525099 ip-10-0-132-82 kubenswrapper[2577]: E1030 19:15:42.525072    2577 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found
Oct 30 19:15:42.525141 ip-10-0-132-82 kubenswrapper[2577]: E1030 19:15:42.525130    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e446ca63-5885-4c66-9cb7-da39afbf742c-cluster-monitoring-operator-tls podName:e446ca63-5885-4c66-9cb7-da39afbf742c nodeName:}" failed. No retries permitted until 2026-10-30 19:15:43.525113993 +0000 UTC m=+33.931034407 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/e446ca63-5885-4c66-9cb7-da39afbf742c-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-884576c57-wkldq" (UID: "e446ca63-5885-4c66-9cb7-da39afbf742c") : secret "cluster-monitoring-operator-tls" not found
Oct 30 19:15:42.525141 ip-10-0-132-82 kubenswrapper[2577]: E1030 19:15:42.525131    2577 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Oct 30 19:15:42.525225 ip-10-0-132-82 kubenswrapper[2577]: E1030 19:15:42.525172    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/fade00ba-24a7-41c9-93f7-23e40971874a-metrics-tls podName:fade00ba-24a7-41c9-93f7-23e40971874a nodeName:}" failed. No retries permitted until 2026-10-30 19:15:43.52516053 +0000 UTC m=+33.931080967 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/fade00ba-24a7-41c9-93f7-23e40971874a-metrics-tls") pod "dns-default-bb9cj" (UID: "fade00ba-24a7-41c9-93f7-23e40971874a") : secret "dns-default-metrics-tls" not found
Oct 30 19:15:42.537725 ip-10-0-132-82 kubenswrapper[2577]: I1030 19:15:42.537703    2577 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-8rwjm"]
Oct 30 19:15:42.539883 ip-10-0-132-82 kubenswrapper[2577]: W1030 19:15:42.539863    2577 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf86c0e38_8144_42fc_b622_25602ada113f.slice/crio-97f8a11334a257518f5e40c7050de6a55a85bab3727386b10b4f04f52c8f0bcd WatchSource:0}: Error finding container 97f8a11334a257518f5e40c7050de6a55a85bab3727386b10b4f04f52c8f0bcd: Status 404 returned error can't find the container with id 97f8a11334a257518f5e40c7050de6a55a85bab3727386b10b4f04f52c8f0bcd
Oct 30 19:15:43.230604 ip-10-0-132-82 kubenswrapper[2577]: I1030 19:15:43.230575    2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console-operator/console-operator-7cd56b8dbc-tg5xj" event={"ID":"11db9cec-653f-4aee-8937-0cf5b367efac","Type":"ContainerStarted","Data":"a203f34f66a5ec4d9297e5f8c1bde08af3007fa46ef54906494feccb340e9a27"}
Oct 30 19:15:43.231690 ip-10-0-132-82 kubenswrapper[2577]: I1030 19:15:43.231666    2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-storage-operator/volume-data-source-validator-58c9f5f46d-p42xv" event={"ID":"4d213db4-73c5-4f50-9930-6d26b880dcf7","Type":"ContainerStarted","Data":"238545c7c7675395e4d64454432551c3daf4585d69f1ee4867b334786a60f6f1"}
Oct 30 19:15:43.233068 ip-10-0-132-82 kubenswrapper[2577]: I1030 19:15:43.233045    2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/kas-connection-checker-685895c7c9-p8kjs" event={"ID":"87da348d-b1ee-4cd3-806d-0560bff3c6f2","Type":"ContainerStarted","Data":"9fec5de4e13bdceff00f3c51477fb76dd8dd7e5d5f06045da218a6300f530ed3"}
Oct 30 19:15:43.233141 ip-10-0-132-82 kubenswrapper[2577]: I1030 19:15:43.233080    2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/kas-connection-checker-685895c7c9-p8kjs" event={"ID":"87da348d-b1ee-4cd3-806d-0560bff3c6f2","Type":"ContainerStarted","Data":"9a980ae6fdff8b6bb898b3b37c651c25cdbecbdcdeacc75bf47c59da9b0302a0"}
Oct 30 19:15:43.234224 ip-10-0-132-82 kubenswrapper[2577]: I1030 19:15:43.234203    2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-source-6f78d59f99-2trjq" event={"ID":"33ce6ceb-02a9-4852-9948-f02e2b39e32f","Type":"ContainerStarted","Data":"1f9b62b0486573df560be923ea7c2199b7827bf627151bdffb9652a48a0aa8b6"}
Oct 30 19:15:43.235275 ip-10-0-132-82 kubenswrapper[2577]: I1030 19:15:43.235258    2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-nbfv2" event={"ID":"4ba601a6-2a44-4408-8c5b-2c29fd096fed","Type":"ContainerStarted","Data":"2edc71ff426362ce04e9fc22239cdc12c104ebc68abb45589d168aa8e283eba0"}
Oct 30 19:15:43.236817 ip-10-0-132-82 kubenswrapper[2577]: I1030 19:15:43.236786    2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/kas-connection-checker-685895c7c9-tvmks" event={"ID":"d260d4d7-b4aa-4f37-92b5-e75b22e13a3b","Type":"ContainerStarted","Data":"4bf3a27f8fb434db99241a4b7f99d6fbbad42a6e55790ab6ff139792ae07adca"}
Oct 30 19:15:43.236920 ip-10-0-132-82 kubenswrapper[2577]: I1030 19:15:43.236839    2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/kas-connection-checker-685895c7c9-tvmks" event={"ID":"d260d4d7-b4aa-4f37-92b5-e75b22e13a3b","Type":"ContainerStarted","Data":"e4dd71d689867fb5f56c774748d9b5a5f2bedffc837d9d7c0ad1723524d414ca"}
Oct 30 19:15:43.238021 ip-10-0-132-82 kubenswrapper[2577]: I1030 19:15:43.237999    2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-scljn" event={"ID":"f9a06263-cb99-4004-bffc-581d66ce5ebf","Type":"ContainerStarted","Data":"f2f64b91fb83a7b6abf1cb091c941acd65de81bf8df0800107508e4e11241e8c"}
Oct 30 19:15:43.239182 ip-10-0-132-82 kubenswrapper[2577]: I1030 19:15:43.239158    2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-operator-7c4986d449-6jdkq" event={"ID":"2bd84dc4-e96a-43f1-85bf-e1ea4005c193","Type":"ContainerStarted","Data":"9e89eaa98efd97ac57e6419b178d76a3452fdc3097e597ea17f038f98d99832e"}
Oct 30 19:15:43.240560 ip-10-0-132-82 kubenswrapper[2577]: I1030 19:15:43.240538    2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-8rwjm" event={"ID":"f86c0e38-8144-42fc-b622-25602ada113f","Type":"ContainerStarted","Data":"97f8a11334a257518f5e40c7050de6a55a85bab3727386b10b4f04f52c8f0bcd"}
Oct 30 19:15:43.242655 ip-10-0-132-82 kubenswrapper[2577]: I1030 19:15:43.242633    2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/kas-connection-checker-685895c7c9-9h85f" event={"ID":"ba2c5571-23a3-46b6-a36d-7e8810a66261","Type":"ContainerStarted","Data":"44900aed69cf30c677998d7ea49c51031f89b797f6d8cfab71842dc780cb0032"}
Oct 30 19:15:43.242764 ip-10-0-132-82 kubenswrapper[2577]: I1030 19:15:43.242674    2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/kas-connection-checker-685895c7c9-9h85f" event={"ID":"ba2c5571-23a3-46b6-a36d-7e8810a66261","Type":"ContainerStarted","Data":"fc2505f6a57fe8bc893233a26b65fea1edffdc09612365750fd0d81c8a31f7d8"}
Oct 30 19:15:43.331584 ip-10-0-132-82 kubenswrapper[2577]: I1030 19:15:43.331554    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/e3206ddf-35de-4b7c-b979-682f5f666a34-registry-tls\") pod \"image-registry-6fcb4848df-8l8cp\" (UID: \"e3206ddf-35de-4b7c-b979-682f5f666a34\") " pod="openshift-image-registry/image-registry-6fcb4848df-8l8cp"
Oct 30 19:15:43.331748 ip-10-0-132-82 kubenswrapper[2577]: I1030 19:15:43.331624    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/87082646-75c3-43b5-864b-3478e9a7373d-metrics-certs\") pod \"router-default-776df59447-8nrhh\" (UID: \"87082646-75c3-43b5-864b-3478e9a7373d\") " pod="openshift-ingress/router-default-776df59447-8nrhh"
Oct 30 19:15:43.331829 ip-10-0-132-82 kubenswrapper[2577]: E1030 19:15:43.331743    2577 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Oct 30 19:15:43.331829 ip-10-0-132-82 kubenswrapper[2577]: E1030 19:15:43.331762    2577 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found
Oct 30 19:15:43.331829 ip-10-0-132-82 kubenswrapper[2577]: E1030 19:15:43.331765    2577 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-6fcb4848df-8l8cp: secret "image-registry-tls" not found
Oct 30 19:15:43.331829 ip-10-0-132-82 kubenswrapper[2577]: E1030 19:15:43.331822    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/87082646-75c3-43b5-864b-3478e9a7373d-metrics-certs podName:87082646-75c3-43b5-864b-3478e9a7373d nodeName:}" failed. No retries permitted until 2026-10-30 19:15:45.331805677 +0000 UTC m=+35.737726110 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/87082646-75c3-43b5-864b-3478e9a7373d-metrics-certs") pod "router-default-776df59447-8nrhh" (UID: "87082646-75c3-43b5-864b-3478e9a7373d") : secret "router-metrics-certs-default" not found
Oct 30 19:15:43.332049 ip-10-0-132-82 kubenswrapper[2577]: I1030 19:15:43.331845    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/87082646-75c3-43b5-864b-3478e9a7373d-service-ca-bundle\") pod \"router-default-776df59447-8nrhh\" (UID: \"87082646-75c3-43b5-864b-3478e9a7373d\") " pod="openshift-ingress/router-default-776df59447-8nrhh"
Oct 30 19:15:43.332049 ip-10-0-132-82 kubenswrapper[2577]: E1030 19:15:43.331859    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/e3206ddf-35de-4b7c-b979-682f5f666a34-registry-tls podName:e3206ddf-35de-4b7c-b979-682f5f666a34 nodeName:}" failed. No retries permitted until 2026-10-30 19:15:45.331844525 +0000 UTC m=+35.737764944 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/e3206ddf-35de-4b7c-b979-682f5f666a34-registry-tls") pod "image-registry-6fcb4848df-8l8cp" (UID: "e3206ddf-35de-4b7c-b979-682f5f666a34") : secret "image-registry-tls" not found
Oct 30 19:15:43.332049 ip-10-0-132-82 kubenswrapper[2577]: I1030 19:15:43.331903    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/56274b23-2400-43d4-9686-c928e20d47de-samples-operator-tls\") pod \"cluster-samples-operator-6ddb6c8b94-np94b\" (UID: \"56274b23-2400-43d4-9686-c928e20d47de\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-np94b"
Oct 30 19:15:43.332049 ip-10-0-132-82 kubenswrapper[2577]: E1030 19:15:43.331910    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/87082646-75c3-43b5-864b-3478e9a7373d-service-ca-bundle podName:87082646-75c3-43b5-864b-3478e9a7373d nodeName:}" failed. No retries permitted until 2026-10-30 19:15:45.331900772 +0000 UTC m=+35.737821207 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/87082646-75c3-43b5-864b-3478e9a7373d-service-ca-bundle") pod "router-default-776df59447-8nrhh" (UID: "87082646-75c3-43b5-864b-3478e9a7373d") : configmap references non-existent config key: service-ca.crt
Oct 30 19:15:43.332049 ip-10-0-132-82 kubenswrapper[2577]: E1030 19:15:43.331951    2577 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found
Oct 30 19:15:43.332049 ip-10-0-132-82 kubenswrapper[2577]: E1030 19:15:43.331998    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/56274b23-2400-43d4-9686-c928e20d47de-samples-operator-tls podName:56274b23-2400-43d4-9686-c928e20d47de nodeName:}" failed. No retries permitted until 2026-10-30 19:15:45.331990396 +0000 UTC m=+35.737910809 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/56274b23-2400-43d4-9686-c928e20d47de-samples-operator-tls") pod "cluster-samples-operator-6ddb6c8b94-np94b" (UID: "56274b23-2400-43d4-9686-c928e20d47de") : secret "samples-operator-tls" not found
Oct 30 19:15:43.433092 ip-10-0-132-82 kubenswrapper[2577]: I1030 19:15:43.432879    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/0e179af5-9f52-4941-b4f5-5e3b8789da11-registry-tls\") pod \"image-registry-d97c68fd4-htsl8\" (UID: \"0e179af5-9f52-4941-b4f5-5e3b8789da11\") " pod="openshift-image-registry/image-registry-d97c68fd4-htsl8"
Oct 30 19:15:43.433092 ip-10-0-132-82 kubenswrapper[2577]: I1030 19:15:43.433028    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/2c7c7060-def6-4737-8fa9-6889309b7463-networking-console-plugin-cert\") pod \"networking-console-plugin-86c478bb88-h4twn\" (UID: \"2c7c7060-def6-4737-8fa9-6889309b7463\") " pod="openshift-network-console/networking-console-plugin-86c478bb88-h4twn"
Oct 30 19:15:43.433536 ip-10-0-132-82 kubenswrapper[2577]: E1030 19:15:43.433183    2577 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found
Oct 30 19:15:43.433536 ip-10-0-132-82 kubenswrapper[2577]: E1030 19:15:43.433287    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/2c7c7060-def6-4737-8fa9-6889309b7463-networking-console-plugin-cert podName:2c7c7060-def6-4737-8fa9-6889309b7463 nodeName:}" failed. No retries permitted until 2026-10-30 19:15:45.433269335 +0000 UTC m=+35.839189757 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/2c7c7060-def6-4737-8fa9-6889309b7463-networking-console-plugin-cert") pod "networking-console-plugin-86c478bb88-h4twn" (UID: "2c7c7060-def6-4737-8fa9-6889309b7463") : secret "networking-console-plugin-cert" not found
Oct 30 19:15:43.433536 ip-10-0-132-82 kubenswrapper[2577]: E1030 19:15:43.433402    2577 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Oct 30 19:15:43.433536 ip-10-0-132-82 kubenswrapper[2577]: E1030 19:15:43.433414    2577 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-d97c68fd4-htsl8: secret "image-registry-tls" not found
Oct 30 19:15:43.433536 ip-10-0-132-82 kubenswrapper[2577]: E1030 19:15:43.433456    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/0e179af5-9f52-4941-b4f5-5e3b8789da11-registry-tls podName:0e179af5-9f52-4941-b4f5-5e3b8789da11 nodeName:}" failed. No retries permitted until 2026-10-30 19:15:45.433444901 +0000 UTC m=+35.839365320 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/0e179af5-9f52-4941-b4f5-5e3b8789da11-registry-tls") pod "image-registry-d97c68fd4-htsl8" (UID: "0e179af5-9f52-4941-b4f5-5e3b8789da11") : secret "image-registry-tls" not found
Oct 30 19:15:43.534436 ip-10-0-132-82 kubenswrapper[2577]: I1030 19:15:43.534352    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/e446ca63-5885-4c66-9cb7-da39afbf742c-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-884576c57-wkldq\" (UID: \"e446ca63-5885-4c66-9cb7-da39afbf742c\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-wkldq"
Oct 30 19:15:43.534436 ip-10-0-132-82 kubenswrapper[2577]: I1030 19:15:43.534429    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/fade00ba-24a7-41c9-93f7-23e40971874a-metrics-tls\") pod \"dns-default-bb9cj\" (UID: \"fade00ba-24a7-41c9-93f7-23e40971874a\") " pod="openshift-dns/dns-default-bb9cj"
Oct 30 19:15:43.534680 ip-10-0-132-82 kubenswrapper[2577]: E1030 19:15:43.534516    2577 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found
Oct 30 19:15:43.534680 ip-10-0-132-82 kubenswrapper[2577]: E1030 19:15:43.534585    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e446ca63-5885-4c66-9cb7-da39afbf742c-cluster-monitoring-operator-tls podName:e446ca63-5885-4c66-9cb7-da39afbf742c nodeName:}" failed. No retries permitted until 2026-10-30 19:15:45.53456464 +0000 UTC m=+35.940485079 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/e446ca63-5885-4c66-9cb7-da39afbf742c-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-884576c57-wkldq" (UID: "e446ca63-5885-4c66-9cb7-da39afbf742c") : secret "cluster-monitoring-operator-tls" not found
Oct 30 19:15:43.534680 ip-10-0-132-82 kubenswrapper[2577]: E1030 19:15:43.534645    2577 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Oct 30 19:15:43.534851 ip-10-0-132-82 kubenswrapper[2577]: E1030 19:15:43.534709    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/fade00ba-24a7-41c9-93f7-23e40971874a-metrics-tls podName:fade00ba-24a7-41c9-93f7-23e40971874a nodeName:}" failed. No retries permitted until 2026-10-30 19:15:45.534693153 +0000 UTC m=+35.940613584 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/fade00ba-24a7-41c9-93f7-23e40971874a-metrics-tls") pod "dns-default-bb9cj" (UID: "fade00ba-24a7-41c9-93f7-23e40971874a") : secret "dns-default-metrics-tls" not found
Oct 30 19:15:43.645820 ip-10-0-132-82 kubenswrapper[2577]: I1030 19:15:43.645752    2577 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kube-system/kas-connection-checker-685895c7c9-p8kjs" podStartSLOduration=420.645730489 podStartE2EDuration="7m0.645730489s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:08:43 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:15:43.643691571 +0000 UTC m=+34.049612010" watchObservedRunningTime="2026-10-30 19:15:43.645730489 +0000 UTC m=+34.051650930"
Oct 30 19:15:43.759612 ip-10-0-132-82 kubenswrapper[2577]: I1030 19:15:43.759563    2577 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kube-system/kas-connection-checker-685895c7c9-9h85f" podStartSLOduration=420.759548423 podStartE2EDuration="7m0.759548423s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:08:43 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:15:43.757956126 +0000 UTC m=+34.163876565" watchObservedRunningTime="2026-10-30 19:15:43.759548423 +0000 UTC m=+34.165468853"
Oct 30 19:15:43.839010 ip-10-0-132-82 kubenswrapper[2577]: I1030 19:15:43.838921    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/c16c0a80-c5ce-4f5d-827a-2e3eb2c3cfb8-original-pull-secret\") pod \"global-pull-secret-syncer-76rht\" (UID: \"c16c0a80-c5ce-4f5d-827a-2e3eb2c3cfb8\") " pod="kube-system/global-pull-secret-syncer-76rht"
Oct 30 19:15:43.839146 ip-10-0-132-82 kubenswrapper[2577]: I1030 19:15:43.839032    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/287ba08f-7b9f-4609-8e35-832c0879c44c-metrics-certs\") pod \"network-metrics-daemon-822gp\" (UID: \"287ba08f-7b9f-4609-8e35-832c0879c44c\") " pod="openshift-multus/network-metrics-daemon-822gp"
Oct 30 19:15:43.839146 ip-10-0-132-82 kubenswrapper[2577]: E1030 19:15:43.839123    2577 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: secret "metrics-daemon-secret" not found
Oct 30 19:15:43.839258 ip-10-0-132-82 kubenswrapper[2577]: E1030 19:15:43.839166    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/287ba08f-7b9f-4609-8e35-832c0879c44c-metrics-certs podName:287ba08f-7b9f-4609-8e35-832c0879c44c nodeName:}" failed. No retries permitted until 2026-10-30 19:16:15.839154661 +0000 UTC m=+66.245075089 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/287ba08f-7b9f-4609-8e35-832c0879c44c-metrics-certs") pod "network-metrics-daemon-822gp" (UID: "287ba08f-7b9f-4609-8e35-832c0879c44c") : secret "metrics-daemon-secret" not found
Oct 30 19:15:43.852929 ip-10-0-132-82 kubenswrapper[2577]: I1030 19:15:43.852901    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/c16c0a80-c5ce-4f5d-827a-2e3eb2c3cfb8-original-pull-secret\") pod \"global-pull-secret-syncer-76rht\" (UID: \"c16c0a80-c5ce-4f5d-827a-2e3eb2c3cfb8\") " pod="kube-system/global-pull-secret-syncer-76rht"
Oct 30 19:15:43.877389 ip-10-0-132-82 kubenswrapper[2577]: I1030 19:15:43.875787    2577 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kube-system/kas-connection-checker-685895c7c9-tvmks" podStartSLOduration=420.875769417 podStartE2EDuration="7m0.875769417s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:08:43 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:15:43.87472188 +0000 UTC m=+34.280642317" watchObservedRunningTime="2026-10-30 19:15:43.875769417 +0000 UTC m=+34.281689855"
Oct 30 19:15:43.880106 ip-10-0-132-82 kubenswrapper[2577]: I1030 19:15:43.880076    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-76rht"
Oct 30 19:15:43.941135 ip-10-0-132-82 kubenswrapper[2577]: I1030 19:15:43.940521    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bzw4s\" (UniqueName: \"kubernetes.io/projected/4ca01b49-9819-4a90-a851-7553ad568679-kube-api-access-bzw4s\") pod \"network-check-target-c24j9\" (UID: \"4ca01b49-9819-4a90-a851-7553ad568679\") " pod="openshift-network-diagnostics/network-check-target-c24j9"
Oct 30 19:15:43.944421 ip-10-0-132-82 kubenswrapper[2577]: I1030 19:15:43.944392    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bzw4s\" (UniqueName: \"kubernetes.io/projected/4ca01b49-9819-4a90-a851-7553ad568679-kube-api-access-bzw4s\") pod \"network-check-target-c24j9\" (UID: \"4ca01b49-9819-4a90-a851-7553ad568679\") " pod="openshift-network-diagnostics/network-check-target-c24j9"
Oct 30 19:15:44.198006 ip-10-0-132-82 kubenswrapper[2577]: I1030 19:15:44.197977    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-c24j9"
Oct 30 19:15:44.278702 ip-10-0-132-82 kubenswrapper[2577]: I1030 19:15:44.278666    2577 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-76rht"]
Oct 30 19:15:44.597610 ip-10-0-132-82 kubenswrapper[2577]: I1030 19:15:44.597524    2577 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-c24j9"]
Oct 30 19:15:45.257232 ip-10-0-132-82 kubenswrapper[2577]: I1030 19:15:45.257135    2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-76rht" event={"ID":"c16c0a80-c5ce-4f5d-827a-2e3eb2c3cfb8","Type":"ContainerStarted","Data":"e2d4de650f3571ac3ab4c420e5b93dc9761667b74a8fe10fb4e7b3decf5ace3f"}
Oct 30 19:15:45.260693 ip-10-0-132-82 kubenswrapper[2577]: I1030 19:15:45.260611    2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-c24j9" event={"ID":"4ca01b49-9819-4a90-a851-7553ad568679","Type":"ContainerStarted","Data":"abc2efb0ba45504bce33f1ce80ee88f9d7e4fc489dd1c5d851c51f8ed53b6b0b"}
Oct 30 19:15:45.356629 ip-10-0-132-82 kubenswrapper[2577]: I1030 19:15:45.356590    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/e3206ddf-35de-4b7c-b979-682f5f666a34-registry-tls\") pod \"image-registry-6fcb4848df-8l8cp\" (UID: \"e3206ddf-35de-4b7c-b979-682f5f666a34\") " pod="openshift-image-registry/image-registry-6fcb4848df-8l8cp"
Oct 30 19:15:45.356789 ip-10-0-132-82 kubenswrapper[2577]: I1030 19:15:45.356669    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/87082646-75c3-43b5-864b-3478e9a7373d-metrics-certs\") pod \"router-default-776df59447-8nrhh\" (UID: \"87082646-75c3-43b5-864b-3478e9a7373d\") " pod="openshift-ingress/router-default-776df59447-8nrhh"
Oct 30 19:15:45.356789 ip-10-0-132-82 kubenswrapper[2577]: I1030 19:15:45.356727    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/87082646-75c3-43b5-864b-3478e9a7373d-service-ca-bundle\") pod \"router-default-776df59447-8nrhh\" (UID: \"87082646-75c3-43b5-864b-3478e9a7373d\") " pod="openshift-ingress/router-default-776df59447-8nrhh"
Oct 30 19:15:45.356789 ip-10-0-132-82 kubenswrapper[2577]: I1030 19:15:45.356763    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/56274b23-2400-43d4-9686-c928e20d47de-samples-operator-tls\") pod \"cluster-samples-operator-6ddb6c8b94-np94b\" (UID: \"56274b23-2400-43d4-9686-c928e20d47de\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-np94b"
Oct 30 19:15:45.357047 ip-10-0-132-82 kubenswrapper[2577]: E1030 19:15:45.357005    2577 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found
Oct 30 19:15:45.357126 ip-10-0-132-82 kubenswrapper[2577]: E1030 19:15:45.357107    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/56274b23-2400-43d4-9686-c928e20d47de-samples-operator-tls podName:56274b23-2400-43d4-9686-c928e20d47de nodeName:}" failed. No retries permitted until 2026-10-30 19:15:49.357088077 +0000 UTC m=+39.763008507 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/56274b23-2400-43d4-9686-c928e20d47de-samples-operator-tls") pod "cluster-samples-operator-6ddb6c8b94-np94b" (UID: "56274b23-2400-43d4-9686-c928e20d47de") : secret "samples-operator-tls" not found
Oct 30 19:15:45.357559 ip-10-0-132-82 kubenswrapper[2577]: E1030 19:15:45.357520    2577 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Oct 30 19:15:45.357559 ip-10-0-132-82 kubenswrapper[2577]: E1030 19:15:45.357538    2577 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-6fcb4848df-8l8cp: secret "image-registry-tls" not found
Oct 30 19:15:45.357695 ip-10-0-132-82 kubenswrapper[2577]: E1030 19:15:45.357583    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/e3206ddf-35de-4b7c-b979-682f5f666a34-registry-tls podName:e3206ddf-35de-4b7c-b979-682f5f666a34 nodeName:}" failed. No retries permitted until 2026-10-30 19:15:49.357569375 +0000 UTC m=+39.763489790 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/e3206ddf-35de-4b7c-b979-682f5f666a34-registry-tls") pod "image-registry-6fcb4848df-8l8cp" (UID: "e3206ddf-35de-4b7c-b979-682f5f666a34") : secret "image-registry-tls" not found
Oct 30 19:15:45.357695 ip-10-0-132-82 kubenswrapper[2577]: E1030 19:15:45.357644    2577 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found
Oct 30 19:15:45.357695 ip-10-0-132-82 kubenswrapper[2577]: E1030 19:15:45.357691    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/87082646-75c3-43b5-864b-3478e9a7373d-metrics-certs podName:87082646-75c3-43b5-864b-3478e9a7373d nodeName:}" failed. No retries permitted until 2026-10-30 19:15:49.357680263 +0000 UTC m=+39.763600679 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/87082646-75c3-43b5-864b-3478e9a7373d-metrics-certs") pod "router-default-776df59447-8nrhh" (UID: "87082646-75c3-43b5-864b-3478e9a7373d") : secret "router-metrics-certs-default" not found
Oct 30 19:15:45.357857 ip-10-0-132-82 kubenswrapper[2577]: E1030 19:15:45.357818    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/87082646-75c3-43b5-864b-3478e9a7373d-service-ca-bundle podName:87082646-75c3-43b5-864b-3478e9a7373d nodeName:}" failed. No retries permitted until 2026-10-30 19:15:49.357806092 +0000 UTC m=+39.763726519 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/87082646-75c3-43b5-864b-3478e9a7373d-service-ca-bundle") pod "router-default-776df59447-8nrhh" (UID: "87082646-75c3-43b5-864b-3478e9a7373d") : configmap references non-existent config key: service-ca.crt
Oct 30 19:15:45.458450 ip-10-0-132-82 kubenswrapper[2577]: I1030 19:15:45.457940    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/0e179af5-9f52-4941-b4f5-5e3b8789da11-registry-tls\") pod \"image-registry-d97c68fd4-htsl8\" (UID: \"0e179af5-9f52-4941-b4f5-5e3b8789da11\") " pod="openshift-image-registry/image-registry-d97c68fd4-htsl8"
Oct 30 19:15:45.458450 ip-10-0-132-82 kubenswrapper[2577]: I1030 19:15:45.458057    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/2c7c7060-def6-4737-8fa9-6889309b7463-networking-console-plugin-cert\") pod \"networking-console-plugin-86c478bb88-h4twn\" (UID: \"2c7c7060-def6-4737-8fa9-6889309b7463\") " pod="openshift-network-console/networking-console-plugin-86c478bb88-h4twn"
Oct 30 19:15:45.458450 ip-10-0-132-82 kubenswrapper[2577]: E1030 19:15:45.458239    2577 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found
Oct 30 19:15:45.458450 ip-10-0-132-82 kubenswrapper[2577]: E1030 19:15:45.458301    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/2c7c7060-def6-4737-8fa9-6889309b7463-networking-console-plugin-cert podName:2c7c7060-def6-4737-8fa9-6889309b7463 nodeName:}" failed. No retries permitted until 2026-10-30 19:15:49.458282495 +0000 UTC m=+39.864202924 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/2c7c7060-def6-4737-8fa9-6889309b7463-networking-console-plugin-cert") pod "networking-console-plugin-86c478bb88-h4twn" (UID: "2c7c7060-def6-4737-8fa9-6889309b7463") : secret "networking-console-plugin-cert" not found
Oct 30 19:15:45.458450 ip-10-0-132-82 kubenswrapper[2577]: E1030 19:15:45.458368    2577 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Oct 30 19:15:45.458450 ip-10-0-132-82 kubenswrapper[2577]: E1030 19:15:45.458379    2577 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-d97c68fd4-htsl8: secret "image-registry-tls" not found
Oct 30 19:15:45.458450 ip-10-0-132-82 kubenswrapper[2577]: E1030 19:15:45.458411    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/0e179af5-9f52-4941-b4f5-5e3b8789da11-registry-tls podName:0e179af5-9f52-4941-b4f5-5e3b8789da11 nodeName:}" failed. No retries permitted until 2026-10-30 19:15:49.458400818 +0000 UTC m=+39.864321247 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/0e179af5-9f52-4941-b4f5-5e3b8789da11-registry-tls") pod "image-registry-d97c68fd4-htsl8" (UID: "0e179af5-9f52-4941-b4f5-5e3b8789da11") : secret "image-registry-tls" not found
Oct 30 19:15:45.559063 ip-10-0-132-82 kubenswrapper[2577]: I1030 19:15:45.559025    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/e446ca63-5885-4c66-9cb7-da39afbf742c-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-884576c57-wkldq\" (UID: \"e446ca63-5885-4c66-9cb7-da39afbf742c\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-wkldq"
Oct 30 19:15:45.559193 ip-10-0-132-82 kubenswrapper[2577]: I1030 19:15:45.559106    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/fade00ba-24a7-41c9-93f7-23e40971874a-metrics-tls\") pod \"dns-default-bb9cj\" (UID: \"fade00ba-24a7-41c9-93f7-23e40971874a\") " pod="openshift-dns/dns-default-bb9cj"
Oct 30 19:15:45.559395 ip-10-0-132-82 kubenswrapper[2577]: E1030 19:15:45.559370    2577 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Oct 30 19:15:45.559482 ip-10-0-132-82 kubenswrapper[2577]: E1030 19:15:45.559440    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/fade00ba-24a7-41c9-93f7-23e40971874a-metrics-tls podName:fade00ba-24a7-41c9-93f7-23e40971874a nodeName:}" failed. No retries permitted until 2026-10-30 19:15:49.559422678 +0000 UTC m=+39.965343104 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/fade00ba-24a7-41c9-93f7-23e40971874a-metrics-tls") pod "dns-default-bb9cj" (UID: "fade00ba-24a7-41c9-93f7-23e40971874a") : secret "dns-default-metrics-tls" not found
Oct 30 19:15:45.559580 ip-10-0-132-82 kubenswrapper[2577]: E1030 19:15:45.559513    2577 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found
Oct 30 19:15:45.559580 ip-10-0-132-82 kubenswrapper[2577]: E1030 19:15:45.559553    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e446ca63-5885-4c66-9cb7-da39afbf742c-cluster-monitoring-operator-tls podName:e446ca63-5885-4c66-9cb7-da39afbf742c nodeName:}" failed. No retries permitted until 2026-10-30 19:15:49.559539359 +0000 UTC m=+39.965459777 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/e446ca63-5885-4c66-9cb7-da39afbf742c-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-884576c57-wkldq" (UID: "e446ca63-5885-4c66-9cb7-da39afbf742c") : secret "cluster-monitoring-operator-tls" not found
Oct 30 19:15:46.268464 ip-10-0-132-82 kubenswrapper[2577]: I1030 19:15:46.268382    2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-storage-operator/volume-data-source-validator-58c9f5f46d-p42xv" event={"ID":"4d213db4-73c5-4f50-9930-6d26b880dcf7","Type":"ContainerStarted","Data":"fe094e78f97f253338267cc1568457b3bf242751b085515d50e9148b634181d8"}
Oct 30 19:15:46.413517 ip-10-0-132-82 kubenswrapper[2577]: I1030 19:15:46.413447    2577 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-cluster-storage-operator/volume-data-source-validator-58c9f5f46d-p42xv" podStartSLOduration=399.873328402 podStartE2EDuration="6m43.413426311s" totalImagesPullingTime="3.540097909s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:09:03 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:15:46.412860908 +0000 UTC m=+36.818781344" watchObservedRunningTime="2026-10-30 19:15:46.413426311 +0000 UTC m=+36.819346747"
Oct 30 19:15:48.294993 ip-10-0-132-82 kubenswrapper[2577]: I1030 19:15:48.281629    2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-scljn" event={"ID":"f9a06263-cb99-4004-bffc-581d66ce5ebf","Type":"ContainerStarted","Data":"4d3b9b7355e8a0ef0308f30c8d15fdb719373def0082085d6a0fa0f9c00888f8"}
Oct 30 19:15:48.294993 ip-10-0-132-82 kubenswrapper[2577]: I1030 19:15:48.283730    2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-operator-7c4986d449-6jdkq" event={"ID":"2bd84dc4-e96a-43f1-85bf-e1ea4005c193","Type":"ContainerStarted","Data":"99317e646b964681db30a7672208a6c1e4f68260f39249e8ede0b027a519c7e8"}
Oct 30 19:15:48.294993 ip-10-0-132-82 kubenswrapper[2577]: I1030 19:15:48.286165    2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console-operator/console-operator-7cd56b8dbc-tg5xj" event={"ID":"11db9cec-653f-4aee-8937-0cf5b367efac","Type":"ContainerStarted","Data":"8646b0d3acccce5150ad2283b22ff5003a8e416b57b8a968dd9891920e8d4c38"}
Oct 30 19:15:48.294993 ip-10-0-132-82 kubenswrapper[2577]: I1030 19:15:48.287326    2577 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console-operator/console-operator-7cd56b8dbc-tg5xj"
Oct 30 19:15:48.294993 ip-10-0-132-82 kubenswrapper[2577]: I1030 19:15:48.288671    2577 patch_prober.go:29] interesting pod/console-operator-7cd56b8dbc-tg5xj container/console-operator namespace/openshift-console-operator: Readiness probe status=failure output="Get \"https://10.132.0.20:8443/readyz\": dial tcp 10.132.0.20:8443: connect: connection refused" start-of-body=
Oct 30 19:15:48.294993 ip-10-0-132-82 kubenswrapper[2577]: I1030 19:15:48.288724    2577 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-console-operator/console-operator-7cd56b8dbc-tg5xj" podUID="11db9cec-653f-4aee-8937-0cf5b367efac" containerName="console-operator" probeResult="failure" output="Get \"https://10.132.0.20:8443/readyz\": dial tcp 10.132.0.20:8443: connect: connection refused"
Oct 30 19:15:48.398078 ip-10-0-132-82 kubenswrapper[2577]: I1030 19:15:48.398024    2577 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-scljn" podStartSLOduration=406.927973776 podStartE2EDuration="6m52.398005729s" totalImagesPullingTime="5.470031953s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:08:56 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:15:48.397916649 +0000 UTC m=+38.803837089" watchObservedRunningTime="2026-10-30 19:15:48.398005729 +0000 UTC m=+38.803926166"
Oct 30 19:15:48.518236 ip-10-0-132-82 kubenswrapper[2577]: I1030 19:15:48.518171    2577 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-insights/insights-operator-7c4986d449-6jdkq" podStartSLOduration=401.996463134 podStartE2EDuration="6m47.518153985s" totalImagesPullingTime="5.521690851s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:09:01 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:15:48.51744013 +0000 UTC m=+38.923360577" watchObservedRunningTime="2026-10-30 19:15:48.518153985 +0000 UTC m=+38.924074422"
Oct 30 19:15:48.636728 ip-10-0-132-82 kubenswrapper[2577]: I1030 19:15:48.636670    2577 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-console-operator/console-operator-7cd56b8dbc-tg5xj" podStartSLOduration=404.282642858 podStartE2EDuration="6m49.636655375s" totalImagesPullingTime="5.354012517s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:08:59 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:15:48.634405051 +0000 UTC m=+39.040325489" watchObservedRunningTime="2026-10-30 19:15:48.636655375 +0000 UTC m=+39.042575810"
Oct 30 19:15:49.293592 ip-10-0-132-82 kubenswrapper[2577]: I1030 19:15:49.292945    2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-nbfv2" event={"ID":"4ba601a6-2a44-4408-8c5b-2c29fd096fed","Type":"ContainerStarted","Data":"b643e2fa8015107d90d3afd9d3a043cb45baf6be1647fb60a60b39d30b7b255d"}
Oct 30 19:15:49.295588 ip-10-0-132-82 kubenswrapper[2577]: I1030 19:15:49.295557    2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-8rwjm" event={"ID":"f86c0e38-8144-42fc-b622-25602ada113f","Type":"ContainerStarted","Data":"e51dad50e9fc1c18fbd5bfc918704243616b5fce58fcfbea11af55e441cf5928"}
Oct 30 19:15:49.298563 ip-10-0-132-82 kubenswrapper[2577]: I1030 19:15:49.297698    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-tg5xj_11db9cec-653f-4aee-8937-0cf5b367efac/console-operator/0.log"
Oct 30 19:15:49.298563 ip-10-0-132-82 kubenswrapper[2577]: I1030 19:15:49.297737    2577 generic.go:356] "Generic (PLEG): container finished" podID="11db9cec-653f-4aee-8937-0cf5b367efac" containerID="8646b0d3acccce5150ad2283b22ff5003a8e416b57b8a968dd9891920e8d4c38" exitCode=255
Oct 30 19:15:49.298563 ip-10-0-132-82 kubenswrapper[2577]: I1030 19:15:49.297984    2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console-operator/console-operator-7cd56b8dbc-tg5xj" event={"ID":"11db9cec-653f-4aee-8937-0cf5b367efac","Type":"ContainerDied","Data":"8646b0d3acccce5150ad2283b22ff5003a8e416b57b8a968dd9891920e8d4c38"}
Oct 30 19:15:49.298563 ip-10-0-132-82 kubenswrapper[2577]: I1030 19:15:49.298245    2577 scope.go:122] "RemoveContainer" containerID="8646b0d3acccce5150ad2283b22ff5003a8e416b57b8a968dd9891920e8d4c38"
Oct 30 19:15:49.402249 ip-10-0-132-82 kubenswrapper[2577]: I1030 19:15:49.402214    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/87082646-75c3-43b5-864b-3478e9a7373d-metrics-certs\") pod \"router-default-776df59447-8nrhh\" (UID: \"87082646-75c3-43b5-864b-3478e9a7373d\") " pod="openshift-ingress/router-default-776df59447-8nrhh"
Oct 30 19:15:49.402368 ip-10-0-132-82 kubenswrapper[2577]: I1030 19:15:49.402349    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/87082646-75c3-43b5-864b-3478e9a7373d-service-ca-bundle\") pod \"router-default-776df59447-8nrhh\" (UID: \"87082646-75c3-43b5-864b-3478e9a7373d\") " pod="openshift-ingress/router-default-776df59447-8nrhh"
Oct 30 19:15:49.402453 ip-10-0-132-82 kubenswrapper[2577]: I1030 19:15:49.402416    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/56274b23-2400-43d4-9686-c928e20d47de-samples-operator-tls\") pod \"cluster-samples-operator-6ddb6c8b94-np94b\" (UID: \"56274b23-2400-43d4-9686-c928e20d47de\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-np94b"
Oct 30 19:15:49.402641 ip-10-0-132-82 kubenswrapper[2577]: I1030 19:15:49.402626    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/e3206ddf-35de-4b7c-b979-682f5f666a34-registry-tls\") pod \"image-registry-6fcb4848df-8l8cp\" (UID: \"e3206ddf-35de-4b7c-b979-682f5f666a34\") " pod="openshift-image-registry/image-registry-6fcb4848df-8l8cp"
Oct 30 19:15:49.402831 ip-10-0-132-82 kubenswrapper[2577]: E1030 19:15:49.402782    2577 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Oct 30 19:15:49.402831 ip-10-0-132-82 kubenswrapper[2577]: E1030 19:15:49.402799    2577 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-6fcb4848df-8l8cp: secret "image-registry-tls" not found
Oct 30 19:15:49.402950 ip-10-0-132-82 kubenswrapper[2577]: E1030 19:15:49.402854    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/e3206ddf-35de-4b7c-b979-682f5f666a34-registry-tls podName:e3206ddf-35de-4b7c-b979-682f5f666a34 nodeName:}" failed. No retries permitted until 2026-10-30 19:15:57.40283718 +0000 UTC m=+47.808757597 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/e3206ddf-35de-4b7c-b979-682f5f666a34-registry-tls") pod "image-registry-6fcb4848df-8l8cp" (UID: "e3206ddf-35de-4b7c-b979-682f5f666a34") : secret "image-registry-tls" not found
Oct 30 19:15:49.404444 ip-10-0-132-82 kubenswrapper[2577]: E1030 19:15:49.403122    2577 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found
Oct 30 19:15:49.404444 ip-10-0-132-82 kubenswrapper[2577]: E1030 19:15:49.403183    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/56274b23-2400-43d4-9686-c928e20d47de-samples-operator-tls podName:56274b23-2400-43d4-9686-c928e20d47de nodeName:}" failed. No retries permitted until 2026-10-30 19:15:57.40316613 +0000 UTC m=+47.809086568 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/56274b23-2400-43d4-9686-c928e20d47de-samples-operator-tls") pod "cluster-samples-operator-6ddb6c8b94-np94b" (UID: "56274b23-2400-43d4-9686-c928e20d47de") : secret "samples-operator-tls" not found
Oct 30 19:15:49.404444 ip-10-0-132-82 kubenswrapper[2577]: E1030 19:15:49.403392    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/87082646-75c3-43b5-864b-3478e9a7373d-service-ca-bundle podName:87082646-75c3-43b5-864b-3478e9a7373d nodeName:}" failed. No retries permitted until 2026-10-30 19:15:57.403378846 +0000 UTC m=+47.809299275 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/87082646-75c3-43b5-864b-3478e9a7373d-service-ca-bundle") pod "router-default-776df59447-8nrhh" (UID: "87082646-75c3-43b5-864b-3478e9a7373d") : configmap references non-existent config key: service-ca.crt
Oct 30 19:15:49.404444 ip-10-0-132-82 kubenswrapper[2577]: E1030 19:15:49.403400    2577 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found
Oct 30 19:15:49.404444 ip-10-0-132-82 kubenswrapper[2577]: E1030 19:15:49.403436    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/87082646-75c3-43b5-864b-3478e9a7373d-metrics-certs podName:87082646-75c3-43b5-864b-3478e9a7373d nodeName:}" failed. No retries permitted until 2026-10-30 19:15:57.403425575 +0000 UTC m=+47.809345997 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/87082646-75c3-43b5-864b-3478e9a7373d-metrics-certs") pod "router-default-776df59447-8nrhh" (UID: "87082646-75c3-43b5-864b-3478e9a7373d") : secret "router-metrics-certs-default" not found
Oct 30 19:15:49.418657 ip-10-0-132-82 kubenswrapper[2577]: I1030 19:15:49.418104    2577 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-nbfv2" podStartSLOduration=404.488073762 podStartE2EDuration="6m50.418087151s" totalImagesPullingTime="5.930013389s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:08:59 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:15:49.417151875 +0000 UTC m=+39.823072312" watchObservedRunningTime="2026-10-30 19:15:49.418087151 +0000 UTC m=+39.824007591"
Oct 30 19:15:49.503695 ip-10-0-132-82 kubenswrapper[2577]: I1030 19:15:49.503649    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/0e179af5-9f52-4941-b4f5-5e3b8789da11-registry-tls\") pod \"image-registry-d97c68fd4-htsl8\" (UID: \"0e179af5-9f52-4941-b4f5-5e3b8789da11\") " pod="openshift-image-registry/image-registry-d97c68fd4-htsl8"
Oct 30 19:15:49.503837 ip-10-0-132-82 kubenswrapper[2577]: I1030 19:15:49.503755    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/2c7c7060-def6-4737-8fa9-6889309b7463-networking-console-plugin-cert\") pod \"networking-console-plugin-86c478bb88-h4twn\" (UID: \"2c7c7060-def6-4737-8fa9-6889309b7463\") " pod="openshift-network-console/networking-console-plugin-86c478bb88-h4twn"
Oct 30 19:15:49.503837 ip-10-0-132-82 kubenswrapper[2577]: E1030 19:15:49.503825    2577 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Oct 30 19:15:49.503932 ip-10-0-132-82 kubenswrapper[2577]: E1030 19:15:49.503849    2577 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-d97c68fd4-htsl8: secret "image-registry-tls" not found
Oct 30 19:15:49.503932 ip-10-0-132-82 kubenswrapper[2577]: E1030 19:15:49.503907    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/0e179af5-9f52-4941-b4f5-5e3b8789da11-registry-tls podName:0e179af5-9f52-4941-b4f5-5e3b8789da11 nodeName:}" failed. No retries permitted until 2026-10-30 19:15:57.503889663 +0000 UTC m=+47.909810092 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/0e179af5-9f52-4941-b4f5-5e3b8789da11-registry-tls") pod "image-registry-d97c68fd4-htsl8" (UID: "0e179af5-9f52-4941-b4f5-5e3b8789da11") : secret "image-registry-tls" not found
Oct 30 19:15:49.503932 ip-10-0-132-82 kubenswrapper[2577]: E1030 19:15:49.503917    2577 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found
Oct 30 19:15:49.504096 ip-10-0-132-82 kubenswrapper[2577]: E1030 19:15:49.504001    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/2c7c7060-def6-4737-8fa9-6889309b7463-networking-console-plugin-cert podName:2c7c7060-def6-4737-8fa9-6889309b7463 nodeName:}" failed. No retries permitted until 2026-10-30 19:15:57.503985889 +0000 UTC m=+47.909906318 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/2c7c7060-def6-4737-8fa9-6889309b7463-networking-console-plugin-cert") pod "networking-console-plugin-86c478bb88-h4twn" (UID: "2c7c7060-def6-4737-8fa9-6889309b7463") : secret "networking-console-plugin-cert" not found
Oct 30 19:15:49.565723 ip-10-0-132-82 kubenswrapper[2577]: I1030 19:15:49.565638    2577 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-insights/insights-runtime-extractor-jrm87"]
Oct 30 19:15:49.585523 ip-10-0-132-82 kubenswrapper[2577]: I1030 19:15:49.585493    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-jrm87"
Oct 30 19:15:49.604631 ip-10-0-132-82 kubenswrapper[2577]: I1030 19:15:49.604597    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/e446ca63-5885-4c66-9cb7-da39afbf742c-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-884576c57-wkldq\" (UID: \"e446ca63-5885-4c66-9cb7-da39afbf742c\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-wkldq"
Oct 30 19:15:49.604760 ip-10-0-132-82 kubenswrapper[2577]: I1030 19:15:49.604687    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/fade00ba-24a7-41c9-93f7-23e40971874a-metrics-tls\") pod \"dns-default-bb9cj\" (UID: \"fade00ba-24a7-41c9-93f7-23e40971874a\") " pod="openshift-dns/dns-default-bb9cj"
Oct 30 19:15:49.604828 ip-10-0-132-82 kubenswrapper[2577]: E1030 19:15:49.604753    2577 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found
Oct 30 19:15:49.604881 ip-10-0-132-82 kubenswrapper[2577]: E1030 19:15:49.604825    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e446ca63-5885-4c66-9cb7-da39afbf742c-cluster-monitoring-operator-tls podName:e446ca63-5885-4c66-9cb7-da39afbf742c nodeName:}" failed. No retries permitted until 2026-10-30 19:15:57.604805693 +0000 UTC m=+48.010726124 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/e446ca63-5885-4c66-9cb7-da39afbf742c-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-884576c57-wkldq" (UID: "e446ca63-5885-4c66-9cb7-da39afbf742c") : secret "cluster-monitoring-operator-tls" not found
Oct 30 19:15:49.604881 ip-10-0-132-82 kubenswrapper[2577]: E1030 19:15:49.604845    2577 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Oct 30 19:15:49.605024 ip-10-0-132-82 kubenswrapper[2577]: E1030 19:15:49.604906    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/fade00ba-24a7-41c9-93f7-23e40971874a-metrics-tls podName:fade00ba-24a7-41c9-93f7-23e40971874a nodeName:}" failed. No retries permitted until 2026-10-30 19:15:57.604888265 +0000 UTC m=+48.010808680 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/fade00ba-24a7-41c9-93f7-23e40971874a-metrics-tls") pod "dns-default-bb9cj" (UID: "fade00ba-24a7-41c9-93f7-23e40971874a") : secret "dns-default-metrics-tls" not found
Oct 30 19:15:49.627556 ip-10-0-132-82 kubenswrapper[2577]: I1030 19:15:49.627451    2577 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-jrm87"]
Oct 30 19:15:49.657950 ip-10-0-132-82 kubenswrapper[2577]: I1030 19:15:49.657893    2577 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress-canary/ingress-canary-8rwjm" podStartSLOduration=2.495118516 podStartE2EDuration="8.657875422s" totalImagesPullingTime="6.162756906s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:15:41 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:15:49.656650575 +0000 UTC m=+40.062571011" watchObservedRunningTime="2026-10-30 19:15:49.657875422 +0000 UTC m=+40.063795858"
Oct 30 19:15:49.693207 ip-10-0-132-82 kubenswrapper[2577]: I1030 19:15:49.693176    2577 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-rbac-proxy\""
Oct 30 19:15:49.693359 ip-10-0-132-82 kubenswrapper[2577]: I1030 19:15:49.693244    2577 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-sa-dockercfg-mnhjf\""
Oct 30 19:15:49.693359 ip-10-0-132-82 kubenswrapper[2577]: I1030 19:15:49.693252    2577 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-tls\""
Oct 30 19:15:49.705838 ip-10-0-132-82 kubenswrapper[2577]: I1030 19:15:49.705807    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/fc4ac297-f510-4439-80e2-487056d3bf06-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-jrm87\" (UID: \"fc4ac297-f510-4439-80e2-487056d3bf06\") " pod="openshift-insights/insights-runtime-extractor-jrm87"
Oct 30 19:15:49.705985 ip-10-0-132-82 kubenswrapper[2577]: I1030 19:15:49.705896    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/fc4ac297-f510-4439-80e2-487056d3bf06-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-jrm87\" (UID: \"fc4ac297-f510-4439-80e2-487056d3bf06\") " pod="openshift-insights/insights-runtime-extractor-jrm87"
Oct 30 19:15:49.705985 ip-10-0-132-82 kubenswrapper[2577]: I1030 19:15:49.705943    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/fc4ac297-f510-4439-80e2-487056d3bf06-data-volume\") pod \"insights-runtime-extractor-jrm87\" (UID: \"fc4ac297-f510-4439-80e2-487056d3bf06\") " pod="openshift-insights/insights-runtime-extractor-jrm87"
Oct 30 19:15:49.706132 ip-10-0-132-82 kubenswrapper[2577]: I1030 19:15:49.706080    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7d4bn\" (UniqueName: \"kubernetes.io/projected/fc4ac297-f510-4439-80e2-487056d3bf06-kube-api-access-7d4bn\") pod \"insights-runtime-extractor-jrm87\" (UID: \"fc4ac297-f510-4439-80e2-487056d3bf06\") " pod="openshift-insights/insights-runtime-extractor-jrm87"
Oct 30 19:15:49.706186 ip-10-0-132-82 kubenswrapper[2577]: I1030 19:15:49.706141    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/fc4ac297-f510-4439-80e2-487056d3bf06-crio-socket\") pod \"insights-runtime-extractor-jrm87\" (UID: \"fc4ac297-f510-4439-80e2-487056d3bf06\") " pod="openshift-insights/insights-runtime-extractor-jrm87"
Oct 30 19:15:49.807200 ip-10-0-132-82 kubenswrapper[2577]: I1030 19:15:49.807162    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/fc4ac297-f510-4439-80e2-487056d3bf06-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-jrm87\" (UID: \"fc4ac297-f510-4439-80e2-487056d3bf06\") " pod="openshift-insights/insights-runtime-extractor-jrm87"
Oct 30 19:15:49.807373 ip-10-0-132-82 kubenswrapper[2577]: I1030 19:15:49.807218    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/fc4ac297-f510-4439-80e2-487056d3bf06-data-volume\") pod \"insights-runtime-extractor-jrm87\" (UID: \"fc4ac297-f510-4439-80e2-487056d3bf06\") " pod="openshift-insights/insights-runtime-extractor-jrm87"
Oct 30 19:15:49.807373 ip-10-0-132-82 kubenswrapper[2577]: I1030 19:15:49.807300    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7d4bn\" (UniqueName: \"kubernetes.io/projected/fc4ac297-f510-4439-80e2-487056d3bf06-kube-api-access-7d4bn\") pod \"insights-runtime-extractor-jrm87\" (UID: \"fc4ac297-f510-4439-80e2-487056d3bf06\") " pod="openshift-insights/insights-runtime-extractor-jrm87"
Oct 30 19:15:49.807373 ip-10-0-132-82 kubenswrapper[2577]: I1030 19:15:49.807345    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/fc4ac297-f510-4439-80e2-487056d3bf06-crio-socket\") pod \"insights-runtime-extractor-jrm87\" (UID: \"fc4ac297-f510-4439-80e2-487056d3bf06\") " pod="openshift-insights/insights-runtime-extractor-jrm87"
Oct 30 19:15:49.807528 ip-10-0-132-82 kubenswrapper[2577]: I1030 19:15:49.807378    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/fc4ac297-f510-4439-80e2-487056d3bf06-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-jrm87\" (UID: \"fc4ac297-f510-4439-80e2-487056d3bf06\") " pod="openshift-insights/insights-runtime-extractor-jrm87"
Oct 30 19:15:49.807528 ip-10-0-132-82 kubenswrapper[2577]: E1030 19:15:49.807452    2577 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Oct 30 19:15:49.807528 ip-10-0-132-82 kubenswrapper[2577]: E1030 19:15:49.807509    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/fc4ac297-f510-4439-80e2-487056d3bf06-kube-rbac-proxy-cm podName:fc4ac297-f510-4439-80e2-487056d3bf06 nodeName:}" failed. No retries permitted until 2026-10-30 19:15:50.307491683 +0000 UTC m=+40.713412113 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/fc4ac297-f510-4439-80e2-487056d3bf06-kube-rbac-proxy-cm") pod "insights-runtime-extractor-jrm87" (UID: "fc4ac297-f510-4439-80e2-487056d3bf06") : configmap "kube-rbac-proxy" not found
Oct 30 19:15:49.807628 ip-10-0-132-82 kubenswrapper[2577]: E1030 19:15:49.807584    2577 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Oct 30 19:15:49.807628 ip-10-0-132-82 kubenswrapper[2577]: E1030 19:15:49.807611    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/fc4ac297-f510-4439-80e2-487056d3bf06-insights-runtime-extractor-tls podName:fc4ac297-f510-4439-80e2-487056d3bf06 nodeName:}" failed. No retries permitted until 2026-10-30 19:15:50.307603936 +0000 UTC m=+40.713524363 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/fc4ac297-f510-4439-80e2-487056d3bf06-insights-runtime-extractor-tls") pod "insights-runtime-extractor-jrm87" (UID: "fc4ac297-f510-4439-80e2-487056d3bf06") : secret "insights-runtime-extractor-tls" not found
Oct 30 19:15:49.807930 ip-10-0-132-82 kubenswrapper[2577]: I1030 19:15:49.807900    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/fc4ac297-f510-4439-80e2-487056d3bf06-data-volume\") pod \"insights-runtime-extractor-jrm87\" (UID: \"fc4ac297-f510-4439-80e2-487056d3bf06\") " pod="openshift-insights/insights-runtime-extractor-jrm87"
Oct 30 19:15:49.808260 ip-10-0-132-82 kubenswrapper[2577]: I1030 19:15:49.808231    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/fc4ac297-f510-4439-80e2-487056d3bf06-crio-socket\") pod \"insights-runtime-extractor-jrm87\" (UID: \"fc4ac297-f510-4439-80e2-487056d3bf06\") " pod="openshift-insights/insights-runtime-extractor-jrm87"
Oct 30 19:15:49.865511 ip-10-0-132-82 kubenswrapper[2577]: I1030 19:15:49.865444    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7d4bn\" (UniqueName: \"kubernetes.io/projected/fc4ac297-f510-4439-80e2-487056d3bf06-kube-api-access-7d4bn\") pod \"insights-runtime-extractor-jrm87\" (UID: \"fc4ac297-f510-4439-80e2-487056d3bf06\") " pod="openshift-insights/insights-runtime-extractor-jrm87"
Oct 30 19:15:50.303146 ip-10-0-132-82 kubenswrapper[2577]: I1030 19:15:50.303123    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-tg5xj_11db9cec-653f-4aee-8937-0cf5b367efac/console-operator/1.log"
Oct 30 19:15:50.303600 ip-10-0-132-82 kubenswrapper[2577]: I1030 19:15:50.303583    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-tg5xj_11db9cec-653f-4aee-8937-0cf5b367efac/console-operator/0.log"
Oct 30 19:15:50.303673 ip-10-0-132-82 kubenswrapper[2577]: I1030 19:15:50.303616    2577 generic.go:356] "Generic (PLEG): container finished" podID="11db9cec-653f-4aee-8937-0cf5b367efac" containerID="dafc11f60068899cfdf6d420a7f0b52ca486082253a85db5cf117594b4979646" exitCode=255
Oct 30 19:15:50.304098 ip-10-0-132-82 kubenswrapper[2577]: I1030 19:15:50.304064    2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console-operator/console-operator-7cd56b8dbc-tg5xj" event={"ID":"11db9cec-653f-4aee-8937-0cf5b367efac","Type":"ContainerDied","Data":"dafc11f60068899cfdf6d420a7f0b52ca486082253a85db5cf117594b4979646"}
Oct 30 19:15:50.304220 ip-10-0-132-82 kubenswrapper[2577]: I1030 19:15:50.304142    2577 scope.go:122] "RemoveContainer" containerID="8646b0d3acccce5150ad2283b22ff5003a8e416b57b8a968dd9891920e8d4c38"
Oct 30 19:15:50.304495 ip-10-0-132-82 kubenswrapper[2577]: I1030 19:15:50.304471    2577 scope.go:122] "RemoveContainer" containerID="dafc11f60068899cfdf6d420a7f0b52ca486082253a85db5cf117594b4979646"
Oct 30 19:15:50.304733 ip-10-0-132-82 kubenswrapper[2577]: E1030 19:15:50.304657    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"console-operator\" with CrashLoopBackOff: \"back-off 10s restarting failed container=console-operator pod=console-operator-7cd56b8dbc-tg5xj_openshift-console-operator(11db9cec-653f-4aee-8937-0cf5b367efac)\"" pod="openshift-console-operator/console-operator-7cd56b8dbc-tg5xj" podUID="11db9cec-653f-4aee-8937-0cf5b367efac"
Oct 30 19:15:50.312528 ip-10-0-132-82 kubenswrapper[2577]: I1030 19:15:50.312504    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/fc4ac297-f510-4439-80e2-487056d3bf06-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-jrm87\" (UID: \"fc4ac297-f510-4439-80e2-487056d3bf06\") " pod="openshift-insights/insights-runtime-extractor-jrm87"
Oct 30 19:15:50.312691 ip-10-0-132-82 kubenswrapper[2577]: I1030 19:15:50.312659    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/fc4ac297-f510-4439-80e2-487056d3bf06-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-jrm87\" (UID: \"fc4ac297-f510-4439-80e2-487056d3bf06\") " pod="openshift-insights/insights-runtime-extractor-jrm87"
Oct 30 19:15:50.312808 ip-10-0-132-82 kubenswrapper[2577]: E1030 19:15:50.312768    2577 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Oct 30 19:15:50.312866 ip-10-0-132-82 kubenswrapper[2577]: E1030 19:15:50.312822    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/fc4ac297-f510-4439-80e2-487056d3bf06-kube-rbac-proxy-cm podName:fc4ac297-f510-4439-80e2-487056d3bf06 nodeName:}" failed. No retries permitted until 2026-10-30 19:15:51.312805285 +0000 UTC m=+41.718725705 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/fc4ac297-f510-4439-80e2-487056d3bf06-kube-rbac-proxy-cm") pod "insights-runtime-extractor-jrm87" (UID: "fc4ac297-f510-4439-80e2-487056d3bf06") : configmap "kube-rbac-proxy" not found
Oct 30 19:15:50.312931 ip-10-0-132-82 kubenswrapper[2577]: E1030 19:15:50.312882    2577 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Oct 30 19:15:50.312931 ip-10-0-132-82 kubenswrapper[2577]: E1030 19:15:50.312906    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/fc4ac297-f510-4439-80e2-487056d3bf06-insights-runtime-extractor-tls podName:fc4ac297-f510-4439-80e2-487056d3bf06 nodeName:}" failed. No retries permitted until 2026-10-30 19:15:51.312899347 +0000 UTC m=+41.718819761 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/fc4ac297-f510-4439-80e2-487056d3bf06-insights-runtime-extractor-tls") pod "insights-runtime-extractor-jrm87" (UID: "fc4ac297-f510-4439-80e2-487056d3bf06") : secret "insights-runtime-extractor-tls" not found
Oct 30 19:15:51.309062 ip-10-0-132-82 kubenswrapper[2577]: I1030 19:15:51.309034    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-tg5xj_11db9cec-653f-4aee-8937-0cf5b367efac/console-operator/1.log"
Oct 30 19:15:51.309488 ip-10-0-132-82 kubenswrapper[2577]: I1030 19:15:51.309416    2577 scope.go:122] "RemoveContainer" containerID="dafc11f60068899cfdf6d420a7f0b52ca486082253a85db5cf117594b4979646"
Oct 30 19:15:51.309698 ip-10-0-132-82 kubenswrapper[2577]: E1030 19:15:51.309659    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"console-operator\" with CrashLoopBackOff: \"back-off 10s restarting failed container=console-operator pod=console-operator-7cd56b8dbc-tg5xj_openshift-console-operator(11db9cec-653f-4aee-8937-0cf5b367efac)\"" pod="openshift-console-operator/console-operator-7cd56b8dbc-tg5xj" podUID="11db9cec-653f-4aee-8937-0cf5b367efac"
Oct 30 19:15:51.310865 ip-10-0-132-82 kubenswrapper[2577]: I1030 19:15:51.310828    2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-source-6f78d59f99-2trjq" event={"ID":"33ce6ceb-02a9-4852-9948-f02e2b39e32f","Type":"ContainerStarted","Data":"422eb30c7c1c64c2e386ed8e8bf2f7c809af668eab1ba767276f92f12171dca5"}
Oct 30 19:15:51.312623 ip-10-0-132-82 kubenswrapper[2577]: I1030 19:15:51.312595    2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-c24j9" event={"ID":"4ca01b49-9819-4a90-a851-7553ad568679","Type":"ContainerStarted","Data":"655c7fff4a34e299631b288704286bb531d65ba03c7966df010837dde6c36c1c"}
Oct 30 19:15:51.312764 ip-10-0-132-82 kubenswrapper[2577]: I1030 19:15:51.312741    2577 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-network-diagnostics/network-check-target-c24j9"
Oct 30 19:15:51.323609 ip-10-0-132-82 kubenswrapper[2577]: I1030 19:15:51.323589    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/fc4ac297-f510-4439-80e2-487056d3bf06-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-jrm87\" (UID: \"fc4ac297-f510-4439-80e2-487056d3bf06\") " pod="openshift-insights/insights-runtime-extractor-jrm87"
Oct 30 19:15:51.323757 ip-10-0-132-82 kubenswrapper[2577]: E1030 19:15:51.323737    2577 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Oct 30 19:15:51.323817 ip-10-0-132-82 kubenswrapper[2577]: E1030 19:15:51.323802    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/fc4ac297-f510-4439-80e2-487056d3bf06-insights-runtime-extractor-tls podName:fc4ac297-f510-4439-80e2-487056d3bf06 nodeName:}" failed. No retries permitted until 2026-10-30 19:15:53.323783267 +0000 UTC m=+43.729703702 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/fc4ac297-f510-4439-80e2-487056d3bf06-insights-runtime-extractor-tls") pod "insights-runtime-extractor-jrm87" (UID: "fc4ac297-f510-4439-80e2-487056d3bf06") : secret "insights-runtime-extractor-tls" not found
Oct 30 19:15:51.323929 ip-10-0-132-82 kubenswrapper[2577]: I1030 19:15:51.323834    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/fc4ac297-f510-4439-80e2-487056d3bf06-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-jrm87\" (UID: \"fc4ac297-f510-4439-80e2-487056d3bf06\") " pod="openshift-insights/insights-runtime-extractor-jrm87"
Oct 30 19:15:51.324249 ip-10-0-132-82 kubenswrapper[2577]: E1030 19:15:51.324233    2577 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Oct 30 19:15:51.324324 ip-10-0-132-82 kubenswrapper[2577]: E1030 19:15:51.324280    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/fc4ac297-f510-4439-80e2-487056d3bf06-kube-rbac-proxy-cm podName:fc4ac297-f510-4439-80e2-487056d3bf06 nodeName:}" failed. No retries permitted until 2026-10-30 19:15:53.324266344 +0000 UTC m=+43.730186776 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/fc4ac297-f510-4439-80e2-487056d3bf06-kube-rbac-proxy-cm") pod "insights-runtime-extractor-jrm87" (UID: "fc4ac297-f510-4439-80e2-487056d3bf06") : configmap "kube-rbac-proxy" not found
Oct 30 19:15:51.544115 ip-10-0-132-82 kubenswrapper[2577]: I1030 19:15:51.544036    2577 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-network-diagnostics/network-check-source-6f78d59f99-2trjq" podStartSLOduration=386.724568061 podStartE2EDuration="6m34.544017521s" totalImagesPullingTime="7.81944946s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:09:17 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:15:51.542024991 +0000 UTC m=+41.947945425" watchObservedRunningTime="2026-10-30 19:15:51.544017521 +0000 UTC m=+41.949937959"
Oct 30 19:15:51.655510 ip-10-0-132-82 kubenswrapper[2577]: I1030 19:15:51.655441    2577 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-network-diagnostics/network-check-target-c24j9" podStartSLOduration=35.476764054 podStartE2EDuration="41.655420638s" totalImagesPullingTime="6.178656584s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:15:10 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:15:51.654920418 +0000 UTC m=+42.060840850" watchObservedRunningTime="2026-10-30 19:15:51.655420638 +0000 UTC m=+42.061341078"
Oct 30 19:15:52.103127 ip-10-0-132-82 kubenswrapper[2577]: I1030 19:15:52.103103    2577 kubelet.go:2727] "SyncLoop (probe)" probe="liveness" status="unhealthy" pod="openshift-console-operator/console-operator-7cd56b8dbc-tg5xj"
Oct 30 19:15:52.320158 ip-10-0-132-82 kubenswrapper[2577]: I1030 19:15:52.320109    2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-76rht" event={"ID":"c16c0a80-c5ce-4f5d-827a-2e3eb2c3cfb8","Type":"ContainerStarted","Data":"003a8218daf906c147ed81ea47b055b77889822c1cabe2db76bcca47bacc94bf"}
Oct 30 19:15:52.320702 ip-10-0-132-82 kubenswrapper[2577]: I1030 19:15:52.320682    2577 scope.go:122] "RemoveContainer" containerID="dafc11f60068899cfdf6d420a7f0b52ca486082253a85db5cf117594b4979646"
Oct 30 19:15:52.320876 ip-10-0-132-82 kubenswrapper[2577]: E1030 19:15:52.320854    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"console-operator\" with CrashLoopBackOff: \"back-off 10s restarting failed container=console-operator pod=console-operator-7cd56b8dbc-tg5xj_openshift-console-operator(11db9cec-653f-4aee-8937-0cf5b367efac)\"" pod="openshift-console-operator/console-operator-7cd56b8dbc-tg5xj" podUID="11db9cec-653f-4aee-8937-0cf5b367efac"
Oct 30 19:15:52.375624 ip-10-0-132-82 kubenswrapper[2577]: I1030 19:15:52.375464    2577 dynamic_cafile_content.go:175] "Shutting down controller" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt"
Oct 30 19:15:52.375656 ip-10-0-132-82 systemd[1]: Stopping Kubernetes Kubelet...
Oct 30 19:15:52.389330 ip-10-0-132-82 systemd[1]: kubelet.service: Deactivated successfully.
Oct 30 19:15:52.389498 ip-10-0-132-82 systemd[1]: Stopped Kubernetes Kubelet.
Oct 30 19:15:52.389550 ip-10-0-132-82 systemd[1]: kubelet.service: Consumed 2.947s CPU time.
Oct 30 19:15:52.398602 ip-10-0-132-82 systemd[1]: Starting Kubernetes Kubelet...
Oct 30 19:15:52.490591 ip-10-0-132-82 kubenswrapper[4892]: Flag --container-runtime-endpoint has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Oct 30 19:15:52.490591 ip-10-0-132-82 kubenswrapper[4892]: Flag --minimum-container-ttl-duration has been deprecated, Use --eviction-hard or --eviction-soft instead. Will be removed in a future version.
Oct 30 19:15:52.490591 ip-10-0-132-82 kubenswrapper[4892]: Flag --volume-plugin-dir has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Oct 30 19:15:52.490591 ip-10-0-132-82 kubenswrapper[4892]: Flag --system-reserved has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Oct 30 19:15:52.492646 ip-10-0-132-82 kubenswrapper[4892]: W1030 19:15:52.492574    4892 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Oct 30 19:15:52.492646 ip-10-0-132-82 kubenswrapper[4892]: W1030 19:15:52.492646    4892 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Oct 30 19:15:52.492755 ip-10-0-132-82 kubenswrapper[4892]: W1030 19:15:52.492651    4892 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Oct 30 19:15:52.492755 ip-10-0-132-82 kubenswrapper[4892]: W1030 19:15:52.492654    4892 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Oct 30 19:15:52.492755 ip-10-0-132-82 kubenswrapper[4892]: W1030 19:15:52.492658    4892 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Oct 30 19:15:52.492755 ip-10-0-132-82 kubenswrapper[4892]: W1030 19:15:52.492661    4892 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Oct 30 19:15:52.492755 ip-10-0-132-82 kubenswrapper[4892]: W1030 19:15:52.492664    4892 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Oct 30 19:15:52.492755 ip-10-0-132-82 kubenswrapper[4892]: W1030 19:15:52.492666    4892 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Oct 30 19:15:52.492755 ip-10-0-132-82 kubenswrapper[4892]: W1030 19:15:52.492669    4892 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Oct 30 19:15:52.492755 ip-10-0-132-82 kubenswrapper[4892]: W1030 19:15:52.492672    4892 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Oct 30 19:15:52.492755 ip-10-0-132-82 kubenswrapper[4892]: W1030 19:15:52.492675    4892 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Oct 30 19:15:52.492755 ip-10-0-132-82 kubenswrapper[4892]: W1030 19:15:52.492677    4892 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Oct 30 19:15:52.492755 ip-10-0-132-82 kubenswrapper[4892]: W1030 19:15:52.492680    4892 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Oct 30 19:15:52.492755 ip-10-0-132-82 kubenswrapper[4892]: W1030 19:15:52.492682    4892 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Oct 30 19:15:52.492755 ip-10-0-132-82 kubenswrapper[4892]: W1030 19:15:52.492685    4892 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Oct 30 19:15:52.492755 ip-10-0-132-82 kubenswrapper[4892]: W1030 19:15:52.492691    4892 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Oct 30 19:15:52.492755 ip-10-0-132-82 kubenswrapper[4892]: W1030 19:15:52.492694    4892 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Oct 30 19:15:52.492755 ip-10-0-132-82 kubenswrapper[4892]: W1030 19:15:52.492696    4892 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Oct 30 19:15:52.492755 ip-10-0-132-82 kubenswrapper[4892]: W1030 19:15:52.492699    4892 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Oct 30 19:15:52.492755 ip-10-0-132-82 kubenswrapper[4892]: W1030 19:15:52.492701    4892 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Oct 30 19:15:52.492755 ip-10-0-132-82 kubenswrapper[4892]: W1030 19:15:52.492704    4892 feature_gate.go:386] unrecognized feature gate: Example
Oct 30 19:15:52.492755 ip-10-0-132-82 kubenswrapper[4892]: W1030 19:15:52.492707    4892 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Oct 30 19:15:52.493337 ip-10-0-132-82 kubenswrapper[4892]: W1030 19:15:52.492709    4892 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Oct 30 19:15:52.493337 ip-10-0-132-82 kubenswrapper[4892]: W1030 19:15:52.492711    4892 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Oct 30 19:15:52.493337 ip-10-0-132-82 kubenswrapper[4892]: W1030 19:15:52.492714    4892 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Oct 30 19:15:52.493337 ip-10-0-132-82 kubenswrapper[4892]: W1030 19:15:52.492716    4892 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Oct 30 19:15:52.493337 ip-10-0-132-82 kubenswrapper[4892]: W1030 19:15:52.492720    4892 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Oct 30 19:15:52.493337 ip-10-0-132-82 kubenswrapper[4892]: W1030 19:15:52.492722    4892 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Oct 30 19:15:52.493337 ip-10-0-132-82 kubenswrapper[4892]: W1030 19:15:52.492725    4892 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Oct 30 19:15:52.493337 ip-10-0-132-82 kubenswrapper[4892]: W1030 19:15:52.492728    4892 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Oct 30 19:15:52.493337 ip-10-0-132-82 kubenswrapper[4892]: W1030 19:15:52.492731    4892 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Oct 30 19:15:52.493337 ip-10-0-132-82 kubenswrapper[4892]: W1030 19:15:52.492733    4892 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Oct 30 19:15:52.493337 ip-10-0-132-82 kubenswrapper[4892]: W1030 19:15:52.492739    4892 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Oct 30 19:15:52.493337 ip-10-0-132-82 kubenswrapper[4892]: W1030 19:15:52.492741    4892 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Oct 30 19:15:52.493337 ip-10-0-132-82 kubenswrapper[4892]: W1030 19:15:52.492744    4892 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Oct 30 19:15:52.493337 ip-10-0-132-82 kubenswrapper[4892]: W1030 19:15:52.492747    4892 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Oct 30 19:15:52.493337 ip-10-0-132-82 kubenswrapper[4892]: W1030 19:15:52.492749    4892 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Oct 30 19:15:52.493337 ip-10-0-132-82 kubenswrapper[4892]: W1030 19:15:52.492752    4892 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Oct 30 19:15:52.493337 ip-10-0-132-82 kubenswrapper[4892]: W1030 19:15:52.492754    4892 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Oct 30 19:15:52.493337 ip-10-0-132-82 kubenswrapper[4892]: W1030 19:15:52.492757    4892 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Oct 30 19:15:52.493337 ip-10-0-132-82 kubenswrapper[4892]: W1030 19:15:52.492759    4892 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Oct 30 19:15:52.493801 ip-10-0-132-82 kubenswrapper[4892]: W1030 19:15:52.492762    4892 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Oct 30 19:15:52.493801 ip-10-0-132-82 kubenswrapper[4892]: W1030 19:15:52.492765    4892 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Oct 30 19:15:52.493801 ip-10-0-132-82 kubenswrapper[4892]: W1030 19:15:52.492768    4892 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Oct 30 19:15:52.493801 ip-10-0-132-82 kubenswrapper[4892]: W1030 19:15:52.492773    4892 feature_gate.go:386] unrecognized feature gate: OVNObservability
Oct 30 19:15:52.493801 ip-10-0-132-82 kubenswrapper[4892]: W1030 19:15:52.492776    4892 feature_gate.go:386] unrecognized feature gate: NewOLM
Oct 30 19:15:52.493801 ip-10-0-132-82 kubenswrapper[4892]: W1030 19:15:52.492779    4892 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Oct 30 19:15:52.493801 ip-10-0-132-82 kubenswrapper[4892]: W1030 19:15:52.492781    4892 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Oct 30 19:15:52.493801 ip-10-0-132-82 kubenswrapper[4892]: W1030 19:15:52.492784    4892 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Oct 30 19:15:52.493801 ip-10-0-132-82 kubenswrapper[4892]: W1030 19:15:52.492787    4892 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Oct 30 19:15:52.493801 ip-10-0-132-82 kubenswrapper[4892]: W1030 19:15:52.492790    4892 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Oct 30 19:15:52.493801 ip-10-0-132-82 kubenswrapper[4892]: W1030 19:15:52.492792    4892 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Oct 30 19:15:52.493801 ip-10-0-132-82 kubenswrapper[4892]: W1030 19:15:52.492796    4892 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Oct 30 19:15:52.493801 ip-10-0-132-82 kubenswrapper[4892]: W1030 19:15:52.492798    4892 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Oct 30 19:15:52.493801 ip-10-0-132-82 kubenswrapper[4892]: W1030 19:15:52.492801    4892 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Oct 30 19:15:52.493801 ip-10-0-132-82 kubenswrapper[4892]: W1030 19:15:52.492804    4892 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Oct 30 19:15:52.493801 ip-10-0-132-82 kubenswrapper[4892]: W1030 19:15:52.492806    4892 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Oct 30 19:15:52.493801 ip-10-0-132-82 kubenswrapper[4892]: W1030 19:15:52.492809    4892 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Oct 30 19:15:52.493801 ip-10-0-132-82 kubenswrapper[4892]: W1030 19:15:52.492811    4892 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Oct 30 19:15:52.493801 ip-10-0-132-82 kubenswrapper[4892]: W1030 19:15:52.492815    4892 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Oct 30 19:15:52.494287 ip-10-0-132-82 kubenswrapper[4892]: W1030 19:15:52.492819    4892 feature_gate.go:386] unrecognized feature gate: DualReplica
Oct 30 19:15:52.494287 ip-10-0-132-82 kubenswrapper[4892]: W1030 19:15:52.492822    4892 feature_gate.go:386] unrecognized feature gate: EVPN
Oct 30 19:15:52.494287 ip-10-0-132-82 kubenswrapper[4892]: W1030 19:15:52.492825    4892 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Oct 30 19:15:52.494287 ip-10-0-132-82 kubenswrapper[4892]: W1030 19:15:52.492828    4892 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Oct 30 19:15:52.494287 ip-10-0-132-82 kubenswrapper[4892]: W1030 19:15:52.492831    4892 feature_gate.go:386] unrecognized feature gate: Example2
Oct 30 19:15:52.494287 ip-10-0-132-82 kubenswrapper[4892]: W1030 19:15:52.492834    4892 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Oct 30 19:15:52.494287 ip-10-0-132-82 kubenswrapper[4892]: W1030 19:15:52.492837    4892 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Oct 30 19:15:52.494287 ip-10-0-132-82 kubenswrapper[4892]: W1030 19:15:52.492839    4892 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Oct 30 19:15:52.494287 ip-10-0-132-82 kubenswrapper[4892]: W1030 19:15:52.492842    4892 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Oct 30 19:15:52.494287 ip-10-0-132-82 kubenswrapper[4892]: W1030 19:15:52.492844    4892 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Oct 30 19:15:52.494287 ip-10-0-132-82 kubenswrapper[4892]: W1030 19:15:52.492847    4892 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Oct 30 19:15:52.494287 ip-10-0-132-82 kubenswrapper[4892]: W1030 19:15:52.492850    4892 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Oct 30 19:15:52.494287 ip-10-0-132-82 kubenswrapper[4892]: W1030 19:15:52.492852    4892 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Oct 30 19:15:52.494287 ip-10-0-132-82 kubenswrapper[4892]: W1030 19:15:52.492855    4892 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Oct 30 19:15:52.494287 ip-10-0-132-82 kubenswrapper[4892]: W1030 19:15:52.492857    4892 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Oct 30 19:15:52.494287 ip-10-0-132-82 kubenswrapper[4892]: W1030 19:15:52.492860    4892 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Oct 30 19:15:52.494287 ip-10-0-132-82 kubenswrapper[4892]: W1030 19:15:52.492862    4892 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Oct 30 19:15:52.494287 ip-10-0-132-82 kubenswrapper[4892]: W1030 19:15:52.492865    4892 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Oct 30 19:15:52.494287 ip-10-0-132-82 kubenswrapper[4892]: W1030 19:15:52.492867    4892 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Oct 30 19:15:52.494287 ip-10-0-132-82 kubenswrapper[4892]: W1030 19:15:52.492870    4892 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Oct 30 19:15:52.494765 ip-10-0-132-82 kubenswrapper[4892]: W1030 19:15:52.492873    4892 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Oct 30 19:15:52.494765 ip-10-0-132-82 kubenswrapper[4892]: W1030 19:15:52.492876    4892 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Oct 30 19:15:52.494765 ip-10-0-132-82 kubenswrapper[4892]: W1030 19:15:52.492879    4892 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Oct 30 19:15:52.494765 ip-10-0-132-82 kubenswrapper[4892]: W1030 19:15:52.492881    4892 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Oct 30 19:15:52.494765 ip-10-0-132-82 kubenswrapper[4892]: W1030 19:15:52.492884    4892 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Oct 30 19:15:52.494765 ip-10-0-132-82 kubenswrapper[4892]: W1030 19:15:52.492886    4892 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Oct 30 19:15:52.494765 ip-10-0-132-82 kubenswrapper[4892]: W1030 19:15:52.492889    4892 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Oct 30 19:15:52.494765 ip-10-0-132-82 kubenswrapper[4892]: W1030 19:15:52.492891    4892 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Oct 30 19:15:52.494765 ip-10-0-132-82 kubenswrapper[4892]: W1030 19:15:52.492894    4892 feature_gate.go:386] unrecognized feature gate: SignatureStores
Oct 30 19:15:52.494765 ip-10-0-132-82 kubenswrapper[4892]: W1030 19:15:52.492898    4892 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Oct 30 19:15:52.494765 ip-10-0-132-82 kubenswrapper[4892]: W1030 19:15:52.492900    4892 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Oct 30 19:15:52.494765 ip-10-0-132-82 kubenswrapper[4892]: W1030 19:15:52.492903    4892 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Oct 30 19:15:52.494765 ip-10-0-132-82 kubenswrapper[4892]: W1030 19:15:52.492905    4892 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Oct 30 19:15:52.494765 ip-10-0-132-82 kubenswrapper[4892]: W1030 19:15:52.492908    4892 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Oct 30 19:15:52.494765 ip-10-0-132-82 kubenswrapper[4892]: W1030 19:15:52.492910    4892 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Oct 30 19:15:52.494765 ip-10-0-132-82 kubenswrapper[4892]: W1030 19:15:52.492913    4892 feature_gate.go:386] unrecognized feature gate: EventTTL
Oct 30 19:15:52.494765 ip-10-0-132-82 kubenswrapper[4892]: W1030 19:15:52.492917    4892 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Oct 30 19:15:52.494765 ip-10-0-132-82 kubenswrapper[4892]: W1030 19:15:52.492919    4892 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Oct 30 19:15:52.494765 ip-10-0-132-82 kubenswrapper[4892]: W1030 19:15:52.492922    4892 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Oct 30 19:15:52.495255 ip-10-0-132-82 kubenswrapper[4892]: W1030 19:15:52.492924    4892 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Oct 30 19:15:52.495255 ip-10-0-132-82 kubenswrapper[4892]: W1030 19:15:52.492927    4892 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Oct 30 19:15:52.495255 ip-10-0-132-82 kubenswrapper[4892]: W1030 19:15:52.492930    4892 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Oct 30 19:15:52.495255 ip-10-0-132-82 kubenswrapper[4892]: W1030 19:15:52.492932    4892 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Oct 30 19:15:52.495255 ip-10-0-132-82 kubenswrapper[4892]: W1030 19:15:52.492935    4892 feature_gate.go:386] unrecognized feature gate: OSStreams
Oct 30 19:15:52.495255 ip-10-0-132-82 kubenswrapper[4892]: W1030 19:15:52.492937    4892 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Oct 30 19:15:52.495255 ip-10-0-132-82 kubenswrapper[4892]: W1030 19:15:52.492940    4892 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Oct 30 19:15:52.495255 ip-10-0-132-82 kubenswrapper[4892]: W1030 19:15:52.493676    4892 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Oct 30 19:15:52.495255 ip-10-0-132-82 kubenswrapper[4892]: W1030 19:15:52.493683    4892 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Oct 30 19:15:52.495255 ip-10-0-132-82 kubenswrapper[4892]: W1030 19:15:52.493685    4892 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Oct 30 19:15:52.495255 ip-10-0-132-82 kubenswrapper[4892]: W1030 19:15:52.493688    4892 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Oct 30 19:15:52.495255 ip-10-0-132-82 kubenswrapper[4892]: W1030 19:15:52.493691    4892 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Oct 30 19:15:52.495255 ip-10-0-132-82 kubenswrapper[4892]: W1030 19:15:52.493693    4892 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Oct 30 19:15:52.495255 ip-10-0-132-82 kubenswrapper[4892]: W1030 19:15:52.493696    4892 feature_gate.go:386] unrecognized feature gate: SignatureStores
Oct 30 19:15:52.495255 ip-10-0-132-82 kubenswrapper[4892]: W1030 19:15:52.493699    4892 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Oct 30 19:15:52.495255 ip-10-0-132-82 kubenswrapper[4892]: W1030 19:15:52.493702    4892 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Oct 30 19:15:52.495255 ip-10-0-132-82 kubenswrapper[4892]: W1030 19:15:52.493705    4892 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Oct 30 19:15:52.495255 ip-10-0-132-82 kubenswrapper[4892]: W1030 19:15:52.493708    4892 feature_gate.go:386] unrecognized feature gate: EVPN
Oct 30 19:15:52.495255 ip-10-0-132-82 kubenswrapper[4892]: W1030 19:15:52.493710    4892 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Oct 30 19:15:52.495255 ip-10-0-132-82 kubenswrapper[4892]: W1030 19:15:52.493713    4892 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Oct 30 19:15:52.495759 ip-10-0-132-82 kubenswrapper[4892]: W1030 19:15:52.493715    4892 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Oct 30 19:15:52.495759 ip-10-0-132-82 kubenswrapper[4892]: W1030 19:15:52.493718    4892 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Oct 30 19:15:52.495759 ip-10-0-132-82 kubenswrapper[4892]: W1030 19:15:52.493721    4892 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Oct 30 19:15:52.495759 ip-10-0-132-82 kubenswrapper[4892]: W1030 19:15:52.493723    4892 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Oct 30 19:15:52.495759 ip-10-0-132-82 kubenswrapper[4892]: W1030 19:15:52.493726    4892 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Oct 30 19:15:52.495759 ip-10-0-132-82 kubenswrapper[4892]: W1030 19:15:52.493729    4892 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Oct 30 19:15:52.495759 ip-10-0-132-82 kubenswrapper[4892]: W1030 19:15:52.493731    4892 feature_gate.go:386] unrecognized feature gate: EventTTL
Oct 30 19:15:52.495759 ip-10-0-132-82 kubenswrapper[4892]: W1030 19:15:52.493734    4892 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Oct 30 19:15:52.495759 ip-10-0-132-82 kubenswrapper[4892]: W1030 19:15:52.493736    4892 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Oct 30 19:15:52.495759 ip-10-0-132-82 kubenswrapper[4892]: W1030 19:15:52.493739    4892 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Oct 30 19:15:52.495759 ip-10-0-132-82 kubenswrapper[4892]: W1030 19:15:52.493742    4892 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Oct 30 19:15:52.495759 ip-10-0-132-82 kubenswrapper[4892]: W1030 19:15:52.493744    4892 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Oct 30 19:15:52.495759 ip-10-0-132-82 kubenswrapper[4892]: W1030 19:15:52.493747    4892 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Oct 30 19:15:52.495759 ip-10-0-132-82 kubenswrapper[4892]: W1030 19:15:52.493750    4892 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Oct 30 19:15:52.495759 ip-10-0-132-82 kubenswrapper[4892]: W1030 19:15:52.493752    4892 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Oct 30 19:15:52.495759 ip-10-0-132-82 kubenswrapper[4892]: W1030 19:15:52.493755    4892 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Oct 30 19:15:52.495759 ip-10-0-132-82 kubenswrapper[4892]: W1030 19:15:52.493757    4892 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Oct 30 19:15:52.495759 ip-10-0-132-82 kubenswrapper[4892]: W1030 19:15:52.493760    4892 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Oct 30 19:15:52.495759 ip-10-0-132-82 kubenswrapper[4892]: W1030 19:15:52.493762    4892 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Oct 30 19:15:52.495759 ip-10-0-132-82 kubenswrapper[4892]: W1030 19:15:52.493765    4892 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Oct 30 19:15:52.496270 ip-10-0-132-82 kubenswrapper[4892]: W1030 19:15:52.493767    4892 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Oct 30 19:15:52.496270 ip-10-0-132-82 kubenswrapper[4892]: W1030 19:15:52.493770    4892 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Oct 30 19:15:52.496270 ip-10-0-132-82 kubenswrapper[4892]: W1030 19:15:52.493772    4892 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Oct 30 19:15:52.496270 ip-10-0-132-82 kubenswrapper[4892]: W1030 19:15:52.493775    4892 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Oct 30 19:15:52.496270 ip-10-0-132-82 kubenswrapper[4892]: W1030 19:15:52.493777    4892 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Oct 30 19:15:52.496270 ip-10-0-132-82 kubenswrapper[4892]: W1030 19:15:52.493780    4892 feature_gate.go:386] unrecognized feature gate: DualReplica
Oct 30 19:15:52.496270 ip-10-0-132-82 kubenswrapper[4892]: W1030 19:15:52.493782    4892 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Oct 30 19:15:52.496270 ip-10-0-132-82 kubenswrapper[4892]: W1030 19:15:52.493785    4892 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Oct 30 19:15:52.496270 ip-10-0-132-82 kubenswrapper[4892]: W1030 19:15:52.493787    4892 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Oct 30 19:15:52.496270 ip-10-0-132-82 kubenswrapper[4892]: W1030 19:15:52.493790    4892 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Oct 30 19:15:52.496270 ip-10-0-132-82 kubenswrapper[4892]: W1030 19:15:52.493792    4892 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Oct 30 19:15:52.496270 ip-10-0-132-82 kubenswrapper[4892]: W1030 19:15:52.493795    4892 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Oct 30 19:15:52.496270 ip-10-0-132-82 kubenswrapper[4892]: W1030 19:15:52.493797    4892 feature_gate.go:386] unrecognized feature gate: Example2
Oct 30 19:15:52.496270 ip-10-0-132-82 kubenswrapper[4892]: W1030 19:15:52.493800    4892 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Oct 30 19:15:52.496270 ip-10-0-132-82 kubenswrapper[4892]: W1030 19:15:52.493802    4892 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Oct 30 19:15:52.496270 ip-10-0-132-82 kubenswrapper[4892]: W1030 19:15:52.493805    4892 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Oct 30 19:15:52.496270 ip-10-0-132-82 kubenswrapper[4892]: W1030 19:15:52.493808    4892 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Oct 30 19:15:52.496270 ip-10-0-132-82 kubenswrapper[4892]: W1030 19:15:52.493811    4892 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Oct 30 19:15:52.496270 ip-10-0-132-82 kubenswrapper[4892]: W1030 19:15:52.493813    4892 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Oct 30 19:15:52.496746 ip-10-0-132-82 kubenswrapper[4892]: W1030 19:15:52.493819    4892 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Oct 30 19:15:52.496746 ip-10-0-132-82 kubenswrapper[4892]: W1030 19:15:52.493822    4892 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Oct 30 19:15:52.496746 ip-10-0-132-82 kubenswrapper[4892]: W1030 19:15:52.493826    4892 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Oct 30 19:15:52.496746 ip-10-0-132-82 kubenswrapper[4892]: W1030 19:15:52.493829    4892 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Oct 30 19:15:52.496746 ip-10-0-132-82 kubenswrapper[4892]: W1030 19:15:52.493831    4892 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Oct 30 19:15:52.496746 ip-10-0-132-82 kubenswrapper[4892]: W1030 19:15:52.493834    4892 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Oct 30 19:15:52.496746 ip-10-0-132-82 kubenswrapper[4892]: W1030 19:15:52.493837    4892 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Oct 30 19:15:52.496746 ip-10-0-132-82 kubenswrapper[4892]: W1030 19:15:52.493839    4892 feature_gate.go:386] unrecognized feature gate: NewOLM
Oct 30 19:15:52.496746 ip-10-0-132-82 kubenswrapper[4892]: W1030 19:15:52.493842    4892 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Oct 30 19:15:52.496746 ip-10-0-132-82 kubenswrapper[4892]: W1030 19:15:52.493844    4892 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Oct 30 19:15:52.496746 ip-10-0-132-82 kubenswrapper[4892]: W1030 19:15:52.493847    4892 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Oct 30 19:15:52.496746 ip-10-0-132-82 kubenswrapper[4892]: W1030 19:15:52.493849    4892 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Oct 30 19:15:52.496746 ip-10-0-132-82 kubenswrapper[4892]: W1030 19:15:52.493852    4892 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Oct 30 19:15:52.496746 ip-10-0-132-82 kubenswrapper[4892]: W1030 19:15:52.493854    4892 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Oct 30 19:15:52.496746 ip-10-0-132-82 kubenswrapper[4892]: W1030 19:15:52.493857    4892 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Oct 30 19:15:52.496746 ip-10-0-132-82 kubenswrapper[4892]: W1030 19:15:52.493859    4892 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Oct 30 19:15:52.496746 ip-10-0-132-82 kubenswrapper[4892]: W1030 19:15:52.493862    4892 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Oct 30 19:15:52.496746 ip-10-0-132-82 kubenswrapper[4892]: W1030 19:15:52.493864    4892 feature_gate.go:386] unrecognized feature gate: OSStreams
Oct 30 19:15:52.496746 ip-10-0-132-82 kubenswrapper[4892]: W1030 19:15:52.493867    4892 feature_gate.go:386] unrecognized feature gate: Example
Oct 30 19:15:52.496746 ip-10-0-132-82 kubenswrapper[4892]: W1030 19:15:52.493869    4892 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Oct 30 19:15:52.497274 ip-10-0-132-82 kubenswrapper[4892]: W1030 19:15:52.493872    4892 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Oct 30 19:15:52.497274 ip-10-0-132-82 kubenswrapper[4892]: W1030 19:15:52.493874    4892 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Oct 30 19:15:52.497274 ip-10-0-132-82 kubenswrapper[4892]: W1030 19:15:52.493877    4892 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Oct 30 19:15:52.497274 ip-10-0-132-82 kubenswrapper[4892]: W1030 19:15:52.493880    4892 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Oct 30 19:15:52.497274 ip-10-0-132-82 kubenswrapper[4892]: W1030 19:15:52.493883    4892 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Oct 30 19:15:52.497274 ip-10-0-132-82 kubenswrapper[4892]: W1030 19:15:52.493885    4892 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Oct 30 19:15:52.497274 ip-10-0-132-82 kubenswrapper[4892]: W1030 19:15:52.493888    4892 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Oct 30 19:15:52.497274 ip-10-0-132-82 kubenswrapper[4892]: W1030 19:15:52.493890    4892 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Oct 30 19:15:52.497274 ip-10-0-132-82 kubenswrapper[4892]: W1030 19:15:52.493893    4892 feature_gate.go:386] unrecognized feature gate: OVNObservability
Oct 30 19:15:52.497274 ip-10-0-132-82 kubenswrapper[4892]: W1030 19:15:52.493896    4892 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Oct 30 19:15:52.497274 ip-10-0-132-82 kubenswrapper[4892]: W1030 19:15:52.493898    4892 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Oct 30 19:15:52.497274 ip-10-0-132-82 kubenswrapper[4892]: W1030 19:15:52.493901    4892 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Oct 30 19:15:52.497274 ip-10-0-132-82 kubenswrapper[4892]: W1030 19:15:52.493905    4892 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Oct 30 19:15:52.497274 ip-10-0-132-82 kubenswrapper[4892]: W1030 19:15:52.493909    4892 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Oct 30 19:15:52.497274 ip-10-0-132-82 kubenswrapper[4892]: W1030 19:15:52.493912    4892 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Oct 30 19:15:52.497274 ip-10-0-132-82 kubenswrapper[4892]: W1030 19:15:52.493915    4892 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Oct 30 19:15:52.497274 ip-10-0-132-82 kubenswrapper[4892]: W1030 19:15:52.493917    4892 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Oct 30 19:15:52.497274 ip-10-0-132-82 kubenswrapper[4892]: W1030 19:15:52.493920    4892 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Oct 30 19:15:52.497274 ip-10-0-132-82 kubenswrapper[4892]: W1030 19:15:52.493922    4892 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Oct 30 19:15:52.497737 ip-10-0-132-82 kubenswrapper[4892]: W1030 19:15:52.493926    4892 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Oct 30 19:15:52.497737 ip-10-0-132-82 kubenswrapper[4892]: W1030 19:15:52.493928    4892 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Oct 30 19:15:52.497737 ip-10-0-132-82 kubenswrapper[4892]: W1030 19:15:52.493931    4892 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Oct 30 19:15:52.497737 ip-10-0-132-82 kubenswrapper[4892]: W1030 19:15:52.493933    4892 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Oct 30 19:15:52.497737 ip-10-0-132-82 kubenswrapper[4892]: W1030 19:15:52.493935    4892 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Oct 30 19:15:52.497737 ip-10-0-132-82 kubenswrapper[4892]: W1030 19:15:52.493938    4892 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Oct 30 19:15:52.497737 ip-10-0-132-82 kubenswrapper[4892]: W1030 19:15:52.493940    4892 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Oct 30 19:15:52.497737 ip-10-0-132-82 kubenswrapper[4892]: W1030 19:15:52.493943    4892 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Oct 30 19:15:52.497737 ip-10-0-132-82 kubenswrapper[4892]: W1030 19:15:52.493945    4892 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Oct 30 19:15:52.497737 ip-10-0-132-82 kubenswrapper[4892]: W1030 19:15:52.493948    4892 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Oct 30 19:15:52.497737 ip-10-0-132-82 kubenswrapper[4892]: W1030 19:15:52.493950    4892 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Oct 30 19:15:52.497737 ip-10-0-132-82 kubenswrapper[4892]: W1030 19:15:52.493953    4892 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Oct 30 19:15:52.497737 ip-10-0-132-82 kubenswrapper[4892]: W1030 19:15:52.493955    4892 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Oct 30 19:15:52.497737 ip-10-0-132-82 kubenswrapper[4892]: W1030 19:15:52.493958    4892 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Oct 30 19:15:52.497737 ip-10-0-132-82 kubenswrapper[4892]: W1030 19:15:52.493975    4892 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Oct 30 19:15:52.497737 ip-10-0-132-82 kubenswrapper[4892]: W1030 19:15:52.494475    4892 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Oct 30 19:15:52.497737 ip-10-0-132-82 kubenswrapper[4892]: W1030 19:15:52.494481    4892 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Oct 30 19:15:52.497737 ip-10-0-132-82 kubenswrapper[4892]: W1030 19:15:52.494484    4892 feature_gate.go:386] unrecognized feature gate: SignatureStores
Oct 30 19:15:52.497737 ip-10-0-132-82 kubenswrapper[4892]: W1030 19:15:52.494488    4892 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Oct 30 19:15:52.498239 ip-10-0-132-82 kubenswrapper[4892]: W1030 19:15:52.494491    4892 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Oct 30 19:15:52.498239 ip-10-0-132-82 kubenswrapper[4892]: W1030 19:15:52.494493    4892 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Oct 30 19:15:52.498239 ip-10-0-132-82 kubenswrapper[4892]: W1030 19:15:52.494496    4892 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Oct 30 19:15:52.498239 ip-10-0-132-82 kubenswrapper[4892]: W1030 19:15:52.494498    4892 feature_gate.go:386] unrecognized feature gate: Example
Oct 30 19:15:52.498239 ip-10-0-132-82 kubenswrapper[4892]: W1030 19:15:52.494501    4892 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Oct 30 19:15:52.498239 ip-10-0-132-82 kubenswrapper[4892]: W1030 19:15:52.494504    4892 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Oct 30 19:15:52.498239 ip-10-0-132-82 kubenswrapper[4892]: W1030 19:15:52.494507    4892 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Oct 30 19:15:52.498239 ip-10-0-132-82 kubenswrapper[4892]: W1030 19:15:52.494510    4892 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Oct 30 19:15:52.498239 ip-10-0-132-82 kubenswrapper[4892]: W1030 19:15:52.494513    4892 feature_gate.go:386] unrecognized feature gate: EVPN
Oct 30 19:15:52.498239 ip-10-0-132-82 kubenswrapper[4892]: W1030 19:15:52.494515    4892 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Oct 30 19:15:52.498239 ip-10-0-132-82 kubenswrapper[4892]: W1030 19:15:52.494518    4892 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Oct 30 19:15:52.498239 ip-10-0-132-82 kubenswrapper[4892]: W1030 19:15:52.494520    4892 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Oct 30 19:15:52.498239 ip-10-0-132-82 kubenswrapper[4892]: W1030 19:15:52.494523    4892 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Oct 30 19:15:52.498239 ip-10-0-132-82 kubenswrapper[4892]: W1030 19:15:52.494526    4892 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Oct 30 19:15:52.498239 ip-10-0-132-82 kubenswrapper[4892]: W1030 19:15:52.494528    4892 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Oct 30 19:15:52.498239 ip-10-0-132-82 kubenswrapper[4892]: W1030 19:15:52.494530    4892 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Oct 30 19:15:52.498239 ip-10-0-132-82 kubenswrapper[4892]: W1030 19:15:52.494533    4892 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Oct 30 19:15:52.498239 ip-10-0-132-82 kubenswrapper[4892]: W1030 19:15:52.494535    4892 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Oct 30 19:15:52.498239 ip-10-0-132-82 kubenswrapper[4892]: W1030 19:15:52.494538    4892 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Oct 30 19:15:52.498239 ip-10-0-132-82 kubenswrapper[4892]: W1030 19:15:52.494540    4892 feature_gate.go:386] unrecognized feature gate: NewOLM
Oct 30 19:15:52.498726 ip-10-0-132-82 kubenswrapper[4892]: W1030 19:15:52.494543    4892 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Oct 30 19:15:52.498726 ip-10-0-132-82 kubenswrapper[4892]: W1030 19:15:52.494545    4892 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Oct 30 19:15:52.498726 ip-10-0-132-82 kubenswrapper[4892]: W1030 19:15:52.494548    4892 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Oct 30 19:15:52.498726 ip-10-0-132-82 kubenswrapper[4892]: W1030 19:15:52.494550    4892 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Oct 30 19:15:52.498726 ip-10-0-132-82 kubenswrapper[4892]: W1030 19:15:52.494553    4892 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Oct 30 19:15:52.498726 ip-10-0-132-82 kubenswrapper[4892]: W1030 19:15:52.494555    4892 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Oct 30 19:15:52.498726 ip-10-0-132-82 kubenswrapper[4892]: W1030 19:15:52.494558    4892 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Oct 30 19:15:52.498726 ip-10-0-132-82 kubenswrapper[4892]: W1030 19:15:52.494561    4892 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Oct 30 19:15:52.498726 ip-10-0-132-82 kubenswrapper[4892]: W1030 19:15:52.494563    4892 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Oct 30 19:15:52.498726 ip-10-0-132-82 kubenswrapper[4892]: W1030 19:15:52.494566    4892 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Oct 30 19:15:52.498726 ip-10-0-132-82 kubenswrapper[4892]: W1030 19:15:52.494569    4892 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Oct 30 19:15:52.498726 ip-10-0-132-82 kubenswrapper[4892]: W1030 19:15:52.494574    4892 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Oct 30 19:15:52.498726 ip-10-0-132-82 kubenswrapper[4892]: W1030 19:15:52.494576    4892 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Oct 30 19:15:52.498726 ip-10-0-132-82 kubenswrapper[4892]: W1030 19:15:52.494579    4892 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Oct 30 19:15:52.498726 ip-10-0-132-82 kubenswrapper[4892]: W1030 19:15:52.494581    4892 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Oct 30 19:15:52.498726 ip-10-0-132-82 kubenswrapper[4892]: W1030 19:15:52.494584    4892 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Oct 30 19:15:52.498726 ip-10-0-132-82 kubenswrapper[4892]: W1030 19:15:52.494586    4892 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Oct 30 19:15:52.498726 ip-10-0-132-82 kubenswrapper[4892]: W1030 19:15:52.494589    4892 feature_gate.go:386] unrecognized feature gate: DualReplica
Oct 30 19:15:52.498726 ip-10-0-132-82 kubenswrapper[4892]: W1030 19:15:52.494592    4892 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Oct 30 19:15:52.498726 ip-10-0-132-82 kubenswrapper[4892]: W1030 19:15:52.494594    4892 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Oct 30 19:15:52.499229 ip-10-0-132-82 kubenswrapper[4892]: W1030 19:15:52.494597    4892 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Oct 30 19:15:52.499229 ip-10-0-132-82 kubenswrapper[4892]: W1030 19:15:52.494599    4892 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Oct 30 19:15:52.499229 ip-10-0-132-82 kubenswrapper[4892]: W1030 19:15:52.494602    4892 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Oct 30 19:15:52.499229 ip-10-0-132-82 kubenswrapper[4892]: W1030 19:15:52.494604    4892 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Oct 30 19:15:52.499229 ip-10-0-132-82 kubenswrapper[4892]: W1030 19:15:52.494607    4892 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Oct 30 19:15:52.499229 ip-10-0-132-82 kubenswrapper[4892]: W1030 19:15:52.494609    4892 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Oct 30 19:15:52.499229 ip-10-0-132-82 kubenswrapper[4892]: W1030 19:15:52.494612    4892 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Oct 30 19:15:52.499229 ip-10-0-132-82 kubenswrapper[4892]: W1030 19:15:52.494615    4892 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Oct 30 19:15:52.499229 ip-10-0-132-82 kubenswrapper[4892]: W1030 19:15:52.494618    4892 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Oct 30 19:15:52.499229 ip-10-0-132-82 kubenswrapper[4892]: W1030 19:15:52.494620    4892 feature_gate.go:386] unrecognized feature gate: EventTTL
Oct 30 19:15:52.499229 ip-10-0-132-82 kubenswrapper[4892]: W1030 19:15:52.494623    4892 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Oct 30 19:15:52.499229 ip-10-0-132-82 kubenswrapper[4892]: W1030 19:15:52.494625    4892 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Oct 30 19:15:52.499229 ip-10-0-132-82 kubenswrapper[4892]: W1030 19:15:52.494628    4892 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Oct 30 19:15:52.499229 ip-10-0-132-82 kubenswrapper[4892]: W1030 19:15:52.494630    4892 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Oct 30 19:15:52.499229 ip-10-0-132-82 kubenswrapper[4892]: W1030 19:15:52.494632    4892 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Oct 30 19:15:52.499229 ip-10-0-132-82 kubenswrapper[4892]: W1030 19:15:52.494636    4892 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Oct 30 19:15:52.499229 ip-10-0-132-82 kubenswrapper[4892]: W1030 19:15:52.494639    4892 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Oct 30 19:15:52.499229 ip-10-0-132-82 kubenswrapper[4892]: W1030 19:15:52.494641    4892 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Oct 30 19:15:52.499229 ip-10-0-132-82 kubenswrapper[4892]: W1030 19:15:52.494644    4892 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Oct 30 19:15:52.499229 ip-10-0-132-82 kubenswrapper[4892]: W1030 19:15:52.494646    4892 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Oct 30 19:15:52.499723 ip-10-0-132-82 kubenswrapper[4892]: W1030 19:15:52.494649    4892 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Oct 30 19:15:52.499723 ip-10-0-132-82 kubenswrapper[4892]: W1030 19:15:52.494651    4892 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Oct 30 19:15:52.499723 ip-10-0-132-82 kubenswrapper[4892]: W1030 19:15:52.494654    4892 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Oct 30 19:15:52.499723 ip-10-0-132-82 kubenswrapper[4892]: W1030 19:15:52.494658    4892 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Oct 30 19:15:52.499723 ip-10-0-132-82 kubenswrapper[4892]: W1030 19:15:52.494660    4892 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Oct 30 19:15:52.499723 ip-10-0-132-82 kubenswrapper[4892]: W1030 19:15:52.494663    4892 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Oct 30 19:15:52.499723 ip-10-0-132-82 kubenswrapper[4892]: W1030 19:15:52.494666    4892 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Oct 30 19:15:52.499723 ip-10-0-132-82 kubenswrapper[4892]: W1030 19:15:52.494668    4892 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Oct 30 19:15:52.499723 ip-10-0-132-82 kubenswrapper[4892]: W1030 19:15:52.494671    4892 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Oct 30 19:15:52.499723 ip-10-0-132-82 kubenswrapper[4892]: W1030 19:15:52.494673    4892 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Oct 30 19:15:52.499723 ip-10-0-132-82 kubenswrapper[4892]: W1030 19:15:52.494676    4892 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Oct 30 19:15:52.499723 ip-10-0-132-82 kubenswrapper[4892]: W1030 19:15:52.494678    4892 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Oct 30 19:15:52.499723 ip-10-0-132-82 kubenswrapper[4892]: W1030 19:15:52.494681    4892 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Oct 30 19:15:52.499723 ip-10-0-132-82 kubenswrapper[4892]: W1030 19:15:52.494683    4892 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Oct 30 19:15:52.499723 ip-10-0-132-82 kubenswrapper[4892]: W1030 19:15:52.494686    4892 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Oct 30 19:15:52.499723 ip-10-0-132-82 kubenswrapper[4892]: W1030 19:15:52.494688    4892 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Oct 30 19:15:52.499723 ip-10-0-132-82 kubenswrapper[4892]: W1030 19:15:52.494691    4892 feature_gate.go:386] unrecognized feature gate: Example2
Oct 30 19:15:52.499723 ip-10-0-132-82 kubenswrapper[4892]: W1030 19:15:52.494693    4892 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Oct 30 19:15:52.499723 ip-10-0-132-82 kubenswrapper[4892]: W1030 19:15:52.494696    4892 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Oct 30 19:15:52.499723 ip-10-0-132-82 kubenswrapper[4892]: W1030 19:15:52.494698    4892 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Oct 30 19:15:52.500231 ip-10-0-132-82 kubenswrapper[4892]: W1030 19:15:52.494701    4892 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Oct 30 19:15:52.500231 ip-10-0-132-82 kubenswrapper[4892]: W1030 19:15:52.494703    4892 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Oct 30 19:15:52.500231 ip-10-0-132-82 kubenswrapper[4892]: W1030 19:15:52.494706    4892 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Oct 30 19:15:52.500231 ip-10-0-132-82 kubenswrapper[4892]: W1030 19:15:52.494708    4892 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Oct 30 19:15:52.500231 ip-10-0-132-82 kubenswrapper[4892]: W1030 19:15:52.494711    4892 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Oct 30 19:15:52.500231 ip-10-0-132-82 kubenswrapper[4892]: W1030 19:15:52.494713    4892 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Oct 30 19:15:52.500231 ip-10-0-132-82 kubenswrapper[4892]: W1030 19:15:52.494716    4892 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Oct 30 19:15:52.500231 ip-10-0-132-82 kubenswrapper[4892]: W1030 19:15:52.494720    4892 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Oct 30 19:15:52.500231 ip-10-0-132-82 kubenswrapper[4892]: W1030 19:15:52.494723    4892 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Oct 30 19:15:52.500231 ip-10-0-132-82 kubenswrapper[4892]: W1030 19:15:52.494726    4892 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Oct 30 19:15:52.500231 ip-10-0-132-82 kubenswrapper[4892]: W1030 19:15:52.494729    4892 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Oct 30 19:15:52.500231 ip-10-0-132-82 kubenswrapper[4892]: W1030 19:15:52.494732    4892 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Oct 30 19:15:52.500231 ip-10-0-132-82 kubenswrapper[4892]: W1030 19:15:52.494736    4892 feature_gate.go:386] unrecognized feature gate: OVNObservability
Oct 30 19:15:52.500231 ip-10-0-132-82 kubenswrapper[4892]: W1030 19:15:52.494740    4892 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Oct 30 19:15:52.500231 ip-10-0-132-82 kubenswrapper[4892]: W1030 19:15:52.494743    4892 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Oct 30 19:15:52.500231 ip-10-0-132-82 kubenswrapper[4892]: W1030 19:15:52.494747    4892 feature_gate.go:386] unrecognized feature gate: OSStreams
Oct 30 19:15:52.500231 ip-10-0-132-82 kubenswrapper[4892]: W1030 19:15:52.494750    4892 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Oct 30 19:15:52.500231 ip-10-0-132-82 kubenswrapper[4892]: W1030 19:15:52.494752    4892 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Oct 30 19:15:52.500675 ip-10-0-132-82 kubenswrapper[4892]: W1030 19:15:52.494755    4892 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Oct 30 19:15:52.500675 ip-10-0-132-82 kubenswrapper[4892]: W1030 19:15:52.494757    4892 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Oct 30 19:15:52.500675 ip-10-0-132-82 kubenswrapper[4892]: W1030 19:15:52.494760    4892 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Oct 30 19:15:52.500675 ip-10-0-132-82 kubenswrapper[4892]: W1030 19:15:52.494762    4892 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Oct 30 19:15:52.500675 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:15:52.494855    4892 flags.go:64] FLAG: --address="0.0.0.0"
Oct 30 19:15:52.500675 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:15:52.494867    4892 flags.go:64] FLAG: --allowed-unsafe-sysctls="[]"
Oct 30 19:15:52.500675 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:15:52.494875    4892 flags.go:64] FLAG: --anonymous-auth="true"
Oct 30 19:15:52.500675 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:15:52.494882    4892 flags.go:64] FLAG: --application-metrics-count-limit="100"
Oct 30 19:15:52.500675 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:15:52.494886    4892 flags.go:64] FLAG: --authentication-token-webhook="false"
Oct 30 19:15:52.500675 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:15:52.494890    4892 flags.go:64] FLAG: --authentication-token-webhook-cache-ttl="2m0s"
Oct 30 19:15:52.500675 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:15:52.494894    4892 flags.go:64] FLAG: --authorization-mode="AlwaysAllow"
Oct 30 19:15:52.500675 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:15:52.494898    4892 flags.go:64] FLAG: --authorization-webhook-cache-authorized-ttl="5m0s"
Oct 30 19:15:52.500675 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:15:52.494902    4892 flags.go:64] FLAG: --authorization-webhook-cache-unauthorized-ttl="30s"
Oct 30 19:15:52.500675 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:15:52.494905    4892 flags.go:64] FLAG: --boot-id-file="/proc/sys/kernel/random/boot_id"
Oct 30 19:15:52.500675 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:15:52.494908    4892 flags.go:64] FLAG: --bootstrap-kubeconfig="/etc/kubernetes/kubeconfig"
Oct 30 19:15:52.500675 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:15:52.494911    4892 flags.go:64] FLAG: --cert-dir="/var/lib/kubelet/pki"
Oct 30 19:15:52.500675 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:15:52.494915    4892 flags.go:64] FLAG: --cgroup-driver="cgroupfs"
Oct 30 19:15:52.500675 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:15:52.494918    4892 flags.go:64] FLAG: --cgroup-root=""
Oct 30 19:15:52.500675 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:15:52.494921    4892 flags.go:64] FLAG: --cgroups-per-qos="true"
Oct 30 19:15:52.500675 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:15:52.494924    4892 flags.go:64] FLAG: --client-ca-file=""
Oct 30 19:15:52.500675 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:15:52.494926    4892 flags.go:64] FLAG: --cloud-provider="external"
Oct 30 19:15:52.500675 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:15:52.494929    4892 flags.go:64] FLAG: --cluster-dns="[]"
Oct 30 19:15:52.500675 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:15:52.494934    4892 flags.go:64] FLAG: --cluster-domain=""
Oct 30 19:15:52.500675 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:15:52.494937    4892 flags.go:64] FLAG: --config="/etc/kubernetes/kubelet.conf"
Oct 30 19:15:52.501291 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:15:52.494940    4892 flags.go:64] FLAG: --config-dir="/etc/openshift/kubelet.conf.d"
Oct 30 19:15:52.501291 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:15:52.494943    4892 flags.go:64] FLAG: --container-hints="/etc/cadvisor/container_hints.json"
Oct 30 19:15:52.501291 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:15:52.494946    4892 flags.go:64] FLAG: --container-log-max-files="5"
Oct 30 19:15:52.501291 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:15:52.494950    4892 flags.go:64] FLAG: --container-log-max-size="10Mi"
Oct 30 19:15:52.501291 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:15:52.494953    4892 flags.go:64] FLAG: --container-runtime-endpoint="/var/run/crio/crio.sock"
Oct 30 19:15:52.501291 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:15:52.494976    4892 flags.go:64] FLAG: --containerd="/run/containerd/containerd.sock"
Oct 30 19:15:52.501291 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:15:52.494982    4892 flags.go:64] FLAG: --containerd-namespace="k8s.io"
Oct 30 19:15:52.501291 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:15:52.494987    4892 flags.go:64] FLAG: --contention-profiling="false"
Oct 30 19:15:52.501291 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:15:52.494991    4892 flags.go:64] FLAG: --cpu-cfs-quota="true"
Oct 30 19:15:52.501291 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:15:52.494996    4892 flags.go:64] FLAG: --cpu-cfs-quota-period="100ms"
Oct 30 19:15:52.501291 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:15:52.494999    4892 flags.go:64] FLAG: --cpu-manager-policy="none"
Oct 30 19:15:52.501291 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:15:52.495002    4892 flags.go:64] FLAG: --cpu-manager-policy-options=""
Oct 30 19:15:52.501291 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:15:52.495006    4892 flags.go:64] FLAG: --cpu-manager-reconcile-period="10s"
Oct 30 19:15:52.501291 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:15:52.495009    4892 flags.go:64] FLAG: --enable-controller-attach-detach="true"
Oct 30 19:15:52.501291 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:15:52.495012    4892 flags.go:64] FLAG: --enable-debugging-handlers="true"
Oct 30 19:15:52.501291 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:15:52.495015    4892 flags.go:64] FLAG: --enable-load-reader="false"
Oct 30 19:15:52.501291 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:15:52.495018    4892 flags.go:64] FLAG: --enable-server="true"
Oct 30 19:15:52.501291 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:15:52.495021    4892 flags.go:64] FLAG: --enforce-node-allocatable="[pods]"
Oct 30 19:15:52.501291 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:15:52.495025    4892 flags.go:64] FLAG: --event-burst="100"
Oct 30 19:15:52.501291 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:15:52.495029    4892 flags.go:64] FLAG: --event-qps="50"
Oct 30 19:15:52.501291 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:15:52.495032    4892 flags.go:64] FLAG: --event-storage-age-limit="default=0"
Oct 30 19:15:52.501291 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:15:52.495034    4892 flags.go:64] FLAG: --event-storage-event-limit="default=0"
Oct 30 19:15:52.501291 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:15:52.495038    4892 flags.go:64] FLAG: --eviction-hard=""
Oct 30 19:15:52.501291 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:15:52.495042    4892 flags.go:64] FLAG: --eviction-max-pod-grace-period="0"
Oct 30 19:15:52.501893 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:15:52.495045    4892 flags.go:64] FLAG: --eviction-minimum-reclaim=""
Oct 30 19:15:52.501893 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:15:52.495048    4892 flags.go:64] FLAG: --eviction-pressure-transition-period="5m0s"
Oct 30 19:15:52.501893 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:15:52.495051    4892 flags.go:64] FLAG: --eviction-soft=""
Oct 30 19:15:52.501893 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:15:52.495054    4892 flags.go:64] FLAG: --eviction-soft-grace-period=""
Oct 30 19:15:52.501893 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:15:52.495057    4892 flags.go:64] FLAG: --exit-on-lock-contention="false"
Oct 30 19:15:52.501893 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:15:52.495060    4892 flags.go:64] FLAG: --experimental-allocatable-ignore-eviction="false"
Oct 30 19:15:52.501893 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:15:52.495063    4892 flags.go:64] FLAG: --experimental-mounter-path=""
Oct 30 19:15:52.501893 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:15:52.495066    4892 flags.go:64] FLAG: --fail-cgroupv1="true"
Oct 30 19:15:52.501893 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:15:52.495069    4892 flags.go:64] FLAG: --fail-swap-on="true"
Oct 30 19:15:52.501893 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:15:52.495072    4892 flags.go:64] FLAG: --feature-gates=""
Oct 30 19:15:52.501893 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:15:52.495076    4892 flags.go:64] FLAG: --file-check-frequency="20s"
Oct 30 19:15:52.501893 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:15:52.495079    4892 flags.go:64] FLAG: --global-housekeeping-interval="1m0s"
Oct 30 19:15:52.501893 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:15:52.495082    4892 flags.go:64] FLAG: --hairpin-mode="promiscuous-bridge"
Oct 30 19:15:52.501893 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:15:52.495087    4892 flags.go:64] FLAG: --healthz-bind-address="127.0.0.1"
Oct 30 19:15:52.501893 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:15:52.495090    4892 flags.go:64] FLAG: --healthz-port="10248"
Oct 30 19:15:52.501893 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:15:52.495093    4892 flags.go:64] FLAG: --help="false"
Oct 30 19:15:52.501893 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:15:52.495096    4892 flags.go:64] FLAG: --hostname-override="ip-10-0-132-82.us-west-1.compute.internal"
Oct 30 19:15:52.501893 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:15:52.495099    4892 flags.go:64] FLAG: --housekeeping-interval="10s"
Oct 30 19:15:52.501893 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:15:52.495102    4892 flags.go:64] FLAG: --http-check-frequency="20s"
Oct 30 19:15:52.501893 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:15:52.495105    4892 flags.go:64] FLAG: --image-credential-provider-bin-dir="/usr/libexec/kubelet-image-credential-provider-plugins"
Oct 30 19:15:52.501893 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:15:52.495108    4892 flags.go:64] FLAG: --image-credential-provider-config="/etc/kubernetes/credential-providers/ecr-credential-provider.yaml"
Oct 30 19:15:52.501893 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:15:52.495111    4892 flags.go:64] FLAG: --image-gc-high-threshold="85"
Oct 30 19:15:52.501893 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:15:52.495114    4892 flags.go:64] FLAG: --image-gc-low-threshold="80"
Oct 30 19:15:52.502491 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:15:52.495117    4892 flags.go:64] FLAG: --image-service-endpoint=""
Oct 30 19:15:52.502491 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:15:52.495120    4892 flags.go:64] FLAG: --kernel-memcg-notification="false"
Oct 30 19:15:52.502491 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:15:52.495123    4892 flags.go:64] FLAG: --kube-api-burst="100"
Oct 30 19:15:52.502491 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:15:52.495126    4892 flags.go:64] FLAG: --kube-api-content-type="application/vnd.kubernetes.protobuf"
Oct 30 19:15:52.502491 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:15:52.495129    4892 flags.go:64] FLAG: --kube-api-qps="50"
Oct 30 19:15:52.502491 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:15:52.495131    4892 flags.go:64] FLAG: --kube-reserved=""
Oct 30 19:15:52.502491 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:15:52.495134    4892 flags.go:64] FLAG: --kube-reserved-cgroup=""
Oct 30 19:15:52.502491 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:15:52.495137    4892 flags.go:64] FLAG: --kubeconfig="/var/lib/kubelet/kubeconfig"
Oct 30 19:15:52.502491 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:15:52.495140    4892 flags.go:64] FLAG: --kubelet-cgroups=""
Oct 30 19:15:52.502491 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:15:52.495143    4892 flags.go:64] FLAG: --local-storage-capacity-isolation="true"
Oct 30 19:15:52.502491 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:15:52.495146    4892 flags.go:64] FLAG: --lock-file=""
Oct 30 19:15:52.502491 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:15:52.495149    4892 flags.go:64] FLAG: --log-cadvisor-usage="false"
Oct 30 19:15:52.502491 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:15:52.495152    4892 flags.go:64] FLAG: --log-flush-frequency="5s"
Oct 30 19:15:52.502491 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:15:52.495155    4892 flags.go:64] FLAG: --log-json-info-buffer-size="0"
Oct 30 19:15:52.502491 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:15:52.495160    4892 flags.go:64] FLAG: --log-json-split-stream="false"
Oct 30 19:15:52.502491 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:15:52.495163    4892 flags.go:64] FLAG: --log-text-info-buffer-size="0"
Oct 30 19:15:52.502491 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:15:52.495166    4892 flags.go:64] FLAG: --log-text-split-stream="false"
Oct 30 19:15:52.502491 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:15:52.495169    4892 flags.go:64] FLAG: --logging-format="text"
Oct 30 19:15:52.502491 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:15:52.495172    4892 flags.go:64] FLAG: --machine-id-file="/etc/machine-id,/var/lib/dbus/machine-id"
Oct 30 19:15:52.502491 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:15:52.495175    4892 flags.go:64] FLAG: --make-iptables-util-chains="true"
Oct 30 19:15:52.502491 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:15:52.495178    4892 flags.go:64] FLAG: --manifest-url=""
Oct 30 19:15:52.502491 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:15:52.495181    4892 flags.go:64] FLAG: --manifest-url-header=""
Oct 30 19:15:52.502491 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:15:52.495186    4892 flags.go:64] FLAG: --max-housekeeping-interval="15s"
Oct 30 19:15:52.502491 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:15:52.495189    4892 flags.go:64] FLAG: --max-open-files="1000000"
Oct 30 19:15:52.502491 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:15:52.495193    4892 flags.go:64] FLAG: --max-pods="110"
Oct 30 19:15:52.503110 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:15:52.495197    4892 flags.go:64] FLAG: --maximum-dead-containers="-1"
Oct 30 19:15:52.503110 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:15:52.495200    4892 flags.go:64] FLAG: --maximum-dead-containers-per-container="1"
Oct 30 19:15:52.503110 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:15:52.495202    4892 flags.go:64] FLAG: --memory-manager-policy="None"
Oct 30 19:15:52.503110 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:15:52.495205    4892 flags.go:64] FLAG: --minimum-container-ttl-duration="6m0s"
Oct 30 19:15:52.503110 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:15:52.495208    4892 flags.go:64] FLAG: --minimum-image-ttl-duration="2m0s"
Oct 30 19:15:52.503110 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:15:52.495211    4892 flags.go:64] FLAG: --node-ip="0.0.0.0"
Oct 30 19:15:52.503110 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:15:52.495214    4892 flags.go:64] FLAG: --node-labels="node-role.kubernetes.io/worker=,node.openshift.io/os_id=rhel"
Oct 30 19:15:52.503110 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:15:52.495220    4892 flags.go:64] FLAG: --node-status-max-images="50"
Oct 30 19:15:52.503110 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:15:52.495223    4892 flags.go:64] FLAG: --node-status-update-frequency="10s"
Oct 30 19:15:52.503110 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:15:52.495226    4892 flags.go:64] FLAG: --oom-score-adj="-999"
Oct 30 19:15:52.503110 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:15:52.495229    4892 flags.go:64] FLAG: --pod-cidr=""
Oct 30 19:15:52.503110 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:15:52.495232    4892 flags.go:64] FLAG: --pod-manifest-path=""
Oct 30 19:15:52.503110 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:15:52.495235    4892 flags.go:64] FLAG: --pod-max-pids="-1"
Oct 30 19:15:52.503110 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:15:52.495238    4892 flags.go:64] FLAG: --pods-per-core="0"
Oct 30 19:15:52.503110 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:15:52.495241    4892 flags.go:64] FLAG: --port="10250"
Oct 30 19:15:52.503110 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:15:52.495244    4892 flags.go:64] FLAG: --protect-kernel-defaults="false"
Oct 30 19:15:52.503110 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:15:52.495247    4892 flags.go:64] FLAG: --provider-id="aws:///us-west-1a/i-0035bfd3429370984"
Oct 30 19:15:52.503110 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:15:52.495250    4892 flags.go:64] FLAG: --qos-reserved=""
Oct 30 19:15:52.503110 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:15:52.495253    4892 flags.go:64] FLAG: --read-only-port="10255"
Oct 30 19:15:52.503110 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:15:52.495256    4892 flags.go:64] FLAG: --register-node="true"
Oct 30 19:15:52.503110 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:15:52.495260    4892 flags.go:64] FLAG: --register-with-taints=""
Oct 30 19:15:52.503110 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:15:52.495264    4892 flags.go:64] FLAG: --registry-burst="10"
Oct 30 19:15:52.503110 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:15:52.495267    4892 flags.go:64] FLAG: --registry-qps="5"
Oct 30 19:15:52.503110 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:15:52.495269    4892 flags.go:64] FLAG: --reserved-cpus=""
Oct 30 19:15:52.503110 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:15:52.495272    4892 flags.go:64] FLAG: --reserved-memory=""
Oct 30 19:15:52.503110 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:15:52.495276    4892 flags.go:64] FLAG: --resolv-conf="/etc/resolv.conf"
Oct 30 19:15:52.503744 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:15:52.495279    4892 flags.go:64] FLAG: --root-dir="/var/lib/kubelet"
Oct 30 19:15:52.503744 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:15:52.495282    4892 flags.go:64] FLAG: --rotate-certificates="false"
Oct 30 19:15:52.503744 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:15:52.495285    4892 flags.go:64] FLAG: --rotate-server-certificates="false"
Oct 30 19:15:52.503744 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:15:52.495289    4892 flags.go:64] FLAG: --runonce="false"
Oct 30 19:15:52.503744 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:15:52.495292    4892 flags.go:64] FLAG: --runtime-cgroups="/system.slice/crio.service"
Oct 30 19:15:52.503744 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:15:52.495295    4892 flags.go:64] FLAG: --runtime-request-timeout="2m0s"
Oct 30 19:15:52.503744 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:15:52.495298    4892 flags.go:64] FLAG: --seccomp-default="false"
Oct 30 19:15:52.503744 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:15:52.495301    4892 flags.go:64] FLAG: --serialize-image-pulls="true"
Oct 30 19:15:52.503744 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:15:52.495304    4892 flags.go:64] FLAG: --storage-driver-buffer-duration="1m0s"
Oct 30 19:15:52.503744 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:15:52.495307    4892 flags.go:64] FLAG: --storage-driver-db="cadvisor"
Oct 30 19:15:52.503744 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:15:52.495310    4892 flags.go:64] FLAG: --storage-driver-host="localhost:8086"
Oct 30 19:15:52.503744 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:15:52.495313    4892 flags.go:64] FLAG: --storage-driver-password="root"
Oct 30 19:15:52.503744 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:15:52.495316    4892 flags.go:64] FLAG: --storage-driver-secure="false"
Oct 30 19:15:52.503744 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:15:52.495319    4892 flags.go:64] FLAG: --storage-driver-table="stats"
Oct 30 19:15:52.503744 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:15:52.495322    4892 flags.go:64] FLAG: --storage-driver-user="root"
Oct 30 19:15:52.503744 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:15:52.495325    4892 flags.go:64] FLAG: --streaming-connection-idle-timeout="4h0m0s"
Oct 30 19:15:52.503744 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:15:52.495328    4892 flags.go:64] FLAG: --sync-frequency="1m0s"
Oct 30 19:15:52.503744 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:15:52.495331    4892 flags.go:64] FLAG: --system-cgroups=""
Oct 30 19:15:52.503744 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:15:52.495334    4892 flags.go:64] FLAG: --system-reserved="cpu=500m,ephemeral-storage=1Gi,memory=1Gi"
Oct 30 19:15:52.503744 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:15:52.495339    4892 flags.go:64] FLAG: --system-reserved-cgroup=""
Oct 30 19:15:52.503744 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:15:52.495342    4892 flags.go:64] FLAG: --tls-cert-file=""
Oct 30 19:15:52.503744 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:15:52.495344    4892 flags.go:64] FLAG: --tls-cipher-suites="[]"
Oct 30 19:15:52.503744 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:15:52.495348    4892 flags.go:64] FLAG: --tls-min-version=""
Oct 30 19:15:52.503744 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:15:52.495351    4892 flags.go:64] FLAG: --tls-private-key-file=""
Oct 30 19:15:52.503744 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:15:52.495354    4892 flags.go:64] FLAG: --topology-manager-policy="none"
Oct 30 19:15:52.504401 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:15:52.495356    4892 flags.go:64] FLAG: --topology-manager-policy-options=""
Oct 30 19:15:52.504401 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:15:52.495359    4892 flags.go:64] FLAG: --topology-manager-scope="container"
Oct 30 19:15:52.504401 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:15:52.495362    4892 flags.go:64] FLAG: --v="2"
Oct 30 19:15:52.504401 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:15:52.495367    4892 flags.go:64] FLAG: --version="false"
Oct 30 19:15:52.504401 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:15:52.495371    4892 flags.go:64] FLAG: --vmodule=""
Oct 30 19:15:52.504401 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:15:52.495375    4892 flags.go:64] FLAG: --volume-plugin-dir="/etc/kubernetes/kubelet-plugins/volume/exec"
Oct 30 19:15:52.504401 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:15:52.495378    4892 flags.go:64] FLAG: --volume-stats-agg-period="1m0s"
Oct 30 19:15:52.504401 ip-10-0-132-82 kubenswrapper[4892]: W1030 19:15:52.495536    4892 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Oct 30 19:15:52.504401 ip-10-0-132-82 kubenswrapper[4892]: W1030 19:15:52.495541    4892 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Oct 30 19:15:52.504401 ip-10-0-132-82 kubenswrapper[4892]: W1030 19:15:52.495544    4892 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Oct 30 19:15:52.504401 ip-10-0-132-82 kubenswrapper[4892]: W1030 19:15:52.495548    4892 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Oct 30 19:15:52.504401 ip-10-0-132-82 kubenswrapper[4892]: W1030 19:15:52.495551    4892 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Oct 30 19:15:52.504401 ip-10-0-132-82 kubenswrapper[4892]: W1030 19:15:52.495554    4892 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Oct 30 19:15:52.504401 ip-10-0-132-82 kubenswrapper[4892]: W1030 19:15:52.495556    4892 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Oct 30 19:15:52.504401 ip-10-0-132-82 kubenswrapper[4892]: W1030 19:15:52.495559    4892 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Oct 30 19:15:52.504401 ip-10-0-132-82 kubenswrapper[4892]: W1030 19:15:52.495562    4892 feature_gate.go:386] unrecognized feature gate: NewOLM
Oct 30 19:15:52.504401 ip-10-0-132-82 kubenswrapper[4892]: W1030 19:15:52.495565    4892 feature_gate.go:386] unrecognized feature gate: EventTTL
Oct 30 19:15:52.504401 ip-10-0-132-82 kubenswrapper[4892]: W1030 19:15:52.495567    4892 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Oct 30 19:15:52.504401 ip-10-0-132-82 kubenswrapper[4892]: W1030 19:15:52.495570    4892 feature_gate.go:386] unrecognized feature gate: OSStreams
Oct 30 19:15:52.504401 ip-10-0-132-82 kubenswrapper[4892]: W1030 19:15:52.495572    4892 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Oct 30 19:15:52.504401 ip-10-0-132-82 kubenswrapper[4892]: W1030 19:15:52.495575    4892 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Oct 30 19:15:52.505162 ip-10-0-132-82 kubenswrapper[4892]: W1030 19:15:52.495577    4892 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Oct 30 19:15:52.505162 ip-10-0-132-82 kubenswrapper[4892]: W1030 19:15:52.495580    4892 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Oct 30 19:15:52.505162 ip-10-0-132-82 kubenswrapper[4892]: W1030 19:15:52.495583    4892 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Oct 30 19:15:52.505162 ip-10-0-132-82 kubenswrapper[4892]: W1030 19:15:52.495585    4892 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Oct 30 19:15:52.505162 ip-10-0-132-82 kubenswrapper[4892]: W1030 19:15:52.495588    4892 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Oct 30 19:15:52.505162 ip-10-0-132-82 kubenswrapper[4892]: W1030 19:15:52.495590    4892 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Oct 30 19:15:52.505162 ip-10-0-132-82 kubenswrapper[4892]: W1030 19:15:52.495593    4892 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Oct 30 19:15:52.505162 ip-10-0-132-82 kubenswrapper[4892]: W1030 19:15:52.495595    4892 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Oct 30 19:15:52.505162 ip-10-0-132-82 kubenswrapper[4892]: W1030 19:15:52.495598    4892 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Oct 30 19:15:52.505162 ip-10-0-132-82 kubenswrapper[4892]: W1030 19:15:52.495601    4892 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Oct 30 19:15:52.505162 ip-10-0-132-82 kubenswrapper[4892]: W1030 19:15:52.495603    4892 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Oct 30 19:15:52.505162 ip-10-0-132-82 kubenswrapper[4892]: W1030 19:15:52.495606    4892 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Oct 30 19:15:52.505162 ip-10-0-132-82 kubenswrapper[4892]: W1030 19:15:52.495609    4892 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Oct 30 19:15:52.505162 ip-10-0-132-82 kubenswrapper[4892]: W1030 19:15:52.495611    4892 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Oct 30 19:15:52.505162 ip-10-0-132-82 kubenswrapper[4892]: W1030 19:15:52.495614    4892 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Oct 30 19:15:52.505162 ip-10-0-132-82 kubenswrapper[4892]: W1030 19:15:52.495616    4892 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Oct 30 19:15:52.505162 ip-10-0-132-82 kubenswrapper[4892]: W1030 19:15:52.495619    4892 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Oct 30 19:15:52.505162 ip-10-0-132-82 kubenswrapper[4892]: W1030 19:15:52.495621    4892 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Oct 30 19:15:52.505162 ip-10-0-132-82 kubenswrapper[4892]: W1030 19:15:52.495638    4892 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Oct 30 19:15:52.505913 ip-10-0-132-82 kubenswrapper[4892]: W1030 19:15:52.495642    4892 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Oct 30 19:15:52.505913 ip-10-0-132-82 kubenswrapper[4892]: W1030 19:15:52.495645    4892 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Oct 30 19:15:52.505913 ip-10-0-132-82 kubenswrapper[4892]: W1030 19:15:52.495650    4892 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Oct 30 19:15:52.505913 ip-10-0-132-82 kubenswrapper[4892]: W1030 19:15:52.495653    4892 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Oct 30 19:15:52.505913 ip-10-0-132-82 kubenswrapper[4892]: W1030 19:15:52.495655    4892 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Oct 30 19:15:52.505913 ip-10-0-132-82 kubenswrapper[4892]: W1030 19:15:52.495658    4892 feature_gate.go:386] unrecognized feature gate: SignatureStores
Oct 30 19:15:52.505913 ip-10-0-132-82 kubenswrapper[4892]: W1030 19:15:52.495660    4892 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Oct 30 19:15:52.505913 ip-10-0-132-82 kubenswrapper[4892]: W1030 19:15:52.495663    4892 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Oct 30 19:15:52.505913 ip-10-0-132-82 kubenswrapper[4892]: W1030 19:15:52.495668    4892 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Oct 30 19:15:52.505913 ip-10-0-132-82 kubenswrapper[4892]: W1030 19:15:52.495672    4892 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Oct 30 19:15:52.505913 ip-10-0-132-82 kubenswrapper[4892]: W1030 19:15:52.495674    4892 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Oct 30 19:15:52.505913 ip-10-0-132-82 kubenswrapper[4892]: W1030 19:15:52.495677    4892 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Oct 30 19:15:52.505913 ip-10-0-132-82 kubenswrapper[4892]: W1030 19:15:52.495680    4892 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Oct 30 19:15:52.505913 ip-10-0-132-82 kubenswrapper[4892]: W1030 19:15:52.495682    4892 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Oct 30 19:15:52.505913 ip-10-0-132-82 kubenswrapper[4892]: W1030 19:15:52.495685    4892 feature_gate.go:386] unrecognized feature gate: Example
Oct 30 19:15:52.505913 ip-10-0-132-82 kubenswrapper[4892]: W1030 19:15:52.495687    4892 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Oct 30 19:15:52.505913 ip-10-0-132-82 kubenswrapper[4892]: W1030 19:15:52.495690    4892 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Oct 30 19:15:52.505913 ip-10-0-132-82 kubenswrapper[4892]: W1030 19:15:52.495692    4892 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Oct 30 19:15:52.505913 ip-10-0-132-82 kubenswrapper[4892]: W1030 19:15:52.495694    4892 feature_gate.go:386] unrecognized feature gate: EVPN
Oct 30 19:15:52.505913 ip-10-0-132-82 kubenswrapper[4892]: W1030 19:15:52.495711    4892 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Oct 30 19:15:52.506713 ip-10-0-132-82 kubenswrapper[4892]: W1030 19:15:52.495715    4892 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Oct 30 19:15:52.506713 ip-10-0-132-82 kubenswrapper[4892]: W1030 19:15:52.495717    4892 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Oct 30 19:15:52.506713 ip-10-0-132-82 kubenswrapper[4892]: W1030 19:15:52.495720    4892 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Oct 30 19:15:52.506713 ip-10-0-132-82 kubenswrapper[4892]: W1030 19:15:52.495723    4892 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Oct 30 19:15:52.506713 ip-10-0-132-82 kubenswrapper[4892]: W1030 19:15:52.495725    4892 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Oct 30 19:15:52.506713 ip-10-0-132-82 kubenswrapper[4892]: W1030 19:15:52.495728    4892 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Oct 30 19:15:52.506713 ip-10-0-132-82 kubenswrapper[4892]: W1030 19:15:52.495730    4892 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Oct 30 19:15:52.506713 ip-10-0-132-82 kubenswrapper[4892]: W1030 19:15:52.495733    4892 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Oct 30 19:15:52.506713 ip-10-0-132-82 kubenswrapper[4892]: W1030 19:15:52.495736    4892 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Oct 30 19:15:52.506713 ip-10-0-132-82 kubenswrapper[4892]: W1030 19:15:52.495738    4892 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Oct 30 19:15:52.506713 ip-10-0-132-82 kubenswrapper[4892]: W1030 19:15:52.495741    4892 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Oct 30 19:15:52.506713 ip-10-0-132-82 kubenswrapper[4892]: W1030 19:15:52.495744    4892 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Oct 30 19:15:52.506713 ip-10-0-132-82 kubenswrapper[4892]: W1030 19:15:52.495746    4892 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Oct 30 19:15:52.506713 ip-10-0-132-82 kubenswrapper[4892]: W1030 19:15:52.495748    4892 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Oct 30 19:15:52.506713 ip-10-0-132-82 kubenswrapper[4892]: W1030 19:15:52.495752    4892 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Oct 30 19:15:52.506713 ip-10-0-132-82 kubenswrapper[4892]: W1030 19:15:52.495755    4892 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Oct 30 19:15:52.506713 ip-10-0-132-82 kubenswrapper[4892]: W1030 19:15:52.495757    4892 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Oct 30 19:15:52.506713 ip-10-0-132-82 kubenswrapper[4892]: W1030 19:15:52.495760    4892 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Oct 30 19:15:52.506713 ip-10-0-132-82 kubenswrapper[4892]: W1030 19:15:52.495763    4892 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Oct 30 19:15:52.507310 ip-10-0-132-82 kubenswrapper[4892]: W1030 19:15:52.495765    4892 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Oct 30 19:15:52.507310 ip-10-0-132-82 kubenswrapper[4892]: W1030 19:15:52.495768    4892 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Oct 30 19:15:52.507310 ip-10-0-132-82 kubenswrapper[4892]: W1030 19:15:52.495771    4892 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Oct 30 19:15:52.507310 ip-10-0-132-82 kubenswrapper[4892]: W1030 19:15:52.495776    4892 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Oct 30 19:15:52.507310 ip-10-0-132-82 kubenswrapper[4892]: W1030 19:15:52.495786    4892 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Oct 30 19:15:52.507310 ip-10-0-132-82 kubenswrapper[4892]: W1030 19:15:52.495790    4892 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Oct 30 19:15:52.507310 ip-10-0-132-82 kubenswrapper[4892]: W1030 19:15:52.495793    4892 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Oct 30 19:15:52.507310 ip-10-0-132-82 kubenswrapper[4892]: W1030 19:15:52.495796    4892 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Oct 30 19:15:52.507310 ip-10-0-132-82 kubenswrapper[4892]: W1030 19:15:52.495799    4892 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Oct 30 19:15:52.507310 ip-10-0-132-82 kubenswrapper[4892]: W1030 19:15:52.495801    4892 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Oct 30 19:15:52.507310 ip-10-0-132-82 kubenswrapper[4892]: W1030 19:15:52.495804    4892 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Oct 30 19:15:52.507310 ip-10-0-132-82 kubenswrapper[4892]: W1030 19:15:52.495806    4892 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Oct 30 19:15:52.507310 ip-10-0-132-82 kubenswrapper[4892]: W1030 19:15:52.495809    4892 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Oct 30 19:15:52.507310 ip-10-0-132-82 kubenswrapper[4892]: W1030 19:15:52.495813    4892 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Oct 30 19:15:52.507310 ip-10-0-132-82 kubenswrapper[4892]: W1030 19:15:52.495816    4892 feature_gate.go:386] unrecognized feature gate: Example2
Oct 30 19:15:52.507310 ip-10-0-132-82 kubenswrapper[4892]: W1030 19:15:52.495819    4892 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Oct 30 19:15:52.507310 ip-10-0-132-82 kubenswrapper[4892]: W1030 19:15:52.495822    4892 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Oct 30 19:15:52.507310 ip-10-0-132-82 kubenswrapper[4892]: W1030 19:15:52.495825    4892 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Oct 30 19:15:52.507310 ip-10-0-132-82 kubenswrapper[4892]: W1030 19:15:52.495828    4892 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Oct 30 19:15:52.507880 ip-10-0-132-82 kubenswrapper[4892]: W1030 19:15:52.495830    4892 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Oct 30 19:15:52.507880 ip-10-0-132-82 kubenswrapper[4892]: W1030 19:15:52.495833    4892 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Oct 30 19:15:52.507880 ip-10-0-132-82 kubenswrapper[4892]: W1030 19:15:52.495835    4892 feature_gate.go:386] unrecognized feature gate: DualReplica
Oct 30 19:15:52.507880 ip-10-0-132-82 kubenswrapper[4892]: W1030 19:15:52.495838    4892 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Oct 30 19:15:52.507880 ip-10-0-132-82 kubenswrapper[4892]: W1030 19:15:52.495840    4892 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Oct 30 19:15:52.507880 ip-10-0-132-82 kubenswrapper[4892]: W1030 19:15:52.495843    4892 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Oct 30 19:15:52.507880 ip-10-0-132-82 kubenswrapper[4892]: W1030 19:15:52.495845    4892 feature_gate.go:386] unrecognized feature gate: OVNObservability
Oct 30 19:15:52.507880 ip-10-0-132-82 kubenswrapper[4892]: W1030 19:15:52.495848    4892 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Oct 30 19:15:52.507880 ip-10-0-132-82 kubenswrapper[4892]: W1030 19:15:52.495852    4892 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Oct 30 19:15:52.507880 ip-10-0-132-82 kubenswrapper[4892]: W1030 19:15:52.495854    4892 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Oct 30 19:15:52.507880 ip-10-0-132-82 kubenswrapper[4892]: W1030 19:15:52.495857    4892 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Oct 30 19:15:52.507880 ip-10-0-132-82 kubenswrapper[4892]: W1030 19:15:52.495859    4892 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Oct 30 19:15:52.507880 ip-10-0-132-82 kubenswrapper[4892]: W1030 19:15:52.495862    4892 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Oct 30 19:15:52.507880 ip-10-0-132-82 kubenswrapper[4892]: W1030 19:15:52.495864    4892 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Oct 30 19:15:52.507880 ip-10-0-132-82 kubenswrapper[4892]: W1030 19:15:52.495867    4892 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Oct 30 19:15:52.507880 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:15:52.495892    4892 feature_gate.go:465] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false DRAPartitionableDevices:false EventedPLEG:false KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:true MutatingAdmissionPolicy:false RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true UserNamespacesSupport:true]}
Oct 30 19:15:52.508613 ip-10-0-132-82 kubenswrapper[4892]: W1030 19:15:52.495926    4892 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Oct 30 19:15:52.508613 ip-10-0-132-82 kubenswrapper[4892]: W1030 19:15:52.495930    4892 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Oct 30 19:15:52.508613 ip-10-0-132-82 kubenswrapper[4892]: W1030 19:15:52.495933    4892 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Oct 30 19:15:52.508613 ip-10-0-132-82 kubenswrapper[4892]: W1030 19:15:52.495936    4892 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Oct 30 19:15:52.508613 ip-10-0-132-82 kubenswrapper[4892]: W1030 19:15:52.495939    4892 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Oct 30 19:15:52.508613 ip-10-0-132-82 kubenswrapper[4892]: W1030 19:15:52.495943    4892 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Oct 30 19:15:52.508613 ip-10-0-132-82 kubenswrapper[4892]: W1030 19:15:52.495947    4892 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Oct 30 19:15:52.508613 ip-10-0-132-82 kubenswrapper[4892]: W1030 19:15:52.495950    4892 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Oct 30 19:15:52.508613 ip-10-0-132-82 kubenswrapper[4892]: W1030 19:15:52.495953    4892 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Oct 30 19:15:52.508613 ip-10-0-132-82 kubenswrapper[4892]: W1030 19:15:52.495956    4892 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Oct 30 19:15:52.508613 ip-10-0-132-82 kubenswrapper[4892]: W1030 19:15:52.495974    4892 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Oct 30 19:15:52.508613 ip-10-0-132-82 kubenswrapper[4892]: W1030 19:15:52.495977    4892 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Oct 30 19:15:52.508613 ip-10-0-132-82 kubenswrapper[4892]: W1030 19:15:52.495979    4892 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Oct 30 19:15:52.508613 ip-10-0-132-82 kubenswrapper[4892]: W1030 19:15:52.495982    4892 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Oct 30 19:15:52.508613 ip-10-0-132-82 kubenswrapper[4892]: W1030 19:15:52.495985    4892 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Oct 30 19:15:52.508613 ip-10-0-132-82 kubenswrapper[4892]: W1030 19:15:52.495988    4892 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Oct 30 19:15:52.508613 ip-10-0-132-82 kubenswrapper[4892]: W1030 19:15:52.495990    4892 feature_gate.go:386] unrecognized feature gate: Example2
Oct 30 19:15:52.508613 ip-10-0-132-82 kubenswrapper[4892]: W1030 19:15:52.495994    4892 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Oct 30 19:15:52.508613 ip-10-0-132-82 kubenswrapper[4892]: W1030 19:15:52.495998    4892 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Oct 30 19:15:52.509516 ip-10-0-132-82 kubenswrapper[4892]: W1030 19:15:52.496000    4892 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Oct 30 19:15:52.509516 ip-10-0-132-82 kubenswrapper[4892]: W1030 19:15:52.496003    4892 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Oct 30 19:15:52.509516 ip-10-0-132-82 kubenswrapper[4892]: W1030 19:15:52.496006    4892 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Oct 30 19:15:52.509516 ip-10-0-132-82 kubenswrapper[4892]: W1030 19:15:52.496008    4892 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Oct 30 19:15:52.509516 ip-10-0-132-82 kubenswrapper[4892]: W1030 19:15:52.496011    4892 feature_gate.go:386] unrecognized feature gate: DualReplica
Oct 30 19:15:52.509516 ip-10-0-132-82 kubenswrapper[4892]: W1030 19:15:52.496013    4892 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Oct 30 19:15:52.509516 ip-10-0-132-82 kubenswrapper[4892]: W1030 19:15:52.496017    4892 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Oct 30 19:15:52.509516 ip-10-0-132-82 kubenswrapper[4892]: W1030 19:15:52.496020    4892 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Oct 30 19:15:52.509516 ip-10-0-132-82 kubenswrapper[4892]: W1030 19:15:52.496022    4892 feature_gate.go:386] unrecognized feature gate: OVNObservability
Oct 30 19:15:52.509516 ip-10-0-132-82 kubenswrapper[4892]: W1030 19:15:52.496025    4892 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Oct 30 19:15:52.509516 ip-10-0-132-82 kubenswrapper[4892]: W1030 19:15:52.496028    4892 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Oct 30 19:15:52.509516 ip-10-0-132-82 kubenswrapper[4892]: W1030 19:15:52.496030    4892 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Oct 30 19:15:52.509516 ip-10-0-132-82 kubenswrapper[4892]: W1030 19:15:52.496033    4892 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Oct 30 19:15:52.509516 ip-10-0-132-82 kubenswrapper[4892]: W1030 19:15:52.496035    4892 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Oct 30 19:15:52.509516 ip-10-0-132-82 kubenswrapper[4892]: W1030 19:15:52.496038    4892 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Oct 30 19:15:52.509516 ip-10-0-132-82 kubenswrapper[4892]: W1030 19:15:52.496041    4892 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Oct 30 19:15:52.509516 ip-10-0-132-82 kubenswrapper[4892]: W1030 19:15:52.496044    4892 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Oct 30 19:15:52.509516 ip-10-0-132-82 kubenswrapper[4892]: W1030 19:15:52.496046    4892 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Oct 30 19:15:52.509516 ip-10-0-132-82 kubenswrapper[4892]: W1030 19:15:52.496049    4892 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Oct 30 19:15:52.509516 ip-10-0-132-82 kubenswrapper[4892]: W1030 19:15:52.496051    4892 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Oct 30 19:15:52.510118 ip-10-0-132-82 kubenswrapper[4892]: W1030 19:15:52.496054    4892 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Oct 30 19:15:52.510118 ip-10-0-132-82 kubenswrapper[4892]: W1030 19:15:52.496056    4892 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Oct 30 19:15:52.510118 ip-10-0-132-82 kubenswrapper[4892]: W1030 19:15:52.496059    4892 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Oct 30 19:15:52.510118 ip-10-0-132-82 kubenswrapper[4892]: W1030 19:15:52.496063    4892 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Oct 30 19:15:52.510118 ip-10-0-132-82 kubenswrapper[4892]: W1030 19:15:52.496066    4892 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Oct 30 19:15:52.510118 ip-10-0-132-82 kubenswrapper[4892]: W1030 19:15:52.496068    4892 feature_gate.go:386] unrecognized feature gate: NewOLM
Oct 30 19:15:52.510118 ip-10-0-132-82 kubenswrapper[4892]: W1030 19:15:52.496071    4892 feature_gate.go:386] unrecognized feature gate: EventTTL
Oct 30 19:15:52.510118 ip-10-0-132-82 kubenswrapper[4892]: W1030 19:15:52.496073    4892 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Oct 30 19:15:52.510118 ip-10-0-132-82 kubenswrapper[4892]: W1030 19:15:52.496075    4892 feature_gate.go:386] unrecognized feature gate: OSStreams
Oct 30 19:15:52.510118 ip-10-0-132-82 kubenswrapper[4892]: W1030 19:15:52.496078    4892 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Oct 30 19:15:52.510118 ip-10-0-132-82 kubenswrapper[4892]: W1030 19:15:52.496081    4892 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Oct 30 19:15:52.510118 ip-10-0-132-82 kubenswrapper[4892]: W1030 19:15:52.496083    4892 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Oct 30 19:15:52.510118 ip-10-0-132-82 kubenswrapper[4892]: W1030 19:15:52.496086    4892 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Oct 30 19:15:52.510118 ip-10-0-132-82 kubenswrapper[4892]: W1030 19:15:52.496089    4892 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Oct 30 19:15:52.510118 ip-10-0-132-82 kubenswrapper[4892]: W1030 19:15:52.496091    4892 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Oct 30 19:15:52.510118 ip-10-0-132-82 kubenswrapper[4892]: W1030 19:15:52.496094    4892 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Oct 30 19:15:52.510118 ip-10-0-132-82 kubenswrapper[4892]: W1030 19:15:52.496097    4892 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Oct 30 19:15:52.510118 ip-10-0-132-82 kubenswrapper[4892]: W1030 19:15:52.496099    4892 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Oct 30 19:15:52.510118 ip-10-0-132-82 kubenswrapper[4892]: W1030 19:15:52.496103    4892 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Oct 30 19:15:52.510689 ip-10-0-132-82 kubenswrapper[4892]: W1030 19:15:52.496105    4892 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Oct 30 19:15:52.510689 ip-10-0-132-82 kubenswrapper[4892]: W1030 19:15:52.496108    4892 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Oct 30 19:15:52.510689 ip-10-0-132-82 kubenswrapper[4892]: W1030 19:15:52.496111    4892 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Oct 30 19:15:52.510689 ip-10-0-132-82 kubenswrapper[4892]: W1030 19:15:52.496114    4892 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Oct 30 19:15:52.510689 ip-10-0-132-82 kubenswrapper[4892]: W1030 19:15:52.496116    4892 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Oct 30 19:15:52.510689 ip-10-0-132-82 kubenswrapper[4892]: W1030 19:15:52.496119    4892 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Oct 30 19:15:52.510689 ip-10-0-132-82 kubenswrapper[4892]: W1030 19:15:52.496121    4892 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Oct 30 19:15:52.510689 ip-10-0-132-82 kubenswrapper[4892]: W1030 19:15:52.496124    4892 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Oct 30 19:15:52.510689 ip-10-0-132-82 kubenswrapper[4892]: W1030 19:15:52.496127    4892 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Oct 30 19:15:52.510689 ip-10-0-132-82 kubenswrapper[4892]: W1030 19:15:52.496129    4892 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Oct 30 19:15:52.510689 ip-10-0-132-82 kubenswrapper[4892]: W1030 19:15:52.496132    4892 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Oct 30 19:15:52.510689 ip-10-0-132-82 kubenswrapper[4892]: W1030 19:15:52.496134    4892 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Oct 30 19:15:52.510689 ip-10-0-132-82 kubenswrapper[4892]: W1030 19:15:52.496136    4892 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Oct 30 19:15:52.510689 ip-10-0-132-82 kubenswrapper[4892]: W1030 19:15:52.496139    4892 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Oct 30 19:15:52.510689 ip-10-0-132-82 kubenswrapper[4892]: W1030 19:15:52.496141    4892 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Oct 30 19:15:52.510689 ip-10-0-132-82 kubenswrapper[4892]: W1030 19:15:52.496144    4892 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Oct 30 19:15:52.510689 ip-10-0-132-82 kubenswrapper[4892]: W1030 19:15:52.496147    4892 feature_gate.go:386] unrecognized feature gate: SignatureStores
Oct 30 19:15:52.510689 ip-10-0-132-82 kubenswrapper[4892]: W1030 19:15:52.496150    4892 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Oct 30 19:15:52.510689 ip-10-0-132-82 kubenswrapper[4892]: W1030 19:15:52.496152    4892 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Oct 30 19:15:52.510689 ip-10-0-132-82 kubenswrapper[4892]: W1030 19:15:52.496155    4892 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Oct 30 19:15:52.511195 ip-10-0-132-82 kubenswrapper[4892]: W1030 19:15:52.496157    4892 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Oct 30 19:15:52.511195 ip-10-0-132-82 kubenswrapper[4892]: W1030 19:15:52.496160    4892 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Oct 30 19:15:52.511195 ip-10-0-132-82 kubenswrapper[4892]: W1030 19:15:52.496162    4892 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Oct 30 19:15:52.511195 ip-10-0-132-82 kubenswrapper[4892]: W1030 19:15:52.496165    4892 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Oct 30 19:15:52.511195 ip-10-0-132-82 kubenswrapper[4892]: W1030 19:15:52.496167    4892 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Oct 30 19:15:52.511195 ip-10-0-132-82 kubenswrapper[4892]: W1030 19:15:52.496170    4892 feature_gate.go:386] unrecognized feature gate: Example
Oct 30 19:15:52.511195 ip-10-0-132-82 kubenswrapper[4892]: W1030 19:15:52.496173    4892 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Oct 30 19:15:52.511195 ip-10-0-132-82 kubenswrapper[4892]: W1030 19:15:52.496175    4892 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Oct 30 19:15:52.511195 ip-10-0-132-82 kubenswrapper[4892]: W1030 19:15:52.496177    4892 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Oct 30 19:15:52.511195 ip-10-0-132-82 kubenswrapper[4892]: W1030 19:15:52.496180    4892 feature_gate.go:386] unrecognized feature gate: EVPN
Oct 30 19:15:52.511195 ip-10-0-132-82 kubenswrapper[4892]: W1030 19:15:52.496182    4892 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Oct 30 19:15:52.511195 ip-10-0-132-82 kubenswrapper[4892]: W1030 19:15:52.496186    4892 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Oct 30 19:15:52.511195 ip-10-0-132-82 kubenswrapper[4892]: W1030 19:15:52.496188    4892 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Oct 30 19:15:52.511195 ip-10-0-132-82 kubenswrapper[4892]: W1030 19:15:52.496191    4892 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Oct 30 19:15:52.511195 ip-10-0-132-82 kubenswrapper[4892]: W1030 19:15:52.496194    4892 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Oct 30 19:15:52.511195 ip-10-0-132-82 kubenswrapper[4892]: W1030 19:15:52.496196    4892 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Oct 30 19:15:52.511195 ip-10-0-132-82 kubenswrapper[4892]: W1030 19:15:52.496199    4892 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Oct 30 19:15:52.511195 ip-10-0-132-82 kubenswrapper[4892]: W1030 19:15:52.496201    4892 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Oct 30 19:15:52.511195 ip-10-0-132-82 kubenswrapper[4892]: W1030 19:15:52.496204    4892 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Oct 30 19:15:52.511195 ip-10-0-132-82 kubenswrapper[4892]: W1030 19:15:52.496208    4892 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Oct 30 19:15:52.511697 ip-10-0-132-82 kubenswrapper[4892]: W1030 19:15:52.496210    4892 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Oct 30 19:15:52.511697 ip-10-0-132-82 kubenswrapper[4892]: W1030 19:15:52.496214    4892 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Oct 30 19:15:52.511697 ip-10-0-132-82 kubenswrapper[4892]: W1030 19:15:52.496216    4892 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Oct 30 19:15:52.511697 ip-10-0-132-82 kubenswrapper[4892]: W1030 19:15:52.496219    4892 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Oct 30 19:15:52.511697 ip-10-0-132-82 kubenswrapper[4892]: W1030 19:15:52.496221    4892 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Oct 30 19:15:52.511697 ip-10-0-132-82 kubenswrapper[4892]: W1030 19:15:52.496224    4892 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Oct 30 19:15:52.511697 ip-10-0-132-82 kubenswrapper[4892]: W1030 19:15:52.496226    4892 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Oct 30 19:15:52.511697 ip-10-0-132-82 kubenswrapper[4892]: W1030 19:15:52.496229    4892 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Oct 30 19:15:52.511697 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:15:52.500245    4892 server.go:525] "Kubelet version" kubeletVersion="v1.35.9"
Oct 30 19:15:52.511697 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:15:52.500261    4892 server.go:527] "Golang settings" GOGC="" GOMAXPROCS="" GOTRACEBACK=""
Oct 30 19:15:52.511697 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:15:52.500279    4892 watchdog_linux.go:95] "Systemd watchdog is not enabled"
Oct 30 19:15:52.511697 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:15:52.500284    4892 watchdog_linux.go:138] "Systemd watchdog is not enabled or the interval is invalid, so health checking will not be started."
Oct 30 19:15:52.511697 ip-10-0-132-82 kubenswrapper[4892]: W1030 19:15:52.500357    4892 feature_gate.go:386] unrecognized feature gate: OSStreams
Oct 30 19:15:52.511697 ip-10-0-132-82 kubenswrapper[4892]: W1030 19:15:52.500362    4892 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Oct 30 19:15:52.511697 ip-10-0-132-82 kubenswrapper[4892]: W1030 19:15:52.500366    4892 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Oct 30 19:15:52.511697 ip-10-0-132-82 kubenswrapper[4892]: W1030 19:15:52.500369    4892 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Oct 30 19:15:52.511697 ip-10-0-132-82 kubenswrapper[4892]: W1030 19:15:52.500371    4892 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Oct 30 19:15:52.511697 ip-10-0-132-82 kubenswrapper[4892]: W1030 19:15:52.500374    4892 feature_gate.go:386] unrecognized feature gate: EventTTL
Oct 30 19:15:52.511697 ip-10-0-132-82 kubenswrapper[4892]: W1030 19:15:52.500377    4892 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Oct 30 19:15:52.512268 ip-10-0-132-82 kubenswrapper[4892]: W1030 19:15:52.500380    4892 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Oct 30 19:15:52.512268 ip-10-0-132-82 kubenswrapper[4892]: W1030 19:15:52.500382    4892 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Oct 30 19:15:52.512268 ip-10-0-132-82 kubenswrapper[4892]: W1030 19:15:52.500385    4892 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Oct 30 19:15:52.512268 ip-10-0-132-82 kubenswrapper[4892]: W1030 19:15:52.500387    4892 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Oct 30 19:15:52.512268 ip-10-0-132-82 kubenswrapper[4892]: W1030 19:15:52.500390    4892 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Oct 30 19:15:52.512268 ip-10-0-132-82 kubenswrapper[4892]: W1030 19:15:52.500393    4892 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Oct 30 19:15:52.512268 ip-10-0-132-82 kubenswrapper[4892]: W1030 19:15:52.500395    4892 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Oct 30 19:15:52.512268 ip-10-0-132-82 kubenswrapper[4892]: W1030 19:15:52.500398    4892 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Oct 30 19:15:52.512268 ip-10-0-132-82 kubenswrapper[4892]: W1030 19:15:52.500400    4892 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Oct 30 19:15:52.512268 ip-10-0-132-82 kubenswrapper[4892]: W1030 19:15:52.500403    4892 feature_gate.go:386] unrecognized feature gate: NewOLM
Oct 30 19:15:52.512268 ip-10-0-132-82 kubenswrapper[4892]: W1030 19:15:52.500406    4892 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Oct 30 19:15:52.512268 ip-10-0-132-82 kubenswrapper[4892]: W1030 19:15:52.500408    4892 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Oct 30 19:15:52.512268 ip-10-0-132-82 kubenswrapper[4892]: W1030 19:15:52.500417    4892 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Oct 30 19:15:52.512268 ip-10-0-132-82 kubenswrapper[4892]: W1030 19:15:52.500420    4892 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Oct 30 19:15:52.512268 ip-10-0-132-82 kubenswrapper[4892]: W1030 19:15:52.500428    4892 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Oct 30 19:15:52.512268 ip-10-0-132-82 kubenswrapper[4892]: W1030 19:15:52.500430    4892 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Oct 30 19:15:52.512268 ip-10-0-132-82 kubenswrapper[4892]: W1030 19:15:52.500433    4892 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Oct 30 19:15:52.512268 ip-10-0-132-82 kubenswrapper[4892]: W1030 19:15:52.500436    4892 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Oct 30 19:15:52.512268 ip-10-0-132-82 kubenswrapper[4892]: W1030 19:15:52.500438    4892 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Oct 30 19:15:52.512725 ip-10-0-132-82 kubenswrapper[4892]: W1030 19:15:52.500441    4892 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Oct 30 19:15:52.512725 ip-10-0-132-82 kubenswrapper[4892]: W1030 19:15:52.500444    4892 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Oct 30 19:15:52.512725 ip-10-0-132-82 kubenswrapper[4892]: W1030 19:15:52.500447    4892 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Oct 30 19:15:52.512725 ip-10-0-132-82 kubenswrapper[4892]: W1030 19:15:52.500450    4892 feature_gate.go:386] unrecognized feature gate: OVNObservability
Oct 30 19:15:52.512725 ip-10-0-132-82 kubenswrapper[4892]: W1030 19:15:52.500453    4892 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Oct 30 19:15:52.512725 ip-10-0-132-82 kubenswrapper[4892]: W1030 19:15:52.500455    4892 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Oct 30 19:15:52.512725 ip-10-0-132-82 kubenswrapper[4892]: W1030 19:15:52.500458    4892 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Oct 30 19:15:52.512725 ip-10-0-132-82 kubenswrapper[4892]: W1030 19:15:52.500462    4892 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Oct 30 19:15:52.512725 ip-10-0-132-82 kubenswrapper[4892]: W1030 19:15:52.500466    4892 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Oct 30 19:15:52.512725 ip-10-0-132-82 kubenswrapper[4892]: W1030 19:15:52.500470    4892 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Oct 30 19:15:52.512725 ip-10-0-132-82 kubenswrapper[4892]: W1030 19:15:52.500473    4892 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Oct 30 19:15:52.512725 ip-10-0-132-82 kubenswrapper[4892]: W1030 19:15:52.500475    4892 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Oct 30 19:15:52.512725 ip-10-0-132-82 kubenswrapper[4892]: W1030 19:15:52.500478    4892 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Oct 30 19:15:52.512725 ip-10-0-132-82 kubenswrapper[4892]: W1030 19:15:52.500481    4892 feature_gate.go:386] unrecognized feature gate: DualReplica
Oct 30 19:15:52.512725 ip-10-0-132-82 kubenswrapper[4892]: W1030 19:15:52.500483    4892 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Oct 30 19:15:52.512725 ip-10-0-132-82 kubenswrapper[4892]: W1030 19:15:52.500486    4892 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Oct 30 19:15:52.512725 ip-10-0-132-82 kubenswrapper[4892]: W1030 19:15:52.500488    4892 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Oct 30 19:15:52.512725 ip-10-0-132-82 kubenswrapper[4892]: W1030 19:15:52.500491    4892 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Oct 30 19:15:52.512725 ip-10-0-132-82 kubenswrapper[4892]: W1030 19:15:52.500494    4892 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Oct 30 19:15:52.513189 ip-10-0-132-82 kubenswrapper[4892]: W1030 19:15:52.500496    4892 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Oct 30 19:15:52.513189 ip-10-0-132-82 kubenswrapper[4892]: W1030 19:15:52.500500    4892 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Oct 30 19:15:52.513189 ip-10-0-132-82 kubenswrapper[4892]: W1030 19:15:52.500502    4892 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Oct 30 19:15:52.513189 ip-10-0-132-82 kubenswrapper[4892]: W1030 19:15:52.500505    4892 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Oct 30 19:15:52.513189 ip-10-0-132-82 kubenswrapper[4892]: W1030 19:15:52.500507    4892 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Oct 30 19:15:52.513189 ip-10-0-132-82 kubenswrapper[4892]: W1030 19:15:52.500510    4892 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Oct 30 19:15:52.513189 ip-10-0-132-82 kubenswrapper[4892]: W1030 19:15:52.500512    4892 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Oct 30 19:15:52.513189 ip-10-0-132-82 kubenswrapper[4892]: W1030 19:15:52.500515    4892 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Oct 30 19:15:52.513189 ip-10-0-132-82 kubenswrapper[4892]: W1030 19:15:52.500518    4892 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Oct 30 19:15:52.513189 ip-10-0-132-82 kubenswrapper[4892]: W1030 19:15:52.500526    4892 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Oct 30 19:15:52.513189 ip-10-0-132-82 kubenswrapper[4892]: W1030 19:15:52.500529    4892 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Oct 30 19:15:52.513189 ip-10-0-132-82 kubenswrapper[4892]: W1030 19:15:52.500532    4892 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Oct 30 19:15:52.513189 ip-10-0-132-82 kubenswrapper[4892]: W1030 19:15:52.500534    4892 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Oct 30 19:15:52.513189 ip-10-0-132-82 kubenswrapper[4892]: W1030 19:15:52.500537    4892 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Oct 30 19:15:52.513189 ip-10-0-132-82 kubenswrapper[4892]: W1030 19:15:52.500540    4892 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Oct 30 19:15:52.513189 ip-10-0-132-82 kubenswrapper[4892]: W1030 19:15:52.500542    4892 feature_gate.go:386] unrecognized feature gate: EVPN
Oct 30 19:15:52.513189 ip-10-0-132-82 kubenswrapper[4892]: W1030 19:15:52.500545    4892 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Oct 30 19:15:52.513189 ip-10-0-132-82 kubenswrapper[4892]: W1030 19:15:52.500547    4892 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Oct 30 19:15:52.513189 ip-10-0-132-82 kubenswrapper[4892]: W1030 19:15:52.500550    4892 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Oct 30 19:15:52.513635 ip-10-0-132-82 kubenswrapper[4892]: W1030 19:15:52.500552    4892 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Oct 30 19:15:52.513635 ip-10-0-132-82 kubenswrapper[4892]: W1030 19:15:52.500555    4892 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Oct 30 19:15:52.513635 ip-10-0-132-82 kubenswrapper[4892]: W1030 19:15:52.500559    4892 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Oct 30 19:15:52.513635 ip-10-0-132-82 kubenswrapper[4892]: W1030 19:15:52.500563    4892 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Oct 30 19:15:52.513635 ip-10-0-132-82 kubenswrapper[4892]: W1030 19:15:52.500567    4892 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Oct 30 19:15:52.513635 ip-10-0-132-82 kubenswrapper[4892]: W1030 19:15:52.500573    4892 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Oct 30 19:15:52.513635 ip-10-0-132-82 kubenswrapper[4892]: W1030 19:15:52.500580    4892 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Oct 30 19:15:52.513635 ip-10-0-132-82 kubenswrapper[4892]: W1030 19:15:52.500584    4892 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Oct 30 19:15:52.513635 ip-10-0-132-82 kubenswrapper[4892]: W1030 19:15:52.500588    4892 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Oct 30 19:15:52.513635 ip-10-0-132-82 kubenswrapper[4892]: W1030 19:15:52.500591    4892 feature_gate.go:386] unrecognized feature gate: Example2
Oct 30 19:15:52.513635 ip-10-0-132-82 kubenswrapper[4892]: W1030 19:15:52.500594    4892 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Oct 30 19:15:52.513635 ip-10-0-132-82 kubenswrapper[4892]: W1030 19:15:52.500596    4892 feature_gate.go:386] unrecognized feature gate: SignatureStores
Oct 30 19:15:52.513635 ip-10-0-132-82 kubenswrapper[4892]: W1030 19:15:52.500599    4892 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Oct 30 19:15:52.513635 ip-10-0-132-82 kubenswrapper[4892]: W1030 19:15:52.500602    4892 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Oct 30 19:15:52.513635 ip-10-0-132-82 kubenswrapper[4892]: W1030 19:15:52.500604    4892 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Oct 30 19:15:52.513635 ip-10-0-132-82 kubenswrapper[4892]: W1030 19:15:52.500606    4892 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Oct 30 19:15:52.513635 ip-10-0-132-82 kubenswrapper[4892]: W1030 19:15:52.500609    4892 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Oct 30 19:15:52.513635 ip-10-0-132-82 kubenswrapper[4892]: W1030 19:15:52.500611    4892 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Oct 30 19:15:52.513635 ip-10-0-132-82 kubenswrapper[4892]: W1030 19:15:52.500614    4892 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Oct 30 19:15:52.514106 ip-10-0-132-82 kubenswrapper[4892]: W1030 19:15:52.500616    4892 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Oct 30 19:15:52.514106 ip-10-0-132-82 kubenswrapper[4892]: W1030 19:15:52.500619    4892 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Oct 30 19:15:52.514106 ip-10-0-132-82 kubenswrapper[4892]: W1030 19:15:52.500621    4892 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Oct 30 19:15:52.514106 ip-10-0-132-82 kubenswrapper[4892]: W1030 19:15:52.500624    4892 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Oct 30 19:15:52.514106 ip-10-0-132-82 kubenswrapper[4892]: W1030 19:15:52.500633    4892 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Oct 30 19:15:52.514106 ip-10-0-132-82 kubenswrapper[4892]: W1030 19:15:52.500636    4892 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Oct 30 19:15:52.514106 ip-10-0-132-82 kubenswrapper[4892]: W1030 19:15:52.500639    4892 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Oct 30 19:15:52.514106 ip-10-0-132-82 kubenswrapper[4892]: W1030 19:15:52.500641    4892 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Oct 30 19:15:52.514106 ip-10-0-132-82 kubenswrapper[4892]: W1030 19:15:52.500644    4892 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Oct 30 19:15:52.514106 ip-10-0-132-82 kubenswrapper[4892]: W1030 19:15:52.500647    4892 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Oct 30 19:15:52.514106 ip-10-0-132-82 kubenswrapper[4892]: W1030 19:15:52.500649    4892 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Oct 30 19:15:52.514106 ip-10-0-132-82 kubenswrapper[4892]: W1030 19:15:52.500652    4892 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Oct 30 19:15:52.514106 ip-10-0-132-82 kubenswrapper[4892]: W1030 19:15:52.500654    4892 feature_gate.go:386] unrecognized feature gate: Example
Oct 30 19:15:52.514106 ip-10-0-132-82 kubenswrapper[4892]: W1030 19:15:52.500656    4892 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Oct 30 19:15:52.514106 ip-10-0-132-82 kubenswrapper[4892]: W1030 19:15:52.500659    4892 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Oct 30 19:15:52.514106 ip-10-0-132-82 kubenswrapper[4892]: W1030 19:15:52.500661    4892 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Oct 30 19:15:52.514106 ip-10-0-132-82 kubenswrapper[4892]: W1030 19:15:52.500664    4892 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Oct 30 19:15:52.514106 ip-10-0-132-82 kubenswrapper[4892]: W1030 19:15:52.500666    4892 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Oct 30 19:15:52.514106 ip-10-0-132-82 kubenswrapper[4892]: W1030 19:15:52.500669    4892 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Oct 30 19:15:52.514106 ip-10-0-132-82 kubenswrapper[4892]: W1030 19:15:52.500671    4892 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Oct 30 19:15:52.514581 ip-10-0-132-82 kubenswrapper[4892]: W1030 19:15:52.500673    4892 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Oct 30 19:15:52.514581 ip-10-0-132-82 kubenswrapper[4892]: W1030 19:15:52.500676    4892 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Oct 30 19:15:52.514581 ip-10-0-132-82 kubenswrapper[4892]: W1030 19:15:52.500679    4892 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Oct 30 19:15:52.514581 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:15:52.500707    4892 feature_gate.go:465] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false DRAPartitionableDevices:false EventedPLEG:false KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:true MutatingAdmissionPolicy:false RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true UserNamespacesSupport:true]}
Oct 30 19:15:52.514581 ip-10-0-132-82 kubenswrapper[4892]: W1030 19:15:52.500892    4892 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Oct 30 19:15:52.514581 ip-10-0-132-82 kubenswrapper[4892]: W1030 19:15:52.500896    4892 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Oct 30 19:15:52.514581 ip-10-0-132-82 kubenswrapper[4892]: W1030 19:15:52.500899    4892 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Oct 30 19:15:52.514581 ip-10-0-132-82 kubenswrapper[4892]: W1030 19:15:52.500902    4892 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Oct 30 19:15:52.514581 ip-10-0-132-82 kubenswrapper[4892]: W1030 19:15:52.500904    4892 feature_gate.go:386] unrecognized feature gate: OVNObservability
Oct 30 19:15:52.514581 ip-10-0-132-82 kubenswrapper[4892]: W1030 19:15:52.500907    4892 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Oct 30 19:15:52.514581 ip-10-0-132-82 kubenswrapper[4892]: W1030 19:15:52.500909    4892 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Oct 30 19:15:52.514581 ip-10-0-132-82 kubenswrapper[4892]: W1030 19:15:52.500912    4892 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Oct 30 19:15:52.514581 ip-10-0-132-82 kubenswrapper[4892]: W1030 19:15:52.500914    4892 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Oct 30 19:15:52.514581 ip-10-0-132-82 kubenswrapper[4892]: W1030 19:15:52.500917    4892 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Oct 30 19:15:52.514581 ip-10-0-132-82 kubenswrapper[4892]: W1030 19:15:52.500919    4892 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Oct 30 19:15:52.514581 ip-10-0-132-82 kubenswrapper[4892]: W1030 19:15:52.500922    4892 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Oct 30 19:15:52.514977 ip-10-0-132-82 kubenswrapper[4892]: W1030 19:15:52.500924    4892 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Oct 30 19:15:52.514977 ip-10-0-132-82 kubenswrapper[4892]: W1030 19:15:52.500934    4892 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Oct 30 19:15:52.514977 ip-10-0-132-82 kubenswrapper[4892]: W1030 19:15:52.500937    4892 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Oct 30 19:15:52.514977 ip-10-0-132-82 kubenswrapper[4892]: W1030 19:15:52.500939    4892 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Oct 30 19:15:52.514977 ip-10-0-132-82 kubenswrapper[4892]: W1030 19:15:52.500942    4892 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Oct 30 19:15:52.514977 ip-10-0-132-82 kubenswrapper[4892]: W1030 19:15:52.500945    4892 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Oct 30 19:15:52.514977 ip-10-0-132-82 kubenswrapper[4892]: W1030 19:15:52.500947    4892 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Oct 30 19:15:52.514977 ip-10-0-132-82 kubenswrapper[4892]: W1030 19:15:52.500950    4892 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Oct 30 19:15:52.514977 ip-10-0-132-82 kubenswrapper[4892]: W1030 19:15:52.500952    4892 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Oct 30 19:15:52.514977 ip-10-0-132-82 kubenswrapper[4892]: W1030 19:15:52.500955    4892 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Oct 30 19:15:52.514977 ip-10-0-132-82 kubenswrapper[4892]: W1030 19:15:52.500957    4892 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Oct 30 19:15:52.514977 ip-10-0-132-82 kubenswrapper[4892]: W1030 19:15:52.500973    4892 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Oct 30 19:15:52.514977 ip-10-0-132-82 kubenswrapper[4892]: W1030 19:15:52.500976    4892 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Oct 30 19:15:52.514977 ip-10-0-132-82 kubenswrapper[4892]: W1030 19:15:52.500978    4892 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Oct 30 19:15:52.514977 ip-10-0-132-82 kubenswrapper[4892]: W1030 19:15:52.500981    4892 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Oct 30 19:15:52.514977 ip-10-0-132-82 kubenswrapper[4892]: W1030 19:15:52.500983    4892 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Oct 30 19:15:52.514977 ip-10-0-132-82 kubenswrapper[4892]: W1030 19:15:52.500986    4892 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Oct 30 19:15:52.514977 ip-10-0-132-82 kubenswrapper[4892]: W1030 19:15:52.500988    4892 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Oct 30 19:15:52.514977 ip-10-0-132-82 kubenswrapper[4892]: W1030 19:15:52.500991    4892 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Oct 30 19:15:52.515442 ip-10-0-132-82 kubenswrapper[4892]: W1030 19:15:52.500993    4892 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Oct 30 19:15:52.515442 ip-10-0-132-82 kubenswrapper[4892]: W1030 19:15:52.500996    4892 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Oct 30 19:15:52.515442 ip-10-0-132-82 kubenswrapper[4892]: W1030 19:15:52.500998    4892 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Oct 30 19:15:52.515442 ip-10-0-132-82 kubenswrapper[4892]: W1030 19:15:52.501001    4892 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Oct 30 19:15:52.515442 ip-10-0-132-82 kubenswrapper[4892]: W1030 19:15:52.501003    4892 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Oct 30 19:15:52.515442 ip-10-0-132-82 kubenswrapper[4892]: W1030 19:15:52.501006    4892 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Oct 30 19:15:52.515442 ip-10-0-132-82 kubenswrapper[4892]: W1030 19:15:52.501008    4892 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Oct 30 19:15:52.515442 ip-10-0-132-82 kubenswrapper[4892]: W1030 19:15:52.501012    4892 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Oct 30 19:15:52.515442 ip-10-0-132-82 kubenswrapper[4892]: W1030 19:15:52.501016    4892 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Oct 30 19:15:52.515442 ip-10-0-132-82 kubenswrapper[4892]: W1030 19:15:52.501019    4892 feature_gate.go:386] unrecognized feature gate: Example2
Oct 30 19:15:52.515442 ip-10-0-132-82 kubenswrapper[4892]: W1030 19:15:52.501021    4892 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Oct 30 19:15:52.515442 ip-10-0-132-82 kubenswrapper[4892]: W1030 19:15:52.501024    4892 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Oct 30 19:15:52.515442 ip-10-0-132-82 kubenswrapper[4892]: W1030 19:15:52.501028    4892 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Oct 30 19:15:52.515442 ip-10-0-132-82 kubenswrapper[4892]: W1030 19:15:52.501030    4892 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Oct 30 19:15:52.515442 ip-10-0-132-82 kubenswrapper[4892]: W1030 19:15:52.501033    4892 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Oct 30 19:15:52.515442 ip-10-0-132-82 kubenswrapper[4892]: W1030 19:15:52.501041    4892 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Oct 30 19:15:52.515442 ip-10-0-132-82 kubenswrapper[4892]: W1030 19:15:52.501044    4892 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Oct 30 19:15:52.515442 ip-10-0-132-82 kubenswrapper[4892]: W1030 19:15:52.501047    4892 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Oct 30 19:15:52.515442 ip-10-0-132-82 kubenswrapper[4892]: W1030 19:15:52.501049    4892 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Oct 30 19:15:52.515920 ip-10-0-132-82 kubenswrapper[4892]: W1030 19:15:52.501052    4892 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Oct 30 19:15:52.515920 ip-10-0-132-82 kubenswrapper[4892]: W1030 19:15:52.501055    4892 feature_gate.go:386] unrecognized feature gate: Example
Oct 30 19:15:52.515920 ip-10-0-132-82 kubenswrapper[4892]: W1030 19:15:52.501058    4892 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Oct 30 19:15:52.515920 ip-10-0-132-82 kubenswrapper[4892]: W1030 19:15:52.501060    4892 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Oct 30 19:15:52.515920 ip-10-0-132-82 kubenswrapper[4892]: W1030 19:15:52.501062    4892 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Oct 30 19:15:52.515920 ip-10-0-132-82 kubenswrapper[4892]: W1030 19:15:52.501065    4892 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Oct 30 19:15:52.515920 ip-10-0-132-82 kubenswrapper[4892]: W1030 19:15:52.501068    4892 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Oct 30 19:15:52.515920 ip-10-0-132-82 kubenswrapper[4892]: W1030 19:15:52.501070    4892 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Oct 30 19:15:52.515920 ip-10-0-132-82 kubenswrapper[4892]: W1030 19:15:52.501073    4892 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Oct 30 19:15:52.515920 ip-10-0-132-82 kubenswrapper[4892]: W1030 19:15:52.501075    4892 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Oct 30 19:15:52.515920 ip-10-0-132-82 kubenswrapper[4892]: W1030 19:15:52.501078    4892 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Oct 30 19:15:52.515920 ip-10-0-132-82 kubenswrapper[4892]: W1030 19:15:52.501080    4892 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Oct 30 19:15:52.515920 ip-10-0-132-82 kubenswrapper[4892]: W1030 19:15:52.501082    4892 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Oct 30 19:15:52.515920 ip-10-0-132-82 kubenswrapper[4892]: W1030 19:15:52.501086    4892 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Oct 30 19:15:52.515920 ip-10-0-132-82 kubenswrapper[4892]: W1030 19:15:52.501090    4892 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Oct 30 19:15:52.515920 ip-10-0-132-82 kubenswrapper[4892]: W1030 19:15:52.501093    4892 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Oct 30 19:15:52.515920 ip-10-0-132-82 kubenswrapper[4892]: W1030 19:15:52.501096    4892 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Oct 30 19:15:52.515920 ip-10-0-132-82 kubenswrapper[4892]: W1030 19:15:52.501099    4892 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Oct 30 19:15:52.515920 ip-10-0-132-82 kubenswrapper[4892]: W1030 19:15:52.501101    4892 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Oct 30 19:15:52.516392 ip-10-0-132-82 kubenswrapper[4892]: W1030 19:15:52.501104    4892 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Oct 30 19:15:52.516392 ip-10-0-132-82 kubenswrapper[4892]: W1030 19:15:52.501106    4892 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Oct 30 19:15:52.516392 ip-10-0-132-82 kubenswrapper[4892]: W1030 19:15:52.501109    4892 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Oct 30 19:15:52.516392 ip-10-0-132-82 kubenswrapper[4892]: W1030 19:15:52.501112    4892 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Oct 30 19:15:52.516392 ip-10-0-132-82 kubenswrapper[4892]: W1030 19:15:52.501114    4892 feature_gate.go:386] unrecognized feature gate: OSStreams
Oct 30 19:15:52.516392 ip-10-0-132-82 kubenswrapper[4892]: W1030 19:15:52.501116    4892 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Oct 30 19:15:52.516392 ip-10-0-132-82 kubenswrapper[4892]: W1030 19:15:52.501119    4892 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Oct 30 19:15:52.516392 ip-10-0-132-82 kubenswrapper[4892]: W1030 19:15:52.501121    4892 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Oct 30 19:15:52.516392 ip-10-0-132-82 kubenswrapper[4892]: W1030 19:15:52.501124    4892 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Oct 30 19:15:52.516392 ip-10-0-132-82 kubenswrapper[4892]: W1030 19:15:52.501126    4892 feature_gate.go:386] unrecognized feature gate: EventTTL
Oct 30 19:15:52.516392 ip-10-0-132-82 kubenswrapper[4892]: W1030 19:15:52.501134    4892 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Oct 30 19:15:52.516392 ip-10-0-132-82 kubenswrapper[4892]: W1030 19:15:52.501137    4892 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Oct 30 19:15:52.516392 ip-10-0-132-82 kubenswrapper[4892]: W1030 19:15:52.501140    4892 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Oct 30 19:15:52.516392 ip-10-0-132-82 kubenswrapper[4892]: W1030 19:15:52.501142    4892 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Oct 30 19:15:52.516392 ip-10-0-132-82 kubenswrapper[4892]: W1030 19:15:52.501144    4892 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Oct 30 19:15:52.516392 ip-10-0-132-82 kubenswrapper[4892]: W1030 19:15:52.501147    4892 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Oct 30 19:15:52.516392 ip-10-0-132-82 kubenswrapper[4892]: W1030 19:15:52.501149    4892 feature_gate.go:386] unrecognized feature gate: SignatureStores
Oct 30 19:15:52.516392 ip-10-0-132-82 kubenswrapper[4892]: W1030 19:15:52.501152    4892 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Oct 30 19:15:52.516392 ip-10-0-132-82 kubenswrapper[4892]: W1030 19:15:52.501154    4892 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Oct 30 19:15:52.516392 ip-10-0-132-82 kubenswrapper[4892]: W1030 19:15:52.501157    4892 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Oct 30 19:15:52.516908 ip-10-0-132-82 kubenswrapper[4892]: W1030 19:15:52.501159    4892 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Oct 30 19:15:52.516908 ip-10-0-132-82 kubenswrapper[4892]: W1030 19:15:52.501162    4892 feature_gate.go:386] unrecognized feature gate: DualReplica
Oct 30 19:15:52.516908 ip-10-0-132-82 kubenswrapper[4892]: W1030 19:15:52.501164    4892 feature_gate.go:386] unrecognized feature gate: NewOLM
Oct 30 19:15:52.516908 ip-10-0-132-82 kubenswrapper[4892]: W1030 19:15:52.501166    4892 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Oct 30 19:15:52.516908 ip-10-0-132-82 kubenswrapper[4892]: W1030 19:15:52.501169    4892 feature_gate.go:386] unrecognized feature gate: EVPN
Oct 30 19:15:52.516908 ip-10-0-132-82 kubenswrapper[4892]: W1030 19:15:52.501172    4892 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Oct 30 19:15:52.516908 ip-10-0-132-82 kubenswrapper[4892]: W1030 19:15:52.501174    4892 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Oct 30 19:15:52.516908 ip-10-0-132-82 kubenswrapper[4892]: W1030 19:15:52.501176    4892 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Oct 30 19:15:52.516908 ip-10-0-132-82 kubenswrapper[4892]: W1030 19:15:52.501179    4892 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Oct 30 19:15:52.516908 ip-10-0-132-82 kubenswrapper[4892]: W1030 19:15:52.501181    4892 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Oct 30 19:15:52.516908 ip-10-0-132-82 kubenswrapper[4892]: W1030 19:15:52.501184    4892 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Oct 30 19:15:52.516908 ip-10-0-132-82 kubenswrapper[4892]: W1030 19:15:52.501186    4892 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Oct 30 19:15:52.516908 ip-10-0-132-82 kubenswrapper[4892]: W1030 19:15:52.501188    4892 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Oct 30 19:15:52.516908 ip-10-0-132-82 kubenswrapper[4892]: W1030 19:15:52.501191    4892 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Oct 30 19:15:52.516908 ip-10-0-132-82 kubenswrapper[4892]: W1030 19:15:52.501193    4892 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Oct 30 19:15:52.516908 ip-10-0-132-82 kubenswrapper[4892]: W1030 19:15:52.501196    4892 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Oct 30 19:15:52.516908 ip-10-0-132-82 kubenswrapper[4892]: W1030 19:15:52.501198    4892 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Oct 30 19:15:52.517394 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:15:52.501221    4892 feature_gate.go:465] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false DRAPartitionableDevices:false EventedPLEG:false KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:true MutatingAdmissionPolicy:false RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true UserNamespacesSupport:true]}
Oct 30 19:15:52.517394 ip-10-0-132-82 kubenswrapper[4892]: W1030 19:15:52.501261    4892 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Oct 30 19:15:52.517394 ip-10-0-132-82 kubenswrapper[4892]: W1030 19:15:52.501265    4892 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Oct 30 19:15:52.517394 ip-10-0-132-82 kubenswrapper[4892]: W1030 19:15:52.501268    4892 feature_gate.go:386] unrecognized feature gate: Example
Oct 30 19:15:52.517394 ip-10-0-132-82 kubenswrapper[4892]: W1030 19:15:52.501271    4892 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Oct 30 19:15:52.517394 ip-10-0-132-82 kubenswrapper[4892]: W1030 19:15:52.501274    4892 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Oct 30 19:15:52.517394 ip-10-0-132-82 kubenswrapper[4892]: W1030 19:15:52.501282    4892 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Oct 30 19:15:52.517394 ip-10-0-132-82 kubenswrapper[4892]: W1030 19:15:52.501285    4892 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Oct 30 19:15:52.517394 ip-10-0-132-82 kubenswrapper[4892]: W1030 19:15:52.501288    4892 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Oct 30 19:15:52.517394 ip-10-0-132-82 kubenswrapper[4892]: W1030 19:15:52.501290    4892 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Oct 30 19:15:52.517394 ip-10-0-132-82 kubenswrapper[4892]: W1030 19:15:52.501293    4892 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Oct 30 19:15:52.517394 ip-10-0-132-82 kubenswrapper[4892]: W1030 19:15:52.501295    4892 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Oct 30 19:15:52.517394 ip-10-0-132-82 kubenswrapper[4892]: W1030 19:15:52.501298    4892 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Oct 30 19:15:52.517394 ip-10-0-132-82 kubenswrapper[4892]: W1030 19:15:52.501301    4892 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Oct 30 19:15:52.517394 ip-10-0-132-82 kubenswrapper[4892]: W1030 19:15:52.501303    4892 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Oct 30 19:15:52.517394 ip-10-0-132-82 kubenswrapper[4892]: W1030 19:15:52.501307    4892 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Oct 30 19:15:52.517789 ip-10-0-132-82 kubenswrapper[4892]: W1030 19:15:52.501311    4892 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Oct 30 19:15:52.517789 ip-10-0-132-82 kubenswrapper[4892]: W1030 19:15:52.501314    4892 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Oct 30 19:15:52.517789 ip-10-0-132-82 kubenswrapper[4892]: W1030 19:15:52.501317    4892 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Oct 30 19:15:52.517789 ip-10-0-132-82 kubenswrapper[4892]: W1030 19:15:52.501319    4892 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Oct 30 19:15:52.517789 ip-10-0-132-82 kubenswrapper[4892]: W1030 19:15:52.501322    4892 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Oct 30 19:15:52.517789 ip-10-0-132-82 kubenswrapper[4892]: W1030 19:15:52.501324    4892 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Oct 30 19:15:52.517789 ip-10-0-132-82 kubenswrapper[4892]: W1030 19:15:52.501326    4892 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Oct 30 19:15:52.517789 ip-10-0-132-82 kubenswrapper[4892]: W1030 19:15:52.501329    4892 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Oct 30 19:15:52.517789 ip-10-0-132-82 kubenswrapper[4892]: W1030 19:15:52.501331    4892 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Oct 30 19:15:52.517789 ip-10-0-132-82 kubenswrapper[4892]: W1030 19:15:52.501334    4892 feature_gate.go:386] unrecognized feature gate: OSStreams
Oct 30 19:15:52.517789 ip-10-0-132-82 kubenswrapper[4892]: W1030 19:15:52.501336    4892 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Oct 30 19:15:52.517789 ip-10-0-132-82 kubenswrapper[4892]: W1030 19:15:52.501338    4892 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Oct 30 19:15:52.517789 ip-10-0-132-82 kubenswrapper[4892]: W1030 19:15:52.501341    4892 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Oct 30 19:15:52.517789 ip-10-0-132-82 kubenswrapper[4892]: W1030 19:15:52.501343    4892 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Oct 30 19:15:52.517789 ip-10-0-132-82 kubenswrapper[4892]: W1030 19:15:52.501346    4892 feature_gate.go:386] unrecognized feature gate: EventTTL
Oct 30 19:15:52.517789 ip-10-0-132-82 kubenswrapper[4892]: W1030 19:15:52.501354    4892 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Oct 30 19:15:52.517789 ip-10-0-132-82 kubenswrapper[4892]: W1030 19:15:52.501357    4892 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Oct 30 19:15:52.517789 ip-10-0-132-82 kubenswrapper[4892]: W1030 19:15:52.501359    4892 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Oct 30 19:15:52.517789 ip-10-0-132-82 kubenswrapper[4892]: W1030 19:15:52.501362    4892 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Oct 30 19:15:52.517789 ip-10-0-132-82 kubenswrapper[4892]: W1030 19:15:52.501364    4892 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Oct 30 19:15:52.518313 ip-10-0-132-82 kubenswrapper[4892]: W1030 19:15:52.501366    4892 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Oct 30 19:15:52.518313 ip-10-0-132-82 kubenswrapper[4892]: W1030 19:15:52.501369    4892 feature_gate.go:386] unrecognized feature gate: SignatureStores
Oct 30 19:15:52.518313 ip-10-0-132-82 kubenswrapper[4892]: W1030 19:15:52.501371    4892 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Oct 30 19:15:52.518313 ip-10-0-132-82 kubenswrapper[4892]: W1030 19:15:52.501381    4892 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Oct 30 19:15:52.518313 ip-10-0-132-82 kubenswrapper[4892]: W1030 19:15:52.501384    4892 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Oct 30 19:15:52.518313 ip-10-0-132-82 kubenswrapper[4892]: W1030 19:15:52.501386    4892 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Oct 30 19:15:52.518313 ip-10-0-132-82 kubenswrapper[4892]: W1030 19:15:52.501388    4892 feature_gate.go:386] unrecognized feature gate: DualReplica
Oct 30 19:15:52.518313 ip-10-0-132-82 kubenswrapper[4892]: W1030 19:15:52.501391    4892 feature_gate.go:386] unrecognized feature gate: NewOLM
Oct 30 19:15:52.518313 ip-10-0-132-82 kubenswrapper[4892]: W1030 19:15:52.501393    4892 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Oct 30 19:15:52.518313 ip-10-0-132-82 kubenswrapper[4892]: W1030 19:15:52.501396    4892 feature_gate.go:386] unrecognized feature gate: EVPN
Oct 30 19:15:52.518313 ip-10-0-132-82 kubenswrapper[4892]: W1030 19:15:52.501398    4892 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Oct 30 19:15:52.518313 ip-10-0-132-82 kubenswrapper[4892]: W1030 19:15:52.501401    4892 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Oct 30 19:15:52.518313 ip-10-0-132-82 kubenswrapper[4892]: W1030 19:15:52.501403    4892 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Oct 30 19:15:52.518313 ip-10-0-132-82 kubenswrapper[4892]: W1030 19:15:52.501408    4892 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Oct 30 19:15:52.518313 ip-10-0-132-82 kubenswrapper[4892]: W1030 19:15:52.501410    4892 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Oct 30 19:15:52.518313 ip-10-0-132-82 kubenswrapper[4892]: W1030 19:15:52.501413    4892 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Oct 30 19:15:52.518313 ip-10-0-132-82 kubenswrapper[4892]: W1030 19:15:52.501415    4892 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Oct 30 19:15:52.518313 ip-10-0-132-82 kubenswrapper[4892]: W1030 19:15:52.501418    4892 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Oct 30 19:15:52.518313 ip-10-0-132-82 kubenswrapper[4892]: W1030 19:15:52.501420    4892 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Oct 30 19:15:52.518313 ip-10-0-132-82 kubenswrapper[4892]: W1030 19:15:52.501423    4892 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Oct 30 19:15:52.518812 ip-10-0-132-82 kubenswrapper[4892]: W1030 19:15:52.501425    4892 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Oct 30 19:15:52.518812 ip-10-0-132-82 kubenswrapper[4892]: W1030 19:15:52.501428    4892 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Oct 30 19:15:52.518812 ip-10-0-132-82 kubenswrapper[4892]: W1030 19:15:52.501430    4892 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Oct 30 19:15:52.518812 ip-10-0-132-82 kubenswrapper[4892]: W1030 19:15:52.501434    4892 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Oct 30 19:15:52.518812 ip-10-0-132-82 kubenswrapper[4892]: W1030 19:15:52.501437    4892 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Oct 30 19:15:52.518812 ip-10-0-132-82 kubenswrapper[4892]: W1030 19:15:52.501440    4892 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Oct 30 19:15:52.518812 ip-10-0-132-82 kubenswrapper[4892]: W1030 19:15:52.501442    4892 feature_gate.go:386] unrecognized feature gate: OVNObservability
Oct 30 19:15:52.518812 ip-10-0-132-82 kubenswrapper[4892]: W1030 19:15:52.501445    4892 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Oct 30 19:15:52.518812 ip-10-0-132-82 kubenswrapper[4892]: W1030 19:15:52.501447    4892 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Oct 30 19:15:52.518812 ip-10-0-132-82 kubenswrapper[4892]: W1030 19:15:52.501450    4892 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Oct 30 19:15:52.518812 ip-10-0-132-82 kubenswrapper[4892]: W1030 19:15:52.501452    4892 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Oct 30 19:15:52.518812 ip-10-0-132-82 kubenswrapper[4892]: W1030 19:15:52.501455    4892 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Oct 30 19:15:52.518812 ip-10-0-132-82 kubenswrapper[4892]: W1030 19:15:52.501457    4892 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Oct 30 19:15:52.518812 ip-10-0-132-82 kubenswrapper[4892]: W1030 19:15:52.501460    4892 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Oct 30 19:15:52.518812 ip-10-0-132-82 kubenswrapper[4892]: W1030 19:15:52.501462    4892 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Oct 30 19:15:52.518812 ip-10-0-132-82 kubenswrapper[4892]: W1030 19:15:52.501465    4892 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Oct 30 19:15:52.518812 ip-10-0-132-82 kubenswrapper[4892]: W1030 19:15:52.501473    4892 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Oct 30 19:15:52.518812 ip-10-0-132-82 kubenswrapper[4892]: W1030 19:15:52.501476    4892 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Oct 30 19:15:52.518812 ip-10-0-132-82 kubenswrapper[4892]: W1030 19:15:52.501478    4892 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Oct 30 19:15:52.519295 ip-10-0-132-82 kubenswrapper[4892]: W1030 19:15:52.501481    4892 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Oct 30 19:15:52.519295 ip-10-0-132-82 kubenswrapper[4892]: W1030 19:15:52.501483    4892 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Oct 30 19:15:52.519295 ip-10-0-132-82 kubenswrapper[4892]: W1030 19:15:52.501486    4892 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Oct 30 19:15:52.519295 ip-10-0-132-82 kubenswrapper[4892]: W1030 19:15:52.501488    4892 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Oct 30 19:15:52.519295 ip-10-0-132-82 kubenswrapper[4892]: W1030 19:15:52.501490    4892 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Oct 30 19:15:52.519295 ip-10-0-132-82 kubenswrapper[4892]: W1030 19:15:52.501493    4892 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Oct 30 19:15:52.519295 ip-10-0-132-82 kubenswrapper[4892]: W1030 19:15:52.501497    4892 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Oct 30 19:15:52.519295 ip-10-0-132-82 kubenswrapper[4892]: W1030 19:15:52.501499    4892 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Oct 30 19:15:52.519295 ip-10-0-132-82 kubenswrapper[4892]: W1030 19:15:52.501502    4892 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Oct 30 19:15:52.519295 ip-10-0-132-82 kubenswrapper[4892]: W1030 19:15:52.501504    4892 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Oct 30 19:15:52.519295 ip-10-0-132-82 kubenswrapper[4892]: W1030 19:15:52.501507    4892 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Oct 30 19:15:52.519295 ip-10-0-132-82 kubenswrapper[4892]: W1030 19:15:52.501509    4892 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Oct 30 19:15:52.519295 ip-10-0-132-82 kubenswrapper[4892]: W1030 19:15:52.501512    4892 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Oct 30 19:15:52.519295 ip-10-0-132-82 kubenswrapper[4892]: W1030 19:15:52.501514    4892 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Oct 30 19:15:52.519295 ip-10-0-132-82 kubenswrapper[4892]: W1030 19:15:52.501516    4892 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Oct 30 19:15:52.519295 ip-10-0-132-82 kubenswrapper[4892]: W1030 19:15:52.501519    4892 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Oct 30 19:15:52.519295 ip-10-0-132-82 kubenswrapper[4892]: W1030 19:15:52.501523    4892 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Oct 30 19:15:52.519295 ip-10-0-132-82 kubenswrapper[4892]: W1030 19:15:52.501525    4892 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Oct 30 19:15:52.519295 ip-10-0-132-82 kubenswrapper[4892]: W1030 19:15:52.501528    4892 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Oct 30 19:15:52.519759 ip-10-0-132-82 kubenswrapper[4892]: W1030 19:15:52.501530    4892 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Oct 30 19:15:52.519759 ip-10-0-132-82 kubenswrapper[4892]: W1030 19:15:52.501533    4892 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Oct 30 19:15:52.519759 ip-10-0-132-82 kubenswrapper[4892]: W1030 19:15:52.501540    4892 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Oct 30 19:15:52.519759 ip-10-0-132-82 kubenswrapper[4892]: W1030 19:15:52.501544    4892 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Oct 30 19:15:52.519759 ip-10-0-132-82 kubenswrapper[4892]: W1030 19:15:52.501548    4892 feature_gate.go:386] unrecognized feature gate: Example2
Oct 30 19:15:52.519759 ip-10-0-132-82 kubenswrapper[4892]: W1030 19:15:52.501551    4892 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Oct 30 19:15:52.519759 ip-10-0-132-82 kubenswrapper[4892]: W1030 19:15:52.501553    4892 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Oct 30 19:15:52.519759 ip-10-0-132-82 kubenswrapper[4892]: W1030 19:15:52.501556    4892 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Oct 30 19:15:52.519759 ip-10-0-132-82 kubenswrapper[4892]: W1030 19:15:52.501558    4892 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Oct 30 19:15:52.519759 ip-10-0-132-82 kubenswrapper[4892]: W1030 19:15:52.501561    4892 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Oct 30 19:15:52.519759 ip-10-0-132-82 kubenswrapper[4892]: W1030 19:15:52.501563    4892 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Oct 30 19:15:52.519759 ip-10-0-132-82 kubenswrapper[4892]: W1030 19:15:52.501572    4892 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Oct 30 19:15:52.519759 ip-10-0-132-82 kubenswrapper[4892]: W1030 19:15:52.501575    4892 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Oct 30 19:15:52.519759 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:15:52.501620    4892 server.go:957] "Client rotation is on, will bootstrap in background"
Oct 30 19:15:52.519759 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:15:52.502638    4892 bootstrap.go:85] "Current kubeconfig file contents are still valid, no bootstrap necessary"
Oct 30 19:15:52.519759 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:15:52.502688    4892 certificate_store.go:147] "Loading cert/key pair from a file" filePath="/var/lib/kubelet/pki/kubelet-client-current.pem"
Oct 30 19:15:52.519759 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:15:52.502868    4892 server.go:1014] "Starting client certificate rotation"
Oct 30 19:15:52.519759 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:15:52.502995    4892 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kube-apiserver-client-kubelet"
Oct 30 19:15:52.520249 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:15:52.503044    4892 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kube-apiserver-client-kubelet" expiration="2028-10-29 19:10:10 +0000 UTC" deadline="2028-05-19 07:46:02.136994246 +0000 UTC"
Oct 30 19:15:52.520249 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:15:52.503069    4892 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kube-apiserver-client-kubelet" sleep="13596h30m9.633927096s"
Oct 30 19:15:52.520249 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:15:52.503377    4892 dynamic_cafile_content.go:123] "Loaded a new CA Bundle and Verifier" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt"
Oct 30 19:15:52.520249 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:15:52.504639    4892 dynamic_cafile_content.go:161] "Starting controller" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt"
Oct 30 19:15:52.520249 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:15:52.507394    4892 log.go:25] "Validated CRI v1 runtime API"
Oct 30 19:15:52.520249 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:15:52.509423    4892 log.go:25] "Validated CRI v1 image API"
Oct 30 19:15:52.520249 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:15:52.510198    4892 server.go:1424] "Using cgroup driver setting received from the CRI runtime" cgroupDriver="systemd"
Oct 30 19:15:52.520249 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:15:52.511940    4892 fs.go:135] Filesystem UUIDs: map[2136bd09-3cc3-4cc1-9474-9c6a9d7fad31:/dev/nvme0n1p3 7B77-95E7:/dev/nvme0n1p2 cb428dfa-9865-415d-9f85-f41e9bc9d06c:/dev/nvme0n1p4]
Oct 30 19:15:52.520575 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:15:52.511986    4892 fs.go:136] Filesystem partitions: map[/dev/nvme0n1p3:{mountpoint:/boot major:259 minor:3 fsType:ext4 blockSize:0} /dev/nvme0n1p4:{mountpoint:/var major:259 minor:4 fsType:xfs blockSize:0} /dev/shm:{mountpoint:/dev/shm major:0 minor:24 fsType:tmpfs blockSize:0} /run:{mountpoint:/run major:0 minor:26 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/0c376387c3362ef564a7120cabcf536b058ed3e285e143402927edceb193291f/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/0c376387c3362ef564a7120cabcf536b058ed3e285e143402927edceb193291f/userdata/shm major:0 minor:84 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/1eeb30806320ba780c1de4b8401be73d6acefab9ba0c36688cdd52db42e2dead/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/1eeb30806320ba780c1de4b8401be73d6acefab9ba0c36688cdd52db42e2dead/userdata/shm major:0 minor:85 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/1f9b62b0486573df560be923ea7c2199b7827bf627151bdffb9652a48a0aa8b6/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/1f9b62b0486573df560be923ea7c2199b7827bf627151bdffb9652a48a0aa8b6/userdata/shm major:0 minor:242 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/238545c7c7675395e4d64454432551c3daf4585d69f1ee4867b334786a60f6f1/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/238545c7c7675395e4d64454432551c3daf4585d69f1ee4867b334786a60f6f1/userdata/shm major:0 minor:231 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/2edc71ff426362ce04e9fc22239cdc12c104ebc68abb45589d168aa8e283eba0/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/2edc71ff426362ce04e9fc22239cdc12c104ebc68abb45589d168aa8e283eba0/userdata/shm major:0 minor:236 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/32595290e48470808e0ab2db048e03df52c1e31b97a08ef92d7a4c00da701711/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/32595290e48470808e0ab2db048e03df52c1e31b97a08ef92d7a4c00da701711/userdata/shm major:0 minor:76 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/3bea07ac8a9e43055344e71268ef13eda8a7b0eb15593518b8076be757e09c3a/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/3bea07ac8a9e43055344e71268ef13eda8a7b0eb15593518b8076be757e09c3a/userdata/shm major:0 minor:48 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/4f8ff29fa81054ed5a37cd86a5ec74649a5651dc5a63c441321ba062767aa578/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/4f8ff29fa81054ed5a37cd86a5ec74649a5651dc5a63c441321ba062767aa578/userdata/shm major:0 minor:68 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/5ef42db8ca33f0169c41836d553c454c6d250a87f4246c19237b64f69a39ef34/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/5ef42db8ca33f0169c41836d553c454c6d250a87f4246c19237b64f69a39ef34/userdata/shm major:0 minor:98 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/71870ffe35cbe4432b34eca4162a3fe61340fcd0c1229a11bee6c775daae713a/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/71870ffe35cbe4432b34eca4162a3fe61340fcd0c1229a11bee6c775daae713a/userdata/shm major:0 minor:94 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/929421d442561c4ba4793828037ab0e15ea8bdf99899887708c893190983a48b/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/929421d442561c4ba4793828037ab0e15ea8bdf99899887708c893190983a48b/userdata/shm major:0 minor:80 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/97f8a11334a257518f5e40c7050de6a55a85bab3727386b10b4f04f52c8f0bcd/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/97f8a11334a257518f5e40c7050de6a55a85bab3727386b10b4f04f52c8f0bcd/userdata/shm major:0 minor:246 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/9a980ae6fdff8b6bb898b3b37c651c25cdbecbdcdeacc75bf47c59da9b0302a0/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/9a980ae6fdff8b6bb898b3b37c651c25cdbecbdcdeacc75bf47c59da9b0302a0/userdata/shm major:0 minor:219 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/9e89eaa98efd97ac57e6419b178d76a3452fdc3097e597ea17f038f98d99832e/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/9e89eaa98efd97ac57e6419b178d76a3452fdc3097e597ea17f038f98d99832e/userdata/shm major:0 minor:225 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/a203f34f66a5ec4d9297e5f8c1bde08af3007fa46ef54906494feccb340e9a27/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/a203f34f66a5ec4d9297e5f8c1bde08af3007fa46ef54906494feccb340e9a27/userdata/shm major:0 minor:244 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/abc2efb0ba45504bce33f1ce80ee88f9d7e4fc489dd1c5d851c51f8ed53b6b0b/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/abc2efb0ba45504bce33f1ce80ee88f9d7e4fc489dd1c5d851c51f8ed53b6b0b/userdata/shm major:0 minor:290 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/b2829769d2e875408df83ae72236364dc323bc32fa4adc204d20ae79308ef283/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/b2829769d2e875408df83ae72236364dc323bc32fa4adc204d20ae79308ef283/userdata/shm major:0 minor:47 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/e2d4de650f3571ac3ab4c420e5b93dc9761667b74a8fe10fb4e7b3decf5ace3f/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/e2d4de650f3571ac3ab4c420e5b93dc9761667b74a8fe10fb4e7b3decf5ace3f/userdata/shm major:0 minor:287 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/e4dd71d689867fb5f56c774748d9b5a5f2bedffc837d9d7c0ad1723524d414ca/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/e4dd71d689867fb5f56c774748d9b5a5f2bedffc837d9d7c0ad1723524d414ca/userdata/shm major:0 minor:208 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/f2f64b91fb83a7b6abf1cb091c941acd65de81bf8df0800107508e4e11241e8c/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/f2f64b91fb83a7b6abf1cb091c941acd65de81bf8df0800107508e4e11241e8c/userdata/shm major:0 minor:233 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/f4a575fbd0a385888d1b5e4d7a3e43290db13e5198927239e5901160a4449ad2/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/f4a575fbd0a385888d1b5e4d7a3e43290db13e5198927239e5901160a4449ad2/userdata/shm major:0 minor:72 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/fc2505f6a57fe8bc893233a26b65fea1edffdc09612365750fd0d81c8a31f7d8/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/fc2505f6a57fe8bc893233a26b65fea1edffdc09612365750fd0d81c8a31f7d8/userdata/shm major:0 minor:216 fsType:tmpfs blockSize:0} /tmp:{mountpoint:/tmp major:0 minor:32 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/08ae8ede-9a96-4a84-8ec8-8d364fd476a3/volume-subpaths/run-systemd/ovnkube-controller/6:{mountpoint:/var/lib/kubelet/pods/08ae8ede-9a96-4a84-8ec8-8d364fd476a3/volume-subpaths/run-systemd/ovnkube-controller/6 major:0 minor:26 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/08ae8ede-9a96-4a84-8ec8-8d364fd476a3/volumes/kubernetes.io~projected/kube-api-access-bnxm6:{mountpoint:/var/lib/kubelet/pods/08ae8ede-9a96-4a84-8ec8-8d364fd476a3/volumes/kubernetes.io~projected/kube-api-access-bnxm6 major:0 minor:62 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/08ae8ede-9a96-4a84-8ec8-8d364fd476a3/volumes/kubernetes.io~secret/ovn-node-metrics-cert:{mountpoint:/var/lib/kubelet/pods/08ae8ede-9a96-4a84-8ec8-8d364fd476a3/volumes/kubernetes.io~secret/ovn-node-metrics-cert major:0 minor:58 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/0e179af5-9f52-4941-b4f5-5e3b8789da11/volumes/kubernetes.io~projected/bound-sa-token:{mountpoint:/var/lib/kubelet/pods/0e179af5-9f52-4941-b4f5-5e3b8789da11/volumes/kubernetes.io~projected/bound-sa-token major:0 minor:228 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/0e179af5-9f52-4941-b4f5-5e3b8789da11/volumes/kubernetes.io~projected/kube-api-access-zj4df:{mountpoint:/var/lib/kubelet/pods/0e179af5-9f52-4941-b4f5-5e3b8789da11/volumes/kubernetes.io~projected/kube-api-access-zj4df major:0 minor:227 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/0e179af5-9f52-4941-b4f5-5e3b8789da11/volumes/kubernetes.io~secret/image-registry-private-configuration:{mountpoint:/var/lib/kubelet/pods/0e179af5-9f52-4941-b4f5-5e3b8789da11/volumes/kubernetes.io~secret/image-registry-private-configuration major:0 minor:223 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/0e179af5-9f52-4941-b4f5-5e3b8789da11/volumes/kubernetes.io~secret/installation-pull-secrets:{mountpoint:/var/lib/kubelet/pods/0e179af5-9f52-4941-b4f5-5e3b8789da11/volumes/kubernetes.io~secret/installation-pull-secrets major:0 minor:224 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/11db9cec-653f-4aee-8937-0cf5b367efac/volumes/kubernetes.io~projected/kube-api-access-5xgwq:{mountpoint:/var/lib/kubelet/pods/11db9cec-653f-4aee-8937-0cf5b367efac/volumes/kubernetes.io~projected/kube-api-access-5xgwq major:0 minor:230 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/11db9cec-653f-4aee-8937-0cf5b367efac/volumes/kubernetes.io~secret/serving-cert:{mountpoint:/var/lib/kubelet/pods/11db9cec-653f-4aee-8937-0cf5b367efac/volumes/kubernetes.io~secret/serving-cert major:0 minor:222 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/287ba08f-7b9f-4609-8e35-832c0879c44c/volumes/kubernetes.io~projected/kube-api-access-v7kvw:{mountpoint:/var/lib/kubelet/pods/287ba08f-7b9f-4609-8e35-832c0879c44c/volumes/kubernetes.io~projected/kube-api-access-v7kvw major:0 minor:65 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/2bd84dc4-e96a-43f1-85bf-e1ea4005c193/volumes/kubernetes.io~projected/kube-api-access-4rmts:{mountpoint:/var/lib/kubelet/pods/2bd84dc4-e96a-43f1-85bf-e1ea4005c193/volumes/kubernetes.io~projected/kube-api-access-4rmts major:0 minor:210 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/2bd84dc4-e96a-43f1-85bf-e1ea4005c193/volumes/kubernetes.io~secret/serving-cert:{mountpoint:/var/lib/kubelet/pods/2bd84dc4-e96a-43f1-85bf-e1ea4005c193/volumes/kubernetes.io~secret/serving-cert major:0 minor:203 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/33ce6ceb-02a9-4852-9948-f02e2b39e32f/volumes/kubernetes.io~projected/kube-api-access-btd8n:{mountpoint:/var/lib/kubelet/pods/33ce6ceb-02a9-4852-9948-f02e2b39e32f/volumes/kubernetes.io~projected/kube-api-access-btd8n major:0 minor:239 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/39bb8c16-8282-442a-841b-0056baae014d/volumes/kubernetes.io~empty-dir/etc-tuned:{mountpoint:/var/lib/kubelet/pods/39bb8c16-8282-442a-841b-0056baae014d/volumes/kubernetes.io~empty-dir/etc-tuned major:0 minor:55 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/39bb8c16-8282-442a-841b-0056baae014d/volumes/kubernetes.io~empty-dir/tmp:{mountpoint:/var/lib/kubelet/pods/39bb8c16-8282-442a-841b-0056baae014d/volumes/kubernetes.io~empty-dir/tmp major:0 minor:56 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/39bb8c16-8282-442a-841b-0056baae014d/volumes/kubernetes.io~projected/kube-api-access-gdq6k:{mountpoint:/var/lib/kubelet/pods/39bb8c16-8282-442a-841b-0056baae014d/volumes/kubernetes.io~projected/kube-api-access-gdq6k major:0 minor:63 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/48386105-d79b-4f92-baa0-f330b9658ad0/volumes/kubernetes.io~projected/kube-api-access-w8zg9:{mountpoint:/var/lib/kubelet/pods/48386105-d79b-4f92-baa0-f330b9658ad0/volumes/kubernetes.io~projected/kube-api-access-w8zg9 major:0 minor:67 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/4ba601a6-2a44-4408-8c5b-2c29fd096fed/volumes/kubernetes.io~projected/kube-api-access-rfxzq:{mountpoint:/var/lib/kubelet/pods/4ba601a6-2a44-4408-8c5b-2c29fd096fed/volumes/kubernetes.io~projected/kube-api-access-rfxzq major:0 minor:229 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/4ba601a6-2a44-4408-8c5b-2c29fd096fed/volumes/kubernetes.io~secret/serving-cert:{mountpoint:/var/lib/kubelet/pods/4ba601a6-2a44-4408-8c5b-2c29fd096fed/volumes/kubernetes.io~secret/serving-cert major:0 minor:221 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/4ca01b49-9819-4a90-a851-7553ad568679/volumes/kubernetes.io~projected/kube-api-access-bzw4s:{mountpoint:/var/lib/kubelet/pods/4ca01b49-9819-4a90-a851-7553ad568679/volumes/kubernetes.io~projected/kube-api-access-bzw4s major:0 minor:289 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/4d213db4-73c5-4f50-9930-6d26b880dcf7/volumes/kubernetes.io~projected/kube-api-access-7xf2g:{mountpoint:/var/lib/kubelet/pods/4d213db4-73c5-4f50-9930-6d26b880dcf7/volumes/kubernetes.io~projected/kube-api-access-7xf2g major:0 minor:214 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/56274b23-2400-43d4-9686-c928e20d47de/volumes/kubernetes.io~projected/kube-api-access-wt5rq:{mountpoint:/var/lib/kubelet/pods/56274b23-2400-43d4-9686-c928e20d47de/volumes/kubernetes.io~projected/kube-api-access-wt5rq major:0 minor:211 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/87082646-75c3-43b5-864b-3478e9a7373d/volumes/kubernetes.io~projected/kube-api-access-8s7g5:{mountpoint:/var/lib/kubelet/pods/87082646-75c3-43b5-864b-3478e9a7373d/volumes/kubernetes.io~projected/kube-api-access-8s7g5 major:0 minor:217 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/87082646-75c3-43b5-864b-3478e9a7373d/volumes/kubernetes.io~secret/default-certificate:{mountpoint:/var/lib/kubelet/pods/87082646-75c3-43b5-864b-3478e9a7373d/volumes/kubernetes.io~secret/default-certificate major:0 minor:204 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/87082646-75c3-43b5-864b-3478e9a7373d/volumes/kubernetes.io~secret/stats-auth:{mountpoint:/var/lib/kubelet/pods/87082646-75c3-43b5-864b-3478e9a7373d/volumes/kubernetes.io~secret/stats-auth major:0 minor:207 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/87da348d-b1ee-4cd3-806d-0560bff3c6f2/volumes/kubernetes.io~projected/kube-api-access-txbm4:{mountpoint:/var/lib/kubelet/pods/87da348d-b1ee-4cd3-806d-0560bff3c6f2/volumes/kubernetes.io~projected/kube-api-access-txbm4 major:0 minor:45 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/af36f4d4-f985-4937-a9f7-9884936f3690/volumes/kubernetes.io~secret/agent-certs:{mountpoint:/var/lib/kubelet/pods/af36f4d4-f985-4937-a9f7-9884936f3690/volumes/kubernetes.io~secret/agent-certs major:0 minor:57 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/ba2c5571-23a3-46b6-a36d-7e8810a66261/volumes/kubernetes.io~projected/kube-api-access-k62g9:{mountpoint:/var/lib/kubelet/pods/ba2c5571-23a3-46b6-a36d-7e8810a66261/volumes/kubernetes.io~projected/kube-api-access-k62g9 major:0 minor:197 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/bab0abf9-46d4-408e-9aa0-afa7279cd9d1/volumes/kubernetes.io~projected/kube-api-access-4rcdn:{mountpoint:/var/lib/kubelet/pods/bab0abf9-46d4-408e-9aa0-afa7279cd9d1/volumes/kubernetes.io~projected/kube-api-access-4rcdn major:0 minor:66 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/c16c0a80-c5ce-4f5d-827a-2e3eb2c3cfb8/volumes/kubernetes.io~secret/original-pull-secret:{mountpoint:/var/lib/kubelet/pods/c16c0a80-c5ce-4f5d-827a-2e3eb2c3cfb8/volumes/kubernetes.io~secret/original-pull-secret major:0 minor:286 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/c3780b96-955e-4728-a2d0-b7ec30d0f6f3/volumes/kubernetes.io~projected/kube-api-access-cdkvt:{mountpoint:/var/lib/kubelet/pods/c3780b96-955e-4728-a2d0-b7ec30d0f6f3/volumes/kubernetes.io~projected/kube-api-access-cdkvt major:0 minor:61 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/cdb750aa-dd7e-4e2f-8297-9028acc0f559/volumes/kubernetes.io~projected/kube-api-access-rvrch:{mountpoint:/var/lib/kubelet/pods/cdb750aa-dd7e-4e2f-8297-9028acc0f559/volumes/kubernetes.io~projected/kube-api-access-rvrch major:0 minor:59 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/d260d4d7-b4aa-4f37-92b5-e75b22e13a3b/volumes/kubernetes.io~projected/kube-api-access-wmkq8:{mountpoint:/var/lib/kubelet/pods/d260d4d7-b4aa-4f37-92b5-e75b22e13a3b/volumes/kubernetes.io~projected/kube-api-access-wmkq8 major:0 minor:201 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/dccdaf98-aac7-4756-b93f-22f62a286a1f/volumes/kubernetes.io~projected/kube-api-access-6bqfr:{mountpoint:/var/lib/kubelet/pods/dccdaf98-aac7-4756-b93f-22f62a286a1f/volumes/kubernetes.io~projected/kube-api-access-6bqfr major:0 minor:64 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/e0221987-8e5e-40d1-8842-c97993f08c6d/volumes/kubernetes.io~projected/kube-api-access-fmnd4:{mountpoint:/var/lib/kubelet/pods/e0221987-8e5e-40d1-8842-c97993f08c6d/volumes/kubernetes.io~projected/kube-api-access-fmnd4 major:0 minor:60 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/e3206ddf-35de-4b7c-b979-682f5f666a34/volumes/kubernetes.io~projected/bound-sa-token:{mountpoint:/var/lib/kubelet/pods/e3206ddf-35de-4b7c-b979-682f5f666a34/volumes/kubernetes.io~projected/bound-sa-token major:0 minor:213 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/e3206ddf-35de-4b7c-b979-682f5f666a34/volumes/kubernetes.io~projected/kube-api-access-w9h4f:{mountpoint:/var/lib/kubelet/pods/e3206ddf-35de-4b7c-b979-682f5f666a34/volumes/kubernetes.io~projected/kube-api-access-w9h4f major:0 minor:215 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/e3206ddf-35de-4b7c-b979-682f5f666a34/volumes/kubernetes.io~secret/image-registry-private-configuration:{mountpoint:/var/lib/kubelet/pods/e3206ddf-35de-4b7c-b979-682f5f666a34/volumes/kubernetes.io~secret/image-registry-private-configuration major:0 minor:202 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/e3206ddf-35de-4b7c-b979-682f5f666a34/volumes/kubernetes.io~secret/installation-pull-secrets:{mountpoint:/var/lib/kubelet/pods/e3206ddf-35de-4b7c-b979-682f5f666a34/volumes/kubernetes.io~secret/installation-pull-secrets major:0 minor:205 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/e446ca63-5885-4c66-9cb7-da39afbf742c/volumes/kubernetes.io~projected/kube-api-access-j7bln:{mountpoint:/var/lib/kubelet/pods/e446ca63-5885-4c66-9cb7-da39afbf742c/volumes/kubernetes.io~projected/kube-api-access-j7bln major:0 minor:241 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/f86c0e38-8144-42fc-b622-25602ada113f/volumes/kubernetes.io~projected/kube-api-access-j4wnm:{mountpoint:/var/lib/kubelet/pods/f86c0e38-8144-42fc-b622-25602ada113f/volumes/kubernetes.io~projected/kube-api-access-j4wnm major:0 minor:240 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/f86c0e38-8144-42fc-b622-25602ada113f/volumes/kubernetes.io~secret/cert:{mountpoint:/var/lib/kubelet/pods/f86c0e38-8144-42fc-b622-25602ada113f/volumes/kubernetes.io~secret/cert major:0 minor:235 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/f9a06263-cb99-4004-bffc-581d66ce5ebf/volumes/kubernetes.io~projected/kube-api-access-qfw7m:{mountpoint:/var/lib/kubelet/pods/f9a06263-cb99-4004-bffc-581d66ce5ebf/volumes/kubernetes.io~projected/kube-api-access-qfw7m major:0 minor:212 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/f9a06263-cb99-4004-bffc-581d66ce5ebf/volumes/kubernetes.io~secret/serving-cert:{mountpoint:/var/lib/kubelet/pods/f9a06263-cb99-4004-bffc-581d66ce5ebf/volumes/kubernetes.io~secret/serving-cert major:0 minor:206 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/fade00ba-24a7-41c9-93f7-23e40971874a/volumes/kubernetes.io~projected/kube-api-access-q667z:{mountpoint:/var/lib/kubelet/pods/fade00ba-24a7-41c9-93f7-23e40971874a/volumes/kubernetes.io~projected/kube-api-access-q667z major:0 minor:238 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/fc4ac297-f510-4439-80e2-487056d3bf06/volumes/kubernetes.io~projected/kube-api-access-7d4bn:{mountpoint:/var/lib/kubelet/pods/fc4ac297-f510-4439-80e2-487056d3bf06/volumes/kubernetes.io~projected/kube-api-access-7d4bn major:0 minor:318 fsType:tmpfs blockSize:0} composefs_0-34:{mountpoint:/ major:0 minor:34 fsType:overlay blockSize:0} overlay_0-100:{mountpoint:/var/lib/containers/storage/overlay/c9c28493173041db3a5eb62a958611f00a4975a456c1920423c5f75078a964cb/merged major:0 minor:100 fsType:overlay blockSize:0} overlay_0-102:{mountpoint:/var/lib/containers/storage/overlay/f29891d5336912705a3f647857f20d6732c7e1ff07f9d29556fa496b42ed6adf/merged major:0 minor:102 fsType:overlay blockSize:0} overlay_0-104:{mountpoint:/var/lib/containers/storage/overlay/c0c12ce7f4b12100fbf50f3630c955947a9e7f5c8d77b5834259a21a12e29bbb/merged major:0 minor:104 fsType:overlay blockSize:0} overlay_0-106:{mountpoint:/var/lib/containers/storage/overlay/e3bb89dc9f3d0e526b96a92d208516cc9a17a6b419a641e0742ab0fa5c9c0480/merged major:0 minor:106 fsType:overlay blockSize:0} overlay_0-111:{mountpoint:/var/lib/containers/storage/overlay/228dd80c57af6cb220a77bead19928c4718782f7ae207df9b1a2a76db5e4156d/merged major:0 minor:111 fsType:overlay blockSize:0} overlay_0-119:{mountpoint:/var/lib/containers/storage/overlay/04c2ef3bec13ae0fde27bae143975f2e74fa7b5ea9690e90bbf36f08aa284be5/merged major:0 minor:119 fsType:overlay blockSize:0} overlay_0-124:{mountpoint:/var/lib/containers/storage/overlay/e225f6957167fe90c2dbc2406b4a7f9d8a3636839e53a8d9d577effa24f51d29/merged major:0 minor:124 fsType:overlay blockSize:0} overlay_0-127:{mountpoint:/var/lib/containers/storage/overlay/e1230a6ef2aeb3203d1c210caf262fedb44774d4eb190445b2eb68bd92626452/merged major:0 minor:127 fsType:overlay blockSize:0} overlay_0-129:{mountpoint:/var/lib/containers/storage/overlay/bfdff8985a732f2cbd580971685e7cd5aca7ba7d6f9a62427b3c731ffd14420c/merged major:0 minor:129 fsType:overlay blockSize:0} overlay_0-138:{mountpoint:/var/lib/containers/storage/overlay/8943ad6e0b8fde249e30c7e3743fd1a4f987cbab7f58f97b5d4104ffa87f3ec0/merged major:0 minor:138 fsType:overlay blockSize:0} overlay_0-142:{mountpoint:/var/lib/containers/storage/overlay/f8371fbc3abd21ee0c9bf982a9ef25e9151652aae719535a203e96d80901b35d/merged major:0 minor:142 fsType:overlay blockSize:0} overlay_0-146:{mountpoint:/var/lib/containers/storage/overlay/3d91e44fb8d2d58fd40c44488cfc228dffa5458a2fc813404dacf6df018fb6c1/merged major:0 minor:146 fsType:overlay blockSize:0} overlay_0-152:{mountpoint:/var/lib/containers/storage/overlay/e4828003d7c27fe198dd7a974bbdeafc8e23ab867d6764b96a0d9f6689b9cb4d/merged major:0 minor:152 fsType:overlay blockSize:0} overlay_0-157:{mountpoint:/var/lib/containers/storage/overlay/ab9be0e127f067f400290afad05c7207dd09486bcaec74e16e6b55574aed875f/merged major:0 minor:157 fsType:overlay blockSize:0} overlay_0-158:{mountpoint:/var/lib/containers/storage/overlay/afdebfa4a4be747b8b3d2e197261313d9336862572393e7f04ca52cd76e55433/merged major:0 minor:158 fsType:overlay blockSize:0} overlay_0-160:{mountpoint:/var/lib/containers/storage/overlay/672bc1722211ca824cc3dde6c61fed6205d40f35981b3cf9f1c97705d80bb4d9/merged major:0 minor:160 fsType:overlay blockSize:0} overlay_0-172:{mountpoint:/var/lib/containers/storage/overlay/930db970d789bfc531034b9599bfea3a6c94540d84c16153a27afa877019cce8/merged major:0 minor:172 fsType:overlay blockSize:0} overlay_0-177:{mountpoint:/var/lib/containers/storage/overlay/bc853768bf8271e67c22eb4697e72059571a796aa2de020dfeec2b91f041699f/merged major:0 minor:177 fsType:overlay blockSize:0} overlay_0-182:{mountpoint:/var/lib/containers/storage/overlay/e4e842fe39ced06e9e0884b7ef653c478e931dd1ceb1790de4b0c9709d78fa58/merged major:0 minor:182 fsType:overlay blockSize:0} overlay_0-187:{mountpoint:/var/lib/containers/storage/overlay/e2fded4bdd6e0a1a73cec6ad5cdf50a199745eca82cef00a430b8af752758c88/merged major:0 minor:187 fsType:overlay blockSize:0} overlay_0-192:{mountpoint:/var/lib/containers/storage/overlay/94845fe3322c2cb5aca88906614d9f2c3484e3fd624e06f71933900daa49a404/merged major:0 minor:192 fsType:overlay blockSize:0} overlay_0-248:{mountpoint:/var/lib/containers/storage/overlay/11fe5c00f745b8f2e817669347f648f6ea1dbf99878f1492f578d3fd1af50a4f/merged major:0 minor:248 fsType:overlay blockSize:0} overlay_0-250:{mountpoint:/var/lib/containers/storage/overlay/f5828c229c8ae9040aec033663b072d75fe786bd2ca576e49bd646aa679e179f/merged major:0 minor:250 fsType:overlay blockSize:0} overlay_0-252:{mountpoint:/var/lib/containers/storage/overlay/8737305fe7412134aa3ce3684211a3d051ec7c0dacaebe5fc8ddfac63b631123/merged major:0 minor:252 fsType:overlay blockSize:0} overlay_0-254:{mountpoint:/var/lib/containers/storage/overlay/b3a317702ac1add5db133a886e1ffd433ee8745e4309aa7e6beee66945f7387f/merged major:0 minor:254 fsType:overlay blockSize:0} overlay_0-256:{mountpoint:/var/lib/containers/storage/overlay/52ab2eacdd073e40745cb3ed1c97e030913f69f597bca8bab13544df76d604c3/merged major:0 minor:256 fsType:overlay blockSize:0} overlay_0-263:{mountpoint:/var/lib/containers/storage/overlay/2a18d4040867497a933a2c3aef9d8a2b10335c192477a220e87f6442516b6ece/merged major:0 minor:263 fsType:overlay blockSize:0} overlay_0-265:{mountpoint:/var/lib/containers/storage/overlay/27f121cb91bd0a07ab8c6c85811d5f40f371d603ff3d19b39ac468de2605f33c/merged major:0 minor:265 fsType:overlay blockSize:0} overlay_0-270:{mountpoint:/var/lib/containers/storage/overlay/39021f8cbf3bb1855cfc9a494bc955166d1cfa6bb86a168f6b64e1c0df7e1ec3/merged major:0 minor:270 fsType:overlay blockSize:0} overlay_0-272:{mountpoint:/var/lib/containers/storage/overlay/ff8c76cec6c70b3080d4797d3b3b03a22e51215c8f5bea59699e7e0c5422bd8b/merged major:0 minor:272 fsType:overlay blockSize:0} overlay_0-274:{mountpoint:/var/lib/containers/storage/overlay/c328e520c1be2e9b4db3d8a57210987b82bd41ab1edce3849b2e71007850d3f6/merged major:0 minor:274 fsType:overlay blockSize:0} overlay_0-276:{mountpoint:/var/lib/containers/storage/overlay/3b09c2434f41e08d6344f9ff54d8ae84fd2975d121059cc6ce993b2b9d147a88/merged major:0 minor:276 fsType:overlay blockSize:0} overlay_0-277:{mountpoint:/var/lib/containers/storage/overlay/280dad01eca5a4ede918fc55e762610b7b025d3a6412d3ab703112ed7f4e73ae/merged major:0 minor:277 fsType:overlay blockSize:0} overlay_0-284:{mountpoint:/var/lib/containers/storage/overlay/508d37abc01e56d2b16d4aa2b137f36dac956b688fbd3a3b029879579813d605/merged major:0 minor:284 fsType:overlay blockSize:0} overlay_0-292:{mountpoint:/var/lib/containers/storage/overlay/a1b04b0a5818ebe207c835ba85f1c27127a7989be87bc8fce7023bbb2387b8b7/merged major:0 minor:292 fsType:overlay blockSize:0} overlay_0-294:{mountpoint:/var/lib/containers/storage/overlay/0564d73ee6e6d30d79a2b50b6244670f4968cd412f51dd9346d1f9e7679b5504/merged major:0 minor:294 fsType:overlay blockSize:0} overlay_0-296:{mountpoint:/var/lib/containers/storage/overlay/5b7a7c228123062352a9855d6e2708c08dd20c7dd5b133d7150bd241a02c3465/merged major:0 minor:296 fsType:overlay blockSize:0} overlay_0-298:{mountpoint:/var/lib/containers/storage/overlay/b6843516fb135fc48105b9ae1e00c540d265d06366f149a4892c22d5e43eaddb/merged major:0 minor:298 fsType:overlay blockSize:0} overlay_0-304:{mountpoint:/var/lib/containers/storage/overlay/fd3779eda1849a165d186122c50cc14ec03520432e962b90e79f6760616068fa/merged major:0 minor:304 fsType:overlay blockSize:0} overlay_0-306:{mountpoint:/var/lib/containers/storage/overlay/6ac1766a9cc538b1af1a041ca86536ea58c464444690ab3e9527a7dcf86d2db6/merged major:0 minor:306 fsType:overlay blockSize:0} overlay_0-308:{mountpoint:/var/lib/containers/storage/overlay/a707cfeaab6d1bbc9203560ba29ba581f90ee63870dc70e2b5b9709e35612233/merged major:0 minor:308 fsType:overlay blockSize:0} overlay_0-327:{mountpoint:/var/lib/containers/storage/overlay/1a727a5a1cabf96f12dce862c6e978a292da6a6c27d0a5c0fb0106d934f12b65/merged major:0 minor:327 fsType:overlay blockSize:0} overlay_0-329:{mountpoint:/var/lib/containers/storage/overlay/6ba830b2ca538f3cb2e39ac7778a9b535fd34e68118ddc8ecbbabdebf7014812/merged major:0 minor:329 fsType:overlay blockSize:0} overlay_0-335:{mountpoint:/var/lib/containers/storage/overlay/882bfb5ed19e4780f91e52fcad714eaa59424ab9e478981fa637522a6f5abe53/merged major:0 minor:335 fsType:overlay blockSize:0} overlay_0-51:{mountpoint:/var/lib/containers/storage/overlay/6463d4f29ca6c39f4cba47aa01979071554dbaef3e1804a454378b52a5dceb30/merged major:0 minor:51 fsType:overlay blockSize:0} overlay_0-53:{mountpoint:/var/lib/containers/storage/overlay/f7bc0002a04e475d95fa111c234a311ec85db28bf6b0e00c1bd2fb9abf174d6e/merged major:0 minor:53 fsType:overlay blockSize:0} overlay_0-70:{mountpoint:/var/lib/containers/storage/overlay/2264cfcf170e82802f7e44cf33434ddad7985561aae73aabaa6e411a95d9e966/merged major:0 minor:70 fsType:overlay blockSize:0} overlay_0-74:{mountpoint:/var/lib/containers/storage/overlay/e675fb1138cb20c1ab390bb3e8986ed6aa5eaa35b6800a8a3aa5681f261c4a36/merged major:0 minor:74 fsType:overlay blockSize:0} overlay_0-78:{mountpoint:/var/lib/containers/storage/overlay/0042fe6fa45496cd2b65bf650a32d70d2653cc2724178f01c7b14c7087304c82/merged major:0 minor:78 fsType:overlay blockSize:0} overlay_0-82:{mountpoint:/var/lib/containers/storage/overlay/71f750a6c54e93dc3bcaff86dc6cca99b762f5da1e3a4ba5fa314682e9191048/merged major:0 minor:82 fsType:overlay blockSize:0} overlay_0-88:{mountpoint:/var/lib/containers/storage/overlay/b55d54f321860f01d45dc45fed85dee53fb1d627737e92c2e4e3749d48e6c97d/merged major:0 minor:88 fsType:overlay blockSize:0} overlay_0-90:{mountpoint:/var/lib/containers/storage/overlay/cdf0e1f0057e08b8917358304bcd2dbe7d57caad48b6e0f6f90654aae6c12975/merged major:0 minor:90 fsType:overlay blockSize:0} overlay_0-92:{mountpoint:/var/lib/containers/storage/overlay/22b7748afcb3a9eb26463bac40729efdeb6675f9d5fd6bc0951dce6433119f0d/merged major:0 minor:92 fsType:overlay blockSize:0} overlay_0-96:{mountpoint:/var/lib/containers/storage/overlay/ea0feae4f7ac209d3b2afd851e8dcd5db9e27f70018f4324d1a0456459b22d9e/merged major:0 minor:96 fsType:overlay blockSize:0}]
Oct 30 19:15:52.520862 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:15:52.519436    4892 manager.go:217] Machine: {Timestamp:2026-10-30 19:15:52.518973586 +0000 UTC m=+0.089439200 CPUVendorID:GenuineIntel NumCores:8 NumPhysicalCores:4 NumSockets:1 CpuFrequency:3121104 MemoryCapacity:32812007424 SwapCapacity:0 MemoryByType:map[] NVMInfo:{MemoryModeCapacity:0 AppDirectModeCapacity:0 AvgPowerBudget:0} HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] MachineID:ec27e6011dbd50892ffc479345e98622 SystemUUID:ec27e601-1dbd-5089-2ffc-479345e98622 BootID:d31c3a64-bf37-4f8c-a6a4-b569507d4ade Filesystems:[{Device:/run/containers/storage/overlay-containers/f4a575fbd0a385888d1b5e4d7a3e43290db13e5198927239e5901160a4449ad2/userdata/shm DeviceMajor:0 DeviceMinor:72 Capacity:67108864 Type:vfs Inodes:4005372 HasInodes:true} {Device:overlay_0-74 DeviceMajor:0 DeviceMinor:74 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-263 DeviceMajor:0 DeviceMinor:263 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-104 DeviceMajor:0 DeviceMinor:104 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-172 DeviceMajor:0 DeviceMinor:172 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/tmp DeviceMajor:0 DeviceMinor:32 Capacity:16406003712 Type:vfs Inodes:1048576 HasInodes:true} {Device:overlay_0-78 DeviceMajor:0 DeviceMinor:78 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-90 DeviceMajor:0 DeviceMinor:90 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/238545c7c7675395e4d64454432551c3daf4585d69f1ee4867b334786a60f6f1/userdata/shm DeviceMajor:0 DeviceMinor:231 Capacity:67108864 Type:vfs Inodes:4005372 HasInodes:true} {Device:overlay_0-160 DeviceMajor:0 DeviceMinor:160 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/e2d4de650f3571ac3ab4c420e5b93dc9761667b74a8fe10fb4e7b3decf5ace3f/userdata/shm DeviceMajor:0 DeviceMinor:287 Capacity:67108864 Type:vfs Inodes:4005372 HasInodes:true} {Device:overlay_0-292 DeviceMajor:0 DeviceMinor:292 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/e3206ddf-35de-4b7c-b979-682f5f666a34/volumes/kubernetes.io~projected/bound-sa-token DeviceMajor:0 DeviceMinor:213 Capacity:31633408000 Type:vfs Inodes:4005372 HasInodes:true} {Device:overlay_0-270 DeviceMajor:0 DeviceMinor:270 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/dev/shm DeviceMajor:0 DeviceMinor:24 Capacity:16406003712 Type:vfs Inodes:4005372 HasInodes:true} {Device:/var/lib/kubelet/pods/08ae8ede-9a96-4a84-8ec8-8d364fd476a3/volume-subpaths/run-systemd/ovnkube-controller/6 DeviceMajor:0 DeviceMinor:26 Capacity:6562402304 Type:vfs Inodes:819200 HasInodes:true} {Device:/var/lib/kubelet/pods/87082646-75c3-43b5-864b-3478e9a7373d/volumes/kubernetes.io~secret/default-certificate DeviceMajor:0 DeviceMinor:204 Capacity:31633408000 Type:vfs Inodes:4005372 HasInodes:true} {Device:/var/lib/kubelet/pods/0e179af5-9f52-4941-b4f5-5e3b8789da11/volumes/kubernetes.io~secret/image-registry-private-configuration DeviceMajor:0 DeviceMinor:223 Capacity:31633408000 Type:vfs Inodes:4005372 HasInodes:true} {Device:/var/lib/kubelet/pods/0e179af5-9f52-4941-b4f5-5e3b8789da11/volumes/kubernetes.io~projected/bound-sa-token DeviceMajor:0 DeviceMinor:228 Capacity:31633408000 Type:vfs Inodes:4005372 HasInodes:true} {Device:/run/containers/storage/overlay-containers/b2829769d2e875408df83ae72236364dc323bc32fa4adc204d20ae79308ef283/userdata/shm DeviceMajor:0 DeviceMinor:47 Capacity:67108864 Type:vfs Inodes:4005372 HasInodes:true} {Device:/run/containers/storage/overlay-containers/929421d442561c4ba4793828037ab0e15ea8bdf99899887708c893190983a48b/userdata/shm DeviceMajor:0 DeviceMinor:80 Capacity:67108864 Type:vfs Inodes:4005372 HasInodes:true} {Device:/run/containers/storage/overlay-containers/1eeb30806320ba780c1de4b8401be73d6acefab9ba0c36688cdd52db42e2dead/userdata/shm DeviceMajor:0 DeviceMinor:85 Capacity:67108864 Type:vfs Inodes:4005372 HasInodes:true} {Device:/var/lib/kubelet/pods/4d213db4-73c5-4f50-9930-6d26b880dcf7/volumes/kubernetes.io~projected/kube-api-access-7xf2g DeviceMajor:0 DeviceMinor:214 Capacity:31633408000 Type:vfs Inodes:4005372 HasInodes:true} {Device:/run/containers/storage/overlay-containers/fc2505f6a57fe8bc893233a26b65fea1edffdc09612365750fd0d81c8a31f7d8/userdata/shm DeviceMajor:0 DeviceMinor:216 Capacity:67108864 Type:vfs Inodes:4005372 HasInodes:true} {Device:overlay_0-329 DeviceMajor:0 DeviceMinor:329 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/287ba08f-7b9f-4609-8e35-832c0879c44c/volumes/kubernetes.io~projected/kube-api-access-v7kvw DeviceMajor:0 DeviceMinor:65 Capacity:31633408000 Type:vfs Inodes:4005372 HasInodes:true} {Device:overlay_0-187 DeviceMajor:0 DeviceMinor:187 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/c16c0a80-c5ce-4f5d-827a-2e3eb2c3cfb8/volumes/kubernetes.io~secret/original-pull-secret DeviceMajor:0 DeviceMinor:286 Capacity:31633408000 Type:vfs Inodes:4005372 HasInodes:true} {Device:/var/lib/kubelet/pods/11db9cec-653f-4aee-8937-0cf5b367efac/volumes/kubernetes.io~projected/kube-api-access-5xgwq DeviceMajor:0 DeviceMinor:230 Capacity:31633408000 Type:vfs Inodes:4005372 HasInodes:true} {Device:overlay_0-100 DeviceMajor:0 DeviceMinor:100 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-119 DeviceMajor:0 DeviceMinor:119 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/2bd84dc4-e96a-43f1-85bf-e1ea4005c193/volumes/kubernetes.io~secret/serving-cert DeviceMajor:0 DeviceMinor:203 Capacity:31633408000 Type:vfs Inodes:4005372 HasInodes:true} {Device:/var/lib/kubelet/pods/e3206ddf-35de-4b7c-b979-682f5f666a34/volumes/kubernetes.io~secret/installation-pull-secrets DeviceMajor:0 DeviceMinor:205 Capacity:31633408000 Type:vfs Inodes:4005372 HasInodes:true} {Device:/run/containers/storage/overlay-containers/e4dd71d689867fb5f56c774748d9b5a5f2bedffc837d9d7c0ad1723524d414ca/userdata/shm DeviceMajor:0 DeviceMinor:208 Capacity:67108864 Type:vfs Inodes:4005372 HasInodes:true} {Device:/var/lib/kubelet/pods/fade00ba-24a7-41c9-93f7-23e40971874a/volumes/kubernetes.io~projected/kube-api-access-q667z DeviceMajor:0 DeviceMinor:238 Capacity:31633408000 Type:vfs Inodes:4005372 HasInodes:true} {Device:overlay_0-70 DeviceMajor:0 DeviceMinor:70 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/39bb8c16-8282-442a-841b-0056baae014d/volumes/kubernetes.io~projected/kube-api-access-gdq6k DeviceMajor:0 DeviceMinor:63 Capacity:31633408000 Type:vfs Inodes:4005372 HasInodes:true} {Device:overlay_0-102 DeviceMajor:0 DeviceMinor:102 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/87082646-75c3-43b5-864b-3478e9a7373d/volumes/kubernetes.io~secret/stats-auth DeviceMajor:0 DeviceMinor:207 Capacity:31633408000 Type:vfs Inodes:4005372 HasInodes:true} {Device:/var/lib/kubelet/pods/f9a06263-cb99-4004-bffc-581d66ce5ebf/volumes/kubernetes.io~projected/kube-api-access-qfw7m DeviceMajor:0 DeviceMinor:212 Capacity:31633408000 Type:vfs Inodes:4005372 HasInodes:true} {Device:/var/lib/kubelet/pods/af36f4d4-f985-4937-a9f7-9884936f3690/volumes/kubernetes.io~secret/agent-certs DeviceMajor:0 DeviceMinor:57 Capacity:31633408000 Type:vfs Inodes:4005372 HasInodes:true} {Device:overlay_0-82 DeviceMajor:0 DeviceMinor:82 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-127 DeviceMajor:0 DeviceMinor:127 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-158 DeviceMajor:0 DeviceMinor:158 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/87082646-75c3-43b5-864b-3478e9a7373d/volumes/kubernetes.io~projected/kube-api-access-8s7g5 DeviceMajor:0 DeviceMinor:217 Capacity:31633408000 Type:vfs Inodes:4005372 HasInodes:true} {Device:overlay_0-252 DeviceMajor:0 DeviceMinor:252 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-306 DeviceMajor:0 DeviceMinor:306 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/87da348d-b1ee-4cd3-806d-0560bff3c6f2/volumes/kubernetes.io~projected/kube-api-access-txbm4 DeviceMajor:0 DeviceMinor:45 Capacity:31633408000 Type:vfs Inodes:4005372 HasInodes:true} {Device:/run/containers/storage/overlay-containers/9e89eaa98efd97ac57e6419b178d76a3452fdc3097e597ea17f038f98d99832e/userdata/shm DeviceMajor:0 DeviceMinor:225 Capacity:67108864 Type:vfs Inodes:4005372 HasInodes:true} {Device:/run/containers/storage/overlay-containers/abc2efb0ba45504bce33f1ce80ee88f9d7e4fc489dd1c5d851c51f8ed53b6b0b/userdata/shm DeviceMajor:0 DeviceMinor:290 Capacity:67108864 Type:vfs Inodes:4005372 HasInodes:true} {Device:/var/lib/kubelet/pods/39bb8c16-8282-442a-841b-0056baae014d/volumes/kubernetes.io~empty-dir/etc-tuned DeviceMajor:0 DeviceMinor:55 Capacity:31633408000 Type:vfs Inodes:4005372 HasInodes:true} {Device:/run/containers/storage/overlay-containers/0c376387c3362ef564a7120cabcf536b058ed3e285e143402927edceb193291f/userdata/shm DeviceMajor:0 DeviceMinor:84 Capacity:67108864 Type:vfs Inodes:4005372 HasInodes:true} {Device:/var/lib/kubelet/pods/e3206ddf-35de-4b7c-b979-682f5f666a34/volumes/kubernetes.io~projected/kube-api-access-w9h4f DeviceMajor:0 DeviceMinor:215 Capacity:31633408000 Type:vfs Inodes:4005372 HasInodes:true} {Device:overlay_0-254 DeviceMajor:0 DeviceMinor:254 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-304 DeviceMajor:0 DeviceMinor:304 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/39bb8c16-8282-442a-841b-0056baae014d/volumes/kubernetes.io~empty-dir/tmp DeviceMajor:0 DeviceMinor:56 Capacity:31633408000 Type:vfs Inodes:4005372 HasInodes:true} {Device:overlay_0-96 DeviceMajor:0 DeviceMinor:96 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/5ef42db8ca33f0169c41836d553c454c6d250a87f4246c19237b64f69a39ef34/userdata/shm DeviceMajor:0 DeviceMinor:98 Capacity:67108864 Type:vfs Inodes:4005372 HasInodes:true} {Device:/var/lib/kubelet/pods/0e179af5-9f52-4941-b4f5-5e3b8789da11/volumes/kubernetes.io~secret/installation-pull-secrets DeviceMajor:0 DeviceMinor:224 Capacity:31633408000 Type:vfs Inodes:4005372 HasInodes:true} {Device:/run/containers/storage/overlay-containers/2edc71ff426362ce04e9fc22239cdc12c104ebc68abb45589d168aa8e283eba0/userdata/shm DeviceMajor:0 DeviceMinor:236 Capacity:67108864 Type:vfs Inodes:4005372 HasInodes:true} {Device:overlay_0-276 DeviceMajor:0 DeviceMinor:276 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/3bea07ac8a9e43055344e71268ef13eda8a7b0eb15593518b8076be757e09c3a/userdata/shm DeviceMajor:0 DeviceMinor:48 Capacity:67108864 Type:vfs Inodes:4005372 HasInodes:true} {Device:/run/containers/storage/overlay-containers/32595290e48470808e0ab2db048e03df52c1e31b97a08ef92d7a4c00da701711/userdata/shm DeviceMajor:0 DeviceMinor:76 Capacity:67108864 Type:vfs Inodes:4005372 HasInodes:true} {Device:overlay_0-129 DeviceMajor:0 DeviceMinor:129 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/0e179af5-9f52-4941-b4f5-5e3b8789da11/volumes/kubernetes.io~projected/kube-api-access-zj4df DeviceMajor:0 DeviceMinor:227 Capacity:31633408000 Type:vfs Inodes:4005372 HasInodes:true} {Device:/run/containers/storage/overlay-containers/a203f34f66a5ec4d9297e5f8c1bde08af3007fa46ef54906494feccb340e9a27/userdata/shm DeviceMajor:0 DeviceMinor:244 Capacity:67108864 Type:vfs Inodes:4005372 HasInodes:true} {Device:overlay_0-294 DeviceMajor:0 DeviceMinor:294 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-192 DeviceMajor:0 DeviceMinor:192 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/ba2c5571-23a3-46b6-a36d-7e8810a66261/volumes/kubernetes.io~projected/kube-api-access-k62g9 DeviceMajor:0 DeviceMinor:197 Capacity:31633408000 Type:vfs Inodes:4005372 HasInodes:true} {Device:/var/lib/kubelet/pods/e3206ddf-35de-4b7c-b979-682f5f666a34/volumes/kubernetes.io~secret/image-registry-private-configuration DeviceMajor:0 DeviceMinor:202 Capacity:31633408000 Type:vfs Inodes:4005372 HasInodes:true} {Device:/var/lib/kubelet/pods/4ba601a6-2a44-4408-8c5b-2c29fd096fed/volumes/kubernetes.io~projected/kube-api-access-rfxzq DeviceMajor:0 DeviceMinor:229 Capacity:31633408000 Type:vfs Inodes:4005372 HasInodes:true} {Device:/run/containers/storage/overlay-containers/f2f64b91fb83a7b6abf1cb091c941acd65de81bf8df0800107508e4e11241e8c/userdata/shm DeviceMajor:0 DeviceMinor:233 Capacity:67108864 Type:vfs Inodes:4005372 HasInodes:true} {Device:overlay_0-111 DeviceMajor:0 DeviceMinor:111 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-124 DeviceMajor:0 DeviceMinor:124 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-152 DeviceMajor:0 DeviceMinor:152 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/cdb750aa-dd7e-4e2f-8297-9028acc0f559/volumes/kubernetes.io~projected/kube-api-access-rvrch DeviceMajor:0 DeviceMinor:59 Capacity:31633408000 Type:vfs Inodes:4005372 HasInodes:true} {Device:overlay_0-250 DeviceMajor:0 DeviceMinor:250 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-256 DeviceMajor:0 DeviceMinor:256 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/dev/nvme0n1p3 DeviceMajor:259 DeviceMinor:3 Capacity:366869504 Type:vfs Inodes:98304 HasInodes:true} {Device:/var/lib/kubelet/pods/08ae8ede-9a96-4a84-8ec8-8d364fd476a3/volumes/kubernetes.io~secret/ovn-node-metrics-cert DeviceMajor:0 DeviceMinor:58 Capacity:31633408000 Type:vfs Inodes:4005372 HasInodes:true} {Device:overlay_0-157 DeviceMajor:0 DeviceMinor:157 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/97f8a11334a257518f5e40c7050de6a55a85bab3727386b10b4f04f52c8f0bcd/userdata/shm DeviceMajor:0 DeviceMinor:246 Capacity:67108864 Type:vfs Inodes:4005372 HasInodes:true} {Device:/run/containers/storage/overlay-containers/9a980ae6fdff8b6bb898b3b37c651c25cdbecbdcdeacc75bf47c59da9b0302a0/userdata/shm DeviceMajor:0 DeviceMinor:219 Capacity:67108864 Type:vfs Inodes:4005372 HasInodes:true} {Device:overlay_0-296 DeviceMajor:0 DeviceMinor:296 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-106 DeviceMajor:0 DeviceMinor:106 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-88 DeviceMajor:0 DeviceMinor:88 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/71870ffe35cbe4432b34eca4162a3fe61340fcd0c1229a11bee6c775daae713a/userdata/shm DeviceMajor:0 DeviceMinor:94 Capacity:67108864 Type:vfs Inodes:4005372 HasInodes:true} {Device:/var/lib/kubelet/pods/4ca01b49-9819-4a90-a851-7553ad568679/volumes/kubernetes.io~projected/kube-api-access-bzw4s DeviceMajor:0 DeviceMinor:289 Capacity:31633408000 Type:vfs Inodes:4005372 HasInodes:true} {Device:overlay_0-274 DeviceMajor:0 DeviceMinor:274 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-265 DeviceMajor:0 DeviceMinor:265 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run DeviceMajor:0 DeviceMinor:26 Capacity:6562402304 Type:vfs Inodes:819200 HasInodes:true} {Device:/var/lib/kubelet/pods/48386105-d79b-4f92-baa0-f330b9658ad0/volumes/kubernetes.io~projected/kube-api-access-w8zg9 DeviceMajor:0 DeviceMinor:67 Capacity:31633408000 Type:vfs Inodes:4005372 HasInodes:true} {Device:/var/lib/kubelet/pods/f86c0e38-8144-42fc-b622-25602ada113f/volumes/kubernetes.io~secret/cert DeviceMajor:0 DeviceMinor:235 Capacity:31633408000 Type:vfs Inodes:4005372 HasInodes:true} {Device:/var/lib/kubelet/pods/33ce6ceb-02a9-4852-9948-f02e2b39e32f/volumes/kubernetes.io~projected/kube-api-access-btd8n DeviceMajor:0 DeviceMinor:239 Capacity:31633408000 Type:vfs Inodes:4005372 HasInodes:true} {Device:overlay_0-277 DeviceMajor:0 DeviceMinor:277 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/08ae8ede-9a96-4a84-8ec8-8d364fd476a3/volumes/kubernetes.io~projected/kube-api-access-bnxm6 DeviceMajor:0 DeviceMinor:62 Capacity:31633408000 Type:vfs Inodes:4005372 HasInodes:true} {Device:/var/lib/kubelet/pods/4ba601a6-2a44-4408-8c5b-2c29fd096fed/volumes/kubernetes.io~secret/serving-cert DeviceMajor:0 DeviceMinor:221 Capacity:31633408000 Type:vfs Inodes:4005372 HasInodes:true} {Device:/var/lib/kubelet/pods/11db9cec-653f-4aee-8937-0cf5b367efac/volumes/kubernetes.io~secret/serving-cert DeviceMajor:0 DeviceMinor:222 Capacity:31633408000 Type:vfs Inodes:4005372 HasInodes:true} {Device:/var/lib/kubelet/pods/e446ca63-5885-4c66-9cb7-da39afbf742c/volumes/kubernetes.io~projected/kube-api-access-j7bln DeviceMajor:0 DeviceMinor:241 Capacity:31633408000 Type:vfs Inodes:4005372 HasInodes:true} {Device:overlay_0-138 DeviceMajor:0 DeviceMinor:138 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-142 DeviceMajor:0 DeviceMinor:142 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/dccdaf98-aac7-4756-b93f-22f62a286a1f/volumes/kubernetes.io~projected/kube-api-access-6bqfr DeviceMajor:0 DeviceMinor:64 Capacity:31633408000 Type:vfs Inodes:4005372 HasInodes:true} {Device:/dev/nvme0n1p4 DeviceMajor:259 DeviceMinor:4 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/c3780b96-955e-4728-a2d0-b7ec30d0f6f3/volumes/kubernetes.io~projected/kube-api-access-cdkvt DeviceMajor:0 DeviceMinor:61 Capacity:31633408000 Type:vfs Inodes:4005372 HasInodes:true} {Device:overlay_0-177 DeviceMajor:0 DeviceMinor:177 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-182 DeviceMajor:0 DeviceMinor:182 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/bab0abf9-46d4-408e-9aa0-afa7279cd9d1/volumes/kubernetes.io~projected/kube-api-access-4rcdn DeviceMajor:0 DeviceMinor:66 Capacity:31633408000 Type:vfs Inodes:4005372 HasInodes:true} {Device:overlay_0-92 DeviceMajor:0 DeviceMinor:92 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/d260d4d7-b4aa-4f37-92b5-e75b22e13a3b/volumes/kubernetes.io~projected/kube-api-access-wmkq8 DeviceMajor:0 DeviceMinor:201 Capacity:31633408000 Type:vfs Inodes:4005372 HasInodes:true} {Device:/var/lib/kubelet/pods/2bd84dc4-e96a-43f1-85bf-e1ea4005c193/volumes/kubernetes.io~projected/kube-api-access-4rmts DeviceMajor:0 DeviceMinor:210 Capacity:31633408000 Type:vfs Inodes:4005372 HasInodes:true} {Device:/var/lib/kubelet/pods/56274b23-2400-43d4-9686-c928e20d47de/volumes/kubernetes.io~projected/kube-api-access-wt5rq DeviceMajor:0 DeviceMinor:211 Capacity:31633408000 Type:vfs Inodes:4005372 HasInodes:true} {Device:overlay_0-248 DeviceMajor:0 DeviceMinor:248 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-284 DeviceMajor:0 DeviceMinor:284 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-298 DeviceMajor:0 DeviceMinor:298 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/1f9b62b0486573df560be923ea7c2199b7827bf627151bdffb9652a48a0aa8b6/userdata/shm DeviceMajor:0 DeviceMinor:242 Capacity:67108864 Type:vfs Inodes:4005372 HasInodes:true} {Device:overlay_0-308 DeviceMajor:0 DeviceMinor:308 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-327 DeviceMajor:0 DeviceMinor:327 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-335 DeviceMajor:0 DeviceMinor:335 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/e0221987-8e5e-40d1-8842-c97993f08c6d/volumes/kubernetes.io~projected/kube-api-access-fmnd4 DeviceMajor:0 DeviceMinor:60 Capacity:31633408000 Type:vfs Inodes:4005372 HasInodes:true} {Device:/run/containers/storage/overlay-containers/4f8ff29fa81054ed5a37cd86a5ec74649a5651dc5a63c441321ba062767aa578/userdata/shm DeviceMajor:0 DeviceMinor:68 Capacity:67108864 Type:vfs Inodes:4005372 HasInodes:true} {Device:/var/lib/kubelet/pods/f86c0e38-8144-42fc-b622-25602ada113f/volumes/kubernetes.io~projected/kube-api-access-j4wnm DeviceMajor:0 DeviceMinor:240 Capacity:31633408000 Type:vfs Inodes:4005372 HasInodes:true} {Device:/var/lib/kubelet/pods/fc4ac297-f510-4439-80e2-487056d3bf06/volumes/kubernetes.io~projected/kube-api-access-7d4bn DeviceMajor:0 DeviceMinor:318 Capacity:31633408000 Type:vfs Inodes:4005372 HasInodes:true} {Device:overlay_0-51 DeviceMajor:0 DeviceMinor:51 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-53 DeviceMajor:0 DeviceMinor:53 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/f9a06263-cb99-4004-bffc-581d66ce5ebf/volumes/kubernetes.io~secret/serving-cert DeviceMajor:0 DeviceMinor:206 Capacity:31633408000 Type:vfs Inodes:4005372 HasInodes:true} {Device:composefs_0-34 DeviceMajor:0 DeviceMinor:34 Capacity:6422528 Type:vfs Inodes:18446744073709551615 HasInodes:true} {Device:overlay_0-146 DeviceMajor:0 DeviceMinor:146 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-272 DeviceMajor:0 DeviceMinor:272 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true}] DiskMap:map[259:0:{Name:nvme0n1 Major:259 Minor:0 Size:128849018880 Scheduler:none}] NetworkDevices:[{Name:1f9b62b0486573d MacAddress:f6:b4:b6:9e:4f:d1 Speed:10000 Mtu:8901} {Name:238545c7c767539 MacAddress:76:64:cf:c2:b5:f0 Speed:10000 Mtu:8901} {Name:2edc71ff426362c MacAddress:26:16:5b:52:1f:bf Speed:10000 Mtu:8901} {Name:97f8a11334a2575 MacAddress:d2:14:b4:19:99:f0 Speed:10000 Mtu:8901} {Name:9a980ae6fdff8b6 MacAddress:96:f4:b5:78:d8:08 Speed:10000 Mtu:8901} {Name:9e89eaa98efd97a MacAddress:d6:41:91:52:82:45 Speed:10000 Mtu:8901} {Name:a203f34f66a5ec4 MacAddress:e2:71:27:e6:31:75 Speed:10000 Mtu:8901} {Name:abc2efb0ba45504 MacAddress:aa:a0:4b:a4:85:01 Speed:10000 Mtu:8901} {Name:br-ex MacAddress:02:58:ed:71:9c:8b Speed:0 Mtu:9001} {Name:br-int MacAddress:0e:72:36:72:d7:2a Speed:0 Mtu:8901} {Name:e2d4de650f3571a MacAddress:92:63:79:e7:3d:c3 Speed:10000 Mtu:8901} {Name:e4dd71d689867fb MacAddress:46:38:99:43:9a:90 Speed:10000 Mtu:8901} {Name:ens5 MacAddress:02:58:ed:71:9c:8b Speed:0 Mtu:9001} {Name:f2f64b91fb83a7b MacAddress:32:32:9e:d4:97:5b Speed:10000 Mtu:8901} {Name:fc2505f6a57fe8b MacAddress:2e:56:3b:38:ac:a6 Speed:10000 Mtu:8901} {Name:genev_sys_6081 MacAddress:a2:45:c9:13:b3:77 Speed:0 Mtu:65000} {Name:ovn-k8s-mp0 MacAddress:0a:58:0a:84:00:02 Speed:0 Mtu:8901} {Name:ovs-system MacAddress:5e:c8:09:01:9f:08 Speed:0 Mtu:1500}] Topology:[{Id:0 Memory:32812007424 HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] Cores:[{Id:0 Threads:[0 4] Caches:[{Id:0 Size:32768 Type:Data Level:1} {Id:0 Size:32768 Type:Instruction Level:1} {Id:0 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:1 Threads:[1 5] Caches:[{Id:1 Size:32768 Type:Data Level:1} {Id:1 Size:32768 Type:Instruction Level:1} {Id:1 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:2 Threads:[2 6] Caches:[{Id:2 Size:32768 Type:Data Level:1} {Id:2 Size:32768 Type:Instruction Level:1} {Id:2 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:3 Threads:[3 7] Caches:[{Id:3 Size:32768 Type:Data Level:1} {Id:3 Size:32768 Type:Instruction Level:1} {Id:3 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:}] Caches:[{Id:0 Size:34603008 Type:Unified Level:3}] Distances:[10]}] CloudProvider:Unknown InstanceType:Unknown InstanceID:None}
Oct 30 19:15:52.520862 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:15:52.519762    4892 manager_no_libpfm.go:29] cAdvisor is build without cgo and/or libpfm support. Perf event counters are not available.
Oct 30 19:15:52.520862 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:15:52.519801    4892 manager.go:233] Version: {KernelVersion:5.14.0-687.60.1.el9_8.x86_64 ContainerOsVersion:Red Hat Enterprise Linux CoreOS 9.8.20261023-0 (Plow) DockerVersion: DockerAPIVersion: CadvisorVersion: CadvisorRevision:}
Oct 30 19:15:52.520862 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:15:52.520089    4892 container_manager_linux.go:275] "Container manager verified user specified cgroup-root exists" cgroupRoot=[]
Oct 30 19:15:52.520862 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:15:52.520109    4892 container_manager_linux.go:280] "Creating Container Manager object based on Node Config" nodeConfig={"NodeName":"ip-10-0-132-82.us-west-1.compute.internal","RuntimeCgroupsName":"/system.slice/crio.service","SystemCgroupsName":"/system.slice","KubeletCgroupsName":"","KubeletOOMScoreAdj":-999,"ContainerRuntime":"","CgroupsPerQOS":true,"CgroupRoot":"/","CgroupDriver":"systemd","KubeletRootDir":"/var/lib/kubelet","ProtectKernelDefaults":true,"KubeReservedCgroupName":"","SystemReservedCgroupName":"/system.slice","ReservedSystemCPUs":{},"EnforceNodeAllocatable":{"pods":{},"system-reserved-compressible":{}},"KubeReserved":null,"SystemReserved":{"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"},"HardEvictionThresholds":[{"Signal":"imagefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.15},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"memory.available","Operator":"LessThan","Value":{"Quantity":"100Mi","Percentage":0},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.1},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null}],"QOSReserved":{},"CPUManagerPolicy":"none","CPUManagerPolicyOptions":null,"TopologyManagerScope":"container","CPUManagerReconcilePeriod":10000000000,"MemoryManagerPolicy":"None","MemoryManagerReservedMemory":null,"PodPidsLimit":4096,"EnforceCPULimits":true,"CPUCFSQuotaPeriod":100000000,"TopologyManagerPolicy":"none","TopologyManagerPolicyOptions":null,"CgroupVersion":2}
Oct 30 19:15:52.520862 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:15:52.520241    4892 topology_manager.go:143] "Creating topology manager with none policy"
Oct 30 19:15:52.520862 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:15:52.520249    4892 container_manager_linux.go:311] "Creating device plugin manager"
Oct 30 19:15:52.520862 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:15:52.520262    4892 manager.go:144] "Creating Device Plugin manager" path="/var/lib/kubelet/device-plugins/kubelet.sock"
Oct 30 19:15:52.520862 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:15:52.520280    4892 server.go:74] "Creating device plugin registration server" version="v1beta1" socket="/var/lib/kubelet/device-plugins/kubelet.sock"
Oct 30 19:15:52.520862 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:15:52.520310    4892 container_manager_linux.go:320] "Creating Dynamic Resource Allocation (DRA) manager"
Oct 30 19:15:52.520862 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:15:52.520480    4892 state_mem.go:41] "Initialized" logger="CPUManager state memory"
Oct 30 19:15:52.520862 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:15:52.520586    4892 server.go:1235] "Using root directory" path="/var/lib/kubelet"
Oct 30 19:15:52.520862 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:15:52.520628    4892 kubelet.go:493] "Attempting to sync node with API server"
Oct 30 19:15:52.520862 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:15:52.520641    4892 kubelet.go:394] "Adding static pod path" path="/etc/kubernetes/manifests"
Oct 30 19:15:52.520862 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:15:52.520654    4892 file.go:69] "Watching path" path="/etc/kubernetes/manifests"
Oct 30 19:15:52.520862 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:15:52.520666    4892 kubelet.go:405] "Adding apiserver pod source"
Oct 30 19:15:52.520862 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:15:52.520678    4892 apiserver.go:42] "Waiting for node sync before watching apiserver pods"
Oct 30 19:15:52.521722 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:15:52.521641    4892 state_checkpoint.go:81] "State checkpoint: restored pod resource state from checkpoint"
Oct 30 19:15:52.521722 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:15:52.521660    4892 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking"
Oct 30 19:15:52.522648 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:15:52.522620    4892 kuberuntime_manager.go:304] "Container runtime initialized" containerRuntime="cri-o" version="1.35.10-3.rhaos4.22.git9383502.el9" apiVersion="v1"
Oct 30 19:15:52.526165 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:15:52.526150    4892 state_checkpoint.go:81] "State checkpoint: restored pod resource state from checkpoint"
Oct 30 19:15:52.526165 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:15:52.526163    4892 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking"
Oct 30 19:15:52.526286 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:15:52.526276    4892 certificate_store.go:147] "Loading cert/key pair from a file" filePath="/var/lib/kubelet/pki/kubelet-server-current.pem"
Oct 30 19:15:52.526472 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:15:52.526461    4892 kubelet.go:963] "Not starting ClusterTrustBundle informer because we are in static kubelet mode or the ClusterTrustBundleProjection featuregate is disabled"
Oct 30 19:15:52.526504 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:15:52.526491    4892 kubelet.go:990] "Not starting PodCertificateRequest manager because we are in static kubelet mode or the PodCertificateProjection feature gate is disabled"
Oct 30 19:15:52.526602 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:15:52.526592    4892 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/portworx-volume"
Oct 30 19:15:52.526631 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:15:52.526608    4892 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/empty-dir"
Oct 30 19:15:52.526631 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:15:52.526615    4892 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/git-repo"
Oct 30 19:15:52.526631 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:15:52.526620    4892 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/host-path"
Oct 30 19:15:52.526631 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:15:52.526628    4892 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/nfs"
Oct 30 19:15:52.526730 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:15:52.526637    4892 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/secret"
Oct 30 19:15:52.526730 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:15:52.526648    4892 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/iscsi"
Oct 30 19:15:52.526730 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:15:52.526655    4892 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/downward-api"
Oct 30 19:15:52.526730 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:15:52.526662    4892 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/fc"
Oct 30 19:15:52.526730 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:15:52.526667    4892 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/configmap"
Oct 30 19:15:52.526730 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:15:52.526676    4892 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/projected"
Oct 30 19:15:52.526730 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:15:52.526686    4892 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/local-volume"
Oct 30 19:15:52.526730 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:15:52.526711    4892 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/csi"
Oct 30 19:15:52.526730 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:15:52.526720    4892 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/image"
Oct 30 19:15:52.530086 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:15:52.530073    4892 server.go:1263] "Started kubelet"
Oct 30 19:15:52.530454 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:15:52.530201    4892 server.go:182] "Starting to listen" address="0.0.0.0" port=10250
Oct 30 19:15:52.530454 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:15:52.530250    4892 ratelimit.go:56] "Setting rate limiting for endpoint" service="podresources" qps=100 burstTokens=10
Oct 30 19:15:52.530454 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:15:52.530319    4892 server_v1.go:49] "podresources" method="list" useActivePods=true
Oct 30 19:15:52.530673 ip-10-0-132-82 systemd[1]: Started Kubernetes Kubelet.
Oct 30 19:15:52.530822 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:15:52.530744    4892 server.go:254] "Starting to serve the podresources API" endpoint="unix:/var/lib/kubelet/pod-resources/kubelet.sock"
Oct 30 19:15:52.532844 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:15:52.532816    4892 server.go:317] "Adding debug handlers to kubelet server"
Oct 30 19:15:52.533941 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:15:52.533924    4892 fs_resource_analyzer.go:69] "Starting FS ResourceAnalyzer"
Oct 30 19:15:52.534088 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:15:52.534070    4892 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kubelet-serving"
Oct 30 19:15:52.536457 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:15:52.534123    4892 volume_manager.go:309] "The desired_state_of_world populator starts"
Oct 30 19:15:52.536543 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:15:52.536461    4892 volume_manager.go:311] "Starting Kubelet Volume Manager"
Oct 30 19:15:52.536593 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:15:52.536537    4892 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-10-29 19:10:12 +0000 UTC" deadline="2028-05-24 11:39:46.768436884 +0000 UTC"
Oct 30 19:15:52.536593 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:15:52.536573    4892 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="13720h23m54.231868104s"
Oct 30 19:15:52.536593 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:15:52.535789    4892 factory.go:55] Registering systemd factory
Oct 30 19:15:52.536593 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:15:52.536593    4892 factory.go:223] Registration of the systemd container factory successfully
Oct 30 19:15:52.536789 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:15:52.534136    4892 desired_state_of_world_populator.go:146] "Desired state populator starts to run"
Oct 30 19:15:52.536789 ip-10-0-132-82 kubenswrapper[4892]: E1030 19:15:52.534440    4892 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-132-82.us-west-1.compute.internal\" not found"
Oct 30 19:15:52.537632 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:15:52.537092    4892 factory.go:153] Registering CRI-O factory
Oct 30 19:15:52.537632 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:15:52.537110    4892 factory.go:223] Registration of the crio container factory successfully
Oct 30 19:15:52.537632 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:15:52.537175    4892 factory.go:221] Registration of the containerd container factory failed: unable to create containerd client: containerd: cannot unix dial containerd api service: dial unix /run/containerd/containerd.sock: connect: no such file or directory
Oct 30 19:15:52.537632 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:15:52.537195    4892 factory.go:103] Registering Raw factory
Oct 30 19:15:52.537632 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:15:52.537211    4892 manager.go:1199] Started watching for new ooms in manager
Oct 30 19:15:52.538642 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:15:52.538622    4892 manager.go:319] Starting recovery of all containers
Oct 30 19:15:52.546566 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:15:52.540900    4892 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="08ae8ede-9a96-4a84-8ec8-8d364fd476a3" volumeName="kubernetes.io/configmap/08ae8ede-9a96-4a84-8ec8-8d364fd476a3-env-overrides" seLinuxMountContext=""
Oct 30 19:15:52.546566 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:15:52.540949    4892 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="87082646-75c3-43b5-864b-3478e9a7373d" volumeName="kubernetes.io/secret/87082646-75c3-43b5-864b-3478e9a7373d-stats-auth" seLinuxMountContext=""
Oct 30 19:15:52.546566 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:15:52.540982    4892 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="ba2c5571-23a3-46b6-a36d-7e8810a66261" volumeName="kubernetes.io/projected/ba2c5571-23a3-46b6-a36d-7e8810a66261-kube-api-access-k62g9" seLinuxMountContext=""
Oct 30 19:15:52.546566 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:15:52.541001    4892 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="0e179af5-9f52-4941-b4f5-5e3b8789da11" volumeName="kubernetes.io/secret/0e179af5-9f52-4941-b4f5-5e3b8789da11-image-registry-private-configuration" seLinuxMountContext=""
Oct 30 19:15:52.546566 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:15:52.541017    4892 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="2c7c7060-def6-4737-8fa9-6889309b7463" volumeName="kubernetes.io/configmap/2c7c7060-def6-4737-8fa9-6889309b7463-nginx-conf" seLinuxMountContext=""
Oct 30 19:15:52.546566 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:15:52.541029    4892 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="48386105-d79b-4f92-baa0-f330b9658ad0" volumeName="kubernetes.io/empty-dir/48386105-d79b-4f92-baa0-f330b9658ad0-tmp-dir" seLinuxMountContext=""
Oct 30 19:15:52.546566 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:15:52.541047    4892 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="cdb750aa-dd7e-4e2f-8297-9028acc0f559" volumeName="kubernetes.io/configmap/cdb750aa-dd7e-4e2f-8297-9028acc0f559-serviceca" seLinuxMountContext=""
Oct 30 19:15:52.546566 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:15:52.541059    4892 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="dccdaf98-aac7-4756-b93f-22f62a286a1f" volumeName="kubernetes.io/configmap/dccdaf98-aac7-4756-b93f-22f62a286a1f-cni-binary-copy" seLinuxMountContext=""
Oct 30 19:15:52.546566 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:15:52.541080    4892 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="e3206ddf-35de-4b7c-b979-682f5f666a34" volumeName="kubernetes.io/configmap/e3206ddf-35de-4b7c-b979-682f5f666a34-registry-certificates" seLinuxMountContext=""
Oct 30 19:15:52.546566 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:15:52.541092    4892 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="e3206ddf-35de-4b7c-b979-682f5f666a34" volumeName="kubernetes.io/projected/e3206ddf-35de-4b7c-b979-682f5f666a34-bound-sa-token" seLinuxMountContext=""
Oct 30 19:15:52.546566 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:15:52.541106    4892 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="f9a06263-cb99-4004-bffc-581d66ce5ebf" volumeName="kubernetes.io/secret/f9a06263-cb99-4004-bffc-581d66ce5ebf-serving-cert" seLinuxMountContext=""
Oct 30 19:15:52.546566 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:15:52.541125    4892 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="0e179af5-9f52-4941-b4f5-5e3b8789da11" volumeName="kubernetes.io/configmap/0e179af5-9f52-4941-b4f5-5e3b8789da11-trusted-ca" seLinuxMountContext=""
Oct 30 19:15:52.546566 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:15:52.541137    4892 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="0e179af5-9f52-4941-b4f5-5e3b8789da11" volumeName="kubernetes.io/configmap/0e179af5-9f52-4941-b4f5-5e3b8789da11-registry-certificates" seLinuxMountContext=""
Oct 30 19:15:52.546566 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:15:52.541155    4892 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="bab0abf9-46d4-408e-9aa0-afa7279cd9d1" volumeName="kubernetes.io/projected/bab0abf9-46d4-408e-9aa0-afa7279cd9d1-kube-api-access-4rcdn" seLinuxMountContext=""
Oct 30 19:15:52.546566 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:15:52.541173    4892 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="08ae8ede-9a96-4a84-8ec8-8d364fd476a3" volumeName="kubernetes.io/configmap/08ae8ede-9a96-4a84-8ec8-8d364fd476a3-ovnkube-script-lib" seLinuxMountContext=""
Oct 30 19:15:52.546566 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:15:52.541190    4892 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="287ba08f-7b9f-4609-8e35-832c0879c44c" volumeName="kubernetes.io/projected/287ba08f-7b9f-4609-8e35-832c0879c44c-kube-api-access-v7kvw" seLinuxMountContext=""
Oct 30 19:15:52.546566 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:15:52.541201    4892 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="c16c0a80-c5ce-4f5d-827a-2e3eb2c3cfb8" volumeName="kubernetes.io/secret/c16c0a80-c5ce-4f5d-827a-2e3eb2c3cfb8-original-pull-secret" seLinuxMountContext=""
Oct 30 19:15:52.546566 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:15:52.541217    4892 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="e3206ddf-35de-4b7c-b979-682f5f666a34" volumeName="kubernetes.io/configmap/e3206ddf-35de-4b7c-b979-682f5f666a34-trusted-ca" seLinuxMountContext=""
Oct 30 19:15:52.546566 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:15:52.541230    4892 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="fade00ba-24a7-41c9-93f7-23e40971874a" volumeName="kubernetes.io/configmap/fade00ba-24a7-41c9-93f7-23e40971874a-config-volume" seLinuxMountContext=""
Oct 30 19:15:52.546566 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:15:52.541243    4892 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="11db9cec-653f-4aee-8937-0cf5b367efac" volumeName="kubernetes.io/secret/11db9cec-653f-4aee-8937-0cf5b367efac-serving-cert" seLinuxMountContext=""
Oct 30 19:15:52.546566 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:15:52.541261    4892 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="48386105-d79b-4f92-baa0-f330b9658ad0" volumeName="kubernetes.io/projected/48386105-d79b-4f92-baa0-f330b9658ad0-kube-api-access-w8zg9" seLinuxMountContext=""
Oct 30 19:15:52.546566 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:15:52.541273    4892 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="4ba601a6-2a44-4408-8c5b-2c29fd096fed" volumeName="kubernetes.io/configmap/4ba601a6-2a44-4408-8c5b-2c29fd096fed-config" seLinuxMountContext=""
Oct 30 19:15:52.546566 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:15:52.541290    4892 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="56274b23-2400-43d4-9686-c928e20d47de" volumeName="kubernetes.io/configmap/56274b23-2400-43d4-9686-c928e20d47de-samples-operator-config" seLinuxMountContext=""
Oct 30 19:15:52.546566 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:15:52.541303    4892 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="f86c0e38-8144-42fc-b622-25602ada113f" volumeName="kubernetes.io/secret/f86c0e38-8144-42fc-b622-25602ada113f-cert" seLinuxMountContext=""
Oct 30 19:15:52.546566 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:15:52.541315    4892 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="fc4ac297-f510-4439-80e2-487056d3bf06" volumeName="kubernetes.io/projected/fc4ac297-f510-4439-80e2-487056d3bf06-kube-api-access-7d4bn" seLinuxMountContext=""
Oct 30 19:15:52.546566 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:15:52.541332    4892 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="0e179af5-9f52-4941-b4f5-5e3b8789da11" volumeName="kubernetes.io/secret/0e179af5-9f52-4941-b4f5-5e3b8789da11-installation-pull-secrets" seLinuxMountContext=""
Oct 30 19:15:52.546566 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:15:52.541350    4892 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="87082646-75c3-43b5-864b-3478e9a7373d" volumeName="kubernetes.io/projected/87082646-75c3-43b5-864b-3478e9a7373d-kube-api-access-8s7g5" seLinuxMountContext=""
Oct 30 19:15:52.546566 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:15:52.541367    4892 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="e446ca63-5885-4c66-9cb7-da39afbf742c" volumeName="kubernetes.io/projected/e446ca63-5885-4c66-9cb7-da39afbf742c-kube-api-access-j7bln" seLinuxMountContext=""
Oct 30 19:15:52.546566 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:15:52.541417    4892 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="4ca01b49-9819-4a90-a851-7553ad568679" volumeName="kubernetes.io/projected/4ca01b49-9819-4a90-a851-7553ad568679-kube-api-access-bzw4s" seLinuxMountContext=""
Oct 30 19:15:52.546566 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:15:52.541457    4892 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="87da348d-b1ee-4cd3-806d-0560bff3c6f2" volumeName="kubernetes.io/projected/87da348d-b1ee-4cd3-806d-0560bff3c6f2-kube-api-access-txbm4" seLinuxMountContext=""
Oct 30 19:15:52.546566 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:15:52.541471    4892 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="af36f4d4-f985-4937-a9f7-9884936f3690" volumeName="kubernetes.io/secret/af36f4d4-f985-4937-a9f7-9884936f3690-agent-certs" seLinuxMountContext=""
Oct 30 19:15:52.546566 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:15:52.541488    4892 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="e3206ddf-35de-4b7c-b979-682f5f666a34" volumeName="kubernetes.io/secret/e3206ddf-35de-4b7c-b979-682f5f666a34-image-registry-private-configuration" seLinuxMountContext=""
Oct 30 19:15:52.546566 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:15:52.541502    4892 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="e446ca63-5885-4c66-9cb7-da39afbf742c" volumeName="kubernetes.io/configmap/e446ca63-5885-4c66-9cb7-da39afbf742c-telemetry-config" seLinuxMountContext=""
Oct 30 19:15:52.546566 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:15:52.541517    4892 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="08ae8ede-9a96-4a84-8ec8-8d364fd476a3" volumeName="kubernetes.io/secret/08ae8ede-9a96-4a84-8ec8-8d364fd476a3-ovn-node-metrics-cert" seLinuxMountContext=""
Oct 30 19:15:52.546566 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:15:52.541534    4892 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="0e179af5-9f52-4941-b4f5-5e3b8789da11" volumeName="kubernetes.io/projected/0e179af5-9f52-4941-b4f5-5e3b8789da11-kube-api-access-zj4df" seLinuxMountContext=""
Oct 30 19:15:52.546566 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:15:52.541546    4892 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="2bd84dc4-e96a-43f1-85bf-e1ea4005c193" volumeName="kubernetes.io/projected/2bd84dc4-e96a-43f1-85bf-e1ea4005c193-kube-api-access-4rmts" seLinuxMountContext=""
Oct 30 19:15:52.546566 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:15:52.541566    4892 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="4ba601a6-2a44-4408-8c5b-2c29fd096fed" volumeName="kubernetes.io/secret/4ba601a6-2a44-4408-8c5b-2c29fd096fed-serving-cert" seLinuxMountContext=""
Oct 30 19:15:52.546566 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:15:52.541580    4892 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="dccdaf98-aac7-4756-b93f-22f62a286a1f" volumeName="kubernetes.io/projected/dccdaf98-aac7-4756-b93f-22f62a286a1f-kube-api-access-6bqfr" seLinuxMountContext=""
Oct 30 19:15:52.546566 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:15:52.541592    4892 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="e0221987-8e5e-40d1-8842-c97993f08c6d" volumeName="kubernetes.io/configmap/e0221987-8e5e-40d1-8842-c97993f08c6d-iptables-alerter-script" seLinuxMountContext=""
Oct 30 19:15:52.546566 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:15:52.541610    4892 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="e3206ddf-35de-4b7c-b979-682f5f666a34" volumeName="kubernetes.io/empty-dir/e3206ddf-35de-4b7c-b979-682f5f666a34-ca-trust-extracted" seLinuxMountContext=""
Oct 30 19:15:52.546566 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:15:52.541622    4892 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="fade00ba-24a7-41c9-93f7-23e40971874a" volumeName="kubernetes.io/projected/fade00ba-24a7-41c9-93f7-23e40971874a-kube-api-access-q667z" seLinuxMountContext=""
Oct 30 19:15:52.546566 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:15:52.541723    4892 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="39bb8c16-8282-442a-841b-0056baae014d" volumeName="kubernetes.io/projected/39bb8c16-8282-442a-841b-0056baae014d-kube-api-access-gdq6k" seLinuxMountContext=""
Oct 30 19:15:52.546566 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:15:52.541765    4892 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="0e179af5-9f52-4941-b4f5-5e3b8789da11" volumeName="kubernetes.io/empty-dir/0e179af5-9f52-4941-b4f5-5e3b8789da11-ca-trust-extracted" seLinuxMountContext=""
Oct 30 19:15:52.546566 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:15:52.541790    4892 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="11db9cec-653f-4aee-8937-0cf5b367efac" volumeName="kubernetes.io/configmap/11db9cec-653f-4aee-8937-0cf5b367efac-trusted-ca" seLinuxMountContext=""
Oct 30 19:15:52.546566 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:15:52.541806    4892 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="56274b23-2400-43d4-9686-c928e20d47de" volumeName="kubernetes.io/projected/56274b23-2400-43d4-9686-c928e20d47de-kube-api-access-wt5rq" seLinuxMountContext=""
Oct 30 19:15:52.546566 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:15:52.541840    4892 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="af36f4d4-f985-4937-a9f7-9884936f3690" volumeName="kubernetes.io/configmap/af36f4d4-f985-4937-a9f7-9884936f3690-konnectivity-ca" seLinuxMountContext=""
Oct 30 19:15:52.546566 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:15:52.541854    4892 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="dccdaf98-aac7-4756-b93f-22f62a286a1f" volumeName="kubernetes.io/configmap/dccdaf98-aac7-4756-b93f-22f62a286a1f-multus-daemon-config" seLinuxMountContext=""
Oct 30 19:15:52.546566 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:15:52.541869    4892 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="fc4ac297-f510-4439-80e2-487056d3bf06" volumeName="kubernetes.io/empty-dir/fc4ac297-f510-4439-80e2-487056d3bf06-data-volume" seLinuxMountContext=""
Oct 30 19:15:52.546566 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:15:52.541887    4892 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="0e179af5-9f52-4941-b4f5-5e3b8789da11" volumeName="kubernetes.io/projected/0e179af5-9f52-4941-b4f5-5e3b8789da11-bound-sa-token" seLinuxMountContext=""
Oct 30 19:15:52.546566 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:15:52.541899    4892 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="2bd84dc4-e96a-43f1-85bf-e1ea4005c193" volumeName="kubernetes.io/configmap/2bd84dc4-e96a-43f1-85bf-e1ea4005c193-trusted-ca-bundle" seLinuxMountContext=""
Oct 30 19:15:52.546566 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:15:52.541915    4892 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="bab0abf9-46d4-408e-9aa0-afa7279cd9d1" volumeName="kubernetes.io/configmap/bab0abf9-46d4-408e-9aa0-afa7279cd9d1-cni-sysctl-allowlist" seLinuxMountContext=""
Oct 30 19:15:52.546566 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:15:52.541930    4892 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="e3206ddf-35de-4b7c-b979-682f5f666a34" volumeName="kubernetes.io/secret/e3206ddf-35de-4b7c-b979-682f5f666a34-installation-pull-secrets" seLinuxMountContext=""
Oct 30 19:15:52.546566 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:15:52.541942    4892 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="f9a06263-cb99-4004-bffc-581d66ce5ebf" volumeName="kubernetes.io/projected/f9a06263-cb99-4004-bffc-581d66ce5ebf-kube-api-access-qfw7m" seLinuxMountContext=""
Oct 30 19:15:52.546566 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:15:52.542183    4892 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="4ba601a6-2a44-4408-8c5b-2c29fd096fed" volumeName="kubernetes.io/projected/4ba601a6-2a44-4408-8c5b-2c29fd096fed-kube-api-access-rfxzq" seLinuxMountContext=""
Oct 30 19:15:52.546566 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:15:52.542221    4892 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="4d213db4-73c5-4f50-9930-6d26b880dcf7" volumeName="kubernetes.io/projected/4d213db4-73c5-4f50-9930-6d26b880dcf7-kube-api-access-7xf2g" seLinuxMountContext=""
Oct 30 19:15:52.546566 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:15:52.542238    4892 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="87082646-75c3-43b5-864b-3478e9a7373d" volumeName="kubernetes.io/secret/87082646-75c3-43b5-864b-3478e9a7373d-default-certificate" seLinuxMountContext=""
Oct 30 19:15:52.546566 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:15:52.542279    4892 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="bab0abf9-46d4-408e-9aa0-afa7279cd9d1" volumeName="kubernetes.io/configmap/bab0abf9-46d4-408e-9aa0-afa7279cd9d1-whereabouts-flatfile-configmap" seLinuxMountContext=""
Oct 30 19:15:52.546566 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:15:52.542291    4892 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="c3780b96-955e-4728-a2d0-b7ec30d0f6f3" volumeName="kubernetes.io/projected/c3780b96-955e-4728-a2d0-b7ec30d0f6f3-kube-api-access-cdkvt" seLinuxMountContext=""
Oct 30 19:15:52.546566 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:15:52.542307    4892 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="e3206ddf-35de-4b7c-b979-682f5f666a34" volumeName="kubernetes.io/projected/e3206ddf-35de-4b7c-b979-682f5f666a34-kube-api-access-w9h4f" seLinuxMountContext=""
Oct 30 19:15:52.546566 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:15:52.542349    4892 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="33ce6ceb-02a9-4852-9948-f02e2b39e32f" volumeName="kubernetes.io/projected/33ce6ceb-02a9-4852-9948-f02e2b39e32f-kube-api-access-btd8n" seLinuxMountContext=""
Oct 30 19:15:52.546566 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:15:52.542363    4892 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="08ae8ede-9a96-4a84-8ec8-8d364fd476a3" volumeName="kubernetes.io/projected/08ae8ede-9a96-4a84-8ec8-8d364fd476a3-kube-api-access-bnxm6" seLinuxMountContext=""
Oct 30 19:15:52.546566 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:15:52.542382    4892 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="11db9cec-653f-4aee-8937-0cf5b367efac" volumeName="kubernetes.io/configmap/11db9cec-653f-4aee-8937-0cf5b367efac-config" seLinuxMountContext=""
Oct 30 19:15:52.546566 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:15:52.542395    4892 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="2bd84dc4-e96a-43f1-85bf-e1ea4005c193" volumeName="kubernetes.io/empty-dir/2bd84dc4-e96a-43f1-85bf-e1ea4005c193-tmp" seLinuxMountContext=""
Oct 30 19:15:52.546566 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:15:52.542412    4892 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="08ae8ede-9a96-4a84-8ec8-8d364fd476a3" volumeName="kubernetes.io/configmap/08ae8ede-9a96-4a84-8ec8-8d364fd476a3-ovnkube-config" seLinuxMountContext=""
Oct 30 19:15:52.546566 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:15:52.542424    4892 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="2bd84dc4-e96a-43f1-85bf-e1ea4005c193" volumeName="kubernetes.io/configmap/2bd84dc4-e96a-43f1-85bf-e1ea4005c193-service-ca-bundle" seLinuxMountContext=""
Oct 30 19:15:52.546566 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:15:52.542436    4892 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="f86c0e38-8144-42fc-b622-25602ada113f" volumeName="kubernetes.io/projected/f86c0e38-8144-42fc-b622-25602ada113f-kube-api-access-j4wnm" seLinuxMountContext=""
Oct 30 19:15:52.546566 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:15:52.542453    4892 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="f9a06263-cb99-4004-bffc-581d66ce5ebf" volumeName="kubernetes.io/configmap/f9a06263-cb99-4004-bffc-581d66ce5ebf-config" seLinuxMountContext=""
Oct 30 19:15:52.546566 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:15:52.542465    4892 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="fade00ba-24a7-41c9-93f7-23e40971874a" volumeName="kubernetes.io/empty-dir/fade00ba-24a7-41c9-93f7-23e40971874a-tmp-dir" seLinuxMountContext=""
Oct 30 19:15:52.546566 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:15:52.542481    4892 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="39bb8c16-8282-442a-841b-0056baae014d" volumeName="kubernetes.io/empty-dir/39bb8c16-8282-442a-841b-0056baae014d-tmp" seLinuxMountContext=""
Oct 30 19:15:52.546566 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:15:52.542630    4892 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="bab0abf9-46d4-408e-9aa0-afa7279cd9d1" volumeName="kubernetes.io/configmap/bab0abf9-46d4-408e-9aa0-afa7279cd9d1-cni-binary-copy" seLinuxMountContext=""
Oct 30 19:15:52.546566 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:15:52.542651    4892 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="cdb750aa-dd7e-4e2f-8297-9028acc0f559" volumeName="kubernetes.io/projected/cdb750aa-dd7e-4e2f-8297-9028acc0f559-kube-api-access-rvrch" seLinuxMountContext=""
Oct 30 19:15:52.546566 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:15:52.542672    4892 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="11db9cec-653f-4aee-8937-0cf5b367efac" volumeName="kubernetes.io/projected/11db9cec-653f-4aee-8937-0cf5b367efac-kube-api-access-5xgwq" seLinuxMountContext=""
Oct 30 19:15:52.546566 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:15:52.542704    4892 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="2bd84dc4-e96a-43f1-85bf-e1ea4005c193" volumeName="kubernetes.io/empty-dir/2bd84dc4-e96a-43f1-85bf-e1ea4005c193-snapshots" seLinuxMountContext=""
Oct 30 19:15:52.546566 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:15:52.542728    4892 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="2bd84dc4-e96a-43f1-85bf-e1ea4005c193" volumeName="kubernetes.io/secret/2bd84dc4-e96a-43f1-85bf-e1ea4005c193-serving-cert" seLinuxMountContext=""
Oct 30 19:15:52.546566 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:15:52.542750    4892 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="d260d4d7-b4aa-4f37-92b5-e75b22e13a3b" volumeName="kubernetes.io/projected/d260d4d7-b4aa-4f37-92b5-e75b22e13a3b-kube-api-access-wmkq8" seLinuxMountContext=""
Oct 30 19:15:52.546566 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:15:52.542763    4892 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="e0221987-8e5e-40d1-8842-c97993f08c6d" volumeName="kubernetes.io/projected/e0221987-8e5e-40d1-8842-c97993f08c6d-kube-api-access-fmnd4" seLinuxMountContext=""
Oct 30 19:15:52.546566 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:15:52.542781    4892 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="39bb8c16-8282-442a-841b-0056baae014d" volumeName="kubernetes.io/empty-dir/39bb8c16-8282-442a-841b-0056baae014d-etc-tuned" seLinuxMountContext=""
Oct 30 19:15:52.546566 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:15:52.542793    4892 reconstruct.go:97] "Volume reconstruction finished"
Oct 30 19:15:52.546566 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:15:52.542801    4892 reconciler.go:29] "Reconciler: start to sync state"
Oct 30 19:15:52.555539 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:15:52.555520    4892 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-tg5xj_11db9cec-653f-4aee-8937-0cf5b367efac/console-operator/1.log"
Oct 30 19:15:52.555857 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:15:52.555837    4892 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-tg5xj_11db9cec-653f-4aee-8937-0cf5b367efac/console-operator/1.log"
Oct 30 19:15:52.559732 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:15:52.559697    4892 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv4"
Oct 30 19:15:52.561492 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:15:52.561472    4892 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv6"
Oct 30 19:15:52.561492 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:15:52.561494    4892 status_manager.go:249] "Starting to sync pod status with apiserver"
Oct 30 19:15:52.561599 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:15:52.561514    4892 kubelet.go:2521] "Starting kubelet main sync loop"
Oct 30 19:15:52.561599 ip-10-0-132-82 kubenswrapper[4892]: E1030 19:15:52.561551    4892 kubelet.go:2545] "Skipping pod synchronization" err="[container runtime status check may not have completed yet, PLEG is not healthy: pleg has yet to be successful]"
Oct 30 19:15:52.577943 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:15:52.577918    4892 generic.go:356] "Generic (PLEG): container finished" podID="bab0abf9-46d4-408e-9aa0-afa7279cd9d1" containerID="12bddeb50edd8fbac71cdb67cadcebd514f07fdd2a4993c4e59997b923ffb564" exitCode=0
Oct 30 19:15:52.577943 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:15:52.577944    4892 generic.go:356] "Generic (PLEG): container finished" podID="bab0abf9-46d4-408e-9aa0-afa7279cd9d1" containerID="ffd8aa686926a2bdb3d5ed4e9fb9d162a05d20856e7f999b77f538b86c2cad85" exitCode=0
Oct 30 19:15:52.578072 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:15:52.577951    4892 generic.go:356] "Generic (PLEG): container finished" podID="bab0abf9-46d4-408e-9aa0-afa7279cd9d1" containerID="4109167b89ef6a0b9bac0a265642b2ea08ed42d0f551fefeb2a79fea2663c221" exitCode=0
Oct 30 19:15:52.578072 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:15:52.577958    4892 generic.go:356] "Generic (PLEG): container finished" podID="bab0abf9-46d4-408e-9aa0-afa7279cd9d1" containerID="d14b9cdbefbe3ac90be1fa17d386d32be0c538df487648aef595307389cb423b" exitCode=0
Oct 30 19:15:52.578072 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:15:52.577986    4892 generic.go:356] "Generic (PLEG): container finished" podID="bab0abf9-46d4-408e-9aa0-afa7279cd9d1" containerID="276e24da2fbd424c8f37607af4179597ed72122d871fd1a0b1d86fa4886c553c" exitCode=0
Oct 30 19:15:52.578072 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:15:52.577992    4892 generic.go:356] "Generic (PLEG): container finished" podID="bab0abf9-46d4-408e-9aa0-afa7279cd9d1" containerID="83463f495ed150855f54c4a9c410dc6b8aea0c70c7017e236ab3def9c6213135" exitCode=0
Oct 30 19:15:52.599323 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:15:52.599293    4892 generic.go:356] "Generic (PLEG): container finished" podID="16007dbb15fdd01822b4d377fac4e8af" containerID="c53bce13b8336e42910636a7d114f4c7f7cfb933eb77fede84f6c63870ca7b1c" exitCode=0
Oct 30 19:15:52.608432 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:15:52.608412    4892 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-tg5xj_11db9cec-653f-4aee-8937-0cf5b367efac/console-operator/1.log"
Oct 30 19:15:52.608506 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:15:52.608447    4892 generic.go:356] "Generic (PLEG): container finished" podID="11db9cec-653f-4aee-8937-0cf5b367efac" containerID="dafc11f60068899cfdf6d420a7f0b52ca486082253a85db5cf117594b4979646" exitCode=255
Oct 30 19:15:52.625932 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:15:52.625889    4892 manager.go:324] Recovery completed
Oct 30 19:15:52.637586 ip-10-0-132-82 kubenswrapper[4892]: E1030 19:15:52.637562    4892 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-132-82.us-west-1.compute.internal\" not found"
Oct 30 19:15:52.644094 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:15:52.644080    4892 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach"
Oct 30 19:15:52.645824 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:15:52.645806    4892 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-132-82.us-west-1.compute.internal" event="NodeHasSufficientMemory"
Oct 30 19:15:52.645925 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:15:52.645840    4892 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-132-82.us-west-1.compute.internal" event="NodeHasNoDiskPressure"
Oct 30 19:15:52.645925 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:15:52.645851    4892 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-132-82.us-west-1.compute.internal" event="NodeHasSufficientPID"
Oct 30 19:15:52.647977 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:15:52.647945    4892 cpu_manager.go:226] "Starting" policy="none"
Oct 30 19:15:52.647977 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:15:52.647977    4892 cpu_manager.go:227] "Reconciling" reconcilePeriod="10s"
Oct 30 19:15:52.648101 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:15:52.647984    4892 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach"
Oct 30 19:15:52.648101 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:15:52.647996    4892 state_mem.go:41] "Initialized" logger="CPUManager state checkpoint.CPUManager state memory"
Oct 30 19:15:52.648168 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:15:52.648143    4892 state_mem.go:94] "Updated default CPUSet" logger="CPUManager state checkpoint.CPUManager state memory" cpuSet=""
Oct 30 19:15:52.648168 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:15:52.648155    4892 state_mem.go:102] "Updated CPUSet assignments" logger="CPUManager state checkpoint.CPUManager state memory" assignments={}
Oct 30 19:15:52.648223 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:15:52.648173    4892 state_checkpoint.go:142] "restored state from checkpoint" logger="CPUManager state checkpoint"
Oct 30 19:15:52.648223 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:15:52.648178    4892 state_checkpoint.go:143] "defaultCPUSet" logger="CPUManager state checkpoint" defaultCpuSet=""
Oct 30 19:15:52.648223 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:15:52.648184    4892 policy_none.go:50] "Start"
Oct 30 19:15:52.648223 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:15:52.648192    4892 memory_manager.go:187] "Starting memorymanager" policy="None"
Oct 30 19:15:52.648223 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:15:52.648202    4892 state_mem.go:36] "Initializing new in-memory state store" logger="Memory Manager state checkpoint"
Oct 30 19:15:52.648884 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:15:52.648871    4892 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-132-82.us-west-1.compute.internal" event="NodeHasSufficientMemory"
Oct 30 19:15:52.648956 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:15:52.648898    4892 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-132-82.us-west-1.compute.internal" event="NodeHasNoDiskPressure"
Oct 30 19:15:52.648956 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:15:52.648908    4892 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-132-82.us-west-1.compute.internal" event="NodeHasSufficientPID"
Oct 30 19:15:52.651017 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:15:52.651004    4892 state_mem.go:77] "Updated machine memory state" logger="Memory Manager state checkpoint"
Oct 30 19:15:52.651052 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:15:52.651020    4892 state_checkpoint.go:85] "State checkpoint: restored state from checkpoint" logger="Memory Manager state checkpoint"
Oct 30 19:15:52.651052 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:15:52.651026    4892 policy_none.go:44] "Start"
Oct 30 19:15:52.655404 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:15:52.655382    4892 node_container_manager_linux.go:130] "Enforcing system reserved compressible on cgroup" cgroupName="/system.slice" limits={"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"}
Oct 30 19:15:52.658383 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:15:52.658371    4892 manager.go:341] "Starting Device Plugin manager"
Oct 30 19:15:52.658430 ip-10-0-132-82 kubenswrapper[4892]: E1030 19:15:52.658394    4892 manager.go:525] "Failed to read data from checkpoint" err="checkpoint is not found" checkpoint="kubelet_internal_checkpoint"
Oct 30 19:15:52.658430 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:15:52.658402    4892 server.go:87] "Starting device plugin registration server"
Oct 30 19:15:52.658703 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:15:52.658692    4892 eviction_manager.go:194] "Eviction manager: starting control loop"
Oct 30 19:15:52.658746 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:15:52.658707    4892 container_log_manager.go:146] "Initializing container log rotate workers" workers=1 monitorPeriod="10s"
Oct 30 19:15:52.658810 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:15:52.658787    4892 plugin_watcher.go:53] "Plugin Watcher Start" path="/var/lib/kubelet/plugins_registry"
Oct 30 19:15:52.658979 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:15:52.658906    4892 plugin_watcher.go:201] "Adding socket path or updating timestamp to desired state cache" path="/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock"
Oct 30 19:15:52.659068 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:15:52.658990    4892 plugin_manager.go:119] "The desired_state_of_world populator (plugin watcher) starts"
Oct 30 19:15:52.659068 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:15:52.658998    4892 plugin_manager.go:121] "Starting Kubelet Plugin Manager"
Oct 30 19:15:52.659163 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:15:52.659049    4892 reconciler.go:167] "OperationExecutor.RegisterPlugin started" plugin={"SocketPath":"/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock","Timestamp":"2026-10-30T19:15:52.658955311Z","UUID":"f148bf36-6fd5-4416-b709-ae187d1cbabe","Handler":null,"Name":"","Endpoint":""}
Oct 30 19:15:52.660443 ip-10-0-132-82 kubenswrapper[4892]: E1030 19:15:52.660419    4892 eviction_manager.go:272] "eviction manager: failed to check if we have separate container filesystem. Ignoring." err="non-existent label \"crio-containers\""
Oct 30 19:15:52.660443 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:15:52.660441    4892 csi_plugin.go:106] kubernetes.io/csi: Trying to validate a new CSI Driver with name: ebs.csi.aws.com endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock versions: 1.0.0
Oct 30 19:15:52.660590 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:15:52.660460    4892 csi_plugin.go:119] kubernetes.io/csi: Register new plugin with name: ebs.csi.aws.com at endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock
Oct 30 19:15:52.660590 ip-10-0-132-82 kubenswrapper[4892]: E1030 19:15:52.660464    4892 eviction_manager.go:297] "Eviction manager: failed to get summary stats" err="failed to get node info: node \"ip-10-0-132-82.us-west-1.compute.internal\" not found"
Oct 30 19:15:52.661652 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:15:52.661613    4892 kubelet.go:2608] "SyncLoop ADD" source="file" pods=["openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-132-82.us-west-1.compute.internal","kube-system/kube-apiserver-proxy-ip-10-0-132-82.us-west-1.compute.internal"]
Oct 30 19:15:52.661759 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:15:52.661746    4892 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach"
Oct 30 19:15:52.662695 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:15:52.662678    4892 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-132-82.us-west-1.compute.internal" event="NodeHasSufficientMemory"
Oct 30 19:15:52.662792 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:15:52.662699    4892 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-132-82.us-west-1.compute.internal" event="NodeHasNoDiskPressure"
Oct 30 19:15:52.662792 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:15:52.662709    4892 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-132-82.us-west-1.compute.internal" event="NodeHasSufficientPID"
Oct 30 19:15:52.663127 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:15:52.663096    4892 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach"
Oct 30 19:15:52.663256 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:15:52.663242    4892 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach"
Oct 30 19:15:52.664170 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:15:52.664154    4892 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-132-82.us-west-1.compute.internal" event="NodeHasSufficientMemory"
Oct 30 19:15:52.664266 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:15:52.664179    4892 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-132-82.us-west-1.compute.internal" event="NodeHasNoDiskPressure"
Oct 30 19:15:52.664266 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:15:52.664193    4892 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-132-82.us-west-1.compute.internal" event="NodeHasSufficientPID"
Oct 30 19:15:52.664266 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:15:52.664201    4892 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-132-82.us-west-1.compute.internal" event="NodeHasSufficientMemory"
Oct 30 19:15:52.664266 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:15:52.664222    4892 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-132-82.us-west-1.compute.internal" event="NodeHasNoDiskPressure"
Oct 30 19:15:52.664266 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:15:52.664236    4892 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-132-82.us-west-1.compute.internal" event="NodeHasSufficientPID"
Oct 30 19:15:52.664504 ip-10-0-132-82 kubenswrapper[4892]: E1030 19:15:52.664358    4892 kubelet.go:3364] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-132-82.us-west-1.compute.internal\" not found" node="ip-10-0-132-82.us-west-1.compute.internal"
Oct 30 19:15:52.664640 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:15:52.664611    4892 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-132-82.us-west-1.compute.internal" event={"ID":"16007dbb15fdd01822b4d377fac4e8af","Type":"ContainerStarted","Data":"068eb92c94d0b342a135d59cbc174e4b303f7d17a164613a22996eeb2e047965"}
Oct 30 19:15:52.664702 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:15:52.664656    4892 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach"
Oct 30 19:15:52.664757 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:15:52.664653    4892 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-132-82.us-west-1.compute.internal" event={"ID":"16007dbb15fdd01822b4d377fac4e8af","Type":"ContainerDied","Data":"c53bce13b8336e42910636a7d114f4c7f7cfb933eb77fede84f6c63870ca7b1c"}
Oct 30 19:15:52.664820 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:15:52.664780    4892 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-132-82.us-west-1.compute.internal" event={"ID":"16007dbb15fdd01822b4d377fac4e8af","Type":"ContainerStarted","Data":"b2829769d2e875408df83ae72236364dc323bc32fa4adc204d20ae79308ef283"}
Oct 30 19:15:52.664820 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:15:52.664806    4892 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-132-82.us-west-1.compute.internal" event={"ID":"f9af2f5b3d8deb871a93a1e3f0011dc6","Type":"ContainerStarted","Data":"a2b2739a883d4be5a965500563e1cc61e2cd3a03a44224b6ec8478014abb02aa"}
Oct 30 19:15:52.664915 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:15:52.664819    4892 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-132-82.us-west-1.compute.internal" event={"ID":"f9af2f5b3d8deb871a93a1e3f0011dc6","Type":"ContainerStarted","Data":"3bea07ac8a9e43055344e71268ef13eda8a7b0eb15593518b8076be757e09c3a"}
Oct 30 19:15:52.665943 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:15:52.665925    4892 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-132-82.us-west-1.compute.internal" event="NodeHasSufficientMemory"
Oct 30 19:15:52.666046 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:15:52.665954    4892 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-132-82.us-west-1.compute.internal" event="NodeHasNoDiskPressure"
Oct 30 19:15:52.666046 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:15:52.666018    4892 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-132-82.us-west-1.compute.internal" event="NodeHasSufficientPID"
Oct 30 19:15:52.666237 ip-10-0-132-82 kubenswrapper[4892]: E1030 19:15:52.666221    4892 kubelet.go:3364] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-132-82.us-west-1.compute.internal\" not found" node="ip-10-0-132-82.us-west-1.compute.internal"
Oct 30 19:15:52.729114 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:15:52.729091    4892 reflector.go:446] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:161"
Oct 30 19:15:52.729251 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:15:52.729235    4892 reconstruct.go:205] "DevicePaths of reconstructed volumes updated"
Oct 30 19:15:52.730236 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:15:52.730191    4892 reflector.go:446] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:161"
Oct 30 19:15:52.730709 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:15:52.730688    4892 reflector.go:446] "Caches populated" type="*v1.Node" reflector="k8s.io/client-go/informers/factory.go:161"
Oct 30 19:15:52.735404 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:15:52.735385    4892 kubelet.go:3368] "Creating a mirror pod for static pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-132-82.us-west-1.compute.internal"
Oct 30 19:15:52.759723 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:15:52.759697    4892 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach"
Oct 30 19:15:52.760472 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:15:52.760453    4892 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-132-82.us-west-1.compute.internal" event="NodeHasSufficientMemory"
Oct 30 19:15:52.760539 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:15:52.760481    4892 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-132-82.us-west-1.compute.internal" event="NodeHasNoDiskPressure"
Oct 30 19:15:52.760539 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:15:52.760491    4892 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-132-82.us-west-1.compute.internal" event="NodeHasSufficientPID"
Oct 30 19:15:52.760539 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:15:52.760526    4892 kubelet_node_status.go:77] "Attempting to register node" node="ip-10-0-132-82.us-west-1.compute.internal"
Oct 30 19:15:52.779818 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:15:52.779801    4892 reflector.go:446] "Caches populated" type="*v1.Service" reflector="k8s.io/client-go/informers/factory.go:161"
Oct 30 19:15:52.820019 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:15:52.819999    4892 warnings.go:107] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]"
Oct 30 19:15:52.820087 ip-10-0-132-82 kubenswrapper[4892]: E1030 19:15:52.820037    4892 kubelet.go:3370] "Failed creating a mirror pod" err="pods \"kube-apiserver-proxy-ip-10-0-132-82.us-west-1.compute.internal\" already exists" pod="kube-system/kube-apiserver-proxy-ip-10-0-132-82.us-west-1.compute.internal"
Oct 30 19:15:52.820087 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:15:52.820047    4892 kubelet.go:3368] "Creating a mirror pod for static pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-132-82.us-west-1.compute.internal"
Oct 30 19:15:52.829834 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:15:52.829816    4892 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/16007dbb15fdd01822b4d377fac4e8af-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-132-82.us-west-1.compute.internal\" (UID: \"16007dbb15fdd01822b4d377fac4e8af\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-132-82.us-west-1.compute.internal"
Oct 30 19:15:52.829925 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:15:52.829841    4892 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/16007dbb15fdd01822b4d377fac4e8af-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-132-82.us-west-1.compute.internal\" (UID: \"16007dbb15fdd01822b4d377fac4e8af\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-132-82.us-west-1.compute.internal"
Oct 30 19:15:52.829925 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:15:52.829863    4892 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/f9af2f5b3d8deb871a93a1e3f0011dc6-config\") pod \"kube-apiserver-proxy-ip-10-0-132-82.us-west-1.compute.internal\" (UID: \"f9af2f5b3d8deb871a93a1e3f0011dc6\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-132-82.us-west-1.compute.internal"
Oct 30 19:15:52.875173 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:15:52.875154    4892 kubelet_node_status.go:126] "Node was previously registered" node="ip-10-0-132-82.us-west-1.compute.internal"
Oct 30 19:15:52.875241 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:15:52.875209    4892 kubelet_node_status.go:80] "Successfully registered node" node="ip-10-0-132-82.us-west-1.compute.internal"
Oct 30 19:15:52.875625 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:15:52.875609    4892 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-132-82.us-west-1.compute.internal" event="NodeNotReady"
Oct 30 19:15:52.875697 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:15:52.875634    4892 setters.go:546] "Node became not ready" node="ip-10-0-132-82.us-west-1.compute.internal" condition={"type":"Ready","status":"False","lastHeartbeatTime":"2026-10-30T19:15:52Z","lastTransitionTime":"2026-10-30T19:15:52Z","reason":"KubeletNotReady","message":"CSINode is not yet initialized"}
Oct 30 19:15:52.896925 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:15:52.896908    4892 warnings.go:107] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]"
Oct 30 19:15:52.897014 ip-10-0-132-82 kubenswrapper[4892]: E1030 19:15:52.896944    4892 kubelet.go:3370] "Failed creating a mirror pod" err="pods \"kube-rbac-proxy-crio-ip-10-0-132-82.us-west-1.compute.internal\" already exists" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-132-82.us-west-1.compute.internal"
Oct 30 19:15:52.897014 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:15:52.896982    4892 nodeinfomanager.go:471] existing CSINode "ip-10-0-132-82.us-west-1.compute.internal" is owned by different node (oldNodeID="e92c42ab-910c-411f-b86d-47ca0a8df3e6", newNodeID=""), cleaning up...
Oct 30 19:15:52.930077 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:15:52.930049    4892 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/16007dbb15fdd01822b4d377fac4e8af-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-132-82.us-west-1.compute.internal\" (UID: \"16007dbb15fdd01822b4d377fac4e8af\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-132-82.us-west-1.compute.internal"
Oct 30 19:15:52.930077 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:15:52.930084    4892 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/16007dbb15fdd01822b4d377fac4e8af-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-132-82.us-west-1.compute.internal\" (UID: \"16007dbb15fdd01822b4d377fac4e8af\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-132-82.us-west-1.compute.internal"
Oct 30 19:15:52.930206 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:15:52.930100    4892 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/f9af2f5b3d8deb871a93a1e3f0011dc6-config\") pod \"kube-apiserver-proxy-ip-10-0-132-82.us-west-1.compute.internal\" (UID: \"f9af2f5b3d8deb871a93a1e3f0011dc6\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-132-82.us-west-1.compute.internal"
Oct 30 19:15:52.930206 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:15:52.930141    4892 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/host-path/f9af2f5b3d8deb871a93a1e3f0011dc6-config\") pod \"kube-apiserver-proxy-ip-10-0-132-82.us-west-1.compute.internal\" (UID: \"f9af2f5b3d8deb871a93a1e3f0011dc6\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-132-82.us-west-1.compute.internal"
Oct 30 19:15:52.930206 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:15:52.930160    4892 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/16007dbb15fdd01822b4d377fac4e8af-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-132-82.us-west-1.compute.internal\" (UID: \"16007dbb15fdd01822b4d377fac4e8af\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-132-82.us-west-1.compute.internal"
Oct 30 19:15:52.930206 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:15:52.930174    4892 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/16007dbb15fdd01822b4d377fac4e8af-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-132-82.us-west-1.compute.internal\" (UID: \"16007dbb15fdd01822b4d377fac4e8af\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-132-82.us-west-1.compute.internal"
Oct 30 19:15:53.521371 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:15:53.521344    4892 apiserver.go:52] "Watching apiserver"
Oct 30 19:15:53.613841 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:15:53.613777    4892 kubelet.go:3368] "Creating a mirror pod for static pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-132-82.us-west-1.compute.internal"
Oct 30 19:15:53.613982 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:15:53.613843    4892 kubelet.go:3368] "Creating a mirror pod for static pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-132-82.us-west-1.compute.internal"
Oct 30 19:15:53.625683 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:15:53.625658    4892 reflector.go:446] "Caches populated" type="*v1.Pod" reflector="pkg/kubelet/config/apiserver.go:66"
Oct 30 19:15:53.626322 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:15:53.626300    4892 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-network-console/networking-console-plugin-86c478bb88-h4twn","kube-system/kas-connection-checker-685895c7c9-p8kjs","openshift-image-registry/image-registry-d97c68fd4-htsl8","openshift-monitoring/cluster-monitoring-operator-884576c57-wkldq","openshift-console-operator/console-operator-7cd56b8dbc-tg5xj","kube-system/kube-apiserver-proxy-ip-10-0-132-82.us-west-1.compute.internal","openshift-ingress-canary/ingress-canary-8rwjm","openshift-cluster-node-tuning-operator/tuned-2fv9z","openshift-ingress/router-default-776df59447-8nrhh","openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-np94b","openshift-ovn-kubernetes/ovnkube-node-5lwkx","kube-system/global-pull-secret-syncer-76rht","openshift-multus/multus-jvbw5","openshift-dns/node-resolver-g9xd6","openshift-cluster-storage-operator/volume-data-source-validator-58c9f5f46d-p42xv","openshift-multus/network-metrics-daemon-822gp","openshift-network-diagnostics/network-check-target-c24j9","openshift-image-registry/image-registry-6fcb4848df-8l8cp","openshift-image-registry/node-ca-kgfvh","openshift-network-diagnostics/network-check-source-6f78d59f99-2trjq","openshift-insights/insights-runtime-extractor-jrm87","kube-system/kas-connection-checker-685895c7c9-9h85f","kube-system/kas-connection-checker-685895c7c9-tvmks","openshift-multus/multus-additional-cni-plugins-4pvgg","openshift-service-ca-operator/service-ca-operator-544d749c8c-nbfv2","openshift-insights/insights-operator-7c4986d449-6jdkq","kube-system/konnectivity-agent-qnm56","openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-132-82.us-west-1.compute.internal","openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-qgv98","openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-scljn","openshift-network-operator/iptables-alerter-wnn66","openshift-dns/dns-default-bb9cj"]
Oct 30 19:15:53.627436 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:15:53.627420    4892 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/router-default-776df59447-8nrhh"
Oct 30 19:15:53.627998 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:15:53.627957    4892 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/cluster-monitoring-operator-884576c57-wkldq"
Oct 30 19:15:53.632127 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:15:53.632106    4892 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-np94b"
Oct 30 19:15:53.632575 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:15:53.632561    4892 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-d97c68fd4-htsl8"
Oct 30 19:15:53.633030 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:15:53.633011    4892 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-console/networking-console-plugin-86c478bb88-h4twn"
Oct 30 19:15:53.634795 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:15:53.634777    4892 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-6fcb4848df-8l8cp"
Oct 30 19:15:53.637377 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:15:53.637361    4892 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-822gp"
Oct 30 19:15:53.640350 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:15:53.640333    4892 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-bb9cj"
Oct 30 19:15:53.641020 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:15:53.641004    4892 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-jrm87"
Oct 30 19:15:53.696583 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:15:53.696559    4892 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console-operator\"/\"serving-cert\""
Oct 30 19:15:53.696583 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:15:53.696564    4892 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"cluster-monitoring-operator-tls\""
Oct 30 19:15:53.696707 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:15:53.696582    4892 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"serving-cert\""
Oct 30 19:15:53.696707 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:15:53.696594    4892 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"default-ingress-cert\""
Oct 30 19:15:53.696707 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:15:53.696623    4892 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"router-metrics-certs-default\""
Oct 30 19:15:53.696707 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:15:53.696624    4892 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-service-ca-operator\"/\"serving-cert\""
Oct 30 19:15:53.696707 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:15:53.696631    4892 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"router-stats-default\""
Oct 30 19:15:53.700088 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:15:53.700074    4892 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"openshift-service-ca.crt\""
Oct 30 19:15:53.700173 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:15:53.700110    4892 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca-operator\"/\"kube-root-ca.crt\""
Oct 30 19:15:53.700230 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:15:53.700172    4892 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca-operator\"/\"service-ca-operator-config\""
Oct 30 19:15:53.700230 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:15:53.700176    4892 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"kube-root-ca.crt\""
Oct 30 19:15:53.700230 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:15:53.700184    4892 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"openshift-service-ca.crt\""
Oct 30 19:15:53.700230 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:15:53.700210    4892 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca-operator\"/\"openshift-service-ca.crt\""
Oct 30 19:15:53.700230 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:15:53.700221    4892 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"openshift-service-ca.crt\""
Oct 30 19:15:53.700467 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:15:53.700257    4892 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"openshift-service-ca.crt\""
Oct 30 19:15:53.700467 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:15:53.700257    4892 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-root-ca.crt\""
Oct 30 19:15:53.700467 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:15:53.700281    4892 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"kube-root-ca.crt\""
Oct 30 19:15:53.700467 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:15:53.700296    4892 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"console-operator-config\""
Oct 30 19:15:53.700467 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:15:53.700291    4892 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"config\""
Oct 30 19:15:53.700467 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:15:53.700292    4892 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-service-ca.crt\""
Oct 30 19:15:53.700467 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:15:53.700316    4892 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"kube-root-ca.crt\""
Oct 30 19:15:53.700467 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:15:53.700278    4892 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"service-ca-bundle\""
Oct 30 19:15:53.700467 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:15:53.700360    4892 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"openshift-service-ca.crt\""
Oct 30 19:15:53.700863 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:15:53.700538    4892 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"kube-root-ca.crt\""
Oct 30 19:15:53.734672 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:15:53.734649    4892 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/39bb8c16-8282-442a-841b-0056baae014d-etc-sysconfig\") pod \"tuned-2fv9z\" (UID: \"39bb8c16-8282-442a-841b-0056baae014d\") " pod="openshift-cluster-node-tuning-operator/tuned-2fv9z"
Oct 30 19:15:53.734672 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:15:53.734672    4892 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/39bb8c16-8282-442a-841b-0056baae014d-etc-sysctl-d\") pod \"tuned-2fv9z\" (UID: \"39bb8c16-8282-442a-841b-0056baae014d\") " pod="openshift-cluster-node-tuning-operator/tuned-2fv9z"
Oct 30 19:15:53.734803 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:15:53.734688    4892 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/dccdaf98-aac7-4756-b93f-22f62a286a1f-host-run-k8s-cni-cncf-io\") pod \"multus-jvbw5\" (UID: \"dccdaf98-aac7-4756-b93f-22f62a286a1f\") " pod="openshift-multus/multus-jvbw5"
Oct 30 19:15:53.734803 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:15:53.734726    4892 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4rmts\" (UniqueName: \"kubernetes.io/projected/2bd84dc4-e96a-43f1-85bf-e1ea4005c193-kube-api-access-4rmts\") pod \"insights-operator-7c4986d449-6jdkq\" (UID: \"2bd84dc4-e96a-43f1-85bf-e1ea4005c193\") " pod="openshift-insights/insights-operator-7c4986d449-6jdkq"
Oct 30 19:15:53.734932 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:15:53.734919    4892 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/08ae8ede-9a96-4a84-8ec8-8d364fd476a3-host-slash\") pod \"ovnkube-node-5lwkx\" (UID: \"08ae8ede-9a96-4a84-8ec8-8d364fd476a3\") " pod="openshift-ovn-kubernetes/ovnkube-node-5lwkx"
Oct 30 19:15:53.734983 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:15:53.734940    4892 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/dccdaf98-aac7-4756-b93f-22f62a286a1f-host-var-lib-cni-bin\") pod \"multus-jvbw5\" (UID: \"dccdaf98-aac7-4756-b93f-22f62a286a1f\") " pod="openshift-multus/multus-jvbw5"
Oct 30 19:15:53.734983 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:15:53.734956    4892 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/dccdaf98-aac7-4756-b93f-22f62a286a1f-host-var-lib-kubelet\") pod \"multus-jvbw5\" (UID: \"dccdaf98-aac7-4756-b93f-22f62a286a1f\") " pod="openshift-multus/multus-jvbw5"
Oct 30 19:15:53.735051 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:15:53.735016    4892 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-btd8n\" (UniqueName: \"kubernetes.io/projected/33ce6ceb-02a9-4852-9948-f02e2b39e32f-kube-api-access-btd8n\") pod \"network-check-source-6f78d59f99-2trjq\" (UID: \"33ce6ceb-02a9-4852-9948-f02e2b39e32f\") " pod="openshift-network-diagnostics/network-check-source-6f78d59f99-2trjq"
Oct 30 19:15:53.735051 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:15:53.735034    4892 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/e0221987-8e5e-40d1-8842-c97993f08c6d-iptables-alerter-script\") pod \"iptables-alerter-wnn66\" (UID: \"e0221987-8e5e-40d1-8842-c97993f08c6d\") " pod="openshift-network-operator/iptables-alerter-wnn66"
Oct 30 19:15:53.735116 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:15:53.735098    4892 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/0e179af5-9f52-4941-b4f5-5e3b8789da11-image-registry-private-configuration\") pod \"image-registry-d97c68fd4-htsl8\" (UID: \"0e179af5-9f52-4941-b4f5-5e3b8789da11\") " pod="openshift-image-registry/image-registry-d97c68fd4-htsl8"
Oct 30 19:15:53.735157 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:15:53.735129    4892 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/08ae8ede-9a96-4a84-8ec8-8d364fd476a3-systemd-units\") pod \"ovnkube-node-5lwkx\" (UID: \"08ae8ede-9a96-4a84-8ec8-8d364fd476a3\") " pod="openshift-ovn-kubernetes/ovnkube-node-5lwkx"
Oct 30 19:15:53.735157 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:15:53.735151    4892 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/48386105-d79b-4f92-baa0-f330b9658ad0-hosts-file\") pod \"node-resolver-g9xd6\" (UID: \"48386105-d79b-4f92-baa0-f330b9658ad0\") " pod="openshift-dns/node-resolver-g9xd6"
Oct 30 19:15:53.735236 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:15:53.735207    4892 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/e3206ddf-35de-4b7c-b979-682f5f666a34-registry-tls\") pod \"image-registry-6fcb4848df-8l8cp\" (UID: \"e3206ddf-35de-4b7c-b979-682f5f666a34\") " pod="openshift-image-registry/image-registry-6fcb4848df-8l8cp"
Oct 30 19:15:53.735274 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:15:53.735237    4892 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-v7kvw\" (UniqueName: \"kubernetes.io/projected/287ba08f-7b9f-4609-8e35-832c0879c44c-kube-api-access-v7kvw\") pod \"network-metrics-daemon-822gp\" (UID: \"287ba08f-7b9f-4609-8e35-832c0879c44c\") " pod="openshift-multus/network-metrics-daemon-822gp"
Oct 30 19:15:53.735304 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:15:53.735278    4892 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/e3206ddf-35de-4b7c-b979-682f5f666a34-trusted-ca\") pod \"image-registry-6fcb4848df-8l8cp\" (UID: \"e3206ddf-35de-4b7c-b979-682f5f666a34\") " pod="openshift-image-registry/image-registry-6fcb4848df-8l8cp"
Oct 30 19:15:53.735377 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:15:53.735357    4892 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-w9h4f\" (UniqueName: \"kubernetes.io/projected/e3206ddf-35de-4b7c-b979-682f5f666a34-kube-api-access-w9h4f\") pod \"image-registry-6fcb4848df-8l8cp\" (UID: \"e3206ddf-35de-4b7c-b979-682f5f666a34\") " pod="openshift-image-registry/image-registry-6fcb4848df-8l8cp"
Oct 30 19:15:53.735435 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:15:53.735395    4892 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/39bb8c16-8282-442a-841b-0056baae014d-lib-modules\") pod \"tuned-2fv9z\" (UID: \"39bb8c16-8282-442a-841b-0056baae014d\") " pod="openshift-cluster-node-tuning-operator/tuned-2fv9z"
Oct 30 19:15:53.735435 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:15:53.735429    4892 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/dccdaf98-aac7-4756-b93f-22f62a286a1f-multus-cni-dir\") pod \"multus-jvbw5\" (UID: \"dccdaf98-aac7-4756-b93f-22f62a286a1f\") " pod="openshift-multus/multus-jvbw5"
Oct 30 19:15:53.735521 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:15:53.735466    4892 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5xgwq\" (UniqueName: \"kubernetes.io/projected/11db9cec-653f-4aee-8937-0cf5b367efac-kube-api-access-5xgwq\") pod \"console-operator-7cd56b8dbc-tg5xj\" (UID: \"11db9cec-653f-4aee-8937-0cf5b367efac\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-tg5xj"
Oct 30 19:15:53.735553 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:15:53.735516    4892 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bnxm6\" (UniqueName: \"kubernetes.io/projected/08ae8ede-9a96-4a84-8ec8-8d364fd476a3-kube-api-access-bnxm6\") pod \"ovnkube-node-5lwkx\" (UID: \"08ae8ede-9a96-4a84-8ec8-8d364fd476a3\") " pod="openshift-ovn-kubernetes/ovnkube-node-5lwkx"
Oct 30 19:15:53.735553 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:15:53.735545    4892 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/e3206ddf-35de-4b7c-b979-682f5f666a34-ca-trust-extracted\") pod \"image-registry-6fcb4848df-8l8cp\" (UID: \"e3206ddf-35de-4b7c-b979-682f5f666a34\") " pod="openshift-image-registry/image-registry-6fcb4848df-8l8cp"
Oct 30 19:15:53.735652 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:15:53.735564    4892 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/39bb8c16-8282-442a-841b-0056baae014d-host\") pod \"tuned-2fv9z\" (UID: \"39bb8c16-8282-442a-841b-0056baae014d\") " pod="openshift-cluster-node-tuning-operator/tuned-2fv9z"
Oct 30 19:15:53.735652 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:15:53.735591    4892 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/287ba08f-7b9f-4609-8e35-832c0879c44c-metrics-certs\") pod \"network-metrics-daemon-822gp\" (UID: \"287ba08f-7b9f-4609-8e35-832c0879c44c\") " pod="openshift-multus/network-metrics-daemon-822gp"
Oct 30 19:15:53.735652 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:15:53.735607    4892 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/2bd84dc4-e96a-43f1-85bf-e1ea4005c193-trusted-ca-bundle\") pod \"insights-operator-7c4986d449-6jdkq\" (UID: \"2bd84dc4-e96a-43f1-85bf-e1ea4005c193\") " pod="openshift-insights/insights-operator-7c4986d449-6jdkq"
Oct 30 19:15:53.735652 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:15:53.735622    4892 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/08ae8ede-9a96-4a84-8ec8-8d364fd476a3-host-run-netns\") pod \"ovnkube-node-5lwkx\" (UID: \"08ae8ede-9a96-4a84-8ec8-8d364fd476a3\") " pod="openshift-ovn-kubernetes/ovnkube-node-5lwkx"
Oct 30 19:15:53.735652 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:15:53.735642    4892 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/08ae8ede-9a96-4a84-8ec8-8d364fd476a3-ovnkube-config\") pod \"ovnkube-node-5lwkx\" (UID: \"08ae8ede-9a96-4a84-8ec8-8d364fd476a3\") " pod="openshift-ovn-kubernetes/ovnkube-node-5lwkx"
Oct 30 19:15:53.735893 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:15:53.735670    4892 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/e3206ddf-35de-4b7c-b979-682f5f666a34-ca-trust-extracted\") pod \"image-registry-6fcb4848df-8l8cp\" (UID: \"e3206ddf-35de-4b7c-b979-682f5f666a34\") " pod="openshift-image-registry/image-registry-6fcb4848df-8l8cp"
Oct 30 19:15:53.735893 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:15:53.735677    4892 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gdq6k\" (UniqueName: \"kubernetes.io/projected/39bb8c16-8282-442a-841b-0056baae014d-kube-api-access-gdq6k\") pod \"tuned-2fv9z\" (UID: \"39bb8c16-8282-442a-841b-0056baae014d\") " pod="openshift-cluster-node-tuning-operator/tuned-2fv9z"
Oct 30 19:15:53.735893 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:15:53.735758    4892 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/dccdaf98-aac7-4756-b93f-22f62a286a1f-host-run-multus-certs\") pod \"multus-jvbw5\" (UID: \"dccdaf98-aac7-4756-b93f-22f62a286a1f\") " pod="openshift-multus/multus-jvbw5"
Oct 30 19:15:53.735893 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:15:53.735775    4892 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/56274b23-2400-43d4-9686-c928e20d47de-samples-operator-tls\") pod \"cluster-samples-operator-6ddb6c8b94-np94b\" (UID: \"56274b23-2400-43d4-9686-c928e20d47de\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-np94b"
Oct 30 19:15:53.735893 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:15:53.735812    4892 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/e0221987-8e5e-40d1-8842-c97993f08c6d-host-slash\") pod \"iptables-alerter-wnn66\" (UID: \"e0221987-8e5e-40d1-8842-c97993f08c6d\") " pod="openshift-network-operator/iptables-alerter-wnn66"
Oct 30 19:15:53.735893 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:15:53.735858    4892 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/dccdaf98-aac7-4756-b93f-22f62a286a1f-system-cni-dir\") pod \"multus-jvbw5\" (UID: \"dccdaf98-aac7-4756-b93f-22f62a286a1f\") " pod="openshift-multus/multus-jvbw5"
Oct 30 19:15:53.735893 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:15:53.735875    4892 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/dccdaf98-aac7-4756-b93f-22f62a286a1f-host-run-netns\") pod \"multus-jvbw5\" (UID: \"dccdaf98-aac7-4756-b93f-22f62a286a1f\") " pod="openshift-multus/multus-jvbw5"
Oct 30 19:15:53.736168 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:15:53.735894    4892 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qfw7m\" (UniqueName: \"kubernetes.io/projected/f9a06263-cb99-4004-bffc-581d66ce5ebf-kube-api-access-qfw7m\") pod \"kube-storage-version-migrator-operator-7d4bc45dd6-scljn\" (UID: \"f9a06263-cb99-4004-bffc-581d66ce5ebf\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-scljn"
Oct 30 19:15:53.736168 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:15:53.735929    4892 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/2bd84dc4-e96a-43f1-85bf-e1ea4005c193-service-ca-bundle\") pod \"insights-operator-7c4986d449-6jdkq\" (UID: \"2bd84dc4-e96a-43f1-85bf-e1ea4005c193\") " pod="openshift-insights/insights-operator-7c4986d449-6jdkq"
Oct 30 19:15:53.736168 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:15:53.735954    4892 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/08ae8ede-9a96-4a84-8ec8-8d364fd476a3-host-kubelet\") pod \"ovnkube-node-5lwkx\" (UID: \"08ae8ede-9a96-4a84-8ec8-8d364fd476a3\") " pod="openshift-ovn-kubernetes/ovnkube-node-5lwkx"
Oct 30 19:15:53.736168 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:15:53.736010    4892 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8s7g5\" (UniqueName: \"kubernetes.io/projected/87082646-75c3-43b5-864b-3478e9a7373d-kube-api-access-8s7g5\") pod \"router-default-776df59447-8nrhh\" (UID: \"87082646-75c3-43b5-864b-3478e9a7373d\") " pod="openshift-ingress/router-default-776df59447-8nrhh"
Oct 30 19:15:53.736168 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:15:53.736078    4892 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/08ae8ede-9a96-4a84-8ec8-8d364fd476a3-env-overrides\") pod \"ovnkube-node-5lwkx\" (UID: \"08ae8ede-9a96-4a84-8ec8-8d364fd476a3\") " pod="openshift-ovn-kubernetes/ovnkube-node-5lwkx"
Oct 30 19:15:53.736168 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:15:53.736121    4892 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-txbm4\" (UniqueName: \"kubernetes.io/projected/87da348d-b1ee-4cd3-806d-0560bff3c6f2-kube-api-access-txbm4\") pod \"kas-connection-checker-685895c7c9-p8kjs\" (UID: \"87da348d-b1ee-4cd3-806d-0560bff3c6f2\") " pod="kube-system/kas-connection-checker-685895c7c9-p8kjs"
Oct 30 19:15:53.736168 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:15:53.736167    4892 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rvrch\" (UniqueName: \"kubernetes.io/projected/cdb750aa-dd7e-4e2f-8297-9028acc0f559-kube-api-access-rvrch\") pod \"node-ca-kgfvh\" (UID: \"cdb750aa-dd7e-4e2f-8297-9028acc0f559\") " pod="openshift-image-registry/node-ca-kgfvh"
Oct 30 19:15:53.736425 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:15:53.736194    4892 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/dccdaf98-aac7-4756-b93f-22f62a286a1f-multus-daemon-config\") pod \"multus-jvbw5\" (UID: \"dccdaf98-aac7-4756-b93f-22f62a286a1f\") " pod="openshift-multus/multus-jvbw5"
Oct 30 19:15:53.736425 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:15:53.736217    4892 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6bqfr\" (UniqueName: \"kubernetes.io/projected/dccdaf98-aac7-4756-b93f-22f62a286a1f-kube-api-access-6bqfr\") pod \"multus-jvbw5\" (UID: \"dccdaf98-aac7-4756-b93f-22f62a286a1f\") " pod="openshift-multus/multus-jvbw5"
Oct 30 19:15:53.736425 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:15:53.736232    4892 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"default-certificate\" (UniqueName: \"kubernetes.io/secret/87082646-75c3-43b5-864b-3478e9a7373d-default-certificate\") pod \"router-default-776df59447-8nrhh\" (UID: \"87082646-75c3-43b5-864b-3478e9a7373d\") " pod="openshift-ingress/router-default-776df59447-8nrhh"
Oct 30 19:15:53.736425 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:15:53.736247    4892 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"stats-auth\" (UniqueName: \"kubernetes.io/secret/87082646-75c3-43b5-864b-3478e9a7373d-stats-auth\") pod \"router-default-776df59447-8nrhh\" (UID: \"87082646-75c3-43b5-864b-3478e9a7373d\") " pod="openshift-ingress/router-default-776df59447-8nrhh"
Oct 30 19:15:53.736425 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:15:53.736269    4892 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/bab0abf9-46d4-408e-9aa0-afa7279cd9d1-cnibin\") pod \"multus-additional-cni-plugins-4pvgg\" (UID: \"bab0abf9-46d4-408e-9aa0-afa7279cd9d1\") " pod="openshift-multus/multus-additional-cni-plugins-4pvgg"
Oct 30 19:15:53.736425 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:15:53.736294    4892 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/bab0abf9-46d4-408e-9aa0-afa7279cd9d1-cni-binary-copy\") pod \"multus-additional-cni-plugins-4pvgg\" (UID: \"bab0abf9-46d4-408e-9aa0-afa7279cd9d1\") " pod="openshift-multus/multus-additional-cni-plugins-4pvgg"
Oct 30 19:15:53.736425 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:15:53.736322    4892 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/48386105-d79b-4f92-baa0-f330b9658ad0-tmp-dir\") pod \"node-resolver-g9xd6\" (UID: \"48386105-d79b-4f92-baa0-f330b9658ad0\") " pod="openshift-dns/node-resolver-g9xd6"
Oct 30 19:15:53.736724 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:15:53.736427    4892 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/48386105-d79b-4f92-baa0-f330b9658ad0-tmp-dir\") pod \"node-resolver-g9xd6\" (UID: \"48386105-d79b-4f92-baa0-f330b9658ad0\") " pod="openshift-dns/node-resolver-g9xd6"
Oct 30 19:15:53.736724 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:15:53.736481    4892 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/e3206ddf-35de-4b7c-b979-682f5f666a34-image-registry-private-configuration\") pod \"image-registry-6fcb4848df-8l8cp\" (UID: \"e3206ddf-35de-4b7c-b979-682f5f666a34\") " pod="openshift-image-registry/image-registry-6fcb4848df-8l8cp"
Oct 30 19:15:53.736724 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:15:53.736514    4892 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/39bb8c16-8282-442a-841b-0056baae014d-etc-kubernetes\") pod \"tuned-2fv9z\" (UID: \"39bb8c16-8282-442a-841b-0056baae014d\") " pod="openshift-cluster-node-tuning-operator/tuned-2fv9z"
Oct 30 19:15:53.736724 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:15:53.736531    4892 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"default-certificate\" (UniqueName: \"kubernetes.io/secret/87082646-75c3-43b5-864b-3478e9a7373d-default-certificate\") pod \"router-default-776df59447-8nrhh\" (UID: \"87082646-75c3-43b5-864b-3478e9a7373d\") " pod="openshift-ingress/router-default-776df59447-8nrhh"
Oct 30 19:15:53.736724 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:15:53.736538    4892 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/39bb8c16-8282-442a-841b-0056baae014d-tmp\") pod \"tuned-2fv9z\" (UID: \"39bb8c16-8282-442a-841b-0056baae014d\") " pod="openshift-cluster-node-tuning-operator/tuned-2fv9z"
Oct 30 19:15:53.736724 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:15:53.736556    4892 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/4ba601a6-2a44-4408-8c5b-2c29fd096fed-config\") pod \"service-ca-operator-544d749c8c-nbfv2\" (UID: \"4ba601a6-2a44-4408-8c5b-2c29fd096fed\") " pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-nbfv2"
Oct 30 19:15:53.736724 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:15:53.736570    4892 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/87082646-75c3-43b5-864b-3478e9a7373d-service-ca-bundle\") pod \"router-default-776df59447-8nrhh\" (UID: \"87082646-75c3-43b5-864b-3478e9a7373d\") " pod="openshift-ingress/router-default-776df59447-8nrhh"
Oct 30 19:15:53.736724 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:15:53.736592    4892 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/4ba601a6-2a44-4408-8c5b-2c29fd096fed-serving-cert\") pod \"service-ca-operator-544d749c8c-nbfv2\" (UID: \"4ba601a6-2a44-4408-8c5b-2c29fd096fed\") " pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-nbfv2"
Oct 30 19:15:53.736724 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:15:53.736630    4892 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/39bb8c16-8282-442a-841b-0056baae014d-tmp\") pod \"tuned-2fv9z\" (UID: \"39bb8c16-8282-442a-841b-0056baae014d\") " pod="openshift-cluster-node-tuning-operator/tuned-2fv9z"
Oct 30 19:15:53.736724 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:15:53.736632    4892 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"stats-auth\" (UniqueName: \"kubernetes.io/secret/87082646-75c3-43b5-864b-3478e9a7373d-stats-auth\") pod \"router-default-776df59447-8nrhh\" (UID: \"87082646-75c3-43b5-864b-3478e9a7373d\") " pod="openshift-ingress/router-default-776df59447-8nrhh"
Oct 30 19:15:53.736724 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:15:53.736683    4892 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/2c7c7060-def6-4737-8fa9-6889309b7463-networking-console-plugin-cert\") pod \"networking-console-plugin-86c478bb88-h4twn\" (UID: \"2c7c7060-def6-4737-8fa9-6889309b7463\") " pod="openshift-network-console/networking-console-plugin-86c478bb88-h4twn"
Oct 30 19:15:53.736724 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:15:53.736716    4892 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/dccdaf98-aac7-4756-b93f-22f62a286a1f-etc-kubernetes\") pod \"multus-jvbw5\" (UID: \"dccdaf98-aac7-4756-b93f-22f62a286a1f\") " pod="openshift-multus/multus-jvbw5"
Oct 30 19:15:53.737226 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:15:53.736787    4892 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/e3206ddf-35de-4b7c-b979-682f5f666a34-registry-certificates\") pod \"image-registry-6fcb4848df-8l8cp\" (UID: \"e3206ddf-35de-4b7c-b979-682f5f666a34\") " pod="openshift-image-registry/image-registry-6fcb4848df-8l8cp"
Oct 30 19:15:53.737226 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:15:53.736838    4892 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/configmap/4ba601a6-2a44-4408-8c5b-2c29fd096fed-config\") pod \"service-ca-operator-544d749c8c-nbfv2\" (UID: \"4ba601a6-2a44-4408-8c5b-2c29fd096fed\") " pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-nbfv2"
Oct 30 19:15:53.737226 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:15:53.736840    4892 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/4ba601a6-2a44-4408-8c5b-2c29fd096fed-serving-cert\") pod \"service-ca-operator-544d749c8c-nbfv2\" (UID: \"4ba601a6-2a44-4408-8c5b-2c29fd096fed\") " pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-nbfv2"
Oct 30 19:15:53.737226 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:15:53.736874    4892 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/dccdaf98-aac7-4756-b93f-22f62a286a1f-cni-binary-copy\") pod \"multus-jvbw5\" (UID: \"dccdaf98-aac7-4756-b93f-22f62a286a1f\") " pod="openshift-multus/multus-jvbw5"
Oct 30 19:15:53.737226 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:15:53.737007    4892 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/f9a06263-cb99-4004-bffc-581d66ce5ebf-serving-cert\") pod \"kube-storage-version-migrator-operator-7d4bc45dd6-scljn\" (UID: \"f9a06263-cb99-4004-bffc-581d66ce5ebf\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-scljn"
Oct 30 19:15:53.737226 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:15:53.737040    4892 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7xf2g\" (UniqueName: \"kubernetes.io/projected/4d213db4-73c5-4f50-9930-6d26b880dcf7-kube-api-access-7xf2g\") pod \"volume-data-source-validator-58c9f5f46d-p42xv\" (UID: \"4d213db4-73c5-4f50-9930-6d26b880dcf7\") " pod="openshift-cluster-storage-operator/volume-data-source-validator-58c9f5f46d-p42xv"
Oct 30 19:15:53.737226 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:15:53.737075    4892 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zj4df\" (UniqueName: \"kubernetes.io/projected/0e179af5-9f52-4941-b4f5-5e3b8789da11-kube-api-access-zj4df\") pod \"image-registry-d97c68fd4-htsl8\" (UID: \"0e179af5-9f52-4941-b4f5-5e3b8789da11\") " pod="openshift-image-registry/image-registry-d97c68fd4-htsl8"
Oct 30 19:15:53.737226 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:15:53.737151    4892 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/e3206ddf-35de-4b7c-b979-682f5f666a34-bound-sa-token\") pod \"image-registry-6fcb4848df-8l8cp\" (UID: \"e3206ddf-35de-4b7c-b979-682f5f666a34\") " pod="openshift-image-registry/image-registry-6fcb4848df-8l8cp"
Oct 30 19:15:53.737540 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:15:53.737247    4892 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/f9a06263-cb99-4004-bffc-581d66ce5ebf-serving-cert\") pod \"kube-storage-version-migrator-operator-7d4bc45dd6-scljn\" (UID: \"f9a06263-cb99-4004-bffc-581d66ce5ebf\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-scljn"
Oct 30 19:15:53.737540 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:15:53.737314    4892 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/dccdaf98-aac7-4756-b93f-22f62a286a1f-cnibin\") pod \"multus-jvbw5\" (UID: \"dccdaf98-aac7-4756-b93f-22f62a286a1f\") " pod="openshift-multus/multus-jvbw5"
Oct 30 19:15:53.737540 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:15:53.737340    4892 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/dccdaf98-aac7-4756-b93f-22f62a286a1f-hostroot\") pod \"multus-jvbw5\" (UID: \"dccdaf98-aac7-4756-b93f-22f62a286a1f\") " pod="openshift-multus/multus-jvbw5"
Oct 30 19:15:53.737540 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:15:53.737384    4892 desired_state_of_world_populator.go:154] "Finished populating initial desired state of world"
Oct 30 19:15:53.737540 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:15:53.737404    4892 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/08ae8ede-9a96-4a84-8ec8-8d364fd476a3-run-systemd\") pod \"ovnkube-node-5lwkx\" (UID: \"08ae8ede-9a96-4a84-8ec8-8d364fd476a3\") " pod="openshift-ovn-kubernetes/ovnkube-node-5lwkx"
Oct 30 19:15:53.737540 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:15:53.737431    4892 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/08ae8ede-9a96-4a84-8ec8-8d364fd476a3-log-socket\") pod \"ovnkube-node-5lwkx\" (UID: \"08ae8ede-9a96-4a84-8ec8-8d364fd476a3\") " pod="openshift-ovn-kubernetes/ovnkube-node-5lwkx"
Oct 30 19:15:53.737540 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:15:53.737478    4892 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/39bb8c16-8282-442a-841b-0056baae014d-etc-modprobe-d\") pod \"tuned-2fv9z\" (UID: \"39bb8c16-8282-442a-841b-0056baae014d\") " pod="openshift-cluster-node-tuning-operator/tuned-2fv9z"
Oct 30 19:15:53.737540 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:15:53.737522    4892 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"snapshots\" (UniqueName: \"kubernetes.io/empty-dir/2bd84dc4-e96a-43f1-85bf-e1ea4005c193-snapshots\") pod \"insights-operator-7c4986d449-6jdkq\" (UID: \"2bd84dc4-e96a-43f1-85bf-e1ea4005c193\") " pod="openshift-insights/insights-operator-7c4986d449-6jdkq"
Oct 30 19:15:53.737821 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:15:53.737568    4892 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/0e179af5-9f52-4941-b4f5-5e3b8789da11-ca-trust-extracted\") pod \"image-registry-d97c68fd4-htsl8\" (UID: \"0e179af5-9f52-4941-b4f5-5e3b8789da11\") " pod="openshift-image-registry/image-registry-d97c68fd4-htsl8"
Oct 30 19:15:53.737821 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:15:53.737612    4892 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"snapshots\" (UniqueName: \"kubernetes.io/empty-dir/2bd84dc4-e96a-43f1-85bf-e1ea4005c193-snapshots\") pod \"insights-operator-7c4986d449-6jdkq\" (UID: \"2bd84dc4-e96a-43f1-85bf-e1ea4005c193\") " pod="openshift-insights/insights-operator-7c4986d449-6jdkq"
Oct 30 19:15:53.737821 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:15:53.737617    4892 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/cdb750aa-dd7e-4e2f-8297-9028acc0f559-serviceca\") pod \"node-ca-kgfvh\" (UID: \"cdb750aa-dd7e-4e2f-8297-9028acc0f559\") " pod="openshift-image-registry/node-ca-kgfvh"
Oct 30 19:15:53.737821 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:15:53.737653    4892 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/0e179af5-9f52-4941-b4f5-5e3b8789da11-ca-trust-extracted\") pod \"image-registry-d97c68fd4-htsl8\" (UID: \"0e179af5-9f52-4941-b4f5-5e3b8789da11\") " pod="openshift-image-registry/image-registry-d97c68fd4-htsl8"
Oct 30 19:15:53.737821 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:15:53.737657    4892 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/39bb8c16-8282-442a-841b-0056baae014d-sys\") pod \"tuned-2fv9z\" (UID: \"39bb8c16-8282-442a-841b-0056baae014d\") " pod="openshift-cluster-node-tuning-operator/tuned-2fv9z"
Oct 30 19:15:53.737821 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:15:53.737690    4892 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/dccdaf98-aac7-4756-b93f-22f62a286a1f-multus-conf-dir\") pod \"multus-jvbw5\" (UID: \"dccdaf98-aac7-4756-b93f-22f62a286a1f\") " pod="openshift-multus/multus-jvbw5"
Oct 30 19:15:53.737821 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:15:53.737741    4892 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/08ae8ede-9a96-4a84-8ec8-8d364fd476a3-run-openvswitch\") pod \"ovnkube-node-5lwkx\" (UID: \"08ae8ede-9a96-4a84-8ec8-8d364fd476a3\") " pod="openshift-ovn-kubernetes/ovnkube-node-5lwkx"
Oct 30 19:15:53.737821 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:15:53.737773    4892 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/08ae8ede-9a96-4a84-8ec8-8d364fd476a3-ovn-node-metrics-cert\") pod \"ovnkube-node-5lwkx\" (UID: \"08ae8ede-9a96-4a84-8ec8-8d364fd476a3\") " pod="openshift-ovn-kubernetes/ovnkube-node-5lwkx"
Oct 30 19:15:53.737821 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:15:53.737820    4892 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rfxzq\" (UniqueName: \"kubernetes.io/projected/4ba601a6-2a44-4408-8c5b-2c29fd096fed-kube-api-access-rfxzq\") pod \"service-ca-operator-544d749c8c-nbfv2\" (UID: \"4ba601a6-2a44-4408-8c5b-2c29fd096fed\") " pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-nbfv2"
Oct 30 19:15:53.738246 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:15:53.738208    4892 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/11db9cec-653f-4aee-8937-0cf5b367efac-config\") pod \"console-operator-7cd56b8dbc-tg5xj\" (UID: \"11db9cec-653f-4aee-8937-0cf5b367efac\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-tg5xj"
Oct 30 19:15:53.738410 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:15:53.738387    4892 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/08ae8ede-9a96-4a84-8ec8-8d364fd476a3-node-log\") pod \"ovnkube-node-5lwkx\" (UID: \"08ae8ede-9a96-4a84-8ec8-8d364fd476a3\") " pod="openshift-ovn-kubernetes/ovnkube-node-5lwkx"
Oct 30 19:15:53.738463 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:15:53.738409    4892 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/configmap/11db9cec-653f-4aee-8937-0cf5b367efac-config\") pod \"console-operator-7cd56b8dbc-tg5xj\" (UID: \"11db9cec-653f-4aee-8937-0cf5b367efac\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-tg5xj"
Oct 30 19:15:53.738512 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:15:53.738460    4892 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/dccdaf98-aac7-4756-b93f-22f62a286a1f-multus-socket-dir-parent\") pod \"multus-jvbw5\" (UID: \"dccdaf98-aac7-4756-b93f-22f62a286a1f\") " pod="openshift-multus/multus-jvbw5"
Oct 30 19:15:53.738565 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:15:53.738540    4892 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wt5rq\" (UniqueName: \"kubernetes.io/projected/56274b23-2400-43d4-9686-c928e20d47de-kube-api-access-wt5rq\") pod \"cluster-samples-operator-6ddb6c8b94-np94b\" (UID: \"56274b23-2400-43d4-9686-c928e20d47de\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-np94b"
Oct 30 19:15:53.738618 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:15:53.738589    4892 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/bab0abf9-46d4-408e-9aa0-afa7279cd9d1-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-4pvgg\" (UID: \"bab0abf9-46d4-408e-9aa0-afa7279cd9d1\") " pod="openshift-multus/multus-additional-cni-plugins-4pvgg"
Oct 30 19:15:53.738663 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:15:53.738630    4892 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/0e179af5-9f52-4941-b4f5-5e3b8789da11-trusted-ca\") pod \"image-registry-d97c68fd4-htsl8\" (UID: \"0e179af5-9f52-4941-b4f5-5e3b8789da11\") " pod="openshift-image-registry/image-registry-d97c68fd4-htsl8"
Oct 30 19:15:53.738748 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:15:53.738731    4892 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-w8zg9\" (UniqueName: \"kubernetes.io/projected/48386105-d79b-4f92-baa0-f330b9658ad0-kube-api-access-w8zg9\") pod \"node-resolver-g9xd6\" (UID: \"48386105-d79b-4f92-baa0-f330b9658ad0\") " pod="openshift-dns/node-resolver-g9xd6"
Oct 30 19:15:53.738821 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:15:53.738804    4892 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/11db9cec-653f-4aee-8937-0cf5b367efac-serving-cert\") pod \"console-operator-7cd56b8dbc-tg5xj\" (UID: \"11db9cec-653f-4aee-8937-0cf5b367efac\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-tg5xj"
Oct 30 19:15:53.738878 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:15:53.738847    4892 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/0e179af5-9f52-4941-b4f5-5e3b8789da11-registry-certificates\") pod \"image-registry-d97c68fd4-htsl8\" (UID: \"0e179af5-9f52-4941-b4f5-5e3b8789da11\") " pod="openshift-image-registry/image-registry-d97c68fd4-htsl8"
Oct 30 19:15:53.738940 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:15:53.738883    4892 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/0e179af5-9f52-4941-b4f5-5e3b8789da11-bound-sa-token\") pod \"image-registry-d97c68fd4-htsl8\" (UID: \"0e179af5-9f52-4941-b4f5-5e3b8789da11\") " pod="openshift-image-registry/image-registry-d97c68fd4-htsl8"
Oct 30 19:15:53.739091 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:15:53.739075    4892 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/e3206ddf-35de-4b7c-b979-682f5f666a34-installation-pull-secrets\") pod \"image-registry-6fcb4848df-8l8cp\" (UID: \"e3206ddf-35de-4b7c-b979-682f5f666a34\") " pod="openshift-image-registry/image-registry-6fcb4848df-8l8cp"
Oct 30 19:15:53.739150 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:15:53.739114    4892 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/0e179af5-9f52-4941-b4f5-5e3b8789da11-installation-pull-secrets\") pod \"image-registry-d97c68fd4-htsl8\" (UID: \"0e179af5-9f52-4941-b4f5-5e3b8789da11\") " pod="openshift-image-registry/image-registry-d97c68fd4-htsl8"
Oct 30 19:15:53.739196 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:15:53.739151    4892 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/11db9cec-653f-4aee-8937-0cf5b367efac-serving-cert\") pod \"console-operator-7cd56b8dbc-tg5xj\" (UID: \"11db9cec-653f-4aee-8937-0cf5b367efac\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-tg5xj"
Oct 30 19:15:53.739196 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:15:53.739189    4892 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/08ae8ede-9a96-4a84-8ec8-8d364fd476a3-run-ovn\") pod \"ovnkube-node-5lwkx\" (UID: \"08ae8ede-9a96-4a84-8ec8-8d364fd476a3\") " pod="openshift-ovn-kubernetes/ovnkube-node-5lwkx"
Oct 30 19:15:53.739302 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:15:53.739234    4892 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/87082646-75c3-43b5-864b-3478e9a7373d-metrics-certs\") pod \"router-default-776df59447-8nrhh\" (UID: \"87082646-75c3-43b5-864b-3478e9a7373d\") " pod="openshift-ingress/router-default-776df59447-8nrhh"
Oct 30 19:15:53.739351 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:15:53.739309    4892 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/bab0abf9-46d4-408e-9aa0-afa7279cd9d1-system-cni-dir\") pod \"multus-additional-cni-plugins-4pvgg\" (UID: \"bab0abf9-46d4-408e-9aa0-afa7279cd9d1\") " pod="openshift-multus/multus-additional-cni-plugins-4pvgg"
Oct 30 19:15:53.739398 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:15:53.739355    4892 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-k62g9\" (UniqueName: \"kubernetes.io/projected/ba2c5571-23a3-46b6-a36d-7e8810a66261-kube-api-access-k62g9\") pod \"kas-connection-checker-685895c7c9-9h85f\" (UID: \"ba2c5571-23a3-46b6-a36d-7e8810a66261\") " pod="kube-system/kas-connection-checker-685895c7c9-9h85f"
Oct 30 19:15:53.739462 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:15:53.739436    4892 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/e446ca63-5885-4c66-9cb7-da39afbf742c-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-884576c57-wkldq\" (UID: \"e446ca63-5885-4c66-9cb7-da39afbf742c\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-wkldq"
Oct 30 19:15:53.739568 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:15:53.739485    4892 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/2bd84dc4-e96a-43f1-85bf-e1ea4005c193-serving-cert\") pod \"insights-operator-7c4986d449-6jdkq\" (UID: \"2bd84dc4-e96a-43f1-85bf-e1ea4005c193\") " pod="openshift-insights/insights-operator-7c4986d449-6jdkq"
Oct 30 19:15:53.739911 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:15:53.739851    4892 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/08ae8ede-9a96-4a84-8ec8-8d364fd476a3-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-5lwkx\" (UID: \"08ae8ede-9a96-4a84-8ec8-8d364fd476a3\") " pod="openshift-ovn-kubernetes/ovnkube-node-5lwkx"
Oct 30 19:15:53.739997 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:15:53.739931    4892 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/08ae8ede-9a96-4a84-8ec8-8d364fd476a3-var-lib-openvswitch\") pod \"ovnkube-node-5lwkx\" (UID: \"08ae8ede-9a96-4a84-8ec8-8d364fd476a3\") " pod="openshift-ovn-kubernetes/ovnkube-node-5lwkx"
Oct 30 19:15:53.740045 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:15:53.740010    4892 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bzw4s\" (UniqueName: \"kubernetes.io/projected/4ca01b49-9819-4a90-a851-7553ad568679-kube-api-access-bzw4s\") pod \"network-check-target-c24j9\" (UID: \"4ca01b49-9819-4a90-a851-7553ad568679\") " pod="openshift-network-diagnostics/network-check-target-c24j9"
Oct 30 19:15:53.740099 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:15:53.740059    4892 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-config\" (UniqueName: \"kubernetes.io/configmap/56274b23-2400-43d4-9686-c928e20d47de-samples-operator-config\") pod \"cluster-samples-operator-6ddb6c8b94-np94b\" (UID: \"56274b23-2400-43d4-9686-c928e20d47de\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-np94b"
Oct 30 19:15:53.740262 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:15:53.740245    4892 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/0e179af5-9f52-4941-b4f5-5e3b8789da11-registry-tls\") pod \"image-registry-d97c68fd4-htsl8\" (UID: \"0e179af5-9f52-4941-b4f5-5e3b8789da11\") " pod="openshift-image-registry/image-registry-d97c68fd4-htsl8"
Oct 30 19:15:53.740333 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:15:53.740287    4892 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/39bb8c16-8282-442a-841b-0056baae014d-etc-sysctl-conf\") pod \"tuned-2fv9z\" (UID: \"39bb8c16-8282-442a-841b-0056baae014d\") " pod="openshift-cluster-node-tuning-operator/tuned-2fv9z"
Oct 30 19:15:53.740378 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:15:53.740358    4892 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4rcdn\" (UniqueName: \"kubernetes.io/projected/bab0abf9-46d4-408e-9aa0-afa7279cd9d1-kube-api-access-4rcdn\") pod \"multus-additional-cni-plugins-4pvgg\" (UID: \"bab0abf9-46d4-408e-9aa0-afa7279cd9d1\") " pod="openshift-multus/multus-additional-cni-plugins-4pvgg"
Oct 30 19:15:53.740422 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:15:53.740389    4892 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fmnd4\" (UniqueName: \"kubernetes.io/projected/e0221987-8e5e-40d1-8842-c97993f08c6d-kube-api-access-fmnd4\") pod \"iptables-alerter-wnn66\" (UID: \"e0221987-8e5e-40d1-8842-c97993f08c6d\") " pod="openshift-network-operator/iptables-alerter-wnn66"
Oct 30 19:15:53.740471 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:15:53.740436    4892 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/11db9cec-653f-4aee-8937-0cf5b367efac-trusted-ca\") pod \"console-operator-7cd56b8dbc-tg5xj\" (UID: \"11db9cec-653f-4aee-8937-0cf5b367efac\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-tg5xj"
Oct 30 19:15:53.740516 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:15:53.740480    4892 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/08ae8ede-9a96-4a84-8ec8-8d364fd476a3-ovnkube-script-lib\") pod \"ovnkube-node-5lwkx\" (UID: \"08ae8ede-9a96-4a84-8ec8-8d364fd476a3\") " pod="openshift-ovn-kubernetes/ovnkube-node-5lwkx"
Oct 30 19:15:53.740559 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:15:53.740513    4892 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/cdb750aa-dd7e-4e2f-8297-9028acc0f559-host\") pod \"node-ca-kgfvh\" (UID: \"cdb750aa-dd7e-4e2f-8297-9028acc0f559\") " pod="openshift-image-registry/node-ca-kgfvh"
Oct 30 19:15:53.740559 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:15:53.740540    4892 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/39bb8c16-8282-442a-841b-0056baae014d-etc-systemd\") pod \"tuned-2fv9z\" (UID: \"39bb8c16-8282-442a-841b-0056baae014d\") " pod="openshift-cluster-node-tuning-operator/tuned-2fv9z"
Oct 30 19:15:53.740667 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:15:53.740623    4892 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/dccdaf98-aac7-4756-b93f-22f62a286a1f-host-var-lib-cni-multus\") pod \"multus-jvbw5\" (UID: \"dccdaf98-aac7-4756-b93f-22f62a286a1f\") " pod="openshift-multus/multus-jvbw5"
Oct 30 19:15:53.740715 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:15:53.740689    4892 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wmkq8\" (UniqueName: \"kubernetes.io/projected/d260d4d7-b4aa-4f37-92b5-e75b22e13a3b-kube-api-access-wmkq8\") pod \"kas-connection-checker-685895c7c9-tvmks\" (UID: \"d260d4d7-b4aa-4f37-92b5-e75b22e13a3b\") " pod="kube-system/kas-connection-checker-685895c7c9-tvmks"
Oct 30 19:15:53.740762 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:15:53.740721    4892 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/f9a06263-cb99-4004-bffc-581d66ce5ebf-config\") pod \"kube-storage-version-migrator-operator-7d4bc45dd6-scljn\" (UID: \"f9a06263-cb99-4004-bffc-581d66ce5ebf\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-scljn"
Oct 30 19:15:53.740812 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:15:53.740782    4892 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"telemetry-config\" (UniqueName: \"kubernetes.io/configmap/e446ca63-5885-4c66-9cb7-da39afbf742c-telemetry-config\") pod \"cluster-monitoring-operator-884576c57-wkldq\" (UID: \"e446ca63-5885-4c66-9cb7-da39afbf742c\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-wkldq"
Oct 30 19:15:53.740877 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:15:53.740826    4892 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/08ae8ede-9a96-4a84-8ec8-8d364fd476a3-etc-openvswitch\") pod \"ovnkube-node-5lwkx\" (UID: \"08ae8ede-9a96-4a84-8ec8-8d364fd476a3\") " pod="openshift-ovn-kubernetes/ovnkube-node-5lwkx"
Oct 30 19:15:53.741309 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:15:53.741237    4892 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/08ae8ede-9a96-4a84-8ec8-8d364fd476a3-host-cni-bin\") pod \"ovnkube-node-5lwkx\" (UID: \"08ae8ede-9a96-4a84-8ec8-8d364fd476a3\") " pod="openshift-ovn-kubernetes/ovnkube-node-5lwkx"
Oct 30 19:15:53.741380 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:15:53.741331    4892 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/39bb8c16-8282-442a-841b-0056baae014d-var-lib-kubelet\") pod \"tuned-2fv9z\" (UID: \"39bb8c16-8282-442a-841b-0056baae014d\") " pod="openshift-cluster-node-tuning-operator/tuned-2fv9z"
Oct 30 19:15:53.741380 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:15:53.741360    4892 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/39bb8c16-8282-442a-841b-0056baae014d-etc-tuned\") pod \"tuned-2fv9z\" (UID: \"39bb8c16-8282-442a-841b-0056baae014d\") " pod="openshift-cluster-node-tuning-operator/tuned-2fv9z"
Oct 30 19:15:53.741595 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:15:53.741547    4892 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/bab0abf9-46d4-408e-9aa0-afa7279cd9d1-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-4pvgg\" (UID: \"bab0abf9-46d4-408e-9aa0-afa7279cd9d1\") " pod="openshift-multus/multus-additional-cni-plugins-4pvgg"
Oct 30 19:15:53.741676 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:15:53.741643    4892 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/39bb8c16-8282-442a-841b-0056baae014d-etc-tuned\") pod \"tuned-2fv9z\" (UID: \"39bb8c16-8282-442a-841b-0056baae014d\") " pod="openshift-cluster-node-tuning-operator/tuned-2fv9z"
Oct 30 19:15:53.741731 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:15:53.741576    4892 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/configmap/f9a06263-cb99-4004-bffc-581d66ce5ebf-config\") pod \"kube-storage-version-migrator-operator-7d4bc45dd6-scljn\" (UID: \"f9a06263-cb99-4004-bffc-581d66ce5ebf\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-scljn"
Oct 30 19:15:53.741821 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:15:53.741805    4892 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/39bb8c16-8282-442a-841b-0056baae014d-run\") pod \"tuned-2fv9z\" (UID: \"39bb8c16-8282-442a-841b-0056baae014d\") " pod="openshift-cluster-node-tuning-operator/tuned-2fv9z"
Oct 30 19:15:53.741992 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:15:53.741958    4892 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/bab0abf9-46d4-408e-9aa0-afa7279cd9d1-tuning-conf-dir\") pod \"multus-additional-cni-plugins-4pvgg\" (UID: \"bab0abf9-46d4-408e-9aa0-afa7279cd9d1\") " pod="openshift-multus/multus-additional-cni-plugins-4pvgg"
Oct 30 19:15:53.742113 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:15:53.742086    4892 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-j7bln\" (UniqueName: \"kubernetes.io/projected/e446ca63-5885-4c66-9cb7-da39afbf742c-kube-api-access-j7bln\") pod \"cluster-monitoring-operator-884576c57-wkldq\" (UID: \"e446ca63-5885-4c66-9cb7-da39afbf742c\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-wkldq"
Oct 30 19:15:53.742244 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:15:53.742223    4892 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/2bd84dc4-e96a-43f1-85bf-e1ea4005c193-tmp\") pod \"insights-operator-7c4986d449-6jdkq\" (UID: \"2bd84dc4-e96a-43f1-85bf-e1ea4005c193\") " pod="openshift-insights/insights-operator-7c4986d449-6jdkq"
Oct 30 19:15:53.742458 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:15:53.742444    4892 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/2c7c7060-def6-4737-8fa9-6889309b7463-nginx-conf\") pod \"networking-console-plugin-86c478bb88-h4twn\" (UID: \"2c7c7060-def6-4737-8fa9-6889309b7463\") " pod="openshift-network-console/networking-console-plugin-86c478bb88-h4twn"
Oct 30 19:15:53.742563 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:15:53.742550    4892 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/08ae8ede-9a96-4a84-8ec8-8d364fd476a3-host-run-ovn-kubernetes\") pod \"ovnkube-node-5lwkx\" (UID: \"08ae8ede-9a96-4a84-8ec8-8d364fd476a3\") " pod="openshift-ovn-kubernetes/ovnkube-node-5lwkx"
Oct 30 19:15:53.742668 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:15:53.742656    4892 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/08ae8ede-9a96-4a84-8ec8-8d364fd476a3-host-cni-netd\") pod \"ovnkube-node-5lwkx\" (UID: \"08ae8ede-9a96-4a84-8ec8-8d364fd476a3\") " pod="openshift-ovn-kubernetes/ovnkube-node-5lwkx"
Oct 30 19:15:53.742839 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:15:53.742792    4892 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/2bd84dc4-e96a-43f1-85bf-e1ea4005c193-tmp\") pod \"insights-operator-7c4986d449-6jdkq\" (UID: \"2bd84dc4-e96a-43f1-85bf-e1ea4005c193\") " pod="openshift-insights/insights-operator-7c4986d449-6jdkq"
Oct 30 19:15:53.749764 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:15:53.749743    4892 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"telemetry-config\""
Oct 30 19:15:53.749845 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:15:53.749828    4892 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"trusted-ca\""
Oct 30 19:15:53.749883 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:15:53.749869    4892 warnings.go:107] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]"
Oct 30 19:15:53.749930 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:15:53.749888    4892 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"trusted-ca-bundle\""
Oct 30 19:15:53.749995 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:15:53.749934    4892 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"service-ca-bundle\""
Oct 30 19:15:53.749995 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:15:53.749941    4892 warnings.go:107] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]"
Oct 30 19:15:53.749995 ip-10-0-132-82 kubenswrapper[4892]: E1030 19:15:53.749977    4892 kubelet.go:3370] "Failed creating a mirror pod" err="pods \"kube-apiserver-proxy-ip-10-0-132-82.us-west-1.compute.internal\" already exists" pod="kube-system/kube-apiserver-proxy-ip-10-0-132-82.us-west-1.compute.internal"
Oct 30 19:15:53.750133 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:15:53.750001    4892 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\""
Oct 30 19:15:53.750133 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:15:53.750010    4892 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-root-ca.crt\""
Oct 30 19:15:53.750133 ip-10-0-132-82 kubenswrapper[4892]: E1030 19:15:53.750010    4892 kubelet.go:3370] "Failed creating a mirror pod" err="pods \"kube-rbac-proxy-crio-ip-10-0-132-82.us-west-1.compute.internal\" already exists" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-132-82.us-west-1.compute.internal"
Oct 30 19:15:53.750133 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:15:53.750038    4892 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-storage-operator\"/\"kube-root-ca.crt\""
Oct 30 19:15:53.750133 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:15:53.750044    4892 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-samples-operator\"/\"samples-operator-config\""
Oct 30 19:15:53.750133 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:15:53.750076    4892 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\""
Oct 30 19:15:53.750133 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:15:53.750105    4892 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-storage-operator\"/\"openshift-service-ca.crt\""
Oct 30 19:15:53.750133 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:15:53.750106    4892 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-samples-operator\"/\"openshift-service-ca.crt\""
Oct 30 19:15:53.750133 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:15:53.750108    4892 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-samples-operator\"/\"kube-root-ca.crt\""
Oct 30 19:15:53.750404 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:15:53.750327    4892 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"samples-operator-config\" (UniqueName: \"kubernetes.io/configmap/56274b23-2400-43d4-9686-c928e20d47de-samples-operator-config\") pod \"cluster-samples-operator-6ddb6c8b94-np94b\" (UID: \"56274b23-2400-43d4-9686-c928e20d47de\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-np94b"
Oct 30 19:15:53.751435 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:15:53.751419    4892 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"telemetry-config\" (UniqueName: \"kubernetes.io/configmap/e446ca63-5885-4c66-9cb7-da39afbf742c-telemetry-config\") pod \"cluster-monitoring-operator-884576c57-wkldq\" (UID: \"e446ca63-5885-4c66-9cb7-da39afbf742c\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-wkldq"
Oct 30 19:15:53.751499 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:15:53.751484    4892 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/11db9cec-653f-4aee-8937-0cf5b367efac-trusted-ca\") pod \"console-operator-7cd56b8dbc-tg5xj\" (UID: \"11db9cec-653f-4aee-8937-0cf5b367efac\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-tg5xj"
Oct 30 19:15:53.756852 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:15:53.756831    4892 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/2bd84dc4-e96a-43f1-85bf-e1ea4005c193-service-ca-bundle\") pod \"insights-operator-7c4986d449-6jdkq\" (UID: \"2bd84dc4-e96a-43f1-85bf-e1ea4005c193\") " pod="openshift-insights/insights-operator-7c4986d449-6jdkq"
Oct 30 19:15:53.757188 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:15:53.757172    4892 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/2bd84dc4-e96a-43f1-85bf-e1ea4005c193-trusted-ca-bundle\") pod \"insights-operator-7c4986d449-6jdkq\" (UID: \"2bd84dc4-e96a-43f1-85bf-e1ea4005c193\") " pod="openshift-insights/insights-operator-7c4986d449-6jdkq"
Oct 30 19:15:53.786328 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:15:53.786302    4892 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"openshift-insights-serving-cert\""
Oct 30 19:15:53.786687 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:15:53.786672    4892 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"canary-serving-cert\""
Oct 30 19:15:53.786757 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:15:53.786723    4892 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"network-diagnostics-dockercfg-q7v7l\""
Oct 30 19:15:53.786845 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:15:53.786780    4892 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-console\"/\"networking-console-plugin-dockercfg-p7m6n\""
Oct 30 19:15:53.786845 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:15:53.786804    4892 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-node-identity-dockercfg-46fzl\""
Oct 30 19:15:53.786845 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:15:53.786828    4892 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-dockercfg-5vtr8\""
Oct 30 19:15:53.787072 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:15:53.787057    4892 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-default-metrics-tls\""
Oct 30 19:15:53.787185 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:15:53.787170    4892 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-console\"/\"networking-console-plugin-cert\""
Oct 30 19:15:53.787621 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:15:53.787609    4892 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-sa-dockercfg-csh58\""
Oct 30 19:15:53.787653 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:15:53.787644    4892 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-metrics-serving-cert\""
Oct 30 19:15:53.788158 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:15:53.788144    4892 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"ingress-canary-dockercfg-qjhd5\""
Oct 30 19:15:53.788228 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:15:53.788214    4892 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-tls\""
Oct 30 19:15:53.788274 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:15:53.788215    4892 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-tls\""
Oct 30 19:15:53.788274 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:15:53.788238    4892 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-private-configuration\""
Oct 30 19:15:53.788274 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:15:53.788245    4892 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-samples-operator\"/\"samples-operator-tls\""
Oct 30 19:15:53.788399 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:15:53.788384    4892 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-node-metrics-cert\""
Oct 30 19:15:53.788558 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:15:53.788546    4892 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\""
Oct 30 19:15:53.788607 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:15:53.788589    4892 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"global-pull-secret-syncer-dockercfg-gt8zl\""
Oct 30 19:15:53.788655 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:15:53.788607    4892 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"node-ca-dockercfg-6brtl\""
Oct 30 19:15:53.788655 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:15:53.788623    4892 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"installation-pull-secrets\""
Oct 30 19:15:53.788655 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:15:53.788624    4892 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-ancillary-tools-dockercfg-4k9jz\""
Oct 30 19:15:53.788655 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:15:53.788624    4892 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent-dockercfg-q7svh\""
Oct 30 19:15:53.788815 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:15:53.788659    4892 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-kubernetes-node-dockercfg-n57d7\""
Oct 30 19:15:53.788815 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:15:53.788667    4892 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-hk6fn\""
Oct 30 19:15:53.788815 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:15:53.788643    4892 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"network-check-target-dockercfg-px7h6\""
Oct 30 19:15:53.788815 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:15:53.788682    4892 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-sa-dockercfg-mnhjf\""
Oct 30 19:15:53.788815 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:15:53.788654    4892 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"registry-dockercfg-2q27v\""
Oct 30 19:15:53.788815 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:15:53.788700    4892 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"tuned-dockercfg-rhxwm\""
Oct 30 19:15:53.788815 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:15:53.788751    4892 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent\""
Oct 30 19:15:53.788815 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:15:53.788772    4892 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-jlfgq\""
Oct 30 19:15:53.789045 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:15:53.788820    4892 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\""
Oct 30 19:15:53.789045 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:15:53.788820    4892 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"node-resolver-dockercfg-xzg6c\""
Oct 30 19:15:53.789877 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:15:53.789863    4892 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/2bd84dc4-e96a-43f1-85bf-e1ea4005c193-serving-cert\") pod \"insights-operator-7c4986d449-6jdkq\" (UID: \"2bd84dc4-e96a-43f1-85bf-e1ea4005c193\") " pod="openshift-insights/insights-operator-7c4986d449-6jdkq"
Oct 30 19:15:53.790052 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:15:53.790037    4892 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/0e179af5-9f52-4941-b4f5-5e3b8789da11-installation-pull-secrets\") pod \"image-registry-d97c68fd4-htsl8\" (UID: \"0e179af5-9f52-4941-b4f5-5e3b8789da11\") " pod="openshift-image-registry/image-registry-d97c68fd4-htsl8"
Oct 30 19:15:53.790084 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:15:53.790037    4892 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/e3206ddf-35de-4b7c-b979-682f5f666a34-installation-pull-secrets\") pod \"image-registry-6fcb4848df-8l8cp\" (UID: \"e3206ddf-35de-4b7c-b979-682f5f666a34\") " pod="openshift-image-registry/image-registry-6fcb4848df-8l8cp"
Oct 30 19:15:53.791485 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:15:53.791470    4892 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4rmts\" (UniqueName: \"kubernetes.io/projected/2bd84dc4-e96a-43f1-85bf-e1ea4005c193-kube-api-access-4rmts\") pod \"insights-operator-7c4986d449-6jdkq\" (UID: \"2bd84dc4-e96a-43f1-85bf-e1ea4005c193\") " pod="openshift-insights/insights-operator-7c4986d449-6jdkq"
Oct 30 19:15:53.795378 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:15:53.795363    4892 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/0e179af5-9f52-4941-b4f5-5e3b8789da11-image-registry-private-configuration\") pod \"image-registry-d97c68fd4-htsl8\" (UID: \"0e179af5-9f52-4941-b4f5-5e3b8789da11\") " pod="openshift-image-registry/image-registry-d97c68fd4-htsl8"
Oct 30 19:15:53.797066 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:15:53.797046    4892 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/e3206ddf-35de-4b7c-b979-682f5f666a34-image-registry-private-configuration\") pod \"image-registry-6fcb4848df-8l8cp\" (UID: \"e3206ddf-35de-4b7c-b979-682f5f666a34\") " pod="openshift-image-registry/image-registry-6fcb4848df-8l8cp"
Oct 30 19:15:53.797470 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:15:53.797452    4892 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qfw7m\" (UniqueName: \"kubernetes.io/projected/f9a06263-cb99-4004-bffc-581d66ce5ebf-kube-api-access-qfw7m\") pod \"kube-storage-version-migrator-operator-7d4bc45dd6-scljn\" (UID: \"f9a06263-cb99-4004-bffc-581d66ce5ebf\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-scljn"
Oct 30 19:15:53.797529 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:15:53.797511    4892 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7xf2g\" (UniqueName: \"kubernetes.io/projected/4d213db4-73c5-4f50-9930-6d26b880dcf7-kube-api-access-7xf2g\") pod \"volume-data-source-validator-58c9f5f46d-p42xv\" (UID: \"4d213db4-73c5-4f50-9930-6d26b880dcf7\") " pod="openshift-cluster-storage-operator/volume-data-source-validator-58c9f5f46d-p42xv"
Oct 30 19:15:53.797580 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:15:53.797563    4892 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5xgwq\" (UniqueName: \"kubernetes.io/projected/11db9cec-653f-4aee-8937-0cf5b367efac-kube-api-access-5xgwq\") pod \"console-operator-7cd56b8dbc-tg5xj\" (UID: \"11db9cec-653f-4aee-8937-0cf5b367efac\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-tg5xj"
Oct 30 19:15:53.798053 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:15:53.798033    4892 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-btd8n\" (UniqueName: \"kubernetes.io/projected/33ce6ceb-02a9-4852-9948-f02e2b39e32f-kube-api-access-btd8n\") pod \"network-check-source-6f78d59f99-2trjq\" (UID: \"33ce6ceb-02a9-4852-9948-f02e2b39e32f\") " pod="openshift-network-diagnostics/network-check-source-6f78d59f99-2trjq"
Oct 30 19:15:53.798445 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:15:53.798428    4892 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/e3206ddf-35de-4b7c-b979-682f5f666a34-bound-sa-token\") pod \"image-registry-6fcb4848df-8l8cp\" (UID: \"e3206ddf-35de-4b7c-b979-682f5f666a34\") " pod="openshift-image-registry/image-registry-6fcb4848df-8l8cp"
Oct 30 19:15:53.798605 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:15:53.798583    4892 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/08ae8ede-9a96-4a84-8ec8-8d364fd476a3-ovn-node-metrics-cert\") pod \"ovnkube-node-5lwkx\" (UID: \"08ae8ede-9a96-4a84-8ec8-8d364fd476a3\") " pod="openshift-ovn-kubernetes/ovnkube-node-5lwkx"
Oct 30 19:15:53.798870 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:15:53.798852    4892 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rfxzq\" (UniqueName: \"kubernetes.io/projected/4ba601a6-2a44-4408-8c5b-2c29fd096fed-kube-api-access-rfxzq\") pod \"service-ca-operator-544d749c8c-nbfv2\" (UID: \"4ba601a6-2a44-4408-8c5b-2c29fd096fed\") " pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-nbfv2"
Oct 30 19:15:53.798927 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:15:53.798880    4892 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wt5rq\" (UniqueName: \"kubernetes.io/projected/56274b23-2400-43d4-9686-c928e20d47de-kube-api-access-wt5rq\") pod \"cluster-samples-operator-6ddb6c8b94-np94b\" (UID: \"56274b23-2400-43d4-9686-c928e20d47de\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-np94b"
Oct 30 19:15:53.799140 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:15:53.799126    4892 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-txbm4\" (UniqueName: \"kubernetes.io/projected/87da348d-b1ee-4cd3-806d-0560bff3c6f2-kube-api-access-txbm4\") pod \"kas-connection-checker-685895c7c9-p8kjs\" (UID: \"87da348d-b1ee-4cd3-806d-0560bff3c6f2\") " pod="kube-system/kas-connection-checker-685895c7c9-p8kjs"
Oct 30 19:15:53.799222 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:15:53.799140    4892 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wmkq8\" (UniqueName: \"kubernetes.io/projected/d260d4d7-b4aa-4f37-92b5-e75b22e13a3b-kube-api-access-wmkq8\") pod \"kas-connection-checker-685895c7c9-tvmks\" (UID: \"d260d4d7-b4aa-4f37-92b5-e75b22e13a3b\") " pod="kube-system/kas-connection-checker-685895c7c9-tvmks"
Oct 30 19:15:53.799222 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:15:53.799142    4892 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8s7g5\" (UniqueName: \"kubernetes.io/projected/87082646-75c3-43b5-864b-3478e9a7373d-kube-api-access-8s7g5\") pod \"router-default-776df59447-8nrhh\" (UID: \"87082646-75c3-43b5-864b-3478e9a7373d\") " pod="openshift-ingress/router-default-776df59447-8nrhh"
Oct 30 19:15:53.802629 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:15:53.802605    4892 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"openshift-service-ca.crt\""
Oct 30 19:15:53.802775 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:15:53.802757    4892 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-console\"/\"networking-console-plugin\""
Oct 30 19:15:53.802842 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:15:53.802823    4892 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"openshift-service-ca.crt\""
Oct 30 19:15:53.802894 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:15:53.802871    4892 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"konnectivity-ca-bundle\""
Oct 30 19:15:53.802955 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:15:53.802897    4892 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"default-cni-sysctl-allowlist\""
Oct 30 19:15:53.802955 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:15:53.802910    4892 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"kube-root-ca.crt\""
Oct 30 19:15:53.802955 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:15:53.802927    4892 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"openshift-service-ca.crt\""
Oct 30 19:15:53.802955 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:15:53.802946    4892 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"multus-daemon-config\""
Oct 30 19:15:53.802955 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:15:53.802870    4892 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"openshift-service-ca.crt\""
Oct 30 19:15:53.803195 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:15:53.802910    4892 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"whereabouts-flatfile-config\""
Oct 30 19:15:53.803195 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:15:53.802957    4892 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/2c7c7060-def6-4737-8fa9-6889309b7463-nginx-conf\") pod \"networking-console-plugin-86c478bb88-h4twn\" (UID: \"2c7c7060-def6-4737-8fa9-6889309b7463\") " pod="openshift-network-console/networking-console-plugin-86c478bb88-h4twn"
Oct 30 19:15:53.803195 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:15:53.803017    4892 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"kube-root-ca.crt\""
Oct 30 19:15:53.803195 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:15:53.803035    4892 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"kube-root-ca.crt\""
Oct 30 19:15:53.803195 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:15:53.803036    4892 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-rbac-proxy\""
Oct 30 19:15:53.803195 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:15:53.803038    4892 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"kube-root-ca.crt\""
Oct 30 19:15:53.803195 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:15:53.803077    4892 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"kube-root-ca.crt\""
Oct 30 19:15:53.803195 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:15:53.803090    4892 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"image-registry-certificates\""
Oct 30 19:15:53.803195 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:15:53.803080    4892 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"openshift-service-ca.crt\""
Oct 30 19:15:53.803195 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:15:53.803017    4892 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"openshift-service-ca.crt\""
Oct 30 19:15:53.803195 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:15:53.803162    4892 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"cni-copy-resources\""
Oct 30 19:15:53.803195 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:15:53.803180    4892 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-config\""
Oct 30 19:15:53.803669 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:15:53.803190    4892 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"openshift-service-ca.crt\""
Oct 30 19:15:53.803669 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:15:53.803219    4892 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"kube-root-ca.crt\""
Oct 30 19:15:53.803669 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:15:53.803226    4892 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"env-overrides\""
Oct 30 19:15:53.803669 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:15:53.803209    4892 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"dns-default\""
Oct 30 19:15:53.803669 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:15:53.803196    4892 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"openshift-service-ca.crt\""
Oct 30 19:15:53.803669 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:15:53.803324    4892 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"kube-root-ca.crt\""
Oct 30 19:15:53.803669 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:15:53.803406    4892 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"kube-root-ca.crt\""
Oct 30 19:15:53.803669 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:15:53.803550    4892 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-j7bln\" (UniqueName: \"kubernetes.io/projected/e446ca63-5885-4c66-9cb7-da39afbf742c-kube-api-access-j7bln\") pod \"cluster-monitoring-operator-884576c57-wkldq\" (UID: \"e446ca63-5885-4c66-9cb7-da39afbf742c\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-wkldq"
Oct 30 19:15:53.803669 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:15:53.803576    4892 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-script\""
Oct 30 19:15:53.803669 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:15:53.803631    4892 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/0e179af5-9f52-4941-b4f5-5e3b8789da11-bound-sa-token\") pod \"image-registry-d97c68fd4-htsl8\" (UID: \"0e179af5-9f52-4941-b4f5-5e3b8789da11\") " pod="openshift-image-registry/image-registry-d97c68fd4-htsl8"
Oct 30 19:15:53.804190 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:15:53.803686    4892 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4rcdn\" (UniqueName: \"kubernetes.io/projected/bab0abf9-46d4-408e-9aa0-afa7279cd9d1-kube-api-access-4rcdn\") pod \"multus-additional-cni-plugins-4pvgg\" (UID: \"bab0abf9-46d4-408e-9aa0-afa7279cd9d1\") " pod="openshift-multus/multus-additional-cni-plugins-4pvgg"
Oct 30 19:15:53.804242 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:15:53.804209    4892 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-script-lib\""
Oct 30 19:15:53.804405 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:15:53.804383    4892 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-k62g9\" (UniqueName: \"kubernetes.io/projected/ba2c5571-23a3-46b6-a36d-7e8810a66261-kube-api-access-k62g9\") pod \"kas-connection-checker-685895c7c9-9h85f\" (UID: \"ba2c5571-23a3-46b6-a36d-7e8810a66261\") " pod="kube-system/kas-connection-checker-685895c7c9-9h85f"
Oct 30 19:15:53.804687 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:15:53.804666    4892 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bzw4s\" (UniqueName: \"kubernetes.io/projected/4ca01b49-9819-4a90-a851-7553ad568679-kube-api-access-bzw4s\") pod \"network-check-target-c24j9\" (UID: \"4ca01b49-9819-4a90-a851-7553ad568679\") " pod="openshift-network-diagnostics/network-check-target-c24j9"
Oct 30 19:15:53.805332 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:15:53.805313    4892 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"trusted-ca\""
Oct 30 19:15:53.805512 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:15:53.805493    4892 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/e0221987-8e5e-40d1-8842-c97993f08c6d-iptables-alerter-script\") pod \"iptables-alerter-wnn66\" (UID: \"e0221987-8e5e-40d1-8842-c97993f08c6d\") " pod="openshift-network-operator/iptables-alerter-wnn66"
Oct 30 19:15:53.805581 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:15:53.805566    4892 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/e3206ddf-35de-4b7c-b979-682f5f666a34-trusted-ca\") pod \"image-registry-6fcb4848df-8l8cp\" (UID: \"e3206ddf-35de-4b7c-b979-682f5f666a34\") " pod="openshift-image-registry/image-registry-6fcb4848df-8l8cp"
Oct 30 19:15:53.806236 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:15:53.806219    4892 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/08ae8ede-9a96-4a84-8ec8-8d364fd476a3-env-overrides\") pod \"ovnkube-node-5lwkx\" (UID: \"08ae8ede-9a96-4a84-8ec8-8d364fd476a3\") " pod="openshift-ovn-kubernetes/ovnkube-node-5lwkx"
Oct 30 19:15:53.806296 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:15:53.806236    4892 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/08ae8ede-9a96-4a84-8ec8-8d364fd476a3-ovnkube-config\") pod \"ovnkube-node-5lwkx\" (UID: \"08ae8ede-9a96-4a84-8ec8-8d364fd476a3\") " pod="openshift-ovn-kubernetes/ovnkube-node-5lwkx"
Oct 30 19:15:53.807804 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:15:53.807377    4892 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/e3206ddf-35de-4b7c-b979-682f5f666a34-registry-certificates\") pod \"image-registry-6fcb4848df-8l8cp\" (UID: \"e3206ddf-35de-4b7c-b979-682f5f666a34\") " pod="openshift-image-registry/image-registry-6fcb4848df-8l8cp"
Oct 30 19:15:53.807804 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:15:53.807501    4892 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/bab0abf9-46d4-408e-9aa0-afa7279cd9d1-cni-binary-copy\") pod \"multus-additional-cni-plugins-4pvgg\" (UID: \"bab0abf9-46d4-408e-9aa0-afa7279cd9d1\") " pod="openshift-multus/multus-additional-cni-plugins-4pvgg"
Oct 30 19:15:53.807804 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:15:53.807518    4892 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/dccdaf98-aac7-4756-b93f-22f62a286a1f-cni-binary-copy\") pod \"multus-jvbw5\" (UID: \"dccdaf98-aac7-4756-b93f-22f62a286a1f\") " pod="openshift-multus/multus-jvbw5"
Oct 30 19:15:53.807804 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:15:53.807522    4892 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/dccdaf98-aac7-4756-b93f-22f62a286a1f-multus-daemon-config\") pod \"multus-jvbw5\" (UID: \"dccdaf98-aac7-4756-b93f-22f62a286a1f\") " pod="openshift-multus/multus-jvbw5"
Oct 30 19:15:53.807804 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:15:53.807643    4892 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-v7kvw\" (UniqueName: \"kubernetes.io/projected/287ba08f-7b9f-4609-8e35-832c0879c44c-kube-api-access-v7kvw\") pod \"network-metrics-daemon-822gp\" (UID: \"287ba08f-7b9f-4609-8e35-832c0879c44c\") " pod="openshift-multus/network-metrics-daemon-822gp"
Oct 30 19:15:53.807804 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:15:53.807770    4892 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6bqfr\" (UniqueName: \"kubernetes.io/projected/dccdaf98-aac7-4756-b93f-22f62a286a1f-kube-api-access-6bqfr\") pod \"multus-jvbw5\" (UID: \"dccdaf98-aac7-4756-b93f-22f62a286a1f\") " pod="openshift-multus/multus-jvbw5"
Oct 30 19:15:53.807804 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:15:53.807777    4892 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bnxm6\" (UniqueName: \"kubernetes.io/projected/08ae8ede-9a96-4a84-8ec8-8d364fd476a3-kube-api-access-bnxm6\") pod \"ovnkube-node-5lwkx\" (UID: \"08ae8ede-9a96-4a84-8ec8-8d364fd476a3\") " pod="openshift-ovn-kubernetes/ovnkube-node-5lwkx"
Oct 30 19:15:53.807804 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:15:53.807785    4892 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/cdb750aa-dd7e-4e2f-8297-9028acc0f559-serviceca\") pod \"node-ca-kgfvh\" (UID: \"cdb750aa-dd7e-4e2f-8297-9028acc0f559\") " pod="openshift-image-registry/node-ca-kgfvh"
Oct 30 19:15:53.807804 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:15:53.807806    4892 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gdq6k\" (UniqueName: \"kubernetes.io/projected/39bb8c16-8282-442a-841b-0056baae014d-kube-api-access-gdq6k\") pod \"tuned-2fv9z\" (UID: \"39bb8c16-8282-442a-841b-0056baae014d\") " pod="openshift-cluster-node-tuning-operator/tuned-2fv9z"
Oct 30 19:15:53.808837 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:15:53.808821    4892 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/bab0abf9-46d4-408e-9aa0-afa7279cd9d1-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-4pvgg\" (UID: \"bab0abf9-46d4-408e-9aa0-afa7279cd9d1\") " pod="openshift-multus/multus-additional-cni-plugins-4pvgg"
Oct 30 19:15:53.808955 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:15:53.808940    4892 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rvrch\" (UniqueName: \"kubernetes.io/projected/cdb750aa-dd7e-4e2f-8297-9028acc0f559-kube-api-access-rvrch\") pod \"node-ca-kgfvh\" (UID: \"cdb750aa-dd7e-4e2f-8297-9028acc0f559\") " pod="openshift-image-registry/node-ca-kgfvh"
Oct 30 19:15:53.809148 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:15:53.809128    4892 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-w8zg9\" (UniqueName: \"kubernetes.io/projected/48386105-d79b-4f92-baa0-f330b9658ad0-kube-api-access-w8zg9\") pod \"node-resolver-g9xd6\" (UID: \"48386105-d79b-4f92-baa0-f330b9658ad0\") " pod="openshift-dns/node-resolver-g9xd6"
Oct 30 19:15:53.809222 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:15:53.809209    4892 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fmnd4\" (UniqueName: \"kubernetes.io/projected/e0221987-8e5e-40d1-8842-c97993f08c6d-kube-api-access-fmnd4\") pod \"iptables-alerter-wnn66\" (UID: \"e0221987-8e5e-40d1-8842-c97993f08c6d\") " pod="openshift-network-operator/iptables-alerter-wnn66"
Oct 30 19:15:53.809267 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:15:53.809233    4892 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/0e179af5-9f52-4941-b4f5-5e3b8789da11-trusted-ca\") pod \"image-registry-d97c68fd4-htsl8\" (UID: \"0e179af5-9f52-4941-b4f5-5e3b8789da11\") " pod="openshift-image-registry/image-registry-d97c68fd4-htsl8"
Oct 30 19:15:53.809518 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:15:53.809497    4892 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zj4df\" (UniqueName: \"kubernetes.io/projected/0e179af5-9f52-4941-b4f5-5e3b8789da11-kube-api-access-zj4df\") pod \"image-registry-d97c68fd4-htsl8\" (UID: \"0e179af5-9f52-4941-b4f5-5e3b8789da11\") " pod="openshift-image-registry/image-registry-d97c68fd4-htsl8"
Oct 30 19:15:53.809615 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:15:53.809603    4892 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/0e179af5-9f52-4941-b4f5-5e3b8789da11-registry-certificates\") pod \"image-registry-d97c68fd4-htsl8\" (UID: \"0e179af5-9f52-4941-b4f5-5e3b8789da11\") " pod="openshift-image-registry/image-registry-d97c68fd4-htsl8"
Oct 30 19:15:53.811663 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:15:53.811649    4892 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/08ae8ede-9a96-4a84-8ec8-8d364fd476a3-ovnkube-script-lib\") pod \"ovnkube-node-5lwkx\" (UID: \"08ae8ede-9a96-4a84-8ec8-8d364fd476a3\") " pod="openshift-ovn-kubernetes/ovnkube-node-5lwkx"
Oct 30 19:15:53.812056 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:15:53.812037    4892 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-w9h4f\" (UniqueName: \"kubernetes.io/projected/e3206ddf-35de-4b7c-b979-682f5f666a34-kube-api-access-w9h4f\") pod \"image-registry-6fcb4848df-8l8cp\" (UID: \"e3206ddf-35de-4b7c-b979-682f5f666a34\") " pod="openshift-image-registry/image-registry-6fcb4848df-8l8cp"
Oct 30 19:15:53.813840 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:15:53.813826    4892 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/bab0abf9-46d4-408e-9aa0-afa7279cd9d1-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-4pvgg\" (UID: \"bab0abf9-46d4-408e-9aa0-afa7279cd9d1\") " pod="openshift-multus/multus-additional-cni-plugins-4pvgg"
Oct 30 19:15:53.843485 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:15:53.843464    4892 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/c3780b96-955e-4728-a2d0-b7ec30d0f6f3-kubelet-dir\") pod \"aws-ebs-csi-driver-node-qgv98\" (UID: \"c3780b96-955e-4728-a2d0-b7ec30d0f6f3\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-qgv98"
Oct 30 19:15:53.843537 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:15:53.843491    4892 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/bab0abf9-46d4-408e-9aa0-afa7279cd9d1-cnibin\") pod \"multus-additional-cni-plugins-4pvgg\" (UID: \"bab0abf9-46d4-408e-9aa0-afa7279cd9d1\") " pod="openshift-multus/multus-additional-cni-plugins-4pvgg"
Oct 30 19:15:53.843537 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:15:53.843508    4892 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/fc4ac297-f510-4439-80e2-487056d3bf06-crio-socket\") pod \"insights-runtime-extractor-jrm87\" (UID: \"fc4ac297-f510-4439-80e2-487056d3bf06\") " pod="openshift-insights/insights-runtime-extractor-jrm87"
Oct 30 19:15:53.843537 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:15:53.843528    4892 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/fc4ac297-f510-4439-80e2-487056d3bf06-data-volume\") pod \"insights-runtime-extractor-jrm87\" (UID: \"fc4ac297-f510-4439-80e2-487056d3bf06\") " pod="openshift-insights/insights-runtime-extractor-jrm87"
Oct 30 19:15:53.843717 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:15:53.843551    4892 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/39bb8c16-8282-442a-841b-0056baae014d-etc-kubernetes\") pod \"tuned-2fv9z\" (UID: \"39bb8c16-8282-442a-841b-0056baae014d\") " pod="openshift-cluster-node-tuning-operator/tuned-2fv9z"
Oct 30 19:15:53.843717 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:15:53.843557    4892 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/bab0abf9-46d4-408e-9aa0-afa7279cd9d1-cnibin\") pod \"multus-additional-cni-plugins-4pvgg\" (UID: \"bab0abf9-46d4-408e-9aa0-afa7279cd9d1\") " pod="openshift-multus/multus-additional-cni-plugins-4pvgg"
Oct 30 19:15:53.843717 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:15:53.843596    4892 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/39bb8c16-8282-442a-841b-0056baae014d-etc-kubernetes\") pod \"tuned-2fv9z\" (UID: \"39bb8c16-8282-442a-841b-0056baae014d\") " pod="openshift-cluster-node-tuning-operator/tuned-2fv9z"
Oct 30 19:15:53.843717 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:15:53.843605    4892 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/87082646-75c3-43b5-864b-3478e9a7373d-service-ca-bundle\") pod \"router-default-776df59447-8nrhh\" (UID: \"87082646-75c3-43b5-864b-3478e9a7373d\") " pod="openshift-ingress/router-default-776df59447-8nrhh"
Oct 30 19:15:53.843717 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:15:53.843630    4892 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/fc4ac297-f510-4439-80e2-487056d3bf06-data-volume\") pod \"insights-runtime-extractor-jrm87\" (UID: \"fc4ac297-f510-4439-80e2-487056d3bf06\") " pod="openshift-insights/insights-runtime-extractor-jrm87"
Oct 30 19:15:53.843900 ip-10-0-132-82 kubenswrapper[4892]: E1030 19:15:53.843731    4892 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/87082646-75c3-43b5-864b-3478e9a7373d-service-ca-bundle podName:87082646-75c3-43b5-864b-3478e9a7373d nodeName:}" failed. No retries permitted until 2026-10-30 19:15:54.343711389 +0000 UTC m=+1.914176983 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/87082646-75c3-43b5-864b-3478e9a7373d-service-ca-bundle") pod "router-default-776df59447-8nrhh" (UID: "87082646-75c3-43b5-864b-3478e9a7373d") : configmap references non-existent config key: service-ca.crt
Oct 30 19:15:53.843900 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:15:53.843752    4892 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/2c7c7060-def6-4737-8fa9-6889309b7463-networking-console-plugin-cert\") pod \"networking-console-plugin-86c478bb88-h4twn\" (UID: \"2c7c7060-def6-4737-8fa9-6889309b7463\") " pod="openshift-network-console/networking-console-plugin-86c478bb88-h4twn"
Oct 30 19:15:53.843900 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:15:53.843770    4892 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/dccdaf98-aac7-4756-b93f-22f62a286a1f-etc-kubernetes\") pod \"multus-jvbw5\" (UID: \"dccdaf98-aac7-4756-b93f-22f62a286a1f\") " pod="openshift-multus/multus-jvbw5"
Oct 30 19:15:53.843900 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:15:53.843787    4892 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/fc4ac297-f510-4439-80e2-487056d3bf06-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-jrm87\" (UID: \"fc4ac297-f510-4439-80e2-487056d3bf06\") " pod="openshift-insights/insights-runtime-extractor-jrm87"
Oct 30 19:15:53.843900 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:15:53.843805    4892 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/dccdaf98-aac7-4756-b93f-22f62a286a1f-cnibin\") pod \"multus-jvbw5\" (UID: \"dccdaf98-aac7-4756-b93f-22f62a286a1f\") " pod="openshift-multus/multus-jvbw5"
Oct 30 19:15:53.843900 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:15:53.843826    4892 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/dccdaf98-aac7-4756-b93f-22f62a286a1f-hostroot\") pod \"multus-jvbw5\" (UID: \"dccdaf98-aac7-4756-b93f-22f62a286a1f\") " pod="openshift-multus/multus-jvbw5"
Oct 30 19:15:53.843900 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:15:53.843844    4892 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/dccdaf98-aac7-4756-b93f-22f62a286a1f-cnibin\") pod \"multus-jvbw5\" (UID: \"dccdaf98-aac7-4756-b93f-22f62a286a1f\") " pod="openshift-multus/multus-jvbw5"
Oct 30 19:15:53.843900 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:15:53.843852    4892 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/af36f4d4-f985-4937-a9f7-9884936f3690-konnectivity-ca\") pod \"konnectivity-agent-qnm56\" (UID: \"af36f4d4-f985-4937-a9f7-9884936f3690\") " pod="kube-system/konnectivity-agent-qnm56"
Oct 30 19:15:53.843900 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:15:53.843868    4892 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/dccdaf98-aac7-4756-b93f-22f62a286a1f-etc-kubernetes\") pod \"multus-jvbw5\" (UID: \"dccdaf98-aac7-4756-b93f-22f62a286a1f\") " pod="openshift-multus/multus-jvbw5"
Oct 30 19:15:53.843900 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:15:53.843869    4892 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/08ae8ede-9a96-4a84-8ec8-8d364fd476a3-run-systemd\") pod \"ovnkube-node-5lwkx\" (UID: \"08ae8ede-9a96-4a84-8ec8-8d364fd476a3\") " pod="openshift-ovn-kubernetes/ovnkube-node-5lwkx"
Oct 30 19:15:53.843900 ip-10-0-132-82 kubenswrapper[4892]: E1030 19:15:53.843886    4892 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found
Oct 30 19:15:53.843900 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:15:53.843891    4892 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/dccdaf98-aac7-4756-b93f-22f62a286a1f-hostroot\") pod \"multus-jvbw5\" (UID: \"dccdaf98-aac7-4756-b93f-22f62a286a1f\") " pod="openshift-multus/multus-jvbw5"
Oct 30 19:15:53.843900 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:15:53.843907    4892 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/08ae8ede-9a96-4a84-8ec8-8d364fd476a3-run-systemd\") pod \"ovnkube-node-5lwkx\" (UID: \"08ae8ede-9a96-4a84-8ec8-8d364fd476a3\") " pod="openshift-ovn-kubernetes/ovnkube-node-5lwkx"
Oct 30 19:15:53.844535 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:15:53.843923    4892 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/08ae8ede-9a96-4a84-8ec8-8d364fd476a3-log-socket\") pod \"ovnkube-node-5lwkx\" (UID: \"08ae8ede-9a96-4a84-8ec8-8d364fd476a3\") " pod="openshift-ovn-kubernetes/ovnkube-node-5lwkx"
Oct 30 19:15:53.844535 ip-10-0-132-82 kubenswrapper[4892]: E1030 19:15:53.843935    4892 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/2c7c7060-def6-4737-8fa9-6889309b7463-networking-console-plugin-cert podName:2c7c7060-def6-4737-8fa9-6889309b7463 nodeName:}" failed. No retries permitted until 2026-10-30 19:15:54.343924157 +0000 UTC m=+1.914389734 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/2c7c7060-def6-4737-8fa9-6889309b7463-networking-console-plugin-cert") pod "networking-console-plugin-86c478bb88-h4twn" (UID: "2c7c7060-def6-4737-8fa9-6889309b7463") : secret "networking-console-plugin-cert" not found
Oct 30 19:15:53.844535 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:15:53.843954    4892 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/39bb8c16-8282-442a-841b-0056baae014d-etc-modprobe-d\") pod \"tuned-2fv9z\" (UID: \"39bb8c16-8282-442a-841b-0056baae014d\") " pod="openshift-cluster-node-tuning-operator/tuned-2fv9z"
Oct 30 19:15:53.844535 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:15:53.843986    4892 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/fc4ac297-f510-4439-80e2-487056d3bf06-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-jrm87\" (UID: \"fc4ac297-f510-4439-80e2-487056d3bf06\") " pod="openshift-insights/insights-runtime-extractor-jrm87"
Oct 30 19:15:53.844535 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:15:53.843990    4892 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/08ae8ede-9a96-4a84-8ec8-8d364fd476a3-log-socket\") pod \"ovnkube-node-5lwkx\" (UID: \"08ae8ede-9a96-4a84-8ec8-8d364fd476a3\") " pod="openshift-ovn-kubernetes/ovnkube-node-5lwkx"
Oct 30 19:15:53.844535 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:15:53.844002    4892 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/39bb8c16-8282-442a-841b-0056baae014d-sys\") pod \"tuned-2fv9z\" (UID: \"39bb8c16-8282-442a-841b-0056baae014d\") " pod="openshift-cluster-node-tuning-operator/tuned-2fv9z"
Oct 30 19:15:53.844535 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:15:53.844017    4892 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/dccdaf98-aac7-4756-b93f-22f62a286a1f-multus-conf-dir\") pod \"multus-jvbw5\" (UID: \"dccdaf98-aac7-4756-b93f-22f62a286a1f\") " pod="openshift-multus/multus-jvbw5"
Oct 30 19:15:53.844535 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:15:53.844044    4892 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/dccdaf98-aac7-4756-b93f-22f62a286a1f-multus-conf-dir\") pod \"multus-jvbw5\" (UID: \"dccdaf98-aac7-4756-b93f-22f62a286a1f\") " pod="openshift-multus/multus-jvbw5"
Oct 30 19:15:53.844535 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:15:53.844054    4892 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/39bb8c16-8282-442a-841b-0056baae014d-etc-modprobe-d\") pod \"tuned-2fv9z\" (UID: \"39bb8c16-8282-442a-841b-0056baae014d\") " pod="openshift-cluster-node-tuning-operator/tuned-2fv9z"
Oct 30 19:15:53.844535 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:15:53.844063    4892 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/39bb8c16-8282-442a-841b-0056baae014d-sys\") pod \"tuned-2fv9z\" (UID: \"39bb8c16-8282-442a-841b-0056baae014d\") " pod="openshift-cluster-node-tuning-operator/tuned-2fv9z"
Oct 30 19:15:53.844535 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:15:53.844093    4892 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/af36f4d4-f985-4937-a9f7-9884936f3690-konnectivity-ca\") pod \"konnectivity-agent-qnm56\" (UID: \"af36f4d4-f985-4937-a9f7-9884936f3690\") " pod="kube-system/konnectivity-agent-qnm56"
Oct 30 19:15:53.844535 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:15:53.844099    4892 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/fade00ba-24a7-41c9-93f7-23e40971874a-config-volume\") pod \"dns-default-bb9cj\" (UID: \"fade00ba-24a7-41c9-93f7-23e40971874a\") " pod="openshift-dns/dns-default-bb9cj"
Oct 30 19:15:53.844535 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:15:53.844137    4892 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/08ae8ede-9a96-4a84-8ec8-8d364fd476a3-run-openvswitch\") pod \"ovnkube-node-5lwkx\" (UID: \"08ae8ede-9a96-4a84-8ec8-8d364fd476a3\") " pod="openshift-ovn-kubernetes/ovnkube-node-5lwkx"
Oct 30 19:15:53.844535 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:15:53.844154    4892 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/08ae8ede-9a96-4a84-8ec8-8d364fd476a3-node-log\") pod \"ovnkube-node-5lwkx\" (UID: \"08ae8ede-9a96-4a84-8ec8-8d364fd476a3\") " pod="openshift-ovn-kubernetes/ovnkube-node-5lwkx"
Oct 30 19:15:53.844535 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:15:53.844162    4892 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/08ae8ede-9a96-4a84-8ec8-8d364fd476a3-run-openvswitch\") pod \"ovnkube-node-5lwkx\" (UID: \"08ae8ede-9a96-4a84-8ec8-8d364fd476a3\") " pod="openshift-ovn-kubernetes/ovnkube-node-5lwkx"
Oct 30 19:15:53.844535 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:15:53.844167    4892 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/dccdaf98-aac7-4756-b93f-22f62a286a1f-multus-socket-dir-parent\") pod \"multus-jvbw5\" (UID: \"dccdaf98-aac7-4756-b93f-22f62a286a1f\") " pod="openshift-multus/multus-jvbw5"
Oct 30 19:15:53.844535 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:15:53.844197    4892 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/dccdaf98-aac7-4756-b93f-22f62a286a1f-multus-socket-dir-parent\") pod \"multus-jvbw5\" (UID: \"dccdaf98-aac7-4756-b93f-22f62a286a1f\") " pod="openshift-multus/multus-jvbw5"
Oct 30 19:15:53.844535 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:15:53.844196    4892 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/fade00ba-24a7-41c9-93f7-23e40971874a-tmp-dir\") pod \"dns-default-bb9cj\" (UID: \"fade00ba-24a7-41c9-93f7-23e40971874a\") " pod="openshift-dns/dns-default-bb9cj"
Oct 30 19:15:53.844535 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:15:53.844221    4892 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/08ae8ede-9a96-4a84-8ec8-8d364fd476a3-node-log\") pod \"ovnkube-node-5lwkx\" (UID: \"08ae8ede-9a96-4a84-8ec8-8d364fd476a3\") " pod="openshift-ovn-kubernetes/ovnkube-node-5lwkx"
Oct 30 19:15:53.844535 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:15:53.844235    4892 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-j4wnm\" (UniqueName: \"kubernetes.io/projected/f86c0e38-8144-42fc-b622-25602ada113f-kube-api-access-j4wnm\") pod \"ingress-canary-8rwjm\" (UID: \"f86c0e38-8144-42fc-b622-25602ada113f\") " pod="openshift-ingress-canary/ingress-canary-8rwjm"
Oct 30 19:15:53.844535 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:15:53.844257    4892 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/fade00ba-24a7-41c9-93f7-23e40971874a-tmp-dir\") pod \"dns-default-bb9cj\" (UID: \"fade00ba-24a7-41c9-93f7-23e40971874a\") " pod="openshift-dns/dns-default-bb9cj"
Oct 30 19:15:53.844535 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:15:53.844263    4892 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/af36f4d4-f985-4937-a9f7-9884936f3690-agent-certs\") pod \"konnectivity-agent-qnm56\" (UID: \"af36f4d4-f985-4937-a9f7-9884936f3690\") " pod="kube-system/konnectivity-agent-qnm56"
Oct 30 19:15:53.844535 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:15:53.844264    4892 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/fade00ba-24a7-41c9-93f7-23e40971874a-config-volume\") pod \"dns-default-bb9cj\" (UID: \"fade00ba-24a7-41c9-93f7-23e40971874a\") " pod="openshift-dns/dns-default-bb9cj"
Oct 30 19:15:53.844535 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:15:53.844295    4892 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/c3780b96-955e-4728-a2d0-b7ec30d0f6f3-socket-dir\") pod \"aws-ebs-csi-driver-node-qgv98\" (UID: \"c3780b96-955e-4728-a2d0-b7ec30d0f6f3\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-qgv98"
Oct 30 19:15:53.844535 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:15:53.844406    4892 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/08ae8ede-9a96-4a84-8ec8-8d364fd476a3-run-ovn\") pod \"ovnkube-node-5lwkx\" (UID: \"08ae8ede-9a96-4a84-8ec8-8d364fd476a3\") " pod="openshift-ovn-kubernetes/ovnkube-node-5lwkx"
Oct 30 19:15:53.844535 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:15:53.844426    4892 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/08ae8ede-9a96-4a84-8ec8-8d364fd476a3-run-ovn\") pod \"ovnkube-node-5lwkx\" (UID: \"08ae8ede-9a96-4a84-8ec8-8d364fd476a3\") " pod="openshift-ovn-kubernetes/ovnkube-node-5lwkx"
Oct 30 19:15:53.844535 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:15:53.844438    4892 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/87082646-75c3-43b5-864b-3478e9a7373d-metrics-certs\") pod \"router-default-776df59447-8nrhh\" (UID: \"87082646-75c3-43b5-864b-3478e9a7373d\") " pod="openshift-ingress/router-default-776df59447-8nrhh"
Oct 30 19:15:53.844535 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:15:53.844461    4892 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/bab0abf9-46d4-408e-9aa0-afa7279cd9d1-system-cni-dir\") pod \"multus-additional-cni-plugins-4pvgg\" (UID: \"bab0abf9-46d4-408e-9aa0-afa7279cd9d1\") " pod="openshift-multus/multus-additional-cni-plugins-4pvgg"
Oct 30 19:15:53.844535 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:15:53.844483    4892 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/e446ca63-5885-4c66-9cb7-da39afbf742c-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-884576c57-wkldq\" (UID: \"e446ca63-5885-4c66-9cb7-da39afbf742c\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-wkldq"
Oct 30 19:15:53.844535 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:15:53.844514    4892 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/bab0abf9-46d4-408e-9aa0-afa7279cd9d1-system-cni-dir\") pod \"multus-additional-cni-plugins-4pvgg\" (UID: \"bab0abf9-46d4-408e-9aa0-afa7279cd9d1\") " pod="openshift-multus/multus-additional-cni-plugins-4pvgg"
Oct 30 19:15:53.844535 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:15:53.844513    4892 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/af36f4d4-f985-4937-a9f7-9884936f3690-agent-certs\") pod \"konnectivity-agent-qnm56\" (UID: \"af36f4d4-f985-4937-a9f7-9884936f3690\") " pod="kube-system/konnectivity-agent-qnm56"
Oct 30 19:15:53.844535 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:15:53.844528    4892 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/08ae8ede-9a96-4a84-8ec8-8d364fd476a3-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-5lwkx\" (UID: \"08ae8ede-9a96-4a84-8ec8-8d364fd476a3\") " pod="openshift-ovn-kubernetes/ovnkube-node-5lwkx"
Oct 30 19:15:53.844535 ip-10-0-132-82 kubenswrapper[4892]: E1030 19:15:53.844545    4892 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found
Oct 30 19:15:53.844535 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:15:53.844553    4892 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/08ae8ede-9a96-4a84-8ec8-8d364fd476a3-var-lib-openvswitch\") pod \"ovnkube-node-5lwkx\" (UID: \"08ae8ede-9a96-4a84-8ec8-8d364fd476a3\") " pod="openshift-ovn-kubernetes/ovnkube-node-5lwkx"
Oct 30 19:15:53.845716 ip-10-0-132-82 kubenswrapper[4892]: E1030 19:15:53.844571    4892 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found
Oct 30 19:15:53.845716 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:15:53.844578    4892 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/0e179af5-9f52-4941-b4f5-5e3b8789da11-registry-tls\") pod \"image-registry-d97c68fd4-htsl8\" (UID: \"0e179af5-9f52-4941-b4f5-5e3b8789da11\") " pod="openshift-image-registry/image-registry-d97c68fd4-htsl8"
Oct 30 19:15:53.845716 ip-10-0-132-82 kubenswrapper[4892]: E1030 19:15:53.844590    4892 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/87082646-75c3-43b5-864b-3478e9a7373d-metrics-certs podName:87082646-75c3-43b5-864b-3478e9a7373d nodeName:}" failed. No retries permitted until 2026-10-30 19:15:54.34457625 +0000 UTC m=+1.915041852 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/87082646-75c3-43b5-864b-3478e9a7373d-metrics-certs") pod "router-default-776df59447-8nrhh" (UID: "87082646-75c3-43b5-864b-3478e9a7373d") : secret "router-metrics-certs-default" not found
Oct 30 19:15:53.845716 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:15:53.844599    4892 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/08ae8ede-9a96-4a84-8ec8-8d364fd476a3-var-lib-openvswitch\") pod \"ovnkube-node-5lwkx\" (UID: \"08ae8ede-9a96-4a84-8ec8-8d364fd476a3\") " pod="openshift-ovn-kubernetes/ovnkube-node-5lwkx"
Oct 30 19:15:53.845716 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:15:53.844568    4892 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/08ae8ede-9a96-4a84-8ec8-8d364fd476a3-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-5lwkx\" (UID: \"08ae8ede-9a96-4a84-8ec8-8d364fd476a3\") " pod="openshift-ovn-kubernetes/ovnkube-node-5lwkx"
Oct 30 19:15:53.845716 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:15:53.844616    4892 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/39bb8c16-8282-442a-841b-0056baae014d-etc-sysctl-conf\") pod \"tuned-2fv9z\" (UID: \"39bb8c16-8282-442a-841b-0056baae014d\") " pod="openshift-cluster-node-tuning-operator/tuned-2fv9z"
Oct 30 19:15:53.845716 ip-10-0-132-82 kubenswrapper[4892]: E1030 19:15:53.844638    4892 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Oct 30 19:15:53.845716 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:15:53.844645    4892 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/c16c0a80-c5ce-4f5d-827a-2e3eb2c3cfb8-dbus\") pod \"global-pull-secret-syncer-76rht\" (UID: \"c16c0a80-c5ce-4f5d-827a-2e3eb2c3cfb8\") " pod="kube-system/global-pull-secret-syncer-76rht"
Oct 30 19:15:53.845716 ip-10-0-132-82 kubenswrapper[4892]: E1030 19:15:53.844650    4892 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-d97c68fd4-htsl8: secret "image-registry-tls" not found
Oct 30 19:15:53.845716 ip-10-0-132-82 kubenswrapper[4892]: E1030 19:15:53.844665    4892 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e446ca63-5885-4c66-9cb7-da39afbf742c-cluster-monitoring-operator-tls podName:e446ca63-5885-4c66-9cb7-da39afbf742c nodeName:}" failed. No retries permitted until 2026-10-30 19:15:54.344651198 +0000 UTC m=+1.915116788 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/e446ca63-5885-4c66-9cb7-da39afbf742c-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-884576c57-wkldq" (UID: "e446ca63-5885-4c66-9cb7-da39afbf742c") : secret "cluster-monitoring-operator-tls" not found
Oct 30 19:15:53.845716 ip-10-0-132-82 kubenswrapper[4892]: E1030 19:15:53.844696    4892 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/0e179af5-9f52-4941-b4f5-5e3b8789da11-registry-tls podName:0e179af5-9f52-4941-b4f5-5e3b8789da11 nodeName:}" failed. No retries permitted until 2026-10-30 19:15:54.344688505 +0000 UTC m=+1.915154082 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/0e179af5-9f52-4941-b4f5-5e3b8789da11-registry-tls") pod "image-registry-d97c68fd4-htsl8" (UID: "0e179af5-9f52-4941-b4f5-5e3b8789da11") : secret "image-registry-tls" not found
Oct 30 19:15:53.845716 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:15:53.844704    4892 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/39bb8c16-8282-442a-841b-0056baae014d-etc-sysctl-conf\") pod \"tuned-2fv9z\" (UID: \"39bb8c16-8282-442a-841b-0056baae014d\") " pod="openshift-cluster-node-tuning-operator/tuned-2fv9z"
Oct 30 19:15:53.845716 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:15:53.844722    4892 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/cdb750aa-dd7e-4e2f-8297-9028acc0f559-host\") pod \"node-ca-kgfvh\" (UID: \"cdb750aa-dd7e-4e2f-8297-9028acc0f559\") " pod="openshift-image-registry/node-ca-kgfvh"
Oct 30 19:15:53.845716 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:15:53.844737    4892 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/39bb8c16-8282-442a-841b-0056baae014d-etc-systemd\") pod \"tuned-2fv9z\" (UID: \"39bb8c16-8282-442a-841b-0056baae014d\") " pod="openshift-cluster-node-tuning-operator/tuned-2fv9z"
Oct 30 19:15:53.845716 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:15:53.844758    4892 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/cdb750aa-dd7e-4e2f-8297-9028acc0f559-host\") pod \"node-ca-kgfvh\" (UID: \"cdb750aa-dd7e-4e2f-8297-9028acc0f559\") " pod="openshift-image-registry/node-ca-kgfvh"
Oct 30 19:15:53.845716 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:15:53.844759    4892 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/dccdaf98-aac7-4756-b93f-22f62a286a1f-host-var-lib-cni-multus\") pod \"multus-jvbw5\" (UID: \"dccdaf98-aac7-4756-b93f-22f62a286a1f\") " pod="openshift-multus/multus-jvbw5"
Oct 30 19:15:53.845716 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:15:53.844776    4892 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/dccdaf98-aac7-4756-b93f-22f62a286a1f-host-var-lib-cni-multus\") pod \"multus-jvbw5\" (UID: \"dccdaf98-aac7-4756-b93f-22f62a286a1f\") " pod="openshift-multus/multus-jvbw5"
Oct 30 19:15:53.845716 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:15:53.844798    4892 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/08ae8ede-9a96-4a84-8ec8-8d364fd476a3-etc-openvswitch\") pod \"ovnkube-node-5lwkx\" (UID: \"08ae8ede-9a96-4a84-8ec8-8d364fd476a3\") " pod="openshift-ovn-kubernetes/ovnkube-node-5lwkx"
Oct 30 19:15:53.845716 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:15:53.844818    4892 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/08ae8ede-9a96-4a84-8ec8-8d364fd476a3-host-cni-bin\") pod \"ovnkube-node-5lwkx\" (UID: \"08ae8ede-9a96-4a84-8ec8-8d364fd476a3\") " pod="openshift-ovn-kubernetes/ovnkube-node-5lwkx"
Oct 30 19:15:53.845716 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:15:53.844808    4892 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/39bb8c16-8282-442a-841b-0056baae014d-etc-systemd\") pod \"tuned-2fv9z\" (UID: \"39bb8c16-8282-442a-841b-0056baae014d\") " pod="openshift-cluster-node-tuning-operator/tuned-2fv9z"
Oct 30 19:15:53.845716 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:15:53.844831    4892 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/39bb8c16-8282-442a-841b-0056baae014d-var-lib-kubelet\") pod \"tuned-2fv9z\" (UID: \"39bb8c16-8282-442a-841b-0056baae014d\") " pod="openshift-cluster-node-tuning-operator/tuned-2fv9z"
Oct 30 19:15:53.845716 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:15:53.844854    4892 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/fade00ba-24a7-41c9-93f7-23e40971874a-metrics-tls\") pod \"dns-default-bb9cj\" (UID: \"fade00ba-24a7-41c9-93f7-23e40971874a\") " pod="openshift-dns/dns-default-bb9cj"
Oct 30 19:15:53.845716 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:15:53.844870    4892 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/08ae8ede-9a96-4a84-8ec8-8d364fd476a3-host-cni-bin\") pod \"ovnkube-node-5lwkx\" (UID: \"08ae8ede-9a96-4a84-8ec8-8d364fd476a3\") " pod="openshift-ovn-kubernetes/ovnkube-node-5lwkx"
Oct 30 19:15:53.845716 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:15:53.844877    4892 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/39bb8c16-8282-442a-841b-0056baae014d-run\") pod \"tuned-2fv9z\" (UID: \"39bb8c16-8282-442a-841b-0056baae014d\") " pod="openshift-cluster-node-tuning-operator/tuned-2fv9z"
Oct 30 19:15:53.845716 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:15:53.844899    4892 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/39bb8c16-8282-442a-841b-0056baae014d-var-lib-kubelet\") pod \"tuned-2fv9z\" (UID: \"39bb8c16-8282-442a-841b-0056baae014d\") " pod="openshift-cluster-node-tuning-operator/tuned-2fv9z"
Oct 30 19:15:53.845716 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:15:53.844898    4892 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/08ae8ede-9a96-4a84-8ec8-8d364fd476a3-etc-openvswitch\") pod \"ovnkube-node-5lwkx\" (UID: \"08ae8ede-9a96-4a84-8ec8-8d364fd476a3\") " pod="openshift-ovn-kubernetes/ovnkube-node-5lwkx"
Oct 30 19:15:53.845716 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:15:53.844913    4892 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/bab0abf9-46d4-408e-9aa0-afa7279cd9d1-tuning-conf-dir\") pod \"multus-additional-cni-plugins-4pvgg\" (UID: \"bab0abf9-46d4-408e-9aa0-afa7279cd9d1\") " pod="openshift-multus/multus-additional-cni-plugins-4pvgg"
Oct 30 19:15:53.845716 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:15:53.844934    4892 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run\" (UniqueName: \"kubernetes.io/host-path/39bb8c16-8282-442a-841b-0056baae014d-run\") pod \"tuned-2fv9z\" (UID: \"39bb8c16-8282-442a-841b-0056baae014d\") " pod="openshift-cluster-node-tuning-operator/tuned-2fv9z"
Oct 30 19:15:53.845716 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:15:53.844957    4892 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-q667z\" (UniqueName: \"kubernetes.io/projected/fade00ba-24a7-41c9-93f7-23e40971874a-kube-api-access-q667z\") pod \"dns-default-bb9cj\" (UID: \"fade00ba-24a7-41c9-93f7-23e40971874a\") " pod="openshift-dns/dns-default-bb9cj"
Oct 30 19:15:53.845716 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:15:53.844988    4892 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7d4bn\" (UniqueName: \"kubernetes.io/projected/fc4ac297-f510-4439-80e2-487056d3bf06-kube-api-access-7d4bn\") pod \"insights-runtime-extractor-jrm87\" (UID: \"fc4ac297-f510-4439-80e2-487056d3bf06\") " pod="openshift-insights/insights-runtime-extractor-jrm87"
Oct 30 19:15:53.845716 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:15:53.845010    4892 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/08ae8ede-9a96-4a84-8ec8-8d364fd476a3-host-run-ovn-kubernetes\") pod \"ovnkube-node-5lwkx\" (UID: \"08ae8ede-9a96-4a84-8ec8-8d364fd476a3\") " pod="openshift-ovn-kubernetes/ovnkube-node-5lwkx"
Oct 30 19:15:53.845716 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:15:53.845024    4892 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/08ae8ede-9a96-4a84-8ec8-8d364fd476a3-host-cni-netd\") pod \"ovnkube-node-5lwkx\" (UID: \"08ae8ede-9a96-4a84-8ec8-8d364fd476a3\") " pod="openshift-ovn-kubernetes/ovnkube-node-5lwkx"
Oct 30 19:15:53.845716 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:15:53.845019    4892 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/bab0abf9-46d4-408e-9aa0-afa7279cd9d1-tuning-conf-dir\") pod \"multus-additional-cni-plugins-4pvgg\" (UID: \"bab0abf9-46d4-408e-9aa0-afa7279cd9d1\") " pod="openshift-multus/multus-additional-cni-plugins-4pvgg"
Oct 30 19:15:53.845716 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:15:53.845039    4892 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/39bb8c16-8282-442a-841b-0056baae014d-etc-sysconfig\") pod \"tuned-2fv9z\" (UID: \"39bb8c16-8282-442a-841b-0056baae014d\") " pod="openshift-cluster-node-tuning-operator/tuned-2fv9z"
Oct 30 19:15:53.845716 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:15:53.845143    4892 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/39bb8c16-8282-442a-841b-0056baae014d-etc-sysctl-d\") pod \"tuned-2fv9z\" (UID: \"39bb8c16-8282-442a-841b-0056baae014d\") " pod="openshift-cluster-node-tuning-operator/tuned-2fv9z"
Oct 30 19:15:53.845716 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:15:53.845159    4892 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/08ae8ede-9a96-4a84-8ec8-8d364fd476a3-host-cni-netd\") pod \"ovnkube-node-5lwkx\" (UID: \"08ae8ede-9a96-4a84-8ec8-8d364fd476a3\") " pod="openshift-ovn-kubernetes/ovnkube-node-5lwkx"
Oct 30 19:15:53.845716 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:15:53.845172    4892 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/dccdaf98-aac7-4756-b93f-22f62a286a1f-host-run-k8s-cni-cncf-io\") pod \"multus-jvbw5\" (UID: \"dccdaf98-aac7-4756-b93f-22f62a286a1f\") " pod="openshift-multus/multus-jvbw5"
Oct 30 19:15:53.845716 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:15:53.845190    4892 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/39bb8c16-8282-442a-841b-0056baae014d-etc-sysconfig\") pod \"tuned-2fv9z\" (UID: \"39bb8c16-8282-442a-841b-0056baae014d\") " pod="openshift-cluster-node-tuning-operator/tuned-2fv9z"
Oct 30 19:15:53.845716 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:15:53.845164    4892 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/08ae8ede-9a96-4a84-8ec8-8d364fd476a3-host-run-ovn-kubernetes\") pod \"ovnkube-node-5lwkx\" (UID: \"08ae8ede-9a96-4a84-8ec8-8d364fd476a3\") " pod="openshift-ovn-kubernetes/ovnkube-node-5lwkx"
Oct 30 19:15:53.845716 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:15:53.845219    4892 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/08ae8ede-9a96-4a84-8ec8-8d364fd476a3-host-slash\") pod \"ovnkube-node-5lwkx\" (UID: \"08ae8ede-9a96-4a84-8ec8-8d364fd476a3\") " pod="openshift-ovn-kubernetes/ovnkube-node-5lwkx"
Oct 30 19:15:53.845716 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:15:53.845223    4892 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/dccdaf98-aac7-4756-b93f-22f62a286a1f-host-run-k8s-cni-cncf-io\") pod \"multus-jvbw5\" (UID: \"dccdaf98-aac7-4756-b93f-22f62a286a1f\") " pod="openshift-multus/multus-jvbw5"
Oct 30 19:15:53.845716 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:15:53.845246    4892 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/08ae8ede-9a96-4a84-8ec8-8d364fd476a3-host-slash\") pod \"ovnkube-node-5lwkx\" (UID: \"08ae8ede-9a96-4a84-8ec8-8d364fd476a3\") " pod="openshift-ovn-kubernetes/ovnkube-node-5lwkx"
Oct 30 19:15:53.845716 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:15:53.845268    4892 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/dccdaf98-aac7-4756-b93f-22f62a286a1f-host-var-lib-cni-bin\") pod \"multus-jvbw5\" (UID: \"dccdaf98-aac7-4756-b93f-22f62a286a1f\") " pod="openshift-multus/multus-jvbw5"
Oct 30 19:15:53.845716 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:15:53.845280    4892 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/39bb8c16-8282-442a-841b-0056baae014d-etc-sysctl-d\") pod \"tuned-2fv9z\" (UID: \"39bb8c16-8282-442a-841b-0056baae014d\") " pod="openshift-cluster-node-tuning-operator/tuned-2fv9z"
Oct 30 19:15:53.845716 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:15:53.845290    4892 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/dccdaf98-aac7-4756-b93f-22f62a286a1f-host-var-lib-kubelet\") pod \"multus-jvbw5\" (UID: \"dccdaf98-aac7-4756-b93f-22f62a286a1f\") " pod="openshift-multus/multus-jvbw5"
Oct 30 19:15:53.845716 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:15:53.845309    4892 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/c16c0a80-c5ce-4f5d-827a-2e3eb2c3cfb8-kubelet-config\") pod \"global-pull-secret-syncer-76rht\" (UID: \"c16c0a80-c5ce-4f5d-827a-2e3eb2c3cfb8\") " pod="kube-system/global-pull-secret-syncer-76rht"
Oct 30 19:15:53.845716 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:15:53.845323    4892 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/dccdaf98-aac7-4756-b93f-22f62a286a1f-host-var-lib-cni-bin\") pod \"multus-jvbw5\" (UID: \"dccdaf98-aac7-4756-b93f-22f62a286a1f\") " pod="openshift-multus/multus-jvbw5"
Oct 30 19:15:53.845716 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:15:53.845326    4892 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/c3780b96-955e-4728-a2d0-b7ec30d0f6f3-registration-dir\") pod \"aws-ebs-csi-driver-node-qgv98\" (UID: \"c3780b96-955e-4728-a2d0-b7ec30d0f6f3\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-qgv98"
Oct 30 19:15:53.845716 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:15:53.845377    4892 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/dccdaf98-aac7-4756-b93f-22f62a286a1f-host-var-lib-kubelet\") pod \"multus-jvbw5\" (UID: \"dccdaf98-aac7-4756-b93f-22f62a286a1f\") " pod="openshift-multus/multus-jvbw5"
Oct 30 19:15:53.845716 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:15:53.845392    4892 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/08ae8ede-9a96-4a84-8ec8-8d364fd476a3-systemd-units\") pod \"ovnkube-node-5lwkx\" (UID: \"08ae8ede-9a96-4a84-8ec8-8d364fd476a3\") " pod="openshift-ovn-kubernetes/ovnkube-node-5lwkx"
Oct 30 19:15:53.845716 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:15:53.845426    4892 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/08ae8ede-9a96-4a84-8ec8-8d364fd476a3-systemd-units\") pod \"ovnkube-node-5lwkx\" (UID: \"08ae8ede-9a96-4a84-8ec8-8d364fd476a3\") " pod="openshift-ovn-kubernetes/ovnkube-node-5lwkx"
Oct 30 19:15:53.845716 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:15:53.845446    4892 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/48386105-d79b-4f92-baa0-f330b9658ad0-hosts-file\") pod \"node-resolver-g9xd6\" (UID: \"48386105-d79b-4f92-baa0-f330b9658ad0\") " pod="openshift-dns/node-resolver-g9xd6"
Oct 30 19:15:53.845716 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:15:53.845474    4892 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/c3780b96-955e-4728-a2d0-b7ec30d0f6f3-device-dir\") pod \"aws-ebs-csi-driver-node-qgv98\" (UID: \"c3780b96-955e-4728-a2d0-b7ec30d0f6f3\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-qgv98"
Oct 30 19:15:53.845716 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:15:53.845501    4892 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/48386105-d79b-4f92-baa0-f330b9658ad0-hosts-file\") pod \"node-resolver-g9xd6\" (UID: \"48386105-d79b-4f92-baa0-f330b9658ad0\") " pod="openshift-dns/node-resolver-g9xd6"
Oct 30 19:15:53.845716 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:15:53.845525    4892 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/e3206ddf-35de-4b7c-b979-682f5f666a34-registry-tls\") pod \"image-registry-6fcb4848df-8l8cp\" (UID: \"e3206ddf-35de-4b7c-b979-682f5f666a34\") " pod="openshift-image-registry/image-registry-6fcb4848df-8l8cp"
Oct 30 19:15:53.845716 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:15:53.845547    4892 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/39bb8c16-8282-442a-841b-0056baae014d-lib-modules\") pod \"tuned-2fv9z\" (UID: \"39bb8c16-8282-442a-841b-0056baae014d\") " pod="openshift-cluster-node-tuning-operator/tuned-2fv9z"
Oct 30 19:15:53.845716 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:15:53.845577    4892 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/dccdaf98-aac7-4756-b93f-22f62a286a1f-multus-cni-dir\") pod \"multus-jvbw5\" (UID: \"dccdaf98-aac7-4756-b93f-22f62a286a1f\") " pod="openshift-multus/multus-jvbw5"
Oct 30 19:15:53.845716 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:15:53.845601    4892 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/c16c0a80-c5ce-4f5d-827a-2e3eb2c3cfb8-original-pull-secret\") pod \"global-pull-secret-syncer-76rht\" (UID: \"c16c0a80-c5ce-4f5d-827a-2e3eb2c3cfb8\") " pod="kube-system/global-pull-secret-syncer-76rht"
Oct 30 19:15:53.845716 ip-10-0-132-82 kubenswrapper[4892]: E1030 19:15:53.845608    4892 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Oct 30 19:15:53.845716 ip-10-0-132-82 kubenswrapper[4892]: E1030 19:15:53.845620    4892 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-6fcb4848df-8l8cp: secret "image-registry-tls" not found
Oct 30 19:15:53.845716 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:15:53.845625    4892 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/39bb8c16-8282-442a-841b-0056baae014d-lib-modules\") pod \"tuned-2fv9z\" (UID: \"39bb8c16-8282-442a-841b-0056baae014d\") " pod="openshift-cluster-node-tuning-operator/tuned-2fv9z"
Oct 30 19:15:53.845716 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:15:53.845627    4892 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/39bb8c16-8282-442a-841b-0056baae014d-host\") pod \"tuned-2fv9z\" (UID: \"39bb8c16-8282-442a-841b-0056baae014d\") " pod="openshift-cluster-node-tuning-operator/tuned-2fv9z"
Oct 30 19:15:53.845716 ip-10-0-132-82 kubenswrapper[4892]: E1030 19:15:53.845666    4892 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/e3206ddf-35de-4b7c-b979-682f5f666a34-registry-tls podName:e3206ddf-35de-4b7c-b979-682f5f666a34 nodeName:}" failed. No retries permitted until 2026-10-30 19:15:54.34565268 +0000 UTC m=+1.916118274 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/e3206ddf-35de-4b7c-b979-682f5f666a34-registry-tls") pod "image-registry-6fcb4848df-8l8cp" (UID: "e3206ddf-35de-4b7c-b979-682f5f666a34") : secret "image-registry-tls" not found
Oct 30 19:15:53.845716 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:15:53.845670    4892 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/39bb8c16-8282-442a-841b-0056baae014d-host\") pod \"tuned-2fv9z\" (UID: \"39bb8c16-8282-442a-841b-0056baae014d\") " pod="openshift-cluster-node-tuning-operator/tuned-2fv9z"
Oct 30 19:15:53.845716 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:15:53.845659    4892 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/287ba08f-7b9f-4609-8e35-832c0879c44c-metrics-certs\") pod \"network-metrics-daemon-822gp\" (UID: \"287ba08f-7b9f-4609-8e35-832c0879c44c\") " pod="openshift-multus/network-metrics-daemon-822gp"
Oct 30 19:15:53.845716 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:15:53.845689    4892 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/dccdaf98-aac7-4756-b93f-22f62a286a1f-multus-cni-dir\") pod \"multus-jvbw5\" (UID: \"dccdaf98-aac7-4756-b93f-22f62a286a1f\") " pod="openshift-multus/multus-jvbw5"
Oct 30 19:15:53.845716 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:15:53.845699    4892 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/08ae8ede-9a96-4a84-8ec8-8d364fd476a3-host-run-netns\") pod \"ovnkube-node-5lwkx\" (UID: \"08ae8ede-9a96-4a84-8ec8-8d364fd476a3\") " pod="openshift-ovn-kubernetes/ovnkube-node-5lwkx"
Oct 30 19:15:53.845716 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:15:53.845716    4892 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/dccdaf98-aac7-4756-b93f-22f62a286a1f-host-run-multus-certs\") pod \"multus-jvbw5\" (UID: \"dccdaf98-aac7-4756-b93f-22f62a286a1f\") " pod="openshift-multus/multus-jvbw5"
Oct 30 19:15:53.845716 ip-10-0-132-82 kubenswrapper[4892]: E1030 19:15:53.845730    4892 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: secret "metrics-daemon-secret" not found
Oct 30 19:15:53.845716 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:15:53.845748    4892 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/08ae8ede-9a96-4a84-8ec8-8d364fd476a3-host-run-netns\") pod \"ovnkube-node-5lwkx\" (UID: \"08ae8ede-9a96-4a84-8ec8-8d364fd476a3\") " pod="openshift-ovn-kubernetes/ovnkube-node-5lwkx"
Oct 30 19:15:53.845716 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:15:53.845751    4892 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/56274b23-2400-43d4-9686-c928e20d47de-samples-operator-tls\") pod \"cluster-samples-operator-6ddb6c8b94-np94b\" (UID: \"56274b23-2400-43d4-9686-c928e20d47de\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-np94b"
Oct 30 19:15:53.845716 ip-10-0-132-82 kubenswrapper[4892]: E1030 19:15:53.845763    4892 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/287ba08f-7b9f-4609-8e35-832c0879c44c-metrics-certs podName:287ba08f-7b9f-4609-8e35-832c0879c44c nodeName:}" failed. No retries permitted until 2026-10-30 19:15:54.345752057 +0000 UTC m=+1.916217665 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/287ba08f-7b9f-4609-8e35-832c0879c44c-metrics-certs") pod "network-metrics-daemon-822gp" (UID: "287ba08f-7b9f-4609-8e35-832c0879c44c") : secret "metrics-daemon-secret" not found
Oct 30 19:15:53.847750 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:15:53.845782    4892 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/e0221987-8e5e-40d1-8842-c97993f08c6d-host-slash\") pod \"iptables-alerter-wnn66\" (UID: \"e0221987-8e5e-40d1-8842-c97993f08c6d\") " pod="openshift-network-operator/iptables-alerter-wnn66"
Oct 30 19:15:53.847750 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:15:53.845791    4892 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/dccdaf98-aac7-4756-b93f-22f62a286a1f-host-run-multus-certs\") pod \"multus-jvbw5\" (UID: \"dccdaf98-aac7-4756-b93f-22f62a286a1f\") " pod="openshift-multus/multus-jvbw5"
Oct 30 19:15:53.847750 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:15:53.845797    4892 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/dccdaf98-aac7-4756-b93f-22f62a286a1f-system-cni-dir\") pod \"multus-jvbw5\" (UID: \"dccdaf98-aac7-4756-b93f-22f62a286a1f\") " pod="openshift-multus/multus-jvbw5"
Oct 30 19:15:53.847750 ip-10-0-132-82 kubenswrapper[4892]: E1030 19:15:53.845795    4892 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found
Oct 30 19:15:53.847750 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:15:53.845822    4892 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/e0221987-8e5e-40d1-8842-c97993f08c6d-host-slash\") pod \"iptables-alerter-wnn66\" (UID: \"e0221987-8e5e-40d1-8842-c97993f08c6d\") " pod="openshift-network-operator/iptables-alerter-wnn66"
Oct 30 19:15:53.847750 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:15:53.845826    4892 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/dccdaf98-aac7-4756-b93f-22f62a286a1f-system-cni-dir\") pod \"multus-jvbw5\" (UID: \"dccdaf98-aac7-4756-b93f-22f62a286a1f\") " pod="openshift-multus/multus-jvbw5"
Oct 30 19:15:53.847750 ip-10-0-132-82 kubenswrapper[4892]: E1030 19:15:53.845841    4892 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/56274b23-2400-43d4-9686-c928e20d47de-samples-operator-tls podName:56274b23-2400-43d4-9686-c928e20d47de nodeName:}" failed. No retries permitted until 2026-10-30 19:15:54.34583005 +0000 UTC m=+1.916295646 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/56274b23-2400-43d4-9686-c928e20d47de-samples-operator-tls") pod "cluster-samples-operator-6ddb6c8b94-np94b" (UID: "56274b23-2400-43d4-9686-c928e20d47de") : secret "samples-operator-tls" not found
Oct 30 19:15:53.847750 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:15:53.845835    4892 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/c16c0a80-c5ce-4f5d-827a-2e3eb2c3cfb8-original-pull-secret\") pod \"global-pull-secret-syncer-76rht\" (UID: \"c16c0a80-c5ce-4f5d-827a-2e3eb2c3cfb8\") " pod="kube-system/global-pull-secret-syncer-76rht"
Oct 30 19:15:53.847750 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:15:53.845858    4892 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/dccdaf98-aac7-4756-b93f-22f62a286a1f-host-run-netns\") pod \"multus-jvbw5\" (UID: \"dccdaf98-aac7-4756-b93f-22f62a286a1f\") " pod="openshift-multus/multus-jvbw5"
Oct 30 19:15:53.847750 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:15:53.845875    4892 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/c3780b96-955e-4728-a2d0-b7ec30d0f6f3-etc-selinux\") pod \"aws-ebs-csi-driver-node-qgv98\" (UID: \"c3780b96-955e-4728-a2d0-b7ec30d0f6f3\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-qgv98"
Oct 30 19:15:53.847750 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:15:53.845894    4892 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/c3780b96-955e-4728-a2d0-b7ec30d0f6f3-sys-fs\") pod \"aws-ebs-csi-driver-node-qgv98\" (UID: \"c3780b96-955e-4728-a2d0-b7ec30d0f6f3\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-qgv98"
Oct 30 19:15:53.847750 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:15:53.845906    4892 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/dccdaf98-aac7-4756-b93f-22f62a286a1f-host-run-netns\") pod \"multus-jvbw5\" (UID: \"dccdaf98-aac7-4756-b93f-22f62a286a1f\") " pod="openshift-multus/multus-jvbw5"
Oct 30 19:15:53.847750 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:15:53.845937    4892 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cdkvt\" (UniqueName: \"kubernetes.io/projected/c3780b96-955e-4728-a2d0-b7ec30d0f6f3-kube-api-access-cdkvt\") pod \"aws-ebs-csi-driver-node-qgv98\" (UID: \"c3780b96-955e-4728-a2d0-b7ec30d0f6f3\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-qgv98"
Oct 30 19:15:53.847750 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:15:53.845992    4892 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/f86c0e38-8144-42fc-b622-25602ada113f-cert\") pod \"ingress-canary-8rwjm\" (UID: \"f86c0e38-8144-42fc-b622-25602ada113f\") " pod="openshift-ingress-canary/ingress-canary-8rwjm"
Oct 30 19:15:53.847750 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:15:53.846017    4892 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/08ae8ede-9a96-4a84-8ec8-8d364fd476a3-host-kubelet\") pod \"ovnkube-node-5lwkx\" (UID: \"08ae8ede-9a96-4a84-8ec8-8d364fd476a3\") " pod="openshift-ovn-kubernetes/ovnkube-node-5lwkx"
Oct 30 19:15:53.847750 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:15:53.846089    4892 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/08ae8ede-9a96-4a84-8ec8-8d364fd476a3-host-kubelet\") pod \"ovnkube-node-5lwkx\" (UID: \"08ae8ede-9a96-4a84-8ec8-8d364fd476a3\") " pod="openshift-ovn-kubernetes/ovnkube-node-5lwkx"
Oct 30 19:15:53.847750 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:15:53.846194    4892 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/f86c0e38-8144-42fc-b622-25602ada113f-cert\") pod \"ingress-canary-8rwjm\" (UID: \"f86c0e38-8144-42fc-b622-25602ada113f\") " pod="openshift-ingress-canary/ingress-canary-8rwjm"
Oct 30 19:15:53.899390 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:15:53.899364    4892 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-j4wnm\" (UniqueName: \"kubernetes.io/projected/f86c0e38-8144-42fc-b622-25602ada113f-kube-api-access-j4wnm\") pod \"ingress-canary-8rwjm\" (UID: \"f86c0e38-8144-42fc-b622-25602ada113f\") " pod="openshift-ingress-canary/ingress-canary-8rwjm"
Oct 30 19:15:53.899619 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:15:53.899601    4892 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-q667z\" (UniqueName: \"kubernetes.io/projected/fade00ba-24a7-41c9-93f7-23e40971874a-kube-api-access-q667z\") pod \"dns-default-bb9cj\" (UID: \"fade00ba-24a7-41c9-93f7-23e40971874a\") " pod="openshift-dns/dns-default-bb9cj"
Oct 30 19:15:53.899720 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:15:53.899691    4892 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7d4bn\" (UniqueName: \"kubernetes.io/projected/fc4ac297-f510-4439-80e2-487056d3bf06-kube-api-access-7d4bn\") pod \"insights-runtime-extractor-jrm87\" (UID: \"fc4ac297-f510-4439-80e2-487056d3bf06\") " pod="openshift-insights/insights-runtime-extractor-jrm87"
Oct 30 19:15:53.900615 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:15:53.900593    4892 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-cdkvt\" (UniqueName: \"kubernetes.io/projected/c3780b96-955e-4728-a2d0-b7ec30d0f6f3-kube-api-access-cdkvt\") pod \"aws-ebs-csi-driver-node-qgv98\" (UID: \"c3780b96-955e-4728-a2d0-b7ec30d0f6f3\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-qgv98"
Oct 30 19:15:53.929190 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:15:53.929167    4892 scope.go:122] "RemoveContainer" containerID="dafc11f60068899cfdf6d420a7f0b52ca486082253a85db5cf117594b4979646"
Oct 30 19:15:53.946344 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:15:53.946321    4892 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/c16c0a80-c5ce-4f5d-827a-2e3eb2c3cfb8-kubelet-config\") pod \"global-pull-secret-syncer-76rht\" (UID: \"c16c0a80-c5ce-4f5d-827a-2e3eb2c3cfb8\") " pod="kube-system/global-pull-secret-syncer-76rht"
Oct 30 19:15:53.946442 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:15:53.946347    4892 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/c3780b96-955e-4728-a2d0-b7ec30d0f6f3-registration-dir\") pod \"aws-ebs-csi-driver-node-qgv98\" (UID: \"c3780b96-955e-4728-a2d0-b7ec30d0f6f3\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-qgv98"
Oct 30 19:15:53.946442 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:15:53.946364    4892 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/c3780b96-955e-4728-a2d0-b7ec30d0f6f3-device-dir\") pod \"aws-ebs-csi-driver-node-qgv98\" (UID: \"c3780b96-955e-4728-a2d0-b7ec30d0f6f3\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-qgv98"
Oct 30 19:15:53.946442 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:15:53.946430    4892 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/c3780b96-955e-4728-a2d0-b7ec30d0f6f3-etc-selinux\") pod \"aws-ebs-csi-driver-node-qgv98\" (UID: \"c3780b96-955e-4728-a2d0-b7ec30d0f6f3\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-qgv98"
Oct 30 19:15:53.946442 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:15:53.946431    4892 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/c16c0a80-c5ce-4f5d-827a-2e3eb2c3cfb8-kubelet-config\") pod \"global-pull-secret-syncer-76rht\" (UID: \"c16c0a80-c5ce-4f5d-827a-2e3eb2c3cfb8\") " pod="kube-system/global-pull-secret-syncer-76rht"
Oct 30 19:15:53.946614 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:15:53.946434    4892 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/c3780b96-955e-4728-a2d0-b7ec30d0f6f3-registration-dir\") pod \"aws-ebs-csi-driver-node-qgv98\" (UID: \"c3780b96-955e-4728-a2d0-b7ec30d0f6f3\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-qgv98"
Oct 30 19:15:53.946614 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:15:53.946456    4892 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/c3780b96-955e-4728-a2d0-b7ec30d0f6f3-sys-fs\") pod \"aws-ebs-csi-driver-node-qgv98\" (UID: \"c3780b96-955e-4728-a2d0-b7ec30d0f6f3\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-qgv98"
Oct 30 19:15:53.946614 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:15:53.946502    4892 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/c3780b96-955e-4728-a2d0-b7ec30d0f6f3-kubelet-dir\") pod \"aws-ebs-csi-driver-node-qgv98\" (UID: \"c3780b96-955e-4728-a2d0-b7ec30d0f6f3\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-qgv98"
Oct 30 19:15:53.946614 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:15:53.946506    4892 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/c3780b96-955e-4728-a2d0-b7ec30d0f6f3-etc-selinux\") pod \"aws-ebs-csi-driver-node-qgv98\" (UID: \"c3780b96-955e-4728-a2d0-b7ec30d0f6f3\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-qgv98"
Oct 30 19:15:53.946614 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:15:53.946512    4892 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/c3780b96-955e-4728-a2d0-b7ec30d0f6f3-sys-fs\") pod \"aws-ebs-csi-driver-node-qgv98\" (UID: \"c3780b96-955e-4728-a2d0-b7ec30d0f6f3\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-qgv98"
Oct 30 19:15:53.946614 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:15:53.946518    4892 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/c3780b96-955e-4728-a2d0-b7ec30d0f6f3-device-dir\") pod \"aws-ebs-csi-driver-node-qgv98\" (UID: \"c3780b96-955e-4728-a2d0-b7ec30d0f6f3\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-qgv98"
Oct 30 19:15:53.946614 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:15:53.946534    4892 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/fc4ac297-f510-4439-80e2-487056d3bf06-crio-socket\") pod \"insights-runtime-extractor-jrm87\" (UID: \"fc4ac297-f510-4439-80e2-487056d3bf06\") " pod="openshift-insights/insights-runtime-extractor-jrm87"
Oct 30 19:15:53.946614 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:15:53.946570    4892 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/fc4ac297-f510-4439-80e2-487056d3bf06-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-jrm87\" (UID: \"fc4ac297-f510-4439-80e2-487056d3bf06\") " pod="openshift-insights/insights-runtime-extractor-jrm87"
Oct 30 19:15:53.946614 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:15:53.946583    4892 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/c3780b96-955e-4728-a2d0-b7ec30d0f6f3-kubelet-dir\") pod \"aws-ebs-csi-driver-node-qgv98\" (UID: \"c3780b96-955e-4728-a2d0-b7ec30d0f6f3\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-qgv98"
Oct 30 19:15:53.946958 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:15:53.946596    4892 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/fc4ac297-f510-4439-80e2-487056d3bf06-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-jrm87\" (UID: \"fc4ac297-f510-4439-80e2-487056d3bf06\") " pod="openshift-insights/insights-runtime-extractor-jrm87"
Oct 30 19:15:53.946958 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:15:53.946629    4892 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/fc4ac297-f510-4439-80e2-487056d3bf06-crio-socket\") pod \"insights-runtime-extractor-jrm87\" (UID: \"fc4ac297-f510-4439-80e2-487056d3bf06\") " pod="openshift-insights/insights-runtime-extractor-jrm87"
Oct 30 19:15:53.946958 ip-10-0-132-82 kubenswrapper[4892]: E1030 19:15:53.946653    4892 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Oct 30 19:15:53.946958 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:15:53.946677    4892 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/c3780b96-955e-4728-a2d0-b7ec30d0f6f3-socket-dir\") pod \"aws-ebs-csi-driver-node-qgv98\" (UID: \"c3780b96-955e-4728-a2d0-b7ec30d0f6f3\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-qgv98"
Oct 30 19:15:53.946958 ip-10-0-132-82 kubenswrapper[4892]: E1030 19:15:53.946695    4892 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/fc4ac297-f510-4439-80e2-487056d3bf06-insights-runtime-extractor-tls podName:fc4ac297-f510-4439-80e2-487056d3bf06 nodeName:}" failed. No retries permitted until 2026-10-30 19:15:54.446682875 +0000 UTC m=+2.017148466 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/fc4ac297-f510-4439-80e2-487056d3bf06-insights-runtime-extractor-tls") pod "insights-runtime-extractor-jrm87" (UID: "fc4ac297-f510-4439-80e2-487056d3bf06") : secret "insights-runtime-extractor-tls" not found
Oct 30 19:15:53.946958 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:15:53.946725    4892 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/c3780b96-955e-4728-a2d0-b7ec30d0f6f3-socket-dir\") pod \"aws-ebs-csi-driver-node-qgv98\" (UID: \"c3780b96-955e-4728-a2d0-b7ec30d0f6f3\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-qgv98"
Oct 30 19:15:53.946958 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:15:53.946768    4892 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/c16c0a80-c5ce-4f5d-827a-2e3eb2c3cfb8-dbus\") pod \"global-pull-secret-syncer-76rht\" (UID: \"c16c0a80-c5ce-4f5d-827a-2e3eb2c3cfb8\") " pod="kube-system/global-pull-secret-syncer-76rht"
Oct 30 19:15:53.946958 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:15:53.946802    4892 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/fade00ba-24a7-41c9-93f7-23e40971874a-metrics-tls\") pod \"dns-default-bb9cj\" (UID: \"fade00ba-24a7-41c9-93f7-23e40971874a\") " pod="openshift-dns/dns-default-bb9cj"
Oct 30 19:15:53.946958 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:15:53.946872    4892 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/c16c0a80-c5ce-4f5d-827a-2e3eb2c3cfb8-dbus\") pod \"global-pull-secret-syncer-76rht\" (UID: \"c16c0a80-c5ce-4f5d-827a-2e3eb2c3cfb8\") " pod="kube-system/global-pull-secret-syncer-76rht"
Oct 30 19:15:53.946958 ip-10-0-132-82 kubenswrapper[4892]: E1030 19:15:53.946895    4892 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Oct 30 19:15:53.946958 ip-10-0-132-82 kubenswrapper[4892]: E1030 19:15:53.946943    4892 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/fade00ba-24a7-41c9-93f7-23e40971874a-metrics-tls podName:fade00ba-24a7-41c9-93f7-23e40971874a nodeName:}" failed. No retries permitted until 2026-10-30 19:15:54.446930234 +0000 UTC m=+2.017395829 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/fade00ba-24a7-41c9-93f7-23e40971874a-metrics-tls") pod "dns-default-bb9cj" (UID: "fade00ba-24a7-41c9-93f7-23e40971874a") : secret "dns-default-metrics-tls" not found
Oct 30 19:15:53.947274 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:15:53.947059    4892 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/fc4ac297-f510-4439-80e2-487056d3bf06-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-jrm87\" (UID: \"fc4ac297-f510-4439-80e2-487056d3bf06\") " pod="openshift-insights/insights-runtime-extractor-jrm87"
Oct 30 19:15:54.350703 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:15:54.350628    4892 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/87082646-75c3-43b5-864b-3478e9a7373d-metrics-certs\") pod \"router-default-776df59447-8nrhh\" (UID: \"87082646-75c3-43b5-864b-3478e9a7373d\") " pod="openshift-ingress/router-default-776df59447-8nrhh"
Oct 30 19:15:54.350703 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:15:54.350665    4892 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/e446ca63-5885-4c66-9cb7-da39afbf742c-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-884576c57-wkldq\" (UID: \"e446ca63-5885-4c66-9cb7-da39afbf742c\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-wkldq"
Oct 30 19:15:54.350703 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:15:54.350686    4892 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/0e179af5-9f52-4941-b4f5-5e3b8789da11-registry-tls\") pod \"image-registry-d97c68fd4-htsl8\" (UID: \"0e179af5-9f52-4941-b4f5-5e3b8789da11\") " pod="openshift-image-registry/image-registry-d97c68fd4-htsl8"
Oct 30 19:15:54.350925 ip-10-0-132-82 kubenswrapper[4892]: E1030 19:15:54.350765    4892 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Oct 30 19:15:54.350925 ip-10-0-132-82 kubenswrapper[4892]: E1030 19:15:54.350776    4892 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-d97c68fd4-htsl8: secret "image-registry-tls" not found
Oct 30 19:15:54.350925 ip-10-0-132-82 kubenswrapper[4892]: E1030 19:15:54.350777    4892 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found
Oct 30 19:15:54.350925 ip-10-0-132-82 kubenswrapper[4892]: E1030 19:15:54.350813    4892 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/0e179af5-9f52-4941-b4f5-5e3b8789da11-registry-tls podName:0e179af5-9f52-4941-b4f5-5e3b8789da11 nodeName:}" failed. No retries permitted until 2026-10-30 19:15:55.350801332 +0000 UTC m=+2.921266910 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/0e179af5-9f52-4941-b4f5-5e3b8789da11-registry-tls") pod "image-registry-d97c68fd4-htsl8" (UID: "0e179af5-9f52-4941-b4f5-5e3b8789da11") : secret "image-registry-tls" not found
Oct 30 19:15:54.350925 ip-10-0-132-82 kubenswrapper[4892]: E1030 19:15:54.350764    4892 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found
Oct 30 19:15:54.350925 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:15:54.350826    4892 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/e3206ddf-35de-4b7c-b979-682f5f666a34-registry-tls\") pod \"image-registry-6fcb4848df-8l8cp\" (UID: \"e3206ddf-35de-4b7c-b979-682f5f666a34\") " pod="openshift-image-registry/image-registry-6fcb4848df-8l8cp"
Oct 30 19:15:54.350925 ip-10-0-132-82 kubenswrapper[4892]: E1030 19:15:54.350852    4892 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/87082646-75c3-43b5-864b-3478e9a7373d-metrics-certs podName:87082646-75c3-43b5-864b-3478e9a7373d nodeName:}" failed. No retries permitted until 2026-10-30 19:15:55.350831357 +0000 UTC m=+2.921296934 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/87082646-75c3-43b5-864b-3478e9a7373d-metrics-certs") pod "router-default-776df59447-8nrhh" (UID: "87082646-75c3-43b5-864b-3478e9a7373d") : secret "router-metrics-certs-default" not found
Oct 30 19:15:54.350925 ip-10-0-132-82 kubenswrapper[4892]: E1030 19:15:54.350872    4892 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e446ca63-5885-4c66-9cb7-da39afbf742c-cluster-monitoring-operator-tls podName:e446ca63-5885-4c66-9cb7-da39afbf742c nodeName:}" failed. No retries permitted until 2026-10-30 19:15:55.350865048 +0000 UTC m=+2.921330625 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/e446ca63-5885-4c66-9cb7-da39afbf742c-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-884576c57-wkldq" (UID: "e446ca63-5885-4c66-9cb7-da39afbf742c") : secret "cluster-monitoring-operator-tls" not found
Oct 30 19:15:54.350925 ip-10-0-132-82 kubenswrapper[4892]: E1030 19:15:54.350872    4892 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Oct 30 19:15:54.350925 ip-10-0-132-82 kubenswrapper[4892]: E1030 19:15:54.350881    4892 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-6fcb4848df-8l8cp: secret "image-registry-tls" not found
Oct 30 19:15:54.350925 ip-10-0-132-82 kubenswrapper[4892]: E1030 19:15:54.350900    4892 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/e3206ddf-35de-4b7c-b979-682f5f666a34-registry-tls podName:e3206ddf-35de-4b7c-b979-682f5f666a34 nodeName:}" failed. No retries permitted until 2026-10-30 19:15:55.350893946 +0000 UTC m=+2.921359580 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/e3206ddf-35de-4b7c-b979-682f5f666a34-registry-tls") pod "image-registry-6fcb4848df-8l8cp" (UID: "e3206ddf-35de-4b7c-b979-682f5f666a34") : secret "image-registry-tls" not found
Oct 30 19:15:54.350925 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:15:54.350899    4892 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/287ba08f-7b9f-4609-8e35-832c0879c44c-metrics-certs\") pod \"network-metrics-daemon-822gp\" (UID: \"287ba08f-7b9f-4609-8e35-832c0879c44c\") " pod="openshift-multus/network-metrics-daemon-822gp"
Oct 30 19:15:54.350925 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:15:54.350921    4892 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/56274b23-2400-43d4-9686-c928e20d47de-samples-operator-tls\") pod \"cluster-samples-operator-6ddb6c8b94-np94b\" (UID: \"56274b23-2400-43d4-9686-c928e20d47de\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-np94b"
Oct 30 19:15:54.351385 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:15:54.350952    4892 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/87082646-75c3-43b5-864b-3478e9a7373d-service-ca-bundle\") pod \"router-default-776df59447-8nrhh\" (UID: \"87082646-75c3-43b5-864b-3478e9a7373d\") " pod="openshift-ingress/router-default-776df59447-8nrhh"
Oct 30 19:15:54.351385 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:15:54.350987    4892 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/2c7c7060-def6-4737-8fa9-6889309b7463-networking-console-plugin-cert\") pod \"networking-console-plugin-86c478bb88-h4twn\" (UID: \"2c7c7060-def6-4737-8fa9-6889309b7463\") " pod="openshift-network-console/networking-console-plugin-86c478bb88-h4twn"
Oct 30 19:15:54.351385 ip-10-0-132-82 kubenswrapper[4892]: E1030 19:15:54.351009    4892 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found
Oct 30 19:15:54.351385 ip-10-0-132-82 kubenswrapper[4892]: E1030 19:15:54.351009    4892 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: secret "metrics-daemon-secret" not found
Oct 30 19:15:54.351385 ip-10-0-132-82 kubenswrapper[4892]: E1030 19:15:54.351031    4892 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/56274b23-2400-43d4-9686-c928e20d47de-samples-operator-tls podName:56274b23-2400-43d4-9686-c928e20d47de nodeName:}" failed. No retries permitted until 2026-10-30 19:15:55.35102549 +0000 UTC m=+2.921491112 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/56274b23-2400-43d4-9686-c928e20d47de-samples-operator-tls") pod "cluster-samples-operator-6ddb6c8b94-np94b" (UID: "56274b23-2400-43d4-9686-c928e20d47de") : secret "samples-operator-tls" not found
Oct 30 19:15:54.351385 ip-10-0-132-82 kubenswrapper[4892]: E1030 19:15:54.351056    4892 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/87082646-75c3-43b5-864b-3478e9a7373d-service-ca-bundle podName:87082646-75c3-43b5-864b-3478e9a7373d nodeName:}" failed. No retries permitted until 2026-10-30 19:15:55.351042395 +0000 UTC m=+2.921507989 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/87082646-75c3-43b5-864b-3478e9a7373d-service-ca-bundle") pod "router-default-776df59447-8nrhh" (UID: "87082646-75c3-43b5-864b-3478e9a7373d") : configmap references non-existent config key: service-ca.crt
Oct 30 19:15:54.351385 ip-10-0-132-82 kubenswrapper[4892]: E1030 19:15:54.351077    4892 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/287ba08f-7b9f-4609-8e35-832c0879c44c-metrics-certs podName:287ba08f-7b9f-4609-8e35-832c0879c44c nodeName:}" failed. No retries permitted until 2026-10-30 19:15:55.351067264 +0000 UTC m=+2.921532843 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/287ba08f-7b9f-4609-8e35-832c0879c44c-metrics-certs") pod "network-metrics-daemon-822gp" (UID: "287ba08f-7b9f-4609-8e35-832c0879c44c") : secret "metrics-daemon-secret" not found
Oct 30 19:15:54.351385 ip-10-0-132-82 kubenswrapper[4892]: E1030 19:15:54.351077    4892 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found
Oct 30 19:15:54.351385 ip-10-0-132-82 kubenswrapper[4892]: E1030 19:15:54.351105    4892 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/2c7c7060-def6-4737-8fa9-6889309b7463-networking-console-plugin-cert podName:2c7c7060-def6-4737-8fa9-6889309b7463 nodeName:}" failed. No retries permitted until 2026-10-30 19:15:55.351098477 +0000 UTC m=+2.921564071 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/2c7c7060-def6-4737-8fa9-6889309b7463-networking-console-plugin-cert") pod "networking-console-plugin-86c478bb88-h4twn" (UID: "2c7c7060-def6-4737-8fa9-6889309b7463") : secret "networking-console-plugin-cert" not found
Oct 30 19:15:54.451443 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:15:54.451418    4892 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/fc4ac297-f510-4439-80e2-487056d3bf06-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-jrm87\" (UID: \"fc4ac297-f510-4439-80e2-487056d3bf06\") " pod="openshift-insights/insights-runtime-extractor-jrm87"
Oct 30 19:15:54.451572 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:15:54.451487    4892 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/fade00ba-24a7-41c9-93f7-23e40971874a-metrics-tls\") pod \"dns-default-bb9cj\" (UID: \"fade00ba-24a7-41c9-93f7-23e40971874a\") " pod="openshift-dns/dns-default-bb9cj"
Oct 30 19:15:54.451572 ip-10-0-132-82 kubenswrapper[4892]: E1030 19:15:54.451528    4892 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Oct 30 19:15:54.451572 ip-10-0-132-82 kubenswrapper[4892]: E1030 19:15:54.451564    4892 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/fc4ac297-f510-4439-80e2-487056d3bf06-insights-runtime-extractor-tls podName:fc4ac297-f510-4439-80e2-487056d3bf06 nodeName:}" failed. No retries permitted until 2026-10-30 19:15:55.451555646 +0000 UTC m=+3.022021223 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/fc4ac297-f510-4439-80e2-487056d3bf06-insights-runtime-extractor-tls") pod "insights-runtime-extractor-jrm87" (UID: "fc4ac297-f510-4439-80e2-487056d3bf06") : secret "insights-runtime-extractor-tls" not found
Oct 30 19:15:54.451572 ip-10-0-132-82 kubenswrapper[4892]: E1030 19:15:54.451565    4892 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Oct 30 19:15:54.451716 ip-10-0-132-82 kubenswrapper[4892]: E1030 19:15:54.451614    4892 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/fade00ba-24a7-41c9-93f7-23e40971874a-metrics-tls podName:fade00ba-24a7-41c9-93f7-23e40971874a nodeName:}" failed. No retries permitted until 2026-10-30 19:15:55.451604775 +0000 UTC m=+3.022070369 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/fade00ba-24a7-41c9-93f7-23e40971874a-metrics-tls") pod "dns-default-bb9cj" (UID: "fade00ba-24a7-41c9-93f7-23e40971874a") : secret "dns-default-metrics-tls" not found
Oct 30 19:15:54.618013 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:15:54.617938    4892 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-tg5xj_11db9cec-653f-4aee-8937-0cf5b367efac/console-operator/2.log"
Oct 30 19:15:54.618384 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:15:54.618370    4892 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-tg5xj_11db9cec-653f-4aee-8937-0cf5b367efac/console-operator/1.log"
Oct 30 19:15:54.618431 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:15:54.618399    4892 generic.go:356] "Generic (PLEG): container finished" podID="11db9cec-653f-4aee-8937-0cf5b367efac" containerID="30c64be9618d96290f5b9636acb5fc4fb024c9b620487f7f493bac70ce4532a1" exitCode=255
Oct 30 19:15:54.618511 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:15:54.618489    4892 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console-operator/console-operator-7cd56b8dbc-tg5xj" event={"ID":"11db9cec-653f-4aee-8937-0cf5b367efac","Type":"ContainerDied","Data":"30c64be9618d96290f5b9636acb5fc4fb024c9b620487f7f493bac70ce4532a1"}
Oct 30 19:15:54.618564 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:15:54.618554    4892 scope.go:122] "RemoveContainer" containerID="dafc11f60068899cfdf6d420a7f0b52ca486082253a85db5cf117594b4979646"
Oct 30 19:15:54.618793 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:15:54.618771    4892 scope.go:122] "RemoveContainer" containerID="30c64be9618d96290f5b9636acb5fc4fb024c9b620487f7f493bac70ce4532a1"
Oct 30 19:15:54.618978 ip-10-0-132-82 kubenswrapper[4892]: E1030 19:15:54.618946    4892 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"console-operator\" with CrashLoopBackOff: \"back-off 10s restarting failed container=console-operator pod=console-operator-7cd56b8dbc-tg5xj_openshift-console-operator(11db9cec-653f-4aee-8937-0cf5b367efac)\"" pod="openshift-console-operator/console-operator-7cd56b8dbc-tg5xj" podUID="11db9cec-653f-4aee-8937-0cf5b367efac"
Oct 30 19:15:55.359381 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:15:55.359341    4892 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/87082646-75c3-43b5-864b-3478e9a7373d-metrics-certs\") pod \"router-default-776df59447-8nrhh\" (UID: \"87082646-75c3-43b5-864b-3478e9a7373d\") " pod="openshift-ingress/router-default-776df59447-8nrhh"
Oct 30 19:15:55.359381 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:15:55.359384    4892 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/e446ca63-5885-4c66-9cb7-da39afbf742c-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-884576c57-wkldq\" (UID: \"e446ca63-5885-4c66-9cb7-da39afbf742c\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-wkldq"
Oct 30 19:15:55.359774 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:15:55.359414    4892 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/0e179af5-9f52-4941-b4f5-5e3b8789da11-registry-tls\") pod \"image-registry-d97c68fd4-htsl8\" (UID: \"0e179af5-9f52-4941-b4f5-5e3b8789da11\") " pod="openshift-image-registry/image-registry-d97c68fd4-htsl8"
Oct 30 19:15:55.359774 ip-10-0-132-82 kubenswrapper[4892]: E1030 19:15:55.359501    4892 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found
Oct 30 19:15:55.359774 ip-10-0-132-82 kubenswrapper[4892]: E1030 19:15:55.359502    4892 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found
Oct 30 19:15:55.359774 ip-10-0-132-82 kubenswrapper[4892]: E1030 19:15:55.359549    4892 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/87082646-75c3-43b5-864b-3478e9a7373d-metrics-certs podName:87082646-75c3-43b5-864b-3478e9a7373d nodeName:}" failed. No retries permitted until 2026-10-30 19:15:57.359537888 +0000 UTC m=+4.930003478 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/87082646-75c3-43b5-864b-3478e9a7373d-metrics-certs") pod "router-default-776df59447-8nrhh" (UID: "87082646-75c3-43b5-864b-3478e9a7373d") : secret "router-metrics-certs-default" not found
Oct 30 19:15:55.359774 ip-10-0-132-82 kubenswrapper[4892]: E1030 19:15:55.359504    4892 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Oct 30 19:15:55.359774 ip-10-0-132-82 kubenswrapper[4892]: E1030 19:15:55.359566    4892 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-d97c68fd4-htsl8: secret "image-registry-tls" not found
Oct 30 19:15:55.359774 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:15:55.359598    4892 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/e3206ddf-35de-4b7c-b979-682f5f666a34-registry-tls\") pod \"image-registry-6fcb4848df-8l8cp\" (UID: \"e3206ddf-35de-4b7c-b979-682f5f666a34\") " pod="openshift-image-registry/image-registry-6fcb4848df-8l8cp"
Oct 30 19:15:55.359774 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:15:55.359627    4892 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/287ba08f-7b9f-4609-8e35-832c0879c44c-metrics-certs\") pod \"network-metrics-daemon-822gp\" (UID: \"287ba08f-7b9f-4609-8e35-832c0879c44c\") " pod="openshift-multus/network-metrics-daemon-822gp"
Oct 30 19:15:55.359774 ip-10-0-132-82 kubenswrapper[4892]: E1030 19:15:55.359661    4892 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/0e179af5-9f52-4941-b4f5-5e3b8789da11-registry-tls podName:0e179af5-9f52-4941-b4f5-5e3b8789da11 nodeName:}" failed. No retries permitted until 2026-10-30 19:15:57.359648973 +0000 UTC m=+4.930114550 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/0e179af5-9f52-4941-b4f5-5e3b8789da11-registry-tls") pod "image-registry-d97c68fd4-htsl8" (UID: "0e179af5-9f52-4941-b4f5-5e3b8789da11") : secret "image-registry-tls" not found
Oct 30 19:15:55.359774 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:15:55.359648    4892 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/56274b23-2400-43d4-9686-c928e20d47de-samples-operator-tls\") pod \"cluster-samples-operator-6ddb6c8b94-np94b\" (UID: \"56274b23-2400-43d4-9686-c928e20d47de\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-np94b"
Oct 30 19:15:55.359774 ip-10-0-132-82 kubenswrapper[4892]: E1030 19:15:55.359684    4892 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e446ca63-5885-4c66-9cb7-da39afbf742c-cluster-monitoring-operator-tls podName:e446ca63-5885-4c66-9cb7-da39afbf742c nodeName:}" failed. No retries permitted until 2026-10-30 19:15:57.359674507 +0000 UTC m=+4.930140084 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/e446ca63-5885-4c66-9cb7-da39afbf742c-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-884576c57-wkldq" (UID: "e446ca63-5885-4c66-9cb7-da39afbf742c") : secret "cluster-monitoring-operator-tls" not found
Oct 30 19:15:55.359774 ip-10-0-132-82 kubenswrapper[4892]: E1030 19:15:55.359685    4892 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Oct 30 19:15:55.359774 ip-10-0-132-82 kubenswrapper[4892]: E1030 19:15:55.359698    4892 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-6fcb4848df-8l8cp: secret "image-registry-tls" not found
Oct 30 19:15:55.359774 ip-10-0-132-82 kubenswrapper[4892]: E1030 19:15:55.359725    4892 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: secret "metrics-daemon-secret" not found
Oct 30 19:15:55.359774 ip-10-0-132-82 kubenswrapper[4892]: E1030 19:15:55.359729    4892 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found
Oct 30 19:15:55.359774 ip-10-0-132-82 kubenswrapper[4892]: E1030 19:15:55.359738    4892 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/e3206ddf-35de-4b7c-b979-682f5f666a34-registry-tls podName:e3206ddf-35de-4b7c-b979-682f5f666a34 nodeName:}" failed. No retries permitted until 2026-10-30 19:15:57.359729306 +0000 UTC m=+4.930194922 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/e3206ddf-35de-4b7c-b979-682f5f666a34-registry-tls") pod "image-registry-6fcb4848df-8l8cp" (UID: "e3206ddf-35de-4b7c-b979-682f5f666a34") : secret "image-registry-tls" not found
Oct 30 19:15:55.359774 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:15:55.359756    4892 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/87082646-75c3-43b5-864b-3478e9a7373d-service-ca-bundle\") pod \"router-default-776df59447-8nrhh\" (UID: \"87082646-75c3-43b5-864b-3478e9a7373d\") " pod="openshift-ingress/router-default-776df59447-8nrhh"
Oct 30 19:15:55.359774 ip-10-0-132-82 kubenswrapper[4892]: E1030 19:15:55.359764    4892 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/287ba08f-7b9f-4609-8e35-832c0879c44c-metrics-certs podName:287ba08f-7b9f-4609-8e35-832c0879c44c nodeName:}" failed. No retries permitted until 2026-10-30 19:15:57.359749643 +0000 UTC m=+4.930215220 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/287ba08f-7b9f-4609-8e35-832c0879c44c-metrics-certs") pod "network-metrics-daemon-822gp" (UID: "287ba08f-7b9f-4609-8e35-832c0879c44c") : secret "metrics-daemon-secret" not found
Oct 30 19:15:55.360460 ip-10-0-132-82 kubenswrapper[4892]: E1030 19:15:55.359794    4892 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/56274b23-2400-43d4-9686-c928e20d47de-samples-operator-tls podName:56274b23-2400-43d4-9686-c928e20d47de nodeName:}" failed. No retries permitted until 2026-10-30 19:15:57.359784908 +0000 UTC m=+4.930250487 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/56274b23-2400-43d4-9686-c928e20d47de-samples-operator-tls") pod "cluster-samples-operator-6ddb6c8b94-np94b" (UID: "56274b23-2400-43d4-9686-c928e20d47de") : secret "samples-operator-tls" not found
Oct 30 19:15:55.360460 ip-10-0-132-82 kubenswrapper[4892]: E1030 19:15:55.359822    4892 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/87082646-75c3-43b5-864b-3478e9a7373d-service-ca-bundle podName:87082646-75c3-43b5-864b-3478e9a7373d nodeName:}" failed. No retries permitted until 2026-10-30 19:15:57.359813431 +0000 UTC m=+4.930279009 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/87082646-75c3-43b5-864b-3478e9a7373d-service-ca-bundle") pod "router-default-776df59447-8nrhh" (UID: "87082646-75c3-43b5-864b-3478e9a7373d") : configmap references non-existent config key: service-ca.crt
Oct 30 19:15:55.360460 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:15:55.359842    4892 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/2c7c7060-def6-4737-8fa9-6889309b7463-networking-console-plugin-cert\") pod \"networking-console-plugin-86c478bb88-h4twn\" (UID: \"2c7c7060-def6-4737-8fa9-6889309b7463\") " pod="openshift-network-console/networking-console-plugin-86c478bb88-h4twn"
Oct 30 19:15:55.360460 ip-10-0-132-82 kubenswrapper[4892]: E1030 19:15:55.359999    4892 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found
Oct 30 19:15:55.360460 ip-10-0-132-82 kubenswrapper[4892]: E1030 19:15:55.360047    4892 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/2c7c7060-def6-4737-8fa9-6889309b7463-networking-console-plugin-cert podName:2c7c7060-def6-4737-8fa9-6889309b7463 nodeName:}" failed. No retries permitted until 2026-10-30 19:15:57.360035038 +0000 UTC m=+4.930500629 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/2c7c7060-def6-4737-8fa9-6889309b7463-networking-console-plugin-cert") pod "networking-console-plugin-86c478bb88-h4twn" (UID: "2c7c7060-def6-4737-8fa9-6889309b7463") : secret "networking-console-plugin-cert" not found
Oct 30 19:15:55.460908 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:15:55.460884    4892 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/fade00ba-24a7-41c9-93f7-23e40971874a-metrics-tls\") pod \"dns-default-bb9cj\" (UID: \"fade00ba-24a7-41c9-93f7-23e40971874a\") " pod="openshift-dns/dns-default-bb9cj"
Oct 30 19:15:55.461034 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:15:55.460990    4892 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/fc4ac297-f510-4439-80e2-487056d3bf06-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-jrm87\" (UID: \"fc4ac297-f510-4439-80e2-487056d3bf06\") " pod="openshift-insights/insights-runtime-extractor-jrm87"
Oct 30 19:15:55.461034 ip-10-0-132-82 kubenswrapper[4892]: E1030 19:15:55.461021    4892 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Oct 30 19:15:55.461102 ip-10-0-132-82 kubenswrapper[4892]: E1030 19:15:55.461066    4892 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/fade00ba-24a7-41c9-93f7-23e40971874a-metrics-tls podName:fade00ba-24a7-41c9-93f7-23e40971874a nodeName:}" failed. No retries permitted until 2026-10-30 19:15:57.461051295 +0000 UTC m=+5.031516871 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/fade00ba-24a7-41c9-93f7-23e40971874a-metrics-tls") pod "dns-default-bb9cj" (UID: "fade00ba-24a7-41c9-93f7-23e40971874a") : secret "dns-default-metrics-tls" not found
Oct 30 19:15:55.461150 ip-10-0-132-82 kubenswrapper[4892]: E1030 19:15:55.461105    4892 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Oct 30 19:15:55.461185 ip-10-0-132-82 kubenswrapper[4892]: E1030 19:15:55.461152    4892 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/fc4ac297-f510-4439-80e2-487056d3bf06-insights-runtime-extractor-tls podName:fc4ac297-f510-4439-80e2-487056d3bf06 nodeName:}" failed. No retries permitted until 2026-10-30 19:15:57.461140792 +0000 UTC m=+5.031606369 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/fc4ac297-f510-4439-80e2-487056d3bf06-insights-runtime-extractor-tls") pod "insights-runtime-extractor-jrm87" (UID: "fc4ac297-f510-4439-80e2-487056d3bf06") : secret "insights-runtime-extractor-tls" not found
Oct 30 19:15:55.626281 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:15:55.626226    4892 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-tg5xj_11db9cec-653f-4aee-8937-0cf5b367efac/console-operator/2.log"
Oct 30 19:15:56.309111 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:15:56.309069    4892 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console-operator/console-operator-7cd56b8dbc-tg5xj"
Oct 30 19:15:56.309450 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:15:56.309436    4892 scope.go:122] "RemoveContainer" containerID="30c64be9618d96290f5b9636acb5fc4fb024c9b620487f7f493bac70ce4532a1"
Oct 30 19:15:56.309630 ip-10-0-132-82 kubenswrapper[4892]: E1030 19:15:56.309611    4892 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"console-operator\" with CrashLoopBackOff: \"back-off 10s restarting failed container=console-operator pod=console-operator-7cd56b8dbc-tg5xj_openshift-console-operator(11db9cec-653f-4aee-8937-0cf5b367efac)\"" pod="openshift-console-operator/console-operator-7cd56b8dbc-tg5xj" podUID="11db9cec-653f-4aee-8937-0cf5b367efac"
Oct 30 19:15:57.380380 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:15:57.380337    4892 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/87082646-75c3-43b5-864b-3478e9a7373d-metrics-certs\") pod \"router-default-776df59447-8nrhh\" (UID: \"87082646-75c3-43b5-864b-3478e9a7373d\") " pod="openshift-ingress/router-default-776df59447-8nrhh"
Oct 30 19:15:57.380380 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:15:57.380380    4892 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/e446ca63-5885-4c66-9cb7-da39afbf742c-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-884576c57-wkldq\" (UID: \"e446ca63-5885-4c66-9cb7-da39afbf742c\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-wkldq"
Oct 30 19:15:57.380929 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:15:57.380404    4892 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/0e179af5-9f52-4941-b4f5-5e3b8789da11-registry-tls\") pod \"image-registry-d97c68fd4-htsl8\" (UID: \"0e179af5-9f52-4941-b4f5-5e3b8789da11\") " pod="openshift-image-registry/image-registry-d97c68fd4-htsl8"
Oct 30 19:15:57.380929 ip-10-0-132-82 kubenswrapper[4892]: E1030 19:15:57.380485    4892 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Oct 30 19:15:57.380929 ip-10-0-132-82 kubenswrapper[4892]: E1030 19:15:57.380485    4892 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found
Oct 30 19:15:57.380929 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:15:57.380533    4892 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/e3206ddf-35de-4b7c-b979-682f5f666a34-registry-tls\") pod \"image-registry-6fcb4848df-8l8cp\" (UID: \"e3206ddf-35de-4b7c-b979-682f5f666a34\") " pod="openshift-image-registry/image-registry-6fcb4848df-8l8cp"
Oct 30 19:15:57.380929 ip-10-0-132-82 kubenswrapper[4892]: E1030 19:15:57.380560    4892 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/87082646-75c3-43b5-864b-3478e9a7373d-metrics-certs podName:87082646-75c3-43b5-864b-3478e9a7373d nodeName:}" failed. No retries permitted until 2026-10-30 19:16:01.380544683 +0000 UTC m=+8.951010314 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/87082646-75c3-43b5-864b-3478e9a7373d-metrics-certs") pod "router-default-776df59447-8nrhh" (UID: "87082646-75c3-43b5-864b-3478e9a7373d") : secret "router-metrics-certs-default" not found
Oct 30 19:15:57.380929 ip-10-0-132-82 kubenswrapper[4892]: E1030 19:15:57.380588    4892 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Oct 30 19:15:57.380929 ip-10-0-132-82 kubenswrapper[4892]: E1030 19:15:57.380485    4892 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found
Oct 30 19:15:57.380929 ip-10-0-132-82 kubenswrapper[4892]: E1030 19:15:57.380598    4892 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-6fcb4848df-8l8cp: secret "image-registry-tls" not found
Oct 30 19:15:57.380929 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:15:57.380594    4892 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/287ba08f-7b9f-4609-8e35-832c0879c44c-metrics-certs\") pod \"network-metrics-daemon-822gp\" (UID: \"287ba08f-7b9f-4609-8e35-832c0879c44c\") " pod="openshift-multus/network-metrics-daemon-822gp"
Oct 30 19:15:57.380929 ip-10-0-132-82 kubenswrapper[4892]: E1030 19:15:57.380496    4892 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-d97c68fd4-htsl8: secret "image-registry-tls" not found
Oct 30 19:15:57.380929 ip-10-0-132-82 kubenswrapper[4892]: E1030 19:15:57.380638    4892 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e446ca63-5885-4c66-9cb7-da39afbf742c-cluster-monitoring-operator-tls podName:e446ca63-5885-4c66-9cb7-da39afbf742c nodeName:}" failed. No retries permitted until 2026-10-30 19:16:01.380625362 +0000 UTC m=+8.951091002 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/e446ca63-5885-4c66-9cb7-da39afbf742c-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-884576c57-wkldq" (UID: "e446ca63-5885-4c66-9cb7-da39afbf742c") : secret "cluster-monitoring-operator-tls" not found
Oct 30 19:15:57.380929 ip-10-0-132-82 kubenswrapper[4892]: E1030 19:15:57.380637    4892 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: secret "metrics-daemon-secret" not found
Oct 30 19:15:57.380929 ip-10-0-132-82 kubenswrapper[4892]: E1030 19:15:57.380655    4892 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/e3206ddf-35de-4b7c-b979-682f5f666a34-registry-tls podName:e3206ddf-35de-4b7c-b979-682f5f666a34 nodeName:}" failed. No retries permitted until 2026-10-30 19:16:01.380647846 +0000 UTC m=+8.951113423 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/e3206ddf-35de-4b7c-b979-682f5f666a34-registry-tls") pod "image-registry-6fcb4848df-8l8cp" (UID: "e3206ddf-35de-4b7c-b979-682f5f666a34") : secret "image-registry-tls" not found
Oct 30 19:15:57.380929 ip-10-0-132-82 kubenswrapper[4892]: E1030 19:15:57.380669    4892 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/287ba08f-7b9f-4609-8e35-832c0879c44c-metrics-certs podName:287ba08f-7b9f-4609-8e35-832c0879c44c nodeName:}" failed. No retries permitted until 2026-10-30 19:16:01.380662646 +0000 UTC m=+8.951128259 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/287ba08f-7b9f-4609-8e35-832c0879c44c-metrics-certs") pod "network-metrics-daemon-822gp" (UID: "287ba08f-7b9f-4609-8e35-832c0879c44c") : secret "metrics-daemon-secret" not found
Oct 30 19:15:57.380929 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:15:57.380682    4892 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/56274b23-2400-43d4-9686-c928e20d47de-samples-operator-tls\") pod \"cluster-samples-operator-6ddb6c8b94-np94b\" (UID: \"56274b23-2400-43d4-9686-c928e20d47de\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-np94b"
Oct 30 19:15:57.380929 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:15:57.380716    4892 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/87082646-75c3-43b5-864b-3478e9a7373d-service-ca-bundle\") pod \"router-default-776df59447-8nrhh\" (UID: \"87082646-75c3-43b5-864b-3478e9a7373d\") " pod="openshift-ingress/router-default-776df59447-8nrhh"
Oct 30 19:15:57.380929 ip-10-0-132-82 kubenswrapper[4892]: E1030 19:15:57.380728    4892 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/0e179af5-9f52-4941-b4f5-5e3b8789da11-registry-tls podName:0e179af5-9f52-4941-b4f5-5e3b8789da11 nodeName:}" failed. No retries permitted until 2026-10-30 19:16:01.380714034 +0000 UTC m=+8.951179610 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/0e179af5-9f52-4941-b4f5-5e3b8789da11-registry-tls") pod "image-registry-d97c68fd4-htsl8" (UID: "0e179af5-9f52-4941-b4f5-5e3b8789da11") : secret "image-registry-tls" not found
Oct 30 19:15:57.380929 ip-10-0-132-82 kubenswrapper[4892]: E1030 19:15:57.380750    4892 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found
Oct 30 19:15:57.380929 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:15:57.380755    4892 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/2c7c7060-def6-4737-8fa9-6889309b7463-networking-console-plugin-cert\") pod \"networking-console-plugin-86c478bb88-h4twn\" (UID: \"2c7c7060-def6-4737-8fa9-6889309b7463\") " pod="openshift-network-console/networking-console-plugin-86c478bb88-h4twn"
Oct 30 19:15:57.380929 ip-10-0-132-82 kubenswrapper[4892]: E1030 19:15:57.380763    4892 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/87082646-75c3-43b5-864b-3478e9a7373d-service-ca-bundle podName:87082646-75c3-43b5-864b-3478e9a7373d nodeName:}" failed. No retries permitted until 2026-10-30 19:16:01.380757253 +0000 UTC m=+8.951222894 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/87082646-75c3-43b5-864b-3478e9a7373d-service-ca-bundle") pod "router-default-776df59447-8nrhh" (UID: "87082646-75c3-43b5-864b-3478e9a7373d") : configmap references non-existent config key: service-ca.crt
Oct 30 19:15:57.380929 ip-10-0-132-82 kubenswrapper[4892]: E1030 19:15:57.380788    4892 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/56274b23-2400-43d4-9686-c928e20d47de-samples-operator-tls podName:56274b23-2400-43d4-9686-c928e20d47de nodeName:}" failed. No retries permitted until 2026-10-30 19:16:01.380770794 +0000 UTC m=+8.951236371 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/56274b23-2400-43d4-9686-c928e20d47de-samples-operator-tls") pod "cluster-samples-operator-6ddb6c8b94-np94b" (UID: "56274b23-2400-43d4-9686-c928e20d47de") : secret "samples-operator-tls" not found
Oct 30 19:15:57.380929 ip-10-0-132-82 kubenswrapper[4892]: E1030 19:15:57.380803    4892 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found
Oct 30 19:15:57.380929 ip-10-0-132-82 kubenswrapper[4892]: E1030 19:15:57.380826    4892 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/2c7c7060-def6-4737-8fa9-6889309b7463-networking-console-plugin-cert podName:2c7c7060-def6-4737-8fa9-6889309b7463 nodeName:}" failed. No retries permitted until 2026-10-30 19:16:01.380818518 +0000 UTC m=+8.951284094 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/2c7c7060-def6-4737-8fa9-6889309b7463-networking-console-plugin-cert") pod "networking-console-plugin-86c478bb88-h4twn" (UID: "2c7c7060-def6-4737-8fa9-6889309b7463") : secret "networking-console-plugin-cert" not found
Oct 30 19:15:57.481671 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:15:57.481640    4892 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/fc4ac297-f510-4439-80e2-487056d3bf06-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-jrm87\" (UID: \"fc4ac297-f510-4439-80e2-487056d3bf06\") " pod="openshift-insights/insights-runtime-extractor-jrm87"
Oct 30 19:15:57.481809 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:15:57.481724    4892 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/fade00ba-24a7-41c9-93f7-23e40971874a-metrics-tls\") pod \"dns-default-bb9cj\" (UID: \"fade00ba-24a7-41c9-93f7-23e40971874a\") " pod="openshift-dns/dns-default-bb9cj"
Oct 30 19:15:57.481882 ip-10-0-132-82 kubenswrapper[4892]: E1030 19:15:57.481799    4892 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Oct 30 19:15:57.481882 ip-10-0-132-82 kubenswrapper[4892]: E1030 19:15:57.481854    4892 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Oct 30 19:15:57.481882 ip-10-0-132-82 kubenswrapper[4892]: E1030 19:15:57.481876    4892 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/fc4ac297-f510-4439-80e2-487056d3bf06-insights-runtime-extractor-tls podName:fc4ac297-f510-4439-80e2-487056d3bf06 nodeName:}" failed. No retries permitted until 2026-10-30 19:16:01.481858495 +0000 UTC m=+9.052324097 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/fc4ac297-f510-4439-80e2-487056d3bf06-insights-runtime-extractor-tls") pod "insights-runtime-extractor-jrm87" (UID: "fc4ac297-f510-4439-80e2-487056d3bf06") : secret "insights-runtime-extractor-tls" not found
Oct 30 19:15:57.482034 ip-10-0-132-82 kubenswrapper[4892]: E1030 19:15:57.481898    4892 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/fade00ba-24a7-41c9-93f7-23e40971874a-metrics-tls podName:fade00ba-24a7-41c9-93f7-23e40971874a nodeName:}" failed. No retries permitted until 2026-10-30 19:16:01.48188606 +0000 UTC m=+9.052351720 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/fade00ba-24a7-41c9-93f7-23e40971874a-metrics-tls") pod "dns-default-bb9cj" (UID: "fade00ba-24a7-41c9-93f7-23e40971874a") : secret "dns-default-metrics-tls" not found
Oct 30 19:15:57.810905 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:15:57.810880    4892 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-5lwkx"
Oct 30 19:15:57.976114 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:15:57.976087    4892 kubelet.go:2727] "SyncLoop (probe)" probe="liveness" status="unhealthy" pod="openshift-console-operator/console-operator-7cd56b8dbc-tg5xj"
Oct 30 19:15:57.976401 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:15:57.976387    4892 scope.go:122] "RemoveContainer" containerID="30c64be9618d96290f5b9636acb5fc4fb024c9b620487f7f493bac70ce4532a1"
Oct 30 19:15:57.976564 ip-10-0-132-82 kubenswrapper[4892]: E1030 19:15:57.976545    4892 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"console-operator\" with CrashLoopBackOff: \"back-off 10s restarting failed container=console-operator pod=console-operator-7cd56b8dbc-tg5xj_openshift-console-operator(11db9cec-653f-4aee-8937-0cf5b367efac)\"" pod="openshift-console-operator/console-operator-7cd56b8dbc-tg5xj" podUID="11db9cec-653f-4aee-8937-0cf5b367efac"
Oct 30 19:15:59.784988 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:15:59.784931    4892 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-g9xd6_48386105-d79b-4f92-baa0-f330b9658ad0/dns-node-resolver/0.log"
Oct 30 19:16:00.785149 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:16:00.785115    4892 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-kgfvh_cdb750aa-dd7e-4e2f-8297-9028acc0f559/node-ca/0.log"
Oct 30 19:16:00.812275 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:16:00.812252    4892 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-network-diagnostics/network-check-target-c24j9"
Oct 30 19:16:01.385191 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:16:01.385147    4892 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-8rwjm_f86c0e38-8144-42fc-b622-25602ada113f/serve-healthcheck-canary/0.log"
Oct 30 19:16:01.418340 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:16:01.418317    4892 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/87082646-75c3-43b5-864b-3478e9a7373d-metrics-certs\") pod \"router-default-776df59447-8nrhh\" (UID: \"87082646-75c3-43b5-864b-3478e9a7373d\") " pod="openshift-ingress/router-default-776df59447-8nrhh"
Oct 30 19:16:01.418438 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:16:01.418350    4892 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/e446ca63-5885-4c66-9cb7-da39afbf742c-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-884576c57-wkldq\" (UID: \"e446ca63-5885-4c66-9cb7-da39afbf742c\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-wkldq"
Oct 30 19:16:01.418438 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:16:01.418376    4892 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/0e179af5-9f52-4941-b4f5-5e3b8789da11-registry-tls\") pod \"image-registry-d97c68fd4-htsl8\" (UID: \"0e179af5-9f52-4941-b4f5-5e3b8789da11\") " pod="openshift-image-registry/image-registry-d97c68fd4-htsl8"
Oct 30 19:16:01.418438 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:16:01.418414    4892 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/e3206ddf-35de-4b7c-b979-682f5f666a34-registry-tls\") pod \"image-registry-6fcb4848df-8l8cp\" (UID: \"e3206ddf-35de-4b7c-b979-682f5f666a34\") " pod="openshift-image-registry/image-registry-6fcb4848df-8l8cp"
Oct 30 19:16:01.418438 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:16:01.418433    4892 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/287ba08f-7b9f-4609-8e35-832c0879c44c-metrics-certs\") pod \"network-metrics-daemon-822gp\" (UID: \"287ba08f-7b9f-4609-8e35-832c0879c44c\") " pod="openshift-multus/network-metrics-daemon-822gp"
Oct 30 19:16:01.418622 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:16:01.418451    4892 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/56274b23-2400-43d4-9686-c928e20d47de-samples-operator-tls\") pod \"cluster-samples-operator-6ddb6c8b94-np94b\" (UID: \"56274b23-2400-43d4-9686-c928e20d47de\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-np94b"
Oct 30 19:16:01.418622 ip-10-0-132-82 kubenswrapper[4892]: E1030 19:16:01.418465    4892 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found
Oct 30 19:16:01.418622 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:16:01.418477    4892 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/87082646-75c3-43b5-864b-3478e9a7373d-service-ca-bundle\") pod \"router-default-776df59447-8nrhh\" (UID: \"87082646-75c3-43b5-864b-3478e9a7373d\") " pod="openshift-ingress/router-default-776df59447-8nrhh"
Oct 30 19:16:01.418622 ip-10-0-132-82 kubenswrapper[4892]: E1030 19:16:01.418486    4892 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found
Oct 30 19:16:01.418622 ip-10-0-132-82 kubenswrapper[4892]: E1030 19:16:01.418532    4892 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/87082646-75c3-43b5-864b-3478e9a7373d-metrics-certs podName:87082646-75c3-43b5-864b-3478e9a7373d nodeName:}" failed. No retries permitted until 2026-10-30 19:16:09.418513258 +0000 UTC m=+16.988978849 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/87082646-75c3-43b5-864b-3478e9a7373d-metrics-certs") pod "router-default-776df59447-8nrhh" (UID: "87082646-75c3-43b5-864b-3478e9a7373d") : secret "router-metrics-certs-default" not found
Oct 30 19:16:01.418622 ip-10-0-132-82 kubenswrapper[4892]: E1030 19:16:01.418545    4892 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Oct 30 19:16:01.418622 ip-10-0-132-82 kubenswrapper[4892]: E1030 19:16:01.418560    4892 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-6fcb4848df-8l8cp: secret "image-registry-tls" not found
Oct 30 19:16:01.418622 ip-10-0-132-82 kubenswrapper[4892]: E1030 19:16:01.418603    4892 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Oct 30 19:16:01.418622 ip-10-0-132-82 kubenswrapper[4892]: E1030 19:16:01.418606    4892 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: secret "metrics-daemon-secret" not found
Oct 30 19:16:01.418622 ip-10-0-132-82 kubenswrapper[4892]: E1030 19:16:01.418621    4892 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-d97c68fd4-htsl8: secret "image-registry-tls" not found
Oct 30 19:16:01.418622 ip-10-0-132-82 kubenswrapper[4892]: E1030 19:16:01.418552    4892 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e446ca63-5885-4c66-9cb7-da39afbf742c-cluster-monitoring-operator-tls podName:e446ca63-5885-4c66-9cb7-da39afbf742c nodeName:}" failed. No retries permitted until 2026-10-30 19:16:09.418542383 +0000 UTC m=+16.989007964 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/e446ca63-5885-4c66-9cb7-da39afbf742c-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-884576c57-wkldq" (UID: "e446ca63-5885-4c66-9cb7-da39afbf742c") : secret "cluster-monitoring-operator-tls" not found
Oct 30 19:16:01.419041 ip-10-0-132-82 kubenswrapper[4892]: E1030 19:16:01.418649    4892 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/87082646-75c3-43b5-864b-3478e9a7373d-service-ca-bundle podName:87082646-75c3-43b5-864b-3478e9a7373d nodeName:}" failed. No retries permitted until 2026-10-30 19:16:09.418636005 +0000 UTC m=+16.989101590 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/87082646-75c3-43b5-864b-3478e9a7373d-service-ca-bundle") pod "router-default-776df59447-8nrhh" (UID: "87082646-75c3-43b5-864b-3478e9a7373d") : configmap references non-existent config key: service-ca.crt
Oct 30 19:16:01.419041 ip-10-0-132-82 kubenswrapper[4892]: E1030 19:16:01.418673    4892 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/287ba08f-7b9f-4609-8e35-832c0879c44c-metrics-certs podName:287ba08f-7b9f-4609-8e35-832c0879c44c nodeName:}" failed. No retries permitted until 2026-10-30 19:16:09.418660535 +0000 UTC m=+16.989126111 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/287ba08f-7b9f-4609-8e35-832c0879c44c-metrics-certs") pod "network-metrics-daemon-822gp" (UID: "287ba08f-7b9f-4609-8e35-832c0879c44c") : secret "metrics-daemon-secret" not found
Oct 30 19:16:01.419041 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:16:01.418670    4892 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/2c7c7060-def6-4737-8fa9-6889309b7463-networking-console-plugin-cert\") pod \"networking-console-plugin-86c478bb88-h4twn\" (UID: \"2c7c7060-def6-4737-8fa9-6889309b7463\") " pod="openshift-network-console/networking-console-plugin-86c478bb88-h4twn"
Oct 30 19:16:01.419041 ip-10-0-132-82 kubenswrapper[4892]: E1030 19:16:01.418692    4892 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/0e179af5-9f52-4941-b4f5-5e3b8789da11-registry-tls podName:0e179af5-9f52-4941-b4f5-5e3b8789da11 nodeName:}" failed. No retries permitted until 2026-10-30 19:16:09.418683016 +0000 UTC m=+16.989148592 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/0e179af5-9f52-4941-b4f5-5e3b8789da11-registry-tls") pod "image-registry-d97c68fd4-htsl8" (UID: "0e179af5-9f52-4941-b4f5-5e3b8789da11") : secret "image-registry-tls" not found
Oct 30 19:16:01.419041 ip-10-0-132-82 kubenswrapper[4892]: E1030 19:16:01.418707    4892 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/e3206ddf-35de-4b7c-b979-682f5f666a34-registry-tls podName:e3206ddf-35de-4b7c-b979-682f5f666a34 nodeName:}" failed. No retries permitted until 2026-10-30 19:16:09.418701644 +0000 UTC m=+16.989167220 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/e3206ddf-35de-4b7c-b979-682f5f666a34-registry-tls") pod "image-registry-6fcb4848df-8l8cp" (UID: "e3206ddf-35de-4b7c-b979-682f5f666a34") : secret "image-registry-tls" not found
Oct 30 19:16:01.419041 ip-10-0-132-82 kubenswrapper[4892]: E1030 19:16:01.418730    4892 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found
Oct 30 19:16:01.419041 ip-10-0-132-82 kubenswrapper[4892]: E1030 19:16:01.418773    4892 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/2c7c7060-def6-4737-8fa9-6889309b7463-networking-console-plugin-cert podName:2c7c7060-def6-4737-8fa9-6889309b7463 nodeName:}" failed. No retries permitted until 2026-10-30 19:16:09.418759223 +0000 UTC m=+16.989224813 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/2c7c7060-def6-4737-8fa9-6889309b7463-networking-console-plugin-cert") pod "networking-console-plugin-86c478bb88-h4twn" (UID: "2c7c7060-def6-4737-8fa9-6889309b7463") : secret "networking-console-plugin-cert" not found
Oct 30 19:16:01.419041 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:16:01.418874    4892 swap_util.go:78] "error creating dir to test if tmpfs noswap is enabled. Assuming not supported" mount path="" error="stat /var/lib/kubelet/plugins/kubernetes.io/empty-dir: no such file or directory"
Oct 30 19:16:01.421180 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:16:01.421155    4892 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/56274b23-2400-43d4-9686-c928e20d47de-samples-operator-tls\") pod \"cluster-samples-operator-6ddb6c8b94-np94b\" (UID: \"56274b23-2400-43d4-9686-c928e20d47de\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-np94b"
Oct 30 19:16:01.433241 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:16:01.433223    4892 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-np94b"
Oct 30 19:16:01.519815 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:16:01.519776    4892 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/fade00ba-24a7-41c9-93f7-23e40971874a-metrics-tls\") pod \"dns-default-bb9cj\" (UID: \"fade00ba-24a7-41c9-93f7-23e40971874a\") " pod="openshift-dns/dns-default-bb9cj"
Oct 30 19:16:01.519982 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:16:01.519873    4892 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/fc4ac297-f510-4439-80e2-487056d3bf06-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-jrm87\" (UID: \"fc4ac297-f510-4439-80e2-487056d3bf06\") " pod="openshift-insights/insights-runtime-extractor-jrm87"
Oct 30 19:16:01.520027 ip-10-0-132-82 kubenswrapper[4892]: E1030 19:16:01.519980    4892 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Oct 30 19:16:01.520061 ip-10-0-132-82 kubenswrapper[4892]: E1030 19:16:01.520043    4892 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/fc4ac297-f510-4439-80e2-487056d3bf06-insights-runtime-extractor-tls podName:fc4ac297-f510-4439-80e2-487056d3bf06 nodeName:}" failed. No retries permitted until 2026-10-30 19:16:09.520029527 +0000 UTC m=+17.090495104 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/fc4ac297-f510-4439-80e2-487056d3bf06-insights-runtime-extractor-tls") pod "insights-runtime-extractor-jrm87" (UID: "fc4ac297-f510-4439-80e2-487056d3bf06") : secret "insights-runtime-extractor-tls" not found
Oct 30 19:16:01.522133 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:16:01.522108    4892 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/fade00ba-24a7-41c9-93f7-23e40971874a-metrics-tls\") pod \"dns-default-bb9cj\" (UID: \"fade00ba-24a7-41c9-93f7-23e40971874a\") " pod="openshift-dns/dns-default-bb9cj"
Oct 30 19:16:01.740834 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:16:01.740750    4892 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-bb9cj"
Oct 30 19:16:01.811524 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:16:01.811488    4892 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-np94b"]
Oct 30 19:16:01.855713 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:16:01.855688    4892 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Oct 30 19:16:02.106716 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:16:02.106636    4892 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-bb9cj"]
Oct 30 19:16:02.109819 ip-10-0-132-82 kubenswrapper[4892]: W1030 19:16:02.109787    4892 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfade00ba_24a7_41c9_93f7_23e40971874a.slice/crio-5e61add4dab9adeb4eb8de5d0e3d7ad4ebcaa9f781f857f35bf96bec6e202f00 WatchSource:0}: Error finding container 5e61add4dab9adeb4eb8de5d0e3d7ad4ebcaa9f781f857f35bf96bec6e202f00: Status 404 returned error can't find the container with id 5e61add4dab9adeb4eb8de5d0e3d7ad4ebcaa9f781f857f35bf96bec6e202f00
Oct 30 19:16:02.386388 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:16:02.386364    4892 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-kube-storage-version-migrator-operator_kube-storage-version-migrator-operator-7d4bc45dd6-scljn_f9a06263-cb99-4004-bffc-581d66ce5ebf/kube-storage-version-migrator-operator/0.log"
Oct 30 19:16:02.651556 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:16:02.651480    4892 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-bb9cj" event={"ID":"fade00ba-24a7-41c9-93f7-23e40971874a","Type":"ContainerStarted","Data":"5e61add4dab9adeb4eb8de5d0e3d7ad4ebcaa9f781f857f35bf96bec6e202f00"}
Oct 30 19:16:02.652658 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:16:02.652632    4892 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-np94b" event={"ID":"56274b23-2400-43d4-9686-c928e20d47de","Type":"ContainerStarted","Data":"fa60f046f7452c34869766d4382a3ca62abd5a2d56edb01ae70585a7b32a4066"}
Oct 30 19:16:02.850037 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:16:02.850006    4892 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-5lwkx"
Oct 30 19:16:02.850428 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:16:02.850211    4892 prober_manager.go:356] "Failed to trigger a manual run" probe="Readiness"
Oct 30 19:16:02.958377 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:16:02.958314    4892 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-132-82.us-west-1.compute.internal" event="NodeReady"
Oct 30 19:16:04.661116 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:16:04.661084    4892 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-bb9cj" event={"ID":"fade00ba-24a7-41c9-93f7-23e40971874a","Type":"ContainerStarted","Data":"f75fefd6ef6727395e9d6f9f6712dafd131d599b5d6794ab4472d666c3d56eb3"}
Oct 30 19:16:04.661516 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:16:04.661131    4892 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-bb9cj" event={"ID":"fade00ba-24a7-41c9-93f7-23e40971874a","Type":"ContainerStarted","Data":"253360a229dcf329e4303c99167e1c338d7beb9b771611bff74101add3c4fe87"}
Oct 30 19:16:04.661516 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:16:04.661153    4892 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-dns/dns-default-bb9cj"
Oct 30 19:16:04.662855 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:16:04.662831    4892 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-np94b" event={"ID":"56274b23-2400-43d4-9686-c928e20d47de","Type":"ContainerStarted","Data":"c23a2d656c71968fda72ec3494179f6cddaf1042eaa8f2e11e8b7fab1a7fbb77"}
Oct 30 19:16:04.663005 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:16:04.662881    4892 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-np94b" event={"ID":"56274b23-2400-43d4-9686-c928e20d47de","Type":"ContainerStarted","Data":"67ced6d5c4b0a60e4095170553e34f793428d2b1528b6636f3bbbb8d70594dcf"}
Oct 30 19:16:05.562612 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:16:05.562586    4892 scope.go:122] "RemoveContainer" containerID="30c64be9618d96290f5b9636acb5fc4fb024c9b620487f7f493bac70ce4532a1"
Oct 30 19:16:05.669311 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:16:05.669286    4892 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-tg5xj_11db9cec-653f-4aee-8937-0cf5b367efac/console-operator/3.log"
Oct 30 19:16:05.670232 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:16:05.670213    4892 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-tg5xj_11db9cec-653f-4aee-8937-0cf5b367efac/console-operator/2.log"
Oct 30 19:16:05.670359 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:16:05.670330    4892 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console-operator/console-operator-7cd56b8dbc-tg5xj" event={"ID":"11db9cec-653f-4aee-8937-0cf5b367efac","Type":"ContainerStarted","Data":"10bcd8d483d179270d9dc9c58b31f4266597d534f55e695933e587887d35d366"}
Oct 30 19:16:05.670860 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:16:05.670846    4892 scope.go:122] "RemoveContainer" containerID="10bcd8d483d179270d9dc9c58b31f4266597d534f55e695933e587887d35d366"
Oct 30 19:16:05.671022 ip-10-0-132-82 kubenswrapper[4892]: E1030 19:16:05.671004    4892 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"console-operator\" with CrashLoopBackOff: \"back-off 20s restarting failed container=console-operator pod=console-operator-7cd56b8dbc-tg5xj_openshift-console-operator(11db9cec-653f-4aee-8937-0cf5b367efac)\"" pod="openshift-console-operator/console-operator-7cd56b8dbc-tg5xj" podUID="11db9cec-653f-4aee-8937-0cf5b367efac"
Oct 30 19:16:06.240739 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:16:06.240712    4892 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="kube-system/konnectivity-agent-qnm56"
Oct 30 19:16:06.309480 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:16:06.309452    4892 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console-operator/console-operator-7cd56b8dbc-tg5xj"
Oct 30 19:16:06.674730 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:16:06.674705    4892 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-tg5xj_11db9cec-653f-4aee-8937-0cf5b367efac/console-operator/3.log"
Oct 30 19:16:06.675178 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:16:06.675161    4892 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-tg5xj_11db9cec-653f-4aee-8937-0cf5b367efac/console-operator/2.log"
Oct 30 19:16:06.675235 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:16:06.675196    4892 generic.go:356] "Generic (PLEG): container finished" podID="11db9cec-653f-4aee-8937-0cf5b367efac" containerID="10bcd8d483d179270d9dc9c58b31f4266597d534f55e695933e587887d35d366" exitCode=255
Oct 30 19:16:06.675292 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:16:06.675247    4892 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console-operator/console-operator-7cd56b8dbc-tg5xj" event={"ID":"11db9cec-653f-4aee-8937-0cf5b367efac","Type":"ContainerDied","Data":"10bcd8d483d179270d9dc9c58b31f4266597d534f55e695933e587887d35d366"}
Oct 30 19:16:06.675344 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:16:06.675294    4892 scope.go:122] "RemoveContainer" containerID="30c64be9618d96290f5b9636acb5fc4fb024c9b620487f7f493bac70ce4532a1"
Oct 30 19:16:06.675556 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:16:06.675525    4892 scope.go:122] "RemoveContainer" containerID="10bcd8d483d179270d9dc9c58b31f4266597d534f55e695933e587887d35d366"
Oct 30 19:16:06.675765 ip-10-0-132-82 kubenswrapper[4892]: E1030 19:16:06.675733    4892 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"console-operator\" with CrashLoopBackOff: \"back-off 20s restarting failed container=console-operator pod=console-operator-7cd56b8dbc-tg5xj_openshift-console-operator(11db9cec-653f-4aee-8937-0cf5b367efac)\"" pod="openshift-console-operator/console-operator-7cd56b8dbc-tg5xj" podUID="11db9cec-653f-4aee-8937-0cf5b367efac"
Oct 30 19:16:07.680535 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:16:07.680507    4892 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-tg5xj_11db9cec-653f-4aee-8937-0cf5b367efac/console-operator/3.log"
Oct 30 19:16:07.680959 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:16:07.680837    4892 scope.go:122] "RemoveContainer" containerID="10bcd8d483d179270d9dc9c58b31f4266597d534f55e695933e587887d35d366"
Oct 30 19:16:07.681080 ip-10-0-132-82 kubenswrapper[4892]: E1030 19:16:07.681055    4892 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"console-operator\" with CrashLoopBackOff: \"back-off 20s restarting failed container=console-operator pod=console-operator-7cd56b8dbc-tg5xj_openshift-console-operator(11db9cec-653f-4aee-8937-0cf5b367efac)\"" pod="openshift-console-operator/console-operator-7cd56b8dbc-tg5xj" podUID="11db9cec-653f-4aee-8937-0cf5b367efac"
Oct 30 19:16:07.977144 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:16:07.977075    4892 kubelet.go:2727] "SyncLoop (probe)" probe="liveness" status="unhealthy" pod="openshift-console-operator/console-operator-7cd56b8dbc-tg5xj"
Oct 30 19:16:08.683893 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:16:08.683866    4892 scope.go:122] "RemoveContainer" containerID="10bcd8d483d179270d9dc9c58b31f4266597d534f55e695933e587887d35d366"
Oct 30 19:16:08.684307 ip-10-0-132-82 kubenswrapper[4892]: E1030 19:16:08.684043    4892 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"console-operator\" with CrashLoopBackOff: \"back-off 20s restarting failed container=console-operator pod=console-operator-7cd56b8dbc-tg5xj_openshift-console-operator(11db9cec-653f-4aee-8937-0cf5b367efac)\"" pod="openshift-console-operator/console-operator-7cd56b8dbc-tg5xj" podUID="11db9cec-653f-4aee-8937-0cf5b367efac"
Oct 30 19:16:09.488420 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:16:09.488387    4892 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/87082646-75c3-43b5-864b-3478e9a7373d-service-ca-bundle\") pod \"router-default-776df59447-8nrhh\" (UID: \"87082646-75c3-43b5-864b-3478e9a7373d\") " pod="openshift-ingress/router-default-776df59447-8nrhh"
Oct 30 19:16:09.488420 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:16:09.488422    4892 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/2c7c7060-def6-4737-8fa9-6889309b7463-networking-console-plugin-cert\") pod \"networking-console-plugin-86c478bb88-h4twn\" (UID: \"2c7c7060-def6-4737-8fa9-6889309b7463\") " pod="openshift-network-console/networking-console-plugin-86c478bb88-h4twn"
Oct 30 19:16:09.488683 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:16:09.488461    4892 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/87082646-75c3-43b5-864b-3478e9a7373d-metrics-certs\") pod \"router-default-776df59447-8nrhh\" (UID: \"87082646-75c3-43b5-864b-3478e9a7373d\") " pod="openshift-ingress/router-default-776df59447-8nrhh"
Oct 30 19:16:09.488683 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:16:09.488478    4892 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/e446ca63-5885-4c66-9cb7-da39afbf742c-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-884576c57-wkldq\" (UID: \"e446ca63-5885-4c66-9cb7-da39afbf742c\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-wkldq"
Oct 30 19:16:09.488683 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:16:09.488500    4892 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/0e179af5-9f52-4941-b4f5-5e3b8789da11-registry-tls\") pod \"image-registry-d97c68fd4-htsl8\" (UID: \"0e179af5-9f52-4941-b4f5-5e3b8789da11\") " pod="openshift-image-registry/image-registry-d97c68fd4-htsl8"
Oct 30 19:16:09.488683 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:16:09.488549    4892 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/e3206ddf-35de-4b7c-b979-682f5f666a34-registry-tls\") pod \"image-registry-6fcb4848df-8l8cp\" (UID: \"e3206ddf-35de-4b7c-b979-682f5f666a34\") " pod="openshift-image-registry/image-registry-6fcb4848df-8l8cp"
Oct 30 19:16:09.488683 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:16:09.488580    4892 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/287ba08f-7b9f-4609-8e35-832c0879c44c-metrics-certs\") pod \"network-metrics-daemon-822gp\" (UID: \"287ba08f-7b9f-4609-8e35-832c0879c44c\") " pod="openshift-multus/network-metrics-daemon-822gp"
Oct 30 19:16:09.489677 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:16:09.489643    4892 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/87082646-75c3-43b5-864b-3478e9a7373d-service-ca-bundle\") pod \"router-default-776df59447-8nrhh\" (UID: \"87082646-75c3-43b5-864b-3478e9a7373d\") " pod="openshift-ingress/router-default-776df59447-8nrhh"
Oct 30 19:16:09.493933 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:16:09.492451    4892 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/e3206ddf-35de-4b7c-b979-682f5f666a34-registry-tls\") pod \"image-registry-6fcb4848df-8l8cp\" (UID: \"e3206ddf-35de-4b7c-b979-682f5f666a34\") " pod="openshift-image-registry/image-registry-6fcb4848df-8l8cp"
Oct 30 19:16:09.493933 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:16:09.492558    4892 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/2c7c7060-def6-4737-8fa9-6889309b7463-networking-console-plugin-cert\") pod \"networking-console-plugin-86c478bb88-h4twn\" (UID: \"2c7c7060-def6-4737-8fa9-6889309b7463\") " pod="openshift-network-console/networking-console-plugin-86c478bb88-h4twn"
Oct 30 19:16:09.493933 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:16:09.492644    4892 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/0e179af5-9f52-4941-b4f5-5e3b8789da11-registry-tls\") pod \"image-registry-d97c68fd4-htsl8\" (UID: \"0e179af5-9f52-4941-b4f5-5e3b8789da11\") " pod="openshift-image-registry/image-registry-d97c68fd4-htsl8"
Oct 30 19:16:09.493933 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:16:09.492675    4892 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/87082646-75c3-43b5-864b-3478e9a7373d-metrics-certs\") pod \"router-default-776df59447-8nrhh\" (UID: \"87082646-75c3-43b5-864b-3478e9a7373d\") " pod="openshift-ingress/router-default-776df59447-8nrhh"
Oct 30 19:16:09.495421 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:16:09.494374    4892 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/287ba08f-7b9f-4609-8e35-832c0879c44c-metrics-certs\") pod \"network-metrics-daemon-822gp\" (UID: \"287ba08f-7b9f-4609-8e35-832c0879c44c\") " pod="openshift-multus/network-metrics-daemon-822gp"
Oct 30 19:16:09.496796 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:16:09.496775    4892 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/e446ca63-5885-4c66-9cb7-da39afbf742c-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-884576c57-wkldq\" (UID: \"e446ca63-5885-4c66-9cb7-da39afbf742c\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-wkldq"
Oct 30 19:16:09.527934 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:16:09.527909    4892 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/router-default-776df59447-8nrhh"
Oct 30 19:16:09.528997 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:16:09.528977    4892 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/cluster-monitoring-operator-884576c57-wkldq"
Oct 30 19:16:09.533639 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:16:09.533623    4892 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-d97c68fd4-htsl8"
Oct 30 19:16:09.533710 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:16:09.533624    4892 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-console/networking-console-plugin-86c478bb88-h4twn"
Oct 30 19:16:09.535494 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:16:09.535471    4892 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-6fcb4848df-8l8cp"
Oct 30 19:16:09.538098 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:16:09.538080    4892 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-822gp"
Oct 30 19:16:09.595014 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:16:09.592229    4892 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/fc4ac297-f510-4439-80e2-487056d3bf06-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-jrm87\" (UID: \"fc4ac297-f510-4439-80e2-487056d3bf06\") " pod="openshift-insights/insights-runtime-extractor-jrm87"
Oct 30 19:16:09.625610 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:16:09.625545    4892 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/fc4ac297-f510-4439-80e2-487056d3bf06-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-jrm87\" (UID: \"fc4ac297-f510-4439-80e2-487056d3bf06\") " pod="openshift-insights/insights-runtime-extractor-jrm87"
Oct 30 19:16:09.841937 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:16:09.841870    4892 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-jrm87"
Oct 30 19:16:09.966933 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:16:09.966903    4892 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-network-console/networking-console-plugin-86c478bb88-h4twn"]
Oct 30 19:16:09.970297 ip-10-0-132-82 kubenswrapper[4892]: W1030 19:16:09.970273    4892 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2c7c7060_def6_4737_8fa9_6889309b7463.slice/crio-375f1cddf3fffb2f80387f1e8b60e863c98003da6713cf18f75a4c7106179309 WatchSource:0}: Error finding container 375f1cddf3fffb2f80387f1e8b60e863c98003da6713cf18f75a4c7106179309: Status 404 returned error can't find the container with id 375f1cddf3fffb2f80387f1e8b60e863c98003da6713cf18f75a4c7106179309
Oct 30 19:16:10.026250 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:16:10.026221    4892 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/router-default-776df59447-8nrhh"]
Oct 30 19:16:10.029423 ip-10-0-132-82 kubenswrapper[4892]: W1030 19:16:10.029398    4892 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod87082646_75c3_43b5_864b_3478e9a7373d.slice/crio-3cb770dce28c1d106d1c20ce6b760da754b09102159f4eac82d7eacb3f238e53 WatchSource:0}: Error finding container 3cb770dce28c1d106d1c20ce6b760da754b09102159f4eac82d7eacb3f238e53: Status 404 returned error can't find the container with id 3cb770dce28c1d106d1c20ce6b760da754b09102159f4eac82d7eacb3f238e53
Oct 30 19:16:10.038654 ip-10-0-132-82 kubenswrapper[4892]: W1030 19:16:10.038630    4892 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod287ba08f_7b9f_4609_8e35_832c0879c44c.slice/crio-ac5ac58443fcc5d4c67d48b2fbef45ce26a7038943650525bb0e4f37655d0e65 WatchSource:0}: Error finding container ac5ac58443fcc5d4c67d48b2fbef45ce26a7038943650525bb0e4f37655d0e65: Status 404 returned error can't find the container with id ac5ac58443fcc5d4c67d48b2fbef45ce26a7038943650525bb0e4f37655d0e65
Oct 30 19:16:10.039392 ip-10-0-132-82 kubenswrapper[4892]: W1030 19:16:10.039367    4892 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode446ca63_5885_4c66_9cb7_da39afbf742c.slice/crio-99d5e7d81158ff8add952532ee3cfbfb052e5f2842f4905cefeb8c708e92bba2 WatchSource:0}: Error finding container 99d5e7d81158ff8add952532ee3cfbfb052e5f2842f4905cefeb8c708e92bba2: Status 404 returned error can't find the container with id 99d5e7d81158ff8add952532ee3cfbfb052e5f2842f4905cefeb8c708e92bba2
Oct 30 19:16:10.040897 ip-10-0-132-82 kubenswrapper[4892]: W1030 19:16:10.040864    4892 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0e179af5_9f52_4941_b4f5_5e3b8789da11.slice/crio-2615df6ab32f3d8685fa6ce15777344c099f3aba6418073a8de5475fcc1dbda2 WatchSource:0}: Error finding container 2615df6ab32f3d8685fa6ce15777344c099f3aba6418073a8de5475fcc1dbda2: Status 404 returned error can't find the container with id 2615df6ab32f3d8685fa6ce15777344c099f3aba6418073a8de5475fcc1dbda2
Oct 30 19:16:10.075638 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:16:10.075611    4892 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/cluster-monitoring-operator-884576c57-wkldq"]
Oct 30 19:16:10.076474 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:16:10.076453    4892 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-d97c68fd4-htsl8"]
Oct 30 19:16:10.079749 ip-10-0-132-82 kubenswrapper[4892]: W1030 19:16:10.079726    4892 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode3206ddf_35de_4b7c_b979_682f5f666a34.slice/crio-2095a5e43334678ddef80aab03c2453ddf309aa6b37285ab24553cb3dcc86b48 WatchSource:0}: Error finding container 2095a5e43334678ddef80aab03c2453ddf309aa6b37285ab24553cb3dcc86b48: Status 404 returned error can't find the container with id 2095a5e43334678ddef80aab03c2453ddf309aa6b37285ab24553cb3dcc86b48
Oct 30 19:16:10.125469 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:16:10.125228    4892 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-822gp"]
Oct 30 19:16:10.126010 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:16:10.125991    4892 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-6fcb4848df-8l8cp"]
Oct 30 19:16:10.221470 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:16:10.221362    4892 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-jrm87"]
Oct 30 19:16:10.223814 ip-10-0-132-82 kubenswrapper[4892]: W1030 19:16:10.223770    4892 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfc4ac297_f510_4439_80e2_487056d3bf06.slice/crio-4c277a332c59fb3c798e6b2380a412063bebf7f8f92e6f11cb7ef22d73c12097 WatchSource:0}: Error finding container 4c277a332c59fb3c798e6b2380a412063bebf7f8f92e6f11cb7ef22d73c12097: Status 404 returned error can't find the container with id 4c277a332c59fb3c798e6b2380a412063bebf7f8f92e6f11cb7ef22d73c12097
Oct 30 19:16:10.692883 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:16:10.692845    4892 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-d97c68fd4-htsl8" event={"ID":"0e179af5-9f52-4941-b4f5-5e3b8789da11","Type":"ContainerStarted","Data":"603f404a1881412797abc80bbc11870012a3ddbadbfd3843a35bba336ceacb11"}
Oct 30 19:16:10.693067 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:16:10.692903    4892 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-d97c68fd4-htsl8" event={"ID":"0e179af5-9f52-4941-b4f5-5e3b8789da11","Type":"ContainerStarted","Data":"2615df6ab32f3d8685fa6ce15777344c099f3aba6418073a8de5475fcc1dbda2"}
Oct 30 19:16:10.693067 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:16:10.692948    4892 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-image-registry/image-registry-d97c68fd4-htsl8"
Oct 30 19:16:10.694510 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:16:10.694484    4892 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-6fcb4848df-8l8cp" event={"ID":"e3206ddf-35de-4b7c-b979-682f5f666a34","Type":"ContainerStarted","Data":"912fe78ec0b952b60d6c051a74a051174f4a894514b2909b7936bbdb14ddd471"}
Oct 30 19:16:10.694626 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:16:10.694532    4892 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-image-registry/image-registry-6fcb4848df-8l8cp"
Oct 30 19:16:10.694626 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:16:10.694551    4892 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-6fcb4848df-8l8cp" event={"ID":"e3206ddf-35de-4b7c-b979-682f5f666a34","Type":"ContainerStarted","Data":"2095a5e43334678ddef80aab03c2453ddf309aa6b37285ab24553cb3dcc86b48"}
Oct 30 19:16:10.696016 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:16:10.695993    4892 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/router-default-776df59447-8nrhh" event={"ID":"87082646-75c3-43b5-864b-3478e9a7373d","Type":"ContainerStarted","Data":"25ecedd75c16e240678239d97804a4796e6a539797095eebe8edfc12df19e898"}
Oct 30 19:16:10.696098 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:16:10.696034    4892 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/router-default-776df59447-8nrhh" event={"ID":"87082646-75c3-43b5-864b-3478e9a7373d","Type":"ContainerStarted","Data":"3cb770dce28c1d106d1c20ce6b760da754b09102159f4eac82d7eacb3f238e53"}
Oct 30 19:16:10.697202 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:16:10.697181    4892 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/cluster-monitoring-operator-884576c57-wkldq" event={"ID":"e446ca63-5885-4c66-9cb7-da39afbf742c","Type":"ContainerStarted","Data":"99d5e7d81158ff8add952532ee3cfbfb052e5f2842f4905cefeb8c708e92bba2"}
Oct 30 19:16:10.698836 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:16:10.698812    4892 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-jrm87" event={"ID":"fc4ac297-f510-4439-80e2-487056d3bf06","Type":"ContainerStarted","Data":"050133cc42fac316546794a5281227e9837400e04bf308f5d220e6a80430f104"}
Oct 30 19:16:10.698915 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:16:10.698862    4892 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-jrm87" event={"ID":"fc4ac297-f510-4439-80e2-487056d3bf06","Type":"ContainerStarted","Data":"4c277a332c59fb3c798e6b2380a412063bebf7f8f92e6f11cb7ef22d73c12097"}
Oct 30 19:16:10.700057 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:16:10.700038    4892 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-822gp" event={"ID":"287ba08f-7b9f-4609-8e35-832c0879c44c","Type":"ContainerStarted","Data":"ac5ac58443fcc5d4c67d48b2fbef45ce26a7038943650525bb0e4f37655d0e65"}
Oct 30 19:16:10.701217 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:16:10.701199    4892 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-network-console/networking-console-plugin-86c478bb88-h4twn" event={"ID":"2c7c7060-def6-4737-8fa9-6889309b7463","Type":"ContainerStarted","Data":"375f1cddf3fffb2f80387f1e8b60e863c98003da6713cf18f75a4c7106179309"}
Oct 30 19:16:10.851529 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:16:10.851496    4892 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-image-registry/image-registry-d97c68fd4-htsl8"]
Oct 30 19:16:11.528133 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:16:11.528102    4892 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/router-default-776df59447-8nrhh"
Oct 30 19:16:11.530978 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:16:11.530941    4892 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/router-default-776df59447-8nrhh"
Oct 30 19:16:11.715888 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:16:11.715845    4892 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-jrm87" event={"ID":"fc4ac297-f510-4439-80e2-487056d3bf06","Type":"ContainerStarted","Data":"695040ec77cdd64864b34c0b16e59e6f21a0f8e2be634cc6363db011cb3aa668"}
Oct 30 19:16:11.718565 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:16:11.718540    4892 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-network-console/networking-console-plugin-86c478bb88-h4twn" event={"ID":"2c7c7060-def6-4737-8fa9-6889309b7463","Type":"ContainerStarted","Data":"fe38aa96488af1094b3bd793e118b10bc7ec3c75fdb06c637950598e5b29372f"}
Oct 30 19:16:11.719090 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:16:11.719068    4892 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/router-default-776df59447-8nrhh"
Oct 30 19:16:11.720472 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:16:11.720421    4892 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/router-default-776df59447-8nrhh"
Oct 30 19:16:12.722484 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:16:12.722447    4892 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/cluster-monitoring-operator-884576c57-wkldq" event={"ID":"e446ca63-5885-4c66-9cb7-da39afbf742c","Type":"ContainerStarted","Data":"dfd66934869ecb6fdb54c9998a79983e4a9e276fd492f04bd11fb70607d37499"}
Oct 30 19:16:12.724355 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:16:12.724288    4892 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-822gp" event={"ID":"287ba08f-7b9f-4609-8e35-832c0879c44c","Type":"ContainerStarted","Data":"27a4dc10c7f194e79195a54fd53346b78f23efd940d11fcf401ab82cdfae81e0"}
Oct 30 19:16:12.724355 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:16:12.724344    4892 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-822gp" event={"ID":"287ba08f-7b9f-4609-8e35-832c0879c44c","Type":"ContainerStarted","Data":"e8137145538743cb1f8bed6417df740647a10b914a7b7c50a219755b44ea40c0"}
Oct 30 19:16:13.730565 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:16:13.730526    4892 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-jrm87" event={"ID":"fc4ac297-f510-4439-80e2-487056d3bf06","Type":"ContainerStarted","Data":"d13a8f6050ae169529cc4e7282a7fdb9a55bc86b32f1f9d5497c54dcd8ee089c"}
Oct 30 19:16:14.673359 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:16:14.673326    4892 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-dns/dns-default-bb9cj"
Oct 30 19:16:20.814643 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:16:20.814612    4892 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-network-diagnostics/network-check-target-c24j9"
Oct 30 19:16:21.136072 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:16:21.136046    4892 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-5lwkx"
Oct 30 19:16:23.949842 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:16:23.949816    4892 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-monitoring/node-exporter-2cghs"]
Oct 30 19:16:23.993353 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:16:23.993329    4892 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-2cghs"
Oct 30 19:16:24.072519 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:16:24.072480    4892 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-kube-rbac-proxy-config\""
Oct 30 19:16:24.094881 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:16:24.094854    4892 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-dockercfg-2dwz4\""
Oct 30 19:16:24.094881 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:16:24.094855    4892 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-tls\""
Oct 30 19:16:24.100173 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:16:24.100155    4892 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/72972ff7-2408-4875-873d-a80c69c4346c-node-exporter-wtmp\") pod \"node-exporter-2cghs\" (UID: \"72972ff7-2408-4875-873d-a80c69c4346c\") " pod="openshift-monitoring/node-exporter-2cghs"
Oct 30 19:16:24.100223 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:16:24.100187    4892 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/72972ff7-2408-4875-873d-a80c69c4346c-node-exporter-tls\") pod \"node-exporter-2cghs\" (UID: \"72972ff7-2408-4875-873d-a80c69c4346c\") " pod="openshift-monitoring/node-exporter-2cghs"
Oct 30 19:16:24.100223 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:16:24.100209    4892 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/72972ff7-2408-4875-873d-a80c69c4346c-root\") pod \"node-exporter-2cghs\" (UID: \"72972ff7-2408-4875-873d-a80c69c4346c\") " pod="openshift-monitoring/node-exporter-2cghs"
Oct 30 19:16:24.100296 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:16:24.100280    4892 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/72972ff7-2408-4875-873d-a80c69c4346c-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-2cghs\" (UID: \"72972ff7-2408-4875-873d-a80c69c4346c\") " pod="openshift-monitoring/node-exporter-2cghs"
Oct 30 19:16:24.100374 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:16:24.100354    4892 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/72972ff7-2408-4875-873d-a80c69c4346c-node-exporter-accelerators-collector-config\") pod \"node-exporter-2cghs\" (UID: \"72972ff7-2408-4875-873d-a80c69c4346c\") " pod="openshift-monitoring/node-exporter-2cghs"
Oct 30 19:16:24.100409 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:16:24.100398    4892 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8k5f7\" (UniqueName: \"kubernetes.io/projected/72972ff7-2408-4875-873d-a80c69c4346c-kube-api-access-8k5f7\") pod \"node-exporter-2cghs\" (UID: \"72972ff7-2408-4875-873d-a80c69c4346c\") " pod="openshift-monitoring/node-exporter-2cghs"
Oct 30 19:16:24.100445 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:16:24.100419    4892 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/72972ff7-2408-4875-873d-a80c69c4346c-sys\") pod \"node-exporter-2cghs\" (UID: \"72972ff7-2408-4875-873d-a80c69c4346c\") " pod="openshift-monitoring/node-exporter-2cghs"
Oct 30 19:16:24.100493 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:16:24.100463    4892 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/72972ff7-2408-4875-873d-a80c69c4346c-node-exporter-textfile\") pod \"node-exporter-2cghs\" (UID: \"72972ff7-2408-4875-873d-a80c69c4346c\") " pod="openshift-monitoring/node-exporter-2cghs"
Oct 30 19:16:24.100582 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:16:24.100562    4892 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/72972ff7-2408-4875-873d-a80c69c4346c-metrics-client-ca\") pod \"node-exporter-2cghs\" (UID: \"72972ff7-2408-4875-873d-a80c69c4346c\") " pod="openshift-monitoring/node-exporter-2cghs"
Oct 30 19:16:24.122146 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:16:24.122121    4892 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"node-exporter-accelerators-collector-config\""
Oct 30 19:16:24.122146 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:16:24.122127    4892 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"metrics-client-ca\""
Oct 30 19:16:24.201698 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:16:24.201638    4892 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/72972ff7-2408-4875-873d-a80c69c4346c-node-exporter-textfile\") pod \"node-exporter-2cghs\" (UID: \"72972ff7-2408-4875-873d-a80c69c4346c\") " pod="openshift-monitoring/node-exporter-2cghs"
Oct 30 19:16:24.201698 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:16:24.201672    4892 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/72972ff7-2408-4875-873d-a80c69c4346c-metrics-client-ca\") pod \"node-exporter-2cghs\" (UID: \"72972ff7-2408-4875-873d-a80c69c4346c\") " pod="openshift-monitoring/node-exporter-2cghs"
Oct 30 19:16:24.201698 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:16:24.201691    4892 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/72972ff7-2408-4875-873d-a80c69c4346c-node-exporter-wtmp\") pod \"node-exporter-2cghs\" (UID: \"72972ff7-2408-4875-873d-a80c69c4346c\") " pod="openshift-monitoring/node-exporter-2cghs"
Oct 30 19:16:24.201902 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:16:24.201716    4892 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/72972ff7-2408-4875-873d-a80c69c4346c-node-exporter-tls\") pod \"node-exporter-2cghs\" (UID: \"72972ff7-2408-4875-873d-a80c69c4346c\") " pod="openshift-monitoring/node-exporter-2cghs"
Oct 30 19:16:24.201902 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:16:24.201748    4892 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/72972ff7-2408-4875-873d-a80c69c4346c-root\") pod \"node-exporter-2cghs\" (UID: \"72972ff7-2408-4875-873d-a80c69c4346c\") " pod="openshift-monitoring/node-exporter-2cghs"
Oct 30 19:16:24.201902 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:16:24.201779    4892 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/72972ff7-2408-4875-873d-a80c69c4346c-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-2cghs\" (UID: \"72972ff7-2408-4875-873d-a80c69c4346c\") " pod="openshift-monitoring/node-exporter-2cghs"
Oct 30 19:16:24.201902 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:16:24.201814    4892 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/72972ff7-2408-4875-873d-a80c69c4346c-node-exporter-accelerators-collector-config\") pod \"node-exporter-2cghs\" (UID: \"72972ff7-2408-4875-873d-a80c69c4346c\") " pod="openshift-monitoring/node-exporter-2cghs"
Oct 30 19:16:24.201902 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:16:24.201833    4892 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/72972ff7-2408-4875-873d-a80c69c4346c-node-exporter-wtmp\") pod \"node-exporter-2cghs\" (UID: \"72972ff7-2408-4875-873d-a80c69c4346c\") " pod="openshift-monitoring/node-exporter-2cghs"
Oct 30 19:16:24.201902 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:16:24.201844    4892 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8k5f7\" (UniqueName: \"kubernetes.io/projected/72972ff7-2408-4875-873d-a80c69c4346c-kube-api-access-8k5f7\") pod \"node-exporter-2cghs\" (UID: \"72972ff7-2408-4875-873d-a80c69c4346c\") " pod="openshift-monitoring/node-exporter-2cghs"
Oct 30 19:16:24.201902 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:16:24.201886    4892 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/72972ff7-2408-4875-873d-a80c69c4346c-sys\") pod \"node-exporter-2cghs\" (UID: \"72972ff7-2408-4875-873d-a80c69c4346c\") " pod="openshift-monitoring/node-exporter-2cghs"
Oct 30 19:16:24.202309 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:16:24.201897    4892 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"root\" (UniqueName: \"kubernetes.io/host-path/72972ff7-2408-4875-873d-a80c69c4346c-root\") pod \"node-exporter-2cghs\" (UID: \"72972ff7-2408-4875-873d-a80c69c4346c\") " pod="openshift-monitoring/node-exporter-2cghs"
Oct 30 19:16:24.202309 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:16:24.201987    4892 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/72972ff7-2408-4875-873d-a80c69c4346c-sys\") pod \"node-exporter-2cghs\" (UID: \"72972ff7-2408-4875-873d-a80c69c4346c\") " pod="openshift-monitoring/node-exporter-2cghs"
Oct 30 19:16:24.202309 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:16:24.202017    4892 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/72972ff7-2408-4875-873d-a80c69c4346c-node-exporter-textfile\") pod \"node-exporter-2cghs\" (UID: \"72972ff7-2408-4875-873d-a80c69c4346c\") " pod="openshift-monitoring/node-exporter-2cghs"
Oct 30 19:16:24.202477 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:16:24.202348    4892 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/72972ff7-2408-4875-873d-a80c69c4346c-metrics-client-ca\") pod \"node-exporter-2cghs\" (UID: \"72972ff7-2408-4875-873d-a80c69c4346c\") " pod="openshift-monitoring/node-exporter-2cghs"
Oct 30 19:16:24.202477 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:16:24.202461    4892 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/72972ff7-2408-4875-873d-a80c69c4346c-node-exporter-accelerators-collector-config\") pod \"node-exporter-2cghs\" (UID: \"72972ff7-2408-4875-873d-a80c69c4346c\") " pod="openshift-monitoring/node-exporter-2cghs"
Oct 30 19:16:24.204207 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:16:24.204187    4892 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/72972ff7-2408-4875-873d-a80c69c4346c-node-exporter-tls\") pod \"node-exporter-2cghs\" (UID: \"72972ff7-2408-4875-873d-a80c69c4346c\") " pod="openshift-monitoring/node-exporter-2cghs"
Oct 30 19:16:24.204365 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:16:24.204348    4892 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/72972ff7-2408-4875-873d-a80c69c4346c-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-2cghs\" (UID: \"72972ff7-2408-4875-873d-a80c69c4346c\") " pod="openshift-monitoring/node-exporter-2cghs"
Oct 30 19:16:24.261509 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:16:24.261482    4892 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8k5f7\" (UniqueName: \"kubernetes.io/projected/72972ff7-2408-4875-873d-a80c69c4346c-kube-api-access-8k5f7\") pod \"node-exporter-2cghs\" (UID: \"72972ff7-2408-4875-873d-a80c69c4346c\") " pod="openshift-monitoring/node-exporter-2cghs"
Oct 30 19:16:24.344225 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:16:24.344195    4892 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-2cghs"
Oct 30 19:16:24.383327 ip-10-0-132-82 kubenswrapper[4892]: W1030 19:16:24.383297    4892 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod72972ff7_2408_4875_873d_a80c69c4346c.slice/crio-f7b0e7a92583cabf53bc7d997dfec067fcdc2c7dbfab4b1a1e35dda9f6c4ac9f WatchSource:0}: Error finding container f7b0e7a92583cabf53bc7d997dfec067fcdc2c7dbfab4b1a1e35dda9f6c4ac9f: Status 404 returned error can't find the container with id f7b0e7a92583cabf53bc7d997dfec067fcdc2c7dbfab4b1a1e35dda9f6c4ac9f
Oct 30 19:16:24.772312 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:16:24.772279    4892 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-2cghs" event={"ID":"72972ff7-2408-4875-873d-a80c69c4346c","Type":"ContainerStarted","Data":"f7b0e7a92583cabf53bc7d997dfec067fcdc2c7dbfab4b1a1e35dda9f6c4ac9f"}
Oct 30 19:16:24.843391 ip-10-0-132-82 kubenswrapper[4892]: E1030 19:16:24.843355    4892 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-jrm87" containerName="extractor"
Oct 30 19:16:26.782603 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:16:26.782572    4892 generic.go:356] "Generic (PLEG): container finished" podID="72972ff7-2408-4875-873d-a80c69c4346c" containerID="fcfd953e0f33333783406d5e988a811c69482cc6810ced07f7ec9dae13b3f8f8" exitCode=0
Oct 30 19:16:26.782904 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:16:26.782649    4892 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-2cghs" event={"ID":"72972ff7-2408-4875-873d-a80c69c4346c","Type":"ContainerDied","Data":"fcfd953e0f33333783406d5e988a811c69482cc6810ced07f7ec9dae13b3f8f8"}
Oct 30 19:16:27.561981 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:16:27.561943    4892 scope.go:122] "RemoveContainer" containerID="10bcd8d483d179270d9dc9c58b31f4266597d534f55e695933e587887d35d366"
Oct 30 19:16:27.788761 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:16:27.788723    4892 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-2cghs" event={"ID":"72972ff7-2408-4875-873d-a80c69c4346c","Type":"ContainerStarted","Data":"8fb63788ff94217d22b028baaffe0cfaf41ca3e7f16238053773865405b196ca"}
Oct 30 19:16:27.789199 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:16:27.788785    4892 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-2cghs" event={"ID":"72972ff7-2408-4875-873d-a80c69c4346c","Type":"ContainerStarted","Data":"342d72aac0d5c93d276fba90ebac82119ac64dda1a119930382dd33329aa421b"}
Oct 30 19:16:27.790649 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:16:27.790631    4892 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-tg5xj_11db9cec-653f-4aee-8937-0cf5b367efac/console-operator/3.log"
Oct 30 19:16:27.790722 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:16:27.790673    4892 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console-operator/console-operator-7cd56b8dbc-tg5xj" event={"ID":"11db9cec-653f-4aee-8937-0cf5b367efac","Type":"ContainerStarted","Data":"919344a4fd0dc651e2df738fadebed65deaa92133f92ff2dd2c822f932752d5b"}
Oct 30 19:16:27.790935 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:16:27.790913    4892 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console-operator/console-operator-7cd56b8dbc-tg5xj"
Oct 30 19:16:27.791809 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:16:27.791787    4892 patch_prober.go:29] interesting pod/console-operator-7cd56b8dbc-tg5xj container/console-operator namespace/openshift-console-operator: Readiness probe status=failure output="Get \"https://10.132.0.20:8443/readyz\": dial tcp 10.132.0.20:8443: connect: connection refused" start-of-body=
Oct 30 19:16:27.791887 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:16:27.791825    4892 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-console-operator/console-operator-7cd56b8dbc-tg5xj" podUID="11db9cec-653f-4aee-8937-0cf5b367efac" containerName="console-operator" probeResult="failure" output="Get \"https://10.132.0.20:8443/readyz\": dial tcp 10.132.0.20:8443: connect: connection refused"
Oct 30 19:16:27.917101 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:16:27.917049    4892 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-monitoring/node-exporter-2cghs" podStartSLOduration=2.681785625 podStartE2EDuration="4.917034715s" totalImagesPullingTime="2.149844666s" totalInitContainerRuntime="85.404424ms" isStatefulPod=true podCreationTimestamp="2026-10-30 19:16:23 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:16:27.916403463 +0000 UTC m=+35.486869066" watchObservedRunningTime="2026-10-30 19:16:27.917034715 +0000 UTC m=+35.487500313"
Oct 30 19:16:28.798583 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:16:28.798552    4892 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console-operator/console-operator-7cd56b8dbc-tg5xj"
Oct 30 19:16:31.722373 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:16:31.722305    4892 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-image-registry/image-registry-6fcb4848df-8l8cp"
Oct 30 19:16:32.730563 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:16:32.730533    4892 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-image-registry/image-registry-d97c68fd4-htsl8"
Oct 30 19:16:33.543797 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:16:33.543762    4892 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-image-registry/image-registry-6fcb4848df-8l8cp"]
Oct 30 19:16:37.744703 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:16:37.744653    4892 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-image-registry/image-registry-d97c68fd4-htsl8" podUID="0e179af5-9f52-4941-b4f5-5e3b8789da11" containerName="registry" containerID="cri-o://603f404a1881412797abc80bbc11870012a3ddbadbfd3843a35bba336ceacb11" gracePeriod=30
Oct 30 19:16:38.002890 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:16:38.002832    4892 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-d97c68fd4-htsl8"
Oct 30 19:16:38.111464 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:16:38.111432    4892 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/0e179af5-9f52-4941-b4f5-5e3b8789da11-installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/0e179af5-9f52-4941-b4f5-5e3b8789da11-installation-pull-secrets\") pod \"0e179af5-9f52-4941-b4f5-5e3b8789da11\" (UID: \"0e179af5-9f52-4941-b4f5-5e3b8789da11\") "
Oct 30 19:16:38.111638 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:16:38.111482    4892 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0e179af5-9f52-4941-b4f5-5e3b8789da11-registry-tls\" (UniqueName: \"kubernetes.io/projected/0e179af5-9f52-4941-b4f5-5e3b8789da11-registry-tls\") pod \"0e179af5-9f52-4941-b4f5-5e3b8789da11\" (UID: \"0e179af5-9f52-4941-b4f5-5e3b8789da11\") "
Oct 30 19:16:38.111638 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:16:38.111503    4892 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0e179af5-9f52-4941-b4f5-5e3b8789da11-bound-sa-token\" (UniqueName: \"kubernetes.io/projected/0e179af5-9f52-4941-b4f5-5e3b8789da11-bound-sa-token\") pod \"0e179af5-9f52-4941-b4f5-5e3b8789da11\" (UID: \"0e179af5-9f52-4941-b4f5-5e3b8789da11\") "
Oct 30 19:16:38.111638 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:16:38.111524    4892 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/0e179af5-9f52-4941-b4f5-5e3b8789da11-trusted-ca\" (UniqueName: \"kubernetes.io/configmap/0e179af5-9f52-4941-b4f5-5e3b8789da11-trusted-ca\") pod \"0e179af5-9f52-4941-b4f5-5e3b8789da11\" (UID: \"0e179af5-9f52-4941-b4f5-5e3b8789da11\") "
Oct 30 19:16:38.111638 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:16:38.111627    4892 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/0e179af5-9f52-4941-b4f5-5e3b8789da11-image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/0e179af5-9f52-4941-b4f5-5e3b8789da11-image-registry-private-configuration\") pod \"0e179af5-9f52-4941-b4f5-5e3b8789da11\" (UID: \"0e179af5-9f52-4941-b4f5-5e3b8789da11\") "
Oct 30 19:16:38.111866 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:16:38.111661    4892 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/0e179af5-9f52-4941-b4f5-5e3b8789da11-registry-certificates\" (UniqueName: \"kubernetes.io/configmap/0e179af5-9f52-4941-b4f5-5e3b8789da11-registry-certificates\") pod \"0e179af5-9f52-4941-b4f5-5e3b8789da11\" (UID: \"0e179af5-9f52-4941-b4f5-5e3b8789da11\") "
Oct 30 19:16:38.111866 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:16:38.111704    4892 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0e179af5-9f52-4941-b4f5-5e3b8789da11-kube-api-access-zj4df\" (UniqueName: \"kubernetes.io/projected/0e179af5-9f52-4941-b4f5-5e3b8789da11-kube-api-access-zj4df\") pod \"0e179af5-9f52-4941-b4f5-5e3b8789da11\" (UID: \"0e179af5-9f52-4941-b4f5-5e3b8789da11\") "
Oct 30 19:16:38.111866 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:16:38.111762    4892 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0e179af5-9f52-4941-b4f5-5e3b8789da11-ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/0e179af5-9f52-4941-b4f5-5e3b8789da11-ca-trust-extracted\") pod \"0e179af5-9f52-4941-b4f5-5e3b8789da11\" (UID: \"0e179af5-9f52-4941-b4f5-5e3b8789da11\") "
Oct 30 19:16:38.112067 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:16:38.111988    4892 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0e179af5-9f52-4941-b4f5-5e3b8789da11-trusted-ca" pod "0e179af5-9f52-4941-b4f5-5e3b8789da11" (UID: "0e179af5-9f52-4941-b4f5-5e3b8789da11"). InnerVolumeSpecName "trusted-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 19:16:38.112067 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:16:38.112041    4892 reconciler_common.go:299] "Volume detached for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/0e179af5-9f52-4941-b4f5-5e3b8789da11-trusted-ca\") on node \"ip-10-0-132-82.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 19:16:38.112546 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:16:38.112513    4892 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0e179af5-9f52-4941-b4f5-5e3b8789da11-registry-certificates" pod "0e179af5-9f52-4941-b4f5-5e3b8789da11" (UID: "0e179af5-9f52-4941-b4f5-5e3b8789da11"). InnerVolumeSpecName "registry-certificates". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 19:16:38.114015 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:16:38.113982    4892 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0e179af5-9f52-4941-b4f5-5e3b8789da11-registry-tls" pod "0e179af5-9f52-4941-b4f5-5e3b8789da11" (UID: "0e179af5-9f52-4941-b4f5-5e3b8789da11"). InnerVolumeSpecName "registry-tls". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:16:38.114126 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:16:38.114059    4892 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0e179af5-9f52-4941-b4f5-5e3b8789da11-installation-pull-secrets" pod "0e179af5-9f52-4941-b4f5-5e3b8789da11" (UID: "0e179af5-9f52-4941-b4f5-5e3b8789da11"). InnerVolumeSpecName "installation-pull-secrets". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 19:16:38.114343 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:16:38.114312    4892 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0e179af5-9f52-4941-b4f5-5e3b8789da11-image-registry-private-configuration" pod "0e179af5-9f52-4941-b4f5-5e3b8789da11" (UID: "0e179af5-9f52-4941-b4f5-5e3b8789da11"). InnerVolumeSpecName "image-registry-private-configuration". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 19:16:38.114441 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:16:38.114358    4892 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0e179af5-9f52-4941-b4f5-5e3b8789da11-bound-sa-token" pod "0e179af5-9f52-4941-b4f5-5e3b8789da11" (UID: "0e179af5-9f52-4941-b4f5-5e3b8789da11"). InnerVolumeSpecName "bound-sa-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:16:38.114819 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:16:38.114789    4892 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0e179af5-9f52-4941-b4f5-5e3b8789da11-kube-api-access-zj4df" pod "0e179af5-9f52-4941-b4f5-5e3b8789da11" (UID: "0e179af5-9f52-4941-b4f5-5e3b8789da11"). InnerVolumeSpecName "kube-api-access-zj4df". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:16:38.119107 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:16:38.119086    4892 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0e179af5-9f52-4941-b4f5-5e3b8789da11-ca-trust-extracted" pod "0e179af5-9f52-4941-b4f5-5e3b8789da11" (UID: "0e179af5-9f52-4941-b4f5-5e3b8789da11"). InnerVolumeSpecName "ca-trust-extracted". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:16:38.212550 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:16:38.212527    4892 reconciler_common.go:299] "Volume detached for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/0e179af5-9f52-4941-b4f5-5e3b8789da11-installation-pull-secrets\") on node \"ip-10-0-132-82.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 19:16:38.212550 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:16:38.212550    4892 reconciler_common.go:299] "Volume detached for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/0e179af5-9f52-4941-b4f5-5e3b8789da11-registry-tls\") on node \"ip-10-0-132-82.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 19:16:38.212688 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:16:38.212560    4892 reconciler_common.go:299] "Volume detached for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/0e179af5-9f52-4941-b4f5-5e3b8789da11-bound-sa-token\") on node \"ip-10-0-132-82.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 19:16:38.212688 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:16:38.212569    4892 reconciler_common.go:299] "Volume detached for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/0e179af5-9f52-4941-b4f5-5e3b8789da11-image-registry-private-configuration\") on node \"ip-10-0-132-82.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 19:16:38.212688 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:16:38.212579    4892 reconciler_common.go:299] "Volume detached for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/0e179af5-9f52-4941-b4f5-5e3b8789da11-registry-certificates\") on node \"ip-10-0-132-82.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 19:16:38.212688 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:16:38.212587    4892 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zj4df\" (UniqueName: \"kubernetes.io/projected/0e179af5-9f52-4941-b4f5-5e3b8789da11-kube-api-access-zj4df\") on node \"ip-10-0-132-82.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 19:16:38.212688 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:16:38.212597    4892 reconciler_common.go:299] "Volume detached for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/0e179af5-9f52-4941-b4f5-5e3b8789da11-ca-trust-extracted\") on node \"ip-10-0-132-82.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 19:16:38.838351 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:16:38.838276    4892 generic.go:356] "Generic (PLEG): container finished" podID="0e179af5-9f52-4941-b4f5-5e3b8789da11" containerID="603f404a1881412797abc80bbc11870012a3ddbadbfd3843a35bba336ceacb11" exitCode=0
Oct 30 19:16:38.838351 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:16:38.838326    4892 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-d97c68fd4-htsl8" event={"ID":"0e179af5-9f52-4941-b4f5-5e3b8789da11","Type":"ContainerDied","Data":"603f404a1881412797abc80bbc11870012a3ddbadbfd3843a35bba336ceacb11"}
Oct 30 19:16:38.838793 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:16:38.838370    4892 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-d97c68fd4-htsl8" event={"ID":"0e179af5-9f52-4941-b4f5-5e3b8789da11","Type":"ContainerDied","Data":"2615df6ab32f3d8685fa6ce15777344c099f3aba6418073a8de5475fcc1dbda2"}
Oct 30 19:16:38.838793 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:16:38.838374    4892 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-d97c68fd4-htsl8"
Oct 30 19:16:38.838793 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:16:38.838391    4892 scope.go:122] "RemoveContainer" containerID="603f404a1881412797abc80bbc11870012a3ddbadbfd3843a35bba336ceacb11"
Oct 30 19:16:38.846613 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:16:38.846583    4892 scope.go:122] "RemoveContainer" containerID="603f404a1881412797abc80bbc11870012a3ddbadbfd3843a35bba336ceacb11"
Oct 30 19:16:38.846885 ip-10-0-132-82 kubenswrapper[4892]: E1030 19:16:38.846861    4892 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"603f404a1881412797abc80bbc11870012a3ddbadbfd3843a35bba336ceacb11\": container with ID starting with 603f404a1881412797abc80bbc11870012a3ddbadbfd3843a35bba336ceacb11 not found: ID does not exist" containerID="603f404a1881412797abc80bbc11870012a3ddbadbfd3843a35bba336ceacb11"
Oct 30 19:16:38.846958 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:16:38.846897    4892 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"603f404a1881412797abc80bbc11870012a3ddbadbfd3843a35bba336ceacb11"} err="failed to get container status \"603f404a1881412797abc80bbc11870012a3ddbadbfd3843a35bba336ceacb11\": rpc error: code = NotFound desc = could not find container \"603f404a1881412797abc80bbc11870012a3ddbadbfd3843a35bba336ceacb11\": container with ID starting with 603f404a1881412797abc80bbc11870012a3ddbadbfd3843a35bba336ceacb11 not found: ID does not exist"
Oct 30 19:16:39.052178 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:16:39.052154    4892 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-image-registry/image-registry-d97c68fd4-htsl8"]
Oct 30 19:16:39.101247 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:16:39.101182    4892 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-image-registry/image-registry-d97c68fd4-htsl8"]
Oct 30 19:16:40.567058 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:16:40.567012    4892 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0e179af5-9f52-4941-b4f5-5e3b8789da11" path="/var/lib/kubelet/pods/0e179af5-9f52-4941-b4f5-5e3b8789da11/volumes"
Oct 30 19:16:54.842703 ip-10-0-132-82 kubenswrapper[4892]: E1030 19:16:54.842667    4892 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-jrm87" containerName="extractor"
Oct 30 19:16:58.563921 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:16:58.563854    4892 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-image-registry/image-registry-6fcb4848df-8l8cp" podUID="e3206ddf-35de-4b7c-b979-682f5f666a34" containerName="registry" containerID="cri-o://912fe78ec0b952b60d6c051a74a051174f4a894514b2909b7936bbdb14ddd471" gracePeriod=30
Oct 30 19:16:58.807863 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:16:58.807835    4892 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-6fcb4848df-8l8cp"
Oct 30 19:16:58.859903 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:16:58.859852    4892 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/e3206ddf-35de-4b7c-b979-682f5f666a34-image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/e3206ddf-35de-4b7c-b979-682f5f666a34-image-registry-private-configuration\") pod \"e3206ddf-35de-4b7c-b979-682f5f666a34\" (UID: \"e3206ddf-35de-4b7c-b979-682f5f666a34\") "
Oct 30 19:16:58.859903 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:16:58.859895    4892 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e3206ddf-35de-4b7c-b979-682f5f666a34-ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/e3206ddf-35de-4b7c-b979-682f5f666a34-ca-trust-extracted\") pod \"e3206ddf-35de-4b7c-b979-682f5f666a34\" (UID: \"e3206ddf-35de-4b7c-b979-682f5f666a34\") "
Oct 30 19:16:58.860086 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:16:58.859929    4892 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e3206ddf-35de-4b7c-b979-682f5f666a34-bound-sa-token\" (UniqueName: \"kubernetes.io/projected/e3206ddf-35de-4b7c-b979-682f5f666a34-bound-sa-token\") pod \"e3206ddf-35de-4b7c-b979-682f5f666a34\" (UID: \"e3206ddf-35de-4b7c-b979-682f5f666a34\") "
Oct 30 19:16:58.860086 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:16:58.860007    4892 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e3206ddf-35de-4b7c-b979-682f5f666a34-kube-api-access-w9h4f\" (UniqueName: \"kubernetes.io/projected/e3206ddf-35de-4b7c-b979-682f5f666a34-kube-api-access-w9h4f\") pod \"e3206ddf-35de-4b7c-b979-682f5f666a34\" (UID: \"e3206ddf-35de-4b7c-b979-682f5f666a34\") "
Oct 30 19:16:58.860086 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:16:58.860037    4892 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e3206ddf-35de-4b7c-b979-682f5f666a34-registry-tls\" (UniqueName: \"kubernetes.io/projected/e3206ddf-35de-4b7c-b979-682f5f666a34-registry-tls\") pod \"e3206ddf-35de-4b7c-b979-682f5f666a34\" (UID: \"e3206ddf-35de-4b7c-b979-682f5f666a34\") "
Oct 30 19:16:58.860086 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:16:58.860081    4892 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e3206ddf-35de-4b7c-b979-682f5f666a34-registry-certificates\" (UniqueName: \"kubernetes.io/configmap/e3206ddf-35de-4b7c-b979-682f5f666a34-registry-certificates\") pod \"e3206ddf-35de-4b7c-b979-682f5f666a34\" (UID: \"e3206ddf-35de-4b7c-b979-682f5f666a34\") "
Oct 30 19:16:58.860284 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:16:58.860129    4892 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e3206ddf-35de-4b7c-b979-682f5f666a34-trusted-ca\" (UniqueName: \"kubernetes.io/configmap/e3206ddf-35de-4b7c-b979-682f5f666a34-trusted-ca\") pod \"e3206ddf-35de-4b7c-b979-682f5f666a34\" (UID: \"e3206ddf-35de-4b7c-b979-682f5f666a34\") "
Oct 30 19:16:58.860284 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:16:58.860165    4892 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/e3206ddf-35de-4b7c-b979-682f5f666a34-installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/e3206ddf-35de-4b7c-b979-682f5f666a34-installation-pull-secrets\") pod \"e3206ddf-35de-4b7c-b979-682f5f666a34\" (UID: \"e3206ddf-35de-4b7c-b979-682f5f666a34\") "
Oct 30 19:16:58.860760 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:16:58.860682    4892 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e3206ddf-35de-4b7c-b979-682f5f666a34-registry-certificates" pod "e3206ddf-35de-4b7c-b979-682f5f666a34" (UID: "e3206ddf-35de-4b7c-b979-682f5f666a34"). InnerVolumeSpecName "registry-certificates". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 19:16:58.860760 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:16:58.860695    4892 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e3206ddf-35de-4b7c-b979-682f5f666a34-trusted-ca" pod "e3206ddf-35de-4b7c-b979-682f5f666a34" (UID: "e3206ddf-35de-4b7c-b979-682f5f666a34"). InnerVolumeSpecName "trusted-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 19:16:58.862528 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:16:58.862500    4892 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e3206ddf-35de-4b7c-b979-682f5f666a34-kube-api-access-w9h4f" pod "e3206ddf-35de-4b7c-b979-682f5f666a34" (UID: "e3206ddf-35de-4b7c-b979-682f5f666a34"). InnerVolumeSpecName "kube-api-access-w9h4f". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:16:58.862832 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:16:58.862810    4892 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e3206ddf-35de-4b7c-b979-682f5f666a34-installation-pull-secrets" pod "e3206ddf-35de-4b7c-b979-682f5f666a34" (UID: "e3206ddf-35de-4b7c-b979-682f5f666a34"). InnerVolumeSpecName "installation-pull-secrets". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 19:16:58.862920 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:16:58.862826    4892 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e3206ddf-35de-4b7c-b979-682f5f666a34-bound-sa-token" pod "e3206ddf-35de-4b7c-b979-682f5f666a34" (UID: "e3206ddf-35de-4b7c-b979-682f5f666a34"). InnerVolumeSpecName "bound-sa-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:16:58.862920 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:16:58.862888    4892 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e3206ddf-35de-4b7c-b979-682f5f666a34-image-registry-private-configuration" pod "e3206ddf-35de-4b7c-b979-682f5f666a34" (UID: "e3206ddf-35de-4b7c-b979-682f5f666a34"). InnerVolumeSpecName "image-registry-private-configuration". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 19:16:58.863164 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:16:58.863147    4892 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e3206ddf-35de-4b7c-b979-682f5f666a34-registry-tls" pod "e3206ddf-35de-4b7c-b979-682f5f666a34" (UID: "e3206ddf-35de-4b7c-b979-682f5f666a34"). InnerVolumeSpecName "registry-tls". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:16:58.867148 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:16:58.867123    4892 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e3206ddf-35de-4b7c-b979-682f5f666a34-ca-trust-extracted" pod "e3206ddf-35de-4b7c-b979-682f5f666a34" (UID: "e3206ddf-35de-4b7c-b979-682f5f666a34"). InnerVolumeSpecName "ca-trust-extracted". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:16:58.912364 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:16:58.912336    4892 generic.go:356] "Generic (PLEG): container finished" podID="e3206ddf-35de-4b7c-b979-682f5f666a34" containerID="912fe78ec0b952b60d6c051a74a051174f4a894514b2909b7936bbdb14ddd471" exitCode=0
Oct 30 19:16:58.912464 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:16:58.912381    4892 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-6fcb4848df-8l8cp"
Oct 30 19:16:58.912464 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:16:58.912414    4892 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-6fcb4848df-8l8cp" event={"ID":"e3206ddf-35de-4b7c-b979-682f5f666a34","Type":"ContainerDied","Data":"912fe78ec0b952b60d6c051a74a051174f4a894514b2909b7936bbdb14ddd471"}
Oct 30 19:16:58.912578 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:16:58.912464    4892 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-6fcb4848df-8l8cp" event={"ID":"e3206ddf-35de-4b7c-b979-682f5f666a34","Type":"ContainerDied","Data":"2095a5e43334678ddef80aab03c2453ddf309aa6b37285ab24553cb3dcc86b48"}
Oct 30 19:16:58.912578 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:16:58.912493    4892 scope.go:122] "RemoveContainer" containerID="912fe78ec0b952b60d6c051a74a051174f4a894514b2909b7936bbdb14ddd471"
Oct 30 19:16:58.920540 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:16:58.920519    4892 scope.go:122] "RemoveContainer" containerID="912fe78ec0b952b60d6c051a74a051174f4a894514b2909b7936bbdb14ddd471"
Oct 30 19:16:58.920785 ip-10-0-132-82 kubenswrapper[4892]: E1030 19:16:58.920763    4892 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"912fe78ec0b952b60d6c051a74a051174f4a894514b2909b7936bbdb14ddd471\": container with ID starting with 912fe78ec0b952b60d6c051a74a051174f4a894514b2909b7936bbdb14ddd471 not found: ID does not exist" containerID="912fe78ec0b952b60d6c051a74a051174f4a894514b2909b7936bbdb14ddd471"
Oct 30 19:16:58.920845 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:16:58.920797    4892 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"912fe78ec0b952b60d6c051a74a051174f4a894514b2909b7936bbdb14ddd471"} err="failed to get container status \"912fe78ec0b952b60d6c051a74a051174f4a894514b2909b7936bbdb14ddd471\": rpc error: code = NotFound desc = could not find container \"912fe78ec0b952b60d6c051a74a051174f4a894514b2909b7936bbdb14ddd471\": container with ID starting with 912fe78ec0b952b60d6c051a74a051174f4a894514b2909b7936bbdb14ddd471 not found: ID does not exist"
Oct 30 19:16:58.961324 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:16:58.961304    4892 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-w9h4f\" (UniqueName: \"kubernetes.io/projected/e3206ddf-35de-4b7c-b979-682f5f666a34-kube-api-access-w9h4f\") on node \"ip-10-0-132-82.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 19:16:58.961436 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:16:58.961330    4892 reconciler_common.go:299] "Volume detached for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/e3206ddf-35de-4b7c-b979-682f5f666a34-registry-tls\") on node \"ip-10-0-132-82.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 19:16:58.961436 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:16:58.961346    4892 reconciler_common.go:299] "Volume detached for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/e3206ddf-35de-4b7c-b979-682f5f666a34-registry-certificates\") on node \"ip-10-0-132-82.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 19:16:58.961436 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:16:58.961359    4892 reconciler_common.go:299] "Volume detached for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/e3206ddf-35de-4b7c-b979-682f5f666a34-trusted-ca\") on node \"ip-10-0-132-82.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 19:16:58.961436 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:16:58.961373    4892 reconciler_common.go:299] "Volume detached for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/e3206ddf-35de-4b7c-b979-682f5f666a34-installation-pull-secrets\") on node \"ip-10-0-132-82.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 19:16:58.961436 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:16:58.961386    4892 reconciler_common.go:299] "Volume detached for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/e3206ddf-35de-4b7c-b979-682f5f666a34-image-registry-private-configuration\") on node \"ip-10-0-132-82.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 19:16:58.961436 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:16:58.961400    4892 reconciler_common.go:299] "Volume detached for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/e3206ddf-35de-4b7c-b979-682f5f666a34-ca-trust-extracted\") on node \"ip-10-0-132-82.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 19:16:58.961436 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:16:58.961413    4892 reconciler_common.go:299] "Volume detached for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/e3206ddf-35de-4b7c-b979-682f5f666a34-bound-sa-token\") on node \"ip-10-0-132-82.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 19:16:59.086540 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:16:59.086507    4892 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-image-registry/image-registry-6fcb4848df-8l8cp"]
Oct 30 19:16:59.090859 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:16:59.090837    4892 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-image-registry/image-registry-6fcb4848df-8l8cp"]
Oct 30 19:17:00.570100 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:17:00.567995    4892 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e3206ddf-35de-4b7c-b979-682f5f666a34" path="/var/lib/kubelet/pods/e3206ddf-35de-4b7c-b979-682f5f666a34/volumes"
Oct 30 19:17:08.948253 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:17:08.948220    4892 generic.go:356] "Generic (PLEG): container finished" podID="4ba601a6-2a44-4408-8c5b-2c29fd096fed" containerID="b643e2fa8015107d90d3afd9d3a043cb45baf6be1647fb60a60b39d30b7b255d" exitCode=0
Oct 30 19:17:08.948558 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:17:08.948271    4892 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-nbfv2" event={"ID":"4ba601a6-2a44-4408-8c5b-2c29fd096fed","Type":"ContainerDied","Data":"b643e2fa8015107d90d3afd9d3a043cb45baf6be1647fb60a60b39d30b7b255d"}
Oct 30 19:17:08.948558 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:17:08.948552    4892 scope.go:122] "RemoveContainer" containerID="b643e2fa8015107d90d3afd9d3a043cb45baf6be1647fb60a60b39d30b7b255d"
Oct 30 19:17:09.953558 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:17:09.953525    4892 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-nbfv2" event={"ID":"4ba601a6-2a44-4408-8c5b-2c29fd096fed","Type":"ContainerStarted","Data":"2a0519196404cb4c7f25c838e279712afdecfae5452d845893eb1985738a5e18"}
Oct 30 19:17:18.986398 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:17:18.986363    4892 generic.go:356] "Generic (PLEG): container finished" podID="2bd84dc4-e96a-43f1-85bf-e1ea4005c193" containerID="99317e646b964681db30a7672208a6c1e4f68260f39249e8ede0b027a519c7e8" exitCode=0
Oct 30 19:17:18.986804 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:17:18.986438    4892 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-operator-7c4986d449-6jdkq" event={"ID":"2bd84dc4-e96a-43f1-85bf-e1ea4005c193","Type":"ContainerDied","Data":"99317e646b964681db30a7672208a6c1e4f68260f39249e8ede0b027a519c7e8"}
Oct 30 19:17:18.986804 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:17:18.986750    4892 scope.go:122] "RemoveContainer" containerID="99317e646b964681db30a7672208a6c1e4f68260f39249e8ede0b027a519c7e8"
Oct 30 19:17:18.988284 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:17:18.988256    4892 generic.go:356] "Generic (PLEG): container finished" podID="f9a06263-cb99-4004-bffc-581d66ce5ebf" containerID="4d3b9b7355e8a0ef0308f30c8d15fdb719373def0082085d6a0fa0f9c00888f8" exitCode=0
Oct 30 19:17:18.988362 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:17:18.988305    4892 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-scljn" event={"ID":"f9a06263-cb99-4004-bffc-581d66ce5ebf","Type":"ContainerDied","Data":"4d3b9b7355e8a0ef0308f30c8d15fdb719373def0082085d6a0fa0f9c00888f8"}
Oct 30 19:17:18.988555 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:17:18.988542    4892 scope.go:122] "RemoveContainer" containerID="4d3b9b7355e8a0ef0308f30c8d15fdb719373def0082085d6a0fa0f9c00888f8"
Oct 30 19:17:19.994045 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:17:19.994003    4892 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-scljn" event={"ID":"f9a06263-cb99-4004-bffc-581d66ce5ebf","Type":"ContainerStarted","Data":"b6be51eb568276b71b551c3db962eda4938bbbd02b9f89ef69a41810e74afd58"}
Oct 30 19:17:19.995809 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:17:19.995782    4892 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-operator-7c4986d449-6jdkq" event={"ID":"2bd84dc4-e96a-43f1-85bf-e1ea4005c193","Type":"ContainerStarted","Data":"5d88168d09df197e81a505e1454c131cd8b50986bddd10bc108ba6cc34f00886"}
Oct 30 19:17:24.842605 ip-10-0-132-82 kubenswrapper[4892]: E1030 19:17:24.842570    4892 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-jrm87" containerName="extractor"
Oct 30 19:17:54.843096 ip-10-0-132-82 kubenswrapper[4892]: E1030 19:17:54.843059    4892 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-jrm87" containerName="extractor"
Oct 30 19:18:24.843439 ip-10-0-132-82 kubenswrapper[4892]: E1030 19:18:24.843403    4892 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-jrm87" containerName="extractor"
Oct 30 19:18:54.844548 ip-10-0-132-82 kubenswrapper[4892]: E1030 19:18:54.844512    4892 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-jrm87" containerName="extractor"
Oct 30 19:19:24.845499 ip-10-0-132-82 kubenswrapper[4892]: E1030 19:19:24.845454    4892 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-jrm87" containerName="extractor"
Oct 30 19:19:37.809485 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:19:37.809409    4892 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651ctwr4d"]
Oct 30 19:19:37.809849 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:19:37.809714    4892 cpu_manager.go:414] "containerMap: removing container" podUID="e3206ddf-35de-4b7c-b979-682f5f666a34" containerName="registry"
Oct 30 19:19:37.809849 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:19:37.809725    4892 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e3206ddf-35de-4b7c-b979-682f5f666a34" containerName="registry"
Oct 30 19:19:37.809849 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:19:37.809747    4892 cpu_manager.go:414] "containerMap: removing container" podUID="0e179af5-9f52-4941-b4f5-5e3b8789da11" containerName="registry"
Oct 30 19:19:37.809849 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:19:37.809755    4892 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0e179af5-9f52-4941-b4f5-5e3b8789da11" containerName="registry"
Oct 30 19:19:37.809849 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:19:37.809818    4892 memory_manager.go:365] "RemoveStaleState removing state" podUID="0e179af5-9f52-4941-b4f5-5e3b8789da11" containerName="registry"
Oct 30 19:19:37.809849 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:19:37.809830    4892 memory_manager.go:365] "RemoveStaleState removing state" podUID="e3206ddf-35de-4b7c-b979-682f5f666a34" containerName="registry"
Oct 30 19:19:37.812795 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:19:37.812773    4892 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651ctwr4d"
Oct 30 19:19:37.864672 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:19:37.864647    4892 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"openshift-service-ca.crt\""
Oct 30 19:19:37.914146 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:19:37.914116    4892 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"kube-root-ca.crt\""
Oct 30 19:19:37.914262 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:19:37.914207    4892 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-marketplace\"/\"c7537007944577f64df34960ace3c69136c416-139ec370-dockercfg-p754r\""
Oct 30 19:19:37.915040 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:19:37.915023    4892 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651ctwr4d"]
Oct 30 19:19:37.932375 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:19:37.932352    4892 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/19013e31-30e4-4611-9b72-40ed4348eec6-bundle\") pod \"c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651ctwr4d\" (UID: \"19013e31-30e4-4611-9b72-40ed4348eec6\") " pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651ctwr4d"
Oct 30 19:19:37.932464 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:19:37.932387    4892 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8df7h\" (UniqueName: \"kubernetes.io/projected/19013e31-30e4-4611-9b72-40ed4348eec6-kube-api-access-8df7h\") pod \"c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651ctwr4d\" (UID: \"19013e31-30e4-4611-9b72-40ed4348eec6\") " pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651ctwr4d"
Oct 30 19:19:37.932464 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:19:37.932417    4892 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/19013e31-30e4-4611-9b72-40ed4348eec6-util\") pod \"c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651ctwr4d\" (UID: \"19013e31-30e4-4611-9b72-40ed4348eec6\") " pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651ctwr4d"
Oct 30 19:19:38.033495 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:19:38.033463    4892 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/19013e31-30e4-4611-9b72-40ed4348eec6-bundle\") pod \"c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651ctwr4d\" (UID: \"19013e31-30e4-4611-9b72-40ed4348eec6\") " pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651ctwr4d"
Oct 30 19:19:38.033642 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:19:38.033502    4892 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8df7h\" (UniqueName: \"kubernetes.io/projected/19013e31-30e4-4611-9b72-40ed4348eec6-kube-api-access-8df7h\") pod \"c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651ctwr4d\" (UID: \"19013e31-30e4-4611-9b72-40ed4348eec6\") " pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651ctwr4d"
Oct 30 19:19:38.033642 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:19:38.033532    4892 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/19013e31-30e4-4611-9b72-40ed4348eec6-util\") pod \"c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651ctwr4d\" (UID: \"19013e31-30e4-4611-9b72-40ed4348eec6\") " pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651ctwr4d"
Oct 30 19:19:38.033848 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:19:38.033833    4892 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/19013e31-30e4-4611-9b72-40ed4348eec6-util\") pod \"c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651ctwr4d\" (UID: \"19013e31-30e4-4611-9b72-40ed4348eec6\") " pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651ctwr4d"
Oct 30 19:19:38.033916 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:19:38.033898    4892 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/19013e31-30e4-4611-9b72-40ed4348eec6-bundle\") pod \"c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651ctwr4d\" (UID: \"19013e31-30e4-4611-9b72-40ed4348eec6\") " pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651ctwr4d"
Oct 30 19:19:38.093086 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:19:38.093025    4892 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8df7h\" (UniqueName: \"kubernetes.io/projected/19013e31-30e4-4611-9b72-40ed4348eec6-kube-api-access-8df7h\") pod \"c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651ctwr4d\" (UID: \"19013e31-30e4-4611-9b72-40ed4348eec6\") " pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651ctwr4d"
Oct 30 19:19:38.127838 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:19:38.127809    4892 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651ctwr4d"
Oct 30 19:19:38.497819 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:19:38.497787    4892 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651ctwr4d"]
Oct 30 19:19:38.500781 ip-10-0-132-82 kubenswrapper[4892]: W1030 19:19:38.500752    4892 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod19013e31_30e4_4611_9b72_40ed4348eec6.slice/crio-2a64631d76d4d6fdf9f296643b3b3c5cdd5c55c6ccc065b29f909f6515940268 WatchSource:0}: Error finding container 2a64631d76d4d6fdf9f296643b3b3c5cdd5c55c6ccc065b29f909f6515940268: Status 404 returned error can't find the container with id 2a64631d76d4d6fdf9f296643b3b3c5cdd5c55c6ccc065b29f909f6515940268
Oct 30 19:19:39.476427 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:19:39.476388    4892 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651ctwr4d" event={"ID":"19013e31-30e4-4611-9b72-40ed4348eec6","Type":"ContainerStarted","Data":"2a64631d76d4d6fdf9f296643b3b3c5cdd5c55c6ccc065b29f909f6515940268"}
Oct 30 19:19:45.501221 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:19:45.501157    4892 generic.go:356] "Generic (PLEG): container finished" podID="19013e31-30e4-4611-9b72-40ed4348eec6" containerID="e99a4b490ba59f5079c96621bf9314688689888d045ea576ac40756cc17e269c" exitCode=0
Oct 30 19:19:45.501515 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:19:45.501234    4892 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651ctwr4d" event={"ID":"19013e31-30e4-4611-9b72-40ed4348eec6","Type":"ContainerDied","Data":"e99a4b490ba59f5079c96621bf9314688689888d045ea576ac40756cc17e269c"}
Oct 30 19:19:49.518930 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:19:49.518894    4892 generic.go:356] "Generic (PLEG): container finished" podID="19013e31-30e4-4611-9b72-40ed4348eec6" containerID="3d22338c26e3d06e1b6a582607d9127b1c9004ad7611661912b79115eb93596c" exitCode=0
Oct 30 19:19:49.519437 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:19:49.519008    4892 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651ctwr4d" event={"ID":"19013e31-30e4-4611-9b72-40ed4348eec6","Type":"ContainerDied","Data":"3d22338c26e3d06e1b6a582607d9127b1c9004ad7611661912b79115eb93596c"}
Oct 30 19:19:54.846390 ip-10-0-132-82 kubenswrapper[4892]: E1030 19:19:54.846340    4892 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-jrm87" containerName="extractor"
Oct 30 19:19:56.551576 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:19:56.551490    4892 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651ctwr4d" event={"ID":"19013e31-30e4-4611-9b72-40ed4348eec6","Type":"ContainerStarted","Data":"a87b8a7fe834efab3a3aef93de8257fa776234cfe247169e46e5e081edc37c4a"}
Oct 30 19:19:56.667205 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:19:56.667157    4892 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651ctwr4d" podStartSLOduration=3.044716082 podStartE2EDuration="19.667144339s" totalImagesPullingTime="16.622428257s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:19:37 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:19:56.666925571 +0000 UTC m=+244.237391194" watchObservedRunningTime="2026-10-30 19:19:56.667144339 +0000 UTC m=+244.237609938"
Oct 30 19:19:57.557313 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:19:57.557276    4892 generic.go:356] "Generic (PLEG): container finished" podID="19013e31-30e4-4611-9b72-40ed4348eec6" containerID="a87b8a7fe834efab3a3aef93de8257fa776234cfe247169e46e5e081edc37c4a" exitCode=0
Oct 30 19:19:57.557684 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:19:57.557353    4892 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651ctwr4d" event={"ID":"19013e31-30e4-4611-9b72-40ed4348eec6","Type":"ContainerDied","Data":"a87b8a7fe834efab3a3aef93de8257fa776234cfe247169e46e5e081edc37c4a"}
Oct 30 19:19:58.685236 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:19:58.685212    4892 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651ctwr4d"
Oct 30 19:19:58.799707 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:19:58.799683    4892 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/19013e31-30e4-4611-9b72-40ed4348eec6-kube-api-access-8df7h\" (UniqueName: \"kubernetes.io/projected/19013e31-30e4-4611-9b72-40ed4348eec6-kube-api-access-8df7h\") pod \"19013e31-30e4-4611-9b72-40ed4348eec6\" (UID: \"19013e31-30e4-4611-9b72-40ed4348eec6\") "
Oct 30 19:19:58.799826 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:19:58.799722    4892 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/19013e31-30e4-4611-9b72-40ed4348eec6-bundle\" (UniqueName: \"kubernetes.io/empty-dir/19013e31-30e4-4611-9b72-40ed4348eec6-bundle\") pod \"19013e31-30e4-4611-9b72-40ed4348eec6\" (UID: \"19013e31-30e4-4611-9b72-40ed4348eec6\") "
Oct 30 19:19:58.799826 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:19:58.799747    4892 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/19013e31-30e4-4611-9b72-40ed4348eec6-util\" (UniqueName: \"kubernetes.io/empty-dir/19013e31-30e4-4611-9b72-40ed4348eec6-util\") pod \"19013e31-30e4-4611-9b72-40ed4348eec6\" (UID: \"19013e31-30e4-4611-9b72-40ed4348eec6\") "
Oct 30 19:19:58.800344 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:19:58.800315    4892 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/19013e31-30e4-4611-9b72-40ed4348eec6-bundle" pod "19013e31-30e4-4611-9b72-40ed4348eec6" (UID: "19013e31-30e4-4611-9b72-40ed4348eec6"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:19:58.803648 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:19:58.802210    4892 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/19013e31-30e4-4611-9b72-40ed4348eec6-kube-api-access-8df7h" pod "19013e31-30e4-4611-9b72-40ed4348eec6" (UID: "19013e31-30e4-4611-9b72-40ed4348eec6"). InnerVolumeSpecName "kube-api-access-8df7h". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:19:58.805467 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:19:58.805439    4892 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/19013e31-30e4-4611-9b72-40ed4348eec6-util" pod "19013e31-30e4-4611-9b72-40ed4348eec6" (UID: "19013e31-30e4-4611-9b72-40ed4348eec6"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:19:58.900936 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:19:58.900901    4892 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8df7h\" (UniqueName: \"kubernetes.io/projected/19013e31-30e4-4611-9b72-40ed4348eec6-kube-api-access-8df7h\") on node \"ip-10-0-132-82.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 19:19:58.900936 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:19:58.900937    4892 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/19013e31-30e4-4611-9b72-40ed4348eec6-bundle\") on node \"ip-10-0-132-82.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 19:19:58.901087 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:19:58.900947    4892 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/19013e31-30e4-4611-9b72-40ed4348eec6-util\") on node \"ip-10-0-132-82.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 19:19:59.566782 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:19:59.566749    4892 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651ctwr4d"
Oct 30 19:19:59.566782 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:19:59.566762    4892 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651ctwr4d" event={"ID":"19013e31-30e4-4611-9b72-40ed4348eec6","Type":"ContainerDied","Data":"2a64631d76d4d6fdf9f296643b3b3c5cdd5c55c6ccc065b29f909f6515940268"}
Oct 30 19:19:59.567008 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:19:59.566812    4892 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="2a64631d76d4d6fdf9f296643b3b3c5cdd5c55c6ccc065b29f909f6515940268"
Oct 30 19:20:13.302695 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:20:13.302664    4892 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-marketplace/61b1ff68c7255ee5c5329fe19a5d96293a95788d9734a0a79829e8dc85dncgl"]
Oct 30 19:20:13.304053 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:20:13.302928    4892 cpu_manager.go:414] "containerMap: removing container" podUID="19013e31-30e4-4611-9b72-40ed4348eec6" containerName="util"
Oct 30 19:20:13.304053 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:20:13.302938    4892 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="19013e31-30e4-4611-9b72-40ed4348eec6" containerName="util"
Oct 30 19:20:13.304053 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:20:13.302949    4892 cpu_manager.go:414] "containerMap: removing container" podUID="19013e31-30e4-4611-9b72-40ed4348eec6" containerName="pull"
Oct 30 19:20:13.304053 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:20:13.302954    4892 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="19013e31-30e4-4611-9b72-40ed4348eec6" containerName="pull"
Oct 30 19:20:13.304053 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:20:13.302998    4892 cpu_manager.go:414] "containerMap: removing container" podUID="19013e31-30e4-4611-9b72-40ed4348eec6" containerName="extract"
Oct 30 19:20:13.304053 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:20:13.303004    4892 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="19013e31-30e4-4611-9b72-40ed4348eec6" containerName="extract"
Oct 30 19:20:13.304053 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:20:13.303064    4892 memory_manager.go:365] "RemoveStaleState removing state" podUID="19013e31-30e4-4611-9b72-40ed4348eec6" containerName="extract"
Oct 30 19:20:13.304665 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:20:13.304648    4892 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/61b1ff68c7255ee5c5329fe19a5d96293a95788d9734a0a79829e8dc85dncgl"
Oct 30 19:20:13.356399 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:20:13.356372    4892 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"openshift-service-ca.crt\""
Oct 30 19:20:13.405918 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:20:13.405881    4892 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"kube-root-ca.crt\""
Oct 30 19:20:13.406069 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:20:13.405926    4892 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-marketplace\"/\"61b1ff68c7255ee5c5329fe19a5d96293a9578-ce91a85d-dockercfg-98sg9\""
Oct 30 19:20:13.406997 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:20:13.406978    4892 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/61b1ff68c7255ee5c5329fe19a5d96293a95788d9734a0a79829e8dc85dncgl"]
Oct 30 19:20:13.500824 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:20:13.500798    4892 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/2e5bafff-671b-4a5d-b129-19f982f9abfb-bundle\") pod \"61b1ff68c7255ee5c5329fe19a5d96293a95788d9734a0a79829e8dc85dncgl\" (UID: \"2e5bafff-671b-4a5d-b129-19f982f9abfb\") " pod="openshift-marketplace/61b1ff68c7255ee5c5329fe19a5d96293a95788d9734a0a79829e8dc85dncgl"
Oct 30 19:20:13.500824 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:20:13.500826    4892 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/2e5bafff-671b-4a5d-b129-19f982f9abfb-util\") pod \"61b1ff68c7255ee5c5329fe19a5d96293a95788d9734a0a79829e8dc85dncgl\" (UID: \"2e5bafff-671b-4a5d-b129-19f982f9abfb\") " pod="openshift-marketplace/61b1ff68c7255ee5c5329fe19a5d96293a95788d9734a0a79829e8dc85dncgl"
Oct 30 19:20:13.500995 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:20:13.500859    4892 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vlhqq\" (UniqueName: \"kubernetes.io/projected/2e5bafff-671b-4a5d-b129-19f982f9abfb-kube-api-access-vlhqq\") pod \"61b1ff68c7255ee5c5329fe19a5d96293a95788d9734a0a79829e8dc85dncgl\" (UID: \"2e5bafff-671b-4a5d-b129-19f982f9abfb\") " pod="openshift-marketplace/61b1ff68c7255ee5c5329fe19a5d96293a95788d9734a0a79829e8dc85dncgl"
Oct 30 19:20:13.602124 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:20:13.602068    4892 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/2e5bafff-671b-4a5d-b129-19f982f9abfb-bundle\") pod \"61b1ff68c7255ee5c5329fe19a5d96293a95788d9734a0a79829e8dc85dncgl\" (UID: \"2e5bafff-671b-4a5d-b129-19f982f9abfb\") " pod="openshift-marketplace/61b1ff68c7255ee5c5329fe19a5d96293a95788d9734a0a79829e8dc85dncgl"
Oct 30 19:20:13.602124 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:20:13.602094    4892 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/2e5bafff-671b-4a5d-b129-19f982f9abfb-util\") pod \"61b1ff68c7255ee5c5329fe19a5d96293a95788d9734a0a79829e8dc85dncgl\" (UID: \"2e5bafff-671b-4a5d-b129-19f982f9abfb\") " pod="openshift-marketplace/61b1ff68c7255ee5c5329fe19a5d96293a95788d9734a0a79829e8dc85dncgl"
Oct 30 19:20:13.602250 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:20:13.602140    4892 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vlhqq\" (UniqueName: \"kubernetes.io/projected/2e5bafff-671b-4a5d-b129-19f982f9abfb-kube-api-access-vlhqq\") pod \"61b1ff68c7255ee5c5329fe19a5d96293a95788d9734a0a79829e8dc85dncgl\" (UID: \"2e5bafff-671b-4a5d-b129-19f982f9abfb\") " pod="openshift-marketplace/61b1ff68c7255ee5c5329fe19a5d96293a95788d9734a0a79829e8dc85dncgl"
Oct 30 19:20:13.602463 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:20:13.602445    4892 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/2e5bafff-671b-4a5d-b129-19f982f9abfb-bundle\") pod \"61b1ff68c7255ee5c5329fe19a5d96293a95788d9734a0a79829e8dc85dncgl\" (UID: \"2e5bafff-671b-4a5d-b129-19f982f9abfb\") " pod="openshift-marketplace/61b1ff68c7255ee5c5329fe19a5d96293a95788d9734a0a79829e8dc85dncgl"
Oct 30 19:20:13.602506 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:20:13.602487    4892 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/2e5bafff-671b-4a5d-b129-19f982f9abfb-util\") pod \"61b1ff68c7255ee5c5329fe19a5d96293a95788d9734a0a79829e8dc85dncgl\" (UID: \"2e5bafff-671b-4a5d-b129-19f982f9abfb\") " pod="openshift-marketplace/61b1ff68c7255ee5c5329fe19a5d96293a95788d9734a0a79829e8dc85dncgl"
Oct 30 19:20:13.668063 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:20:13.668036    4892 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vlhqq\" (UniqueName: \"kubernetes.io/projected/2e5bafff-671b-4a5d-b129-19f982f9abfb-kube-api-access-vlhqq\") pod \"61b1ff68c7255ee5c5329fe19a5d96293a95788d9734a0a79829e8dc85dncgl\" (UID: \"2e5bafff-671b-4a5d-b129-19f982f9abfb\") " pod="openshift-marketplace/61b1ff68c7255ee5c5329fe19a5d96293a95788d9734a0a79829e8dc85dncgl"
Oct 30 19:20:13.916815 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:20:13.916788    4892 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/61b1ff68c7255ee5c5329fe19a5d96293a95788d9734a0a79829e8dc85dncgl"
Oct 30 19:20:14.283778 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:20:14.283707    4892 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/61b1ff68c7255ee5c5329fe19a5d96293a95788d9734a0a79829e8dc85dncgl"]
Oct 30 19:20:14.286946 ip-10-0-132-82 kubenswrapper[4892]: W1030 19:20:14.286918    4892 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2e5bafff_671b_4a5d_b129_19f982f9abfb.slice/crio-3fe03dea6724cdaf9f3185da8f0746c538efe53d11771cd29787f2af2fc32839 WatchSource:0}: Error finding container 3fe03dea6724cdaf9f3185da8f0746c538efe53d11771cd29787f2af2fc32839: Status 404 returned error can't find the container with id 3fe03dea6724cdaf9f3185da8f0746c538efe53d11771cd29787f2af2fc32839
Oct 30 19:20:14.618153 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:20:14.618068    4892 generic.go:356] "Generic (PLEG): container finished" podID="2e5bafff-671b-4a5d-b129-19f982f9abfb" containerID="41438dac024b0d506652c14a87a3d300aaedb1c2107621310aeb02488849ff57" exitCode=0
Oct 30 19:20:14.618153 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:20:14.618120    4892 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/61b1ff68c7255ee5c5329fe19a5d96293a95788d9734a0a79829e8dc85dncgl" event={"ID":"2e5bafff-671b-4a5d-b129-19f982f9abfb","Type":"ContainerDied","Data":"41438dac024b0d506652c14a87a3d300aaedb1c2107621310aeb02488849ff57"}
Oct 30 19:20:14.618594 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:20:14.618160    4892 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/61b1ff68c7255ee5c5329fe19a5d96293a95788d9734a0a79829e8dc85dncgl" event={"ID":"2e5bafff-671b-4a5d-b129-19f982f9abfb","Type":"ContainerStarted","Data":"3fe03dea6724cdaf9f3185da8f0746c538efe53d11771cd29787f2af2fc32839"}
Oct 30 19:20:14.913720 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:20:14.913689    4892 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["cert-manager/cert-manager-cainjector-5c57bbd4fc-lvhhj"]
Oct 30 19:20:14.915719 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:20:14.915704    4892 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager/cert-manager-cainjector-5c57bbd4fc-lvhhj"
Oct 30 19:20:14.967331 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:20:14.967301    4892 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"cert-manager\"/\"cert-manager-cainjector-dockercfg-xlbd6\""
Oct 30 19:20:14.967331 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:20:14.967301    4892 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"cert-manager\"/\"openshift-service-ca.crt\""
Oct 30 19:20:14.967475 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:20:14.967409    4892 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"cert-manager\"/\"kube-root-ca.crt\""
Oct 30 19:20:14.974017 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:20:14.973987    4892 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["cert-manager/cert-manager-cainjector-5c57bbd4fc-lvhhj"]
Oct 30 19:20:15.014606 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:20:15.014573    4892 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/d19ede36-411d-42e3-a2b8-b605c3bdc138-bound-sa-token\") pod \"cert-manager-cainjector-5c57bbd4fc-lvhhj\" (UID: \"d19ede36-411d-42e3-a2b8-b605c3bdc138\") " pod="cert-manager/cert-manager-cainjector-5c57bbd4fc-lvhhj"
Oct 30 19:20:15.018821 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:20:15.014748    4892 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rzdcq\" (UniqueName: \"kubernetes.io/projected/d19ede36-411d-42e3-a2b8-b605c3bdc138-kube-api-access-rzdcq\") pod \"cert-manager-cainjector-5c57bbd4fc-lvhhj\" (UID: \"d19ede36-411d-42e3-a2b8-b605c3bdc138\") " pod="cert-manager/cert-manager-cainjector-5c57bbd4fc-lvhhj"
Oct 30 19:20:15.115921 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:20:15.115894    4892 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/d19ede36-411d-42e3-a2b8-b605c3bdc138-bound-sa-token\") pod \"cert-manager-cainjector-5c57bbd4fc-lvhhj\" (UID: \"d19ede36-411d-42e3-a2b8-b605c3bdc138\") " pod="cert-manager/cert-manager-cainjector-5c57bbd4fc-lvhhj"
Oct 30 19:20:15.116069 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:20:15.115940    4892 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rzdcq\" (UniqueName: \"kubernetes.io/projected/d19ede36-411d-42e3-a2b8-b605c3bdc138-kube-api-access-rzdcq\") pod \"cert-manager-cainjector-5c57bbd4fc-lvhhj\" (UID: \"d19ede36-411d-42e3-a2b8-b605c3bdc138\") " pod="cert-manager/cert-manager-cainjector-5c57bbd4fc-lvhhj"
Oct 30 19:20:15.174759 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:20:15.174696    4892 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/d19ede36-411d-42e3-a2b8-b605c3bdc138-bound-sa-token\") pod \"cert-manager-cainjector-5c57bbd4fc-lvhhj\" (UID: \"d19ede36-411d-42e3-a2b8-b605c3bdc138\") " pod="cert-manager/cert-manager-cainjector-5c57bbd4fc-lvhhj"
Oct 30 19:20:15.174858 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:20:15.174762    4892 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rzdcq\" (UniqueName: \"kubernetes.io/projected/d19ede36-411d-42e3-a2b8-b605c3bdc138-kube-api-access-rzdcq\") pod \"cert-manager-cainjector-5c57bbd4fc-lvhhj\" (UID: \"d19ede36-411d-42e3-a2b8-b605c3bdc138\") " pod="cert-manager/cert-manager-cainjector-5c57bbd4fc-lvhhj"
Oct 30 19:20:15.238461 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:20:15.238436    4892 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager/cert-manager-cainjector-5c57bbd4fc-lvhhj"
Oct 30 19:20:15.603569 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:20:15.603492    4892 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["cert-manager/cert-manager-cainjector-5c57bbd4fc-lvhhj"]
Oct 30 19:20:15.606458 ip-10-0-132-82 kubenswrapper[4892]: W1030 19:20:15.606431    4892 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podd19ede36_411d_42e3_a2b8_b605c3bdc138.slice/crio-f54d445a9ca2a48e9bd745525e63c02e51d214606bfe723faadb34a37e857f4d WatchSource:0}: Error finding container f54d445a9ca2a48e9bd745525e63c02e51d214606bfe723faadb34a37e857f4d: Status 404 returned error can't find the container with id f54d445a9ca2a48e9bd745525e63c02e51d214606bfe723faadb34a37e857f4d
Oct 30 19:20:15.622572 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:20:15.622551    4892 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-cainjector-5c57bbd4fc-lvhhj" event={"ID":"d19ede36-411d-42e3-a2b8-b605c3bdc138","Type":"ContainerStarted","Data":"f54d445a9ca2a48e9bd745525e63c02e51d214606bfe723faadb34a37e857f4d"}
Oct 30 19:20:18.637290 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:20:18.637260    4892 generic.go:356] "Generic (PLEG): container finished" podID="2e5bafff-671b-4a5d-b129-19f982f9abfb" containerID="35b09e107c6f3f7db74e1c950942fb4fd2762220edf07c5016c06e7f65d8932e" exitCode=0
Oct 30 19:20:18.637674 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:20:18.637345    4892 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/61b1ff68c7255ee5c5329fe19a5d96293a95788d9734a0a79829e8dc85dncgl" event={"ID":"2e5bafff-671b-4a5d-b129-19f982f9abfb","Type":"ContainerDied","Data":"35b09e107c6f3f7db74e1c950942fb4fd2762220edf07c5016c06e7f65d8932e"}
Oct 30 19:20:19.645234 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:20:19.645196    4892 generic.go:356] "Generic (PLEG): container finished" podID="2e5bafff-671b-4a5d-b129-19f982f9abfb" containerID="4bbfe0e3384508c523cfc9fbd9e43c090ed73ea6b15f0c8c95dfa93b8c019d6b" exitCode=0
Oct 30 19:20:19.645650 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:20:19.645284    4892 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/61b1ff68c7255ee5c5329fe19a5d96293a95788d9734a0a79829e8dc85dncgl" event={"ID":"2e5bafff-671b-4a5d-b129-19f982f9abfb","Type":"ContainerDied","Data":"4bbfe0e3384508c523cfc9fbd9e43c090ed73ea6b15f0c8c95dfa93b8c019d6b"}
Oct 30 19:20:19.647031 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:20:19.646999    4892 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-cainjector-5c57bbd4fc-lvhhj" event={"ID":"d19ede36-411d-42e3-a2b8-b605c3bdc138","Type":"ContainerStarted","Data":"564ed4ca4b47d1644ab28f3c18938ba0a8825e8ecf3399f10d7dfe9e93da114f"}
Oct 30 19:20:19.884786 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:20:19.884736    4892 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="cert-manager/cert-manager-cainjector-5c57bbd4fc-lvhhj" podStartSLOduration=2.234246611 podStartE2EDuration="5.884724095s" totalImagesPullingTime="3.650477484s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:20:14 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:20:19.884092738 +0000 UTC m=+267.454558337" watchObservedRunningTime="2026-10-30 19:20:19.884724095 +0000 UTC m=+267.455189691"
Oct 30 19:20:20.769662 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:20:20.769638    4892 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/61b1ff68c7255ee5c5329fe19a5d96293a95788d9734a0a79829e8dc85dncgl"
Oct 30 19:20:20.856630 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:20:20.856603    4892 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2e5bafff-671b-4a5d-b129-19f982f9abfb-bundle\" (UniqueName: \"kubernetes.io/empty-dir/2e5bafff-671b-4a5d-b129-19f982f9abfb-bundle\") pod \"2e5bafff-671b-4a5d-b129-19f982f9abfb\" (UID: \"2e5bafff-671b-4a5d-b129-19f982f9abfb\") "
Oct 30 19:20:20.856744 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:20:20.856648    4892 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2e5bafff-671b-4a5d-b129-19f982f9abfb-util\" (UniqueName: \"kubernetes.io/empty-dir/2e5bafff-671b-4a5d-b129-19f982f9abfb-util\") pod \"2e5bafff-671b-4a5d-b129-19f982f9abfb\" (UID: \"2e5bafff-671b-4a5d-b129-19f982f9abfb\") "
Oct 30 19:20:20.856744 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:20:20.856679    4892 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2e5bafff-671b-4a5d-b129-19f982f9abfb-kube-api-access-vlhqq\" (UniqueName: \"kubernetes.io/projected/2e5bafff-671b-4a5d-b129-19f982f9abfb-kube-api-access-vlhqq\") pod \"2e5bafff-671b-4a5d-b129-19f982f9abfb\" (UID: \"2e5bafff-671b-4a5d-b129-19f982f9abfb\") "
Oct 30 19:20:20.857054 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:20:20.857031    4892 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2e5bafff-671b-4a5d-b129-19f982f9abfb-bundle" pod "2e5bafff-671b-4a5d-b129-19f982f9abfb" (UID: "2e5bafff-671b-4a5d-b129-19f982f9abfb"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:20:20.858828 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:20:20.858802    4892 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2e5bafff-671b-4a5d-b129-19f982f9abfb-kube-api-access-vlhqq" pod "2e5bafff-671b-4a5d-b129-19f982f9abfb" (UID: "2e5bafff-671b-4a5d-b129-19f982f9abfb"). InnerVolumeSpecName "kube-api-access-vlhqq". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:20:20.861335 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:20:20.861301    4892 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2e5bafff-671b-4a5d-b129-19f982f9abfb-util" pod "2e5bafff-671b-4a5d-b129-19f982f9abfb" (UID: "2e5bafff-671b-4a5d-b129-19f982f9abfb"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:20:20.957926 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:20:20.957861    4892 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vlhqq\" (UniqueName: \"kubernetes.io/projected/2e5bafff-671b-4a5d-b129-19f982f9abfb-kube-api-access-vlhqq\") on node \"ip-10-0-132-82.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 19:20:20.957926 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:20:20.957895    4892 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/2e5bafff-671b-4a5d-b129-19f982f9abfb-bundle\") on node \"ip-10-0-132-82.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 19:20:20.957926 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:20:20.957905    4892 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/2e5bafff-671b-4a5d-b129-19f982f9abfb-util\") on node \"ip-10-0-132-82.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 19:20:21.659100 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:20:21.659069    4892 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/61b1ff68c7255ee5c5329fe19a5d96293a95788d9734a0a79829e8dc85dncgl"
Oct 30 19:20:21.659306 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:20:21.659061    4892 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/61b1ff68c7255ee5c5329fe19a5d96293a95788d9734a0a79829e8dc85dncgl" event={"ID":"2e5bafff-671b-4a5d-b129-19f982f9abfb","Type":"ContainerDied","Data":"3fe03dea6724cdaf9f3185da8f0746c538efe53d11771cd29787f2af2fc32839"}
Oct 30 19:20:21.659306 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:20:21.659197    4892 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="3fe03dea6724cdaf9f3185da8f0746c538efe53d11771cd29787f2af2fc32839"
Oct 30 19:20:24.843630 ip-10-0-132-82 kubenswrapper[4892]: E1030 19:20:24.843593    4892 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-jrm87" containerName="extractor"
Oct 30 19:20:28.132708 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:20:28.132667    4892 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-lws-operator/openshift-lws-operator-8465994787-jfmc6"]
Oct 30 19:20:28.133331 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:20:28.133091    4892 cpu_manager.go:414] "containerMap: removing container" podUID="2e5bafff-671b-4a5d-b129-19f982f9abfb" containerName="util"
Oct 30 19:20:28.133331 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:20:28.133108    4892 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2e5bafff-671b-4a5d-b129-19f982f9abfb" containerName="util"
Oct 30 19:20:28.133331 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:20:28.133125    4892 cpu_manager.go:414] "containerMap: removing container" podUID="2e5bafff-671b-4a5d-b129-19f982f9abfb" containerName="extract"
Oct 30 19:20:28.133331 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:20:28.133132    4892 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2e5bafff-671b-4a5d-b129-19f982f9abfb" containerName="extract"
Oct 30 19:20:28.133331 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:20:28.133167    4892 cpu_manager.go:414] "containerMap: removing container" podUID="2e5bafff-671b-4a5d-b129-19f982f9abfb" containerName="pull"
Oct 30 19:20:28.133331 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:20:28.133174    4892 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2e5bafff-671b-4a5d-b129-19f982f9abfb" containerName="pull"
Oct 30 19:20:28.133331 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:20:28.133277    4892 memory_manager.go:365] "RemoveStaleState removing state" podUID="2e5bafff-671b-4a5d-b129-19f982f9abfb" containerName="extract"
Oct 30 19:20:28.136552 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:20:28.136531    4892 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-lws-operator/openshift-lws-operator-8465994787-jfmc6"
Oct 30 19:20:28.208916 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:20:28.208888    4892 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/a4d8b652-fa8c-43a8-87f4-d293ad856b2e-tmp\") pod \"openshift-lws-operator-8465994787-jfmc6\" (UID: \"a4d8b652-fa8c-43a8-87f4-d293ad856b2e\") " pod="openshift-lws-operator/openshift-lws-operator-8465994787-jfmc6"
Oct 30 19:20:28.209035 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:20:28.208942    4892 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hrl7d\" (UniqueName: \"kubernetes.io/projected/a4d8b652-fa8c-43a8-87f4-d293ad856b2e-kube-api-access-hrl7d\") pod \"openshift-lws-operator-8465994787-jfmc6\" (UID: \"a4d8b652-fa8c-43a8-87f4-d293ad856b2e\") " pod="openshift-lws-operator/openshift-lws-operator-8465994787-jfmc6"
Oct 30 19:20:28.240733 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:20:28.240706    4892 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-lws-operator\"/\"openshift-service-ca.crt\""
Oct 30 19:20:28.240861 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:20:28.240715    4892 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-lws-operator\"/\"kube-root-ca.crt\""
Oct 30 19:20:28.240861 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:20:28.240767    4892 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-lws-operator\"/\"openshift-lws-operator-dockercfg-d2mvd\""
Oct 30 19:20:28.241884 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:20:28.241863    4892 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-lws-operator/openshift-lws-operator-8465994787-jfmc6"]
Oct 30 19:20:28.309927 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:20:28.309905    4892 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/a4d8b652-fa8c-43a8-87f4-d293ad856b2e-tmp\") pod \"openshift-lws-operator-8465994787-jfmc6\" (UID: \"a4d8b652-fa8c-43a8-87f4-d293ad856b2e\") " pod="openshift-lws-operator/openshift-lws-operator-8465994787-jfmc6"
Oct 30 19:20:28.310051 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:20:28.309954    4892 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hrl7d\" (UniqueName: \"kubernetes.io/projected/a4d8b652-fa8c-43a8-87f4-d293ad856b2e-kube-api-access-hrl7d\") pod \"openshift-lws-operator-8465994787-jfmc6\" (UID: \"a4d8b652-fa8c-43a8-87f4-d293ad856b2e\") " pod="openshift-lws-operator/openshift-lws-operator-8465994787-jfmc6"
Oct 30 19:20:28.310266 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:20:28.310248    4892 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/a4d8b652-fa8c-43a8-87f4-d293ad856b2e-tmp\") pod \"openshift-lws-operator-8465994787-jfmc6\" (UID: \"a4d8b652-fa8c-43a8-87f4-d293ad856b2e\") " pod="openshift-lws-operator/openshift-lws-operator-8465994787-jfmc6"
Oct 30 19:20:28.367644 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:20:28.367613    4892 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hrl7d\" (UniqueName: \"kubernetes.io/projected/a4d8b652-fa8c-43a8-87f4-d293ad856b2e-kube-api-access-hrl7d\") pod \"openshift-lws-operator-8465994787-jfmc6\" (UID: \"a4d8b652-fa8c-43a8-87f4-d293ad856b2e\") " pod="openshift-lws-operator/openshift-lws-operator-8465994787-jfmc6"
Oct 30 19:20:28.447052 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:20:28.447030    4892 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-lws-operator/openshift-lws-operator-8465994787-jfmc6"
Oct 30 19:20:28.814476 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:20:28.814403    4892 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-lws-operator/openshift-lws-operator-8465994787-jfmc6"]
Oct 30 19:20:28.817952 ip-10-0-132-82 kubenswrapper[4892]: W1030 19:20:28.817927    4892 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-poda4d8b652_fa8c_43a8_87f4_d293ad856b2e.slice/crio-c54fcb5fe96d0d2b407c2de8181bdac51c37e05b80259343fe45a19df0a61800 WatchSource:0}: Error finding container c54fcb5fe96d0d2b407c2de8181bdac51c37e05b80259343fe45a19df0a61800: Status 404 returned error can't find the container with id c54fcb5fe96d0d2b407c2de8181bdac51c37e05b80259343fe45a19df0a61800
Oct 30 19:20:29.691692 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:20:29.691649    4892 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-lws-operator/openshift-lws-operator-8465994787-jfmc6" event={"ID":"a4d8b652-fa8c-43a8-87f4-d293ad856b2e","Type":"ContainerStarted","Data":"c54fcb5fe96d0d2b407c2de8181bdac51c37e05b80259343fe45a19df0a61800"}
Oct 30 19:20:32.678821 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:20:32.678792    4892 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["cert-manager/cert-manager-597b95cb84-cfdr5"]
Oct 30 19:20:32.701412 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:20:32.701393    4892 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager/cert-manager-597b95cb84-cfdr5"
Oct 30 19:20:32.707986 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:20:32.707934    4892 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-lws-operator/openshift-lws-operator-8465994787-jfmc6" event={"ID":"a4d8b652-fa8c-43a8-87f4-d293ad856b2e","Type":"ContainerStarted","Data":"e06d94143dfd6f55bc94e7d0eaeb7c39c6cc81ca27a708cb8f8e4f3d81caf775"}
Oct 30 19:20:32.740291 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:20:32.740266    4892 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["cert-manager/cert-manager-597b95cb84-cfdr5"]
Oct 30 19:20:32.743287 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:20:32.743255    4892 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/717b7d09-3af8-4fc3-876f-d8dd38b0339a-bound-sa-token\") pod \"cert-manager-597b95cb84-cfdr5\" (UID: \"717b7d09-3af8-4fc3-876f-d8dd38b0339a\") " pod="cert-manager/cert-manager-597b95cb84-cfdr5"
Oct 30 19:20:32.743387 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:20:32.743301    4892 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-m7lpv\" (UniqueName: \"kubernetes.io/projected/717b7d09-3af8-4fc3-876f-d8dd38b0339a-kube-api-access-m7lpv\") pod \"cert-manager-597b95cb84-cfdr5\" (UID: \"717b7d09-3af8-4fc3-876f-d8dd38b0339a\") " pod="cert-manager/cert-manager-597b95cb84-cfdr5"
Oct 30 19:20:32.788440 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:20:32.788418    4892 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"cert-manager\"/\"cert-manager-dockercfg-zm6zf\""
Oct 30 19:20:32.843880 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:20:32.843857    4892 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/717b7d09-3af8-4fc3-876f-d8dd38b0339a-bound-sa-token\") pod \"cert-manager-597b95cb84-cfdr5\" (UID: \"717b7d09-3af8-4fc3-876f-d8dd38b0339a\") " pod="cert-manager/cert-manager-597b95cb84-cfdr5"
Oct 30 19:20:32.843993 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:20:32.843892    4892 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-m7lpv\" (UniqueName: \"kubernetes.io/projected/717b7d09-3af8-4fc3-876f-d8dd38b0339a-kube-api-access-m7lpv\") pod \"cert-manager-597b95cb84-cfdr5\" (UID: \"717b7d09-3af8-4fc3-876f-d8dd38b0339a\") " pod="cert-manager/cert-manager-597b95cb84-cfdr5"
Oct 30 19:20:32.900696 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:20:32.900673    4892 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/717b7d09-3af8-4fc3-876f-d8dd38b0339a-bound-sa-token\") pod \"cert-manager-597b95cb84-cfdr5\" (UID: \"717b7d09-3af8-4fc3-876f-d8dd38b0339a\") " pod="cert-manager/cert-manager-597b95cb84-cfdr5"
Oct 30 19:20:32.900782 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:20:32.900741    4892 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-m7lpv\" (UniqueName: \"kubernetes.io/projected/717b7d09-3af8-4fc3-876f-d8dd38b0339a-kube-api-access-m7lpv\") pod \"cert-manager-597b95cb84-cfdr5\" (UID: \"717b7d09-3af8-4fc3-876f-d8dd38b0339a\") " pod="cert-manager/cert-manager-597b95cb84-cfdr5"
Oct 30 19:20:32.929561 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:20:32.929492    4892 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-lws-operator/openshift-lws-operator-8465994787-jfmc6" podStartSLOduration=1.407199256 podStartE2EDuration="4.929481434s" totalImagesPullingTime="3.522282178s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:20:28 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:20:32.928120488 +0000 UTC m=+280.498586088" watchObservedRunningTime="2026-10-30 19:20:32.929481434 +0000 UTC m=+280.499947032"
Oct 30 19:20:33.022169 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:20:33.022146    4892 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager/cert-manager-597b95cb84-cfdr5"
Oct 30 19:20:33.408450 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:20:33.408420    4892 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["cert-manager/cert-manager-597b95cb84-cfdr5"]
Oct 30 19:20:33.411473 ip-10-0-132-82 kubenswrapper[4892]: W1030 19:20:33.411446    4892 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod717b7d09_3af8_4fc3_876f_d8dd38b0339a.slice/crio-e2e5915abc0087ccf487baef8d68bdb9fcf36ea79b46525ae53b2ed4122cf989 WatchSource:0}: Error finding container e2e5915abc0087ccf487baef8d68bdb9fcf36ea79b46525ae53b2ed4122cf989: Status 404 returned error can't find the container with id e2e5915abc0087ccf487baef8d68bdb9fcf36ea79b46525ae53b2ed4122cf989
Oct 30 19:20:33.713736 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:20:33.713651    4892 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-597b95cb84-cfdr5" event={"ID":"717b7d09-3af8-4fc3-876f-d8dd38b0339a","Type":"ContainerStarted","Data":"0f73f11ce37cc111fbc4fb50a76b261d6e118506068248244a193150787ae718"}
Oct 30 19:20:33.713736 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:20:33.713699    4892 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-597b95cb84-cfdr5" event={"ID":"717b7d09-3af8-4fc3-876f-d8dd38b0339a","Type":"ContainerStarted","Data":"e2e5915abc0087ccf487baef8d68bdb9fcf36ea79b46525ae53b2ed4122cf989"}
Oct 30 19:20:33.828163 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:20:33.828114    4892 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="cert-manager/cert-manager-597b95cb84-cfdr5" podStartSLOduration=1.828097717 podStartE2EDuration="1.828097717s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:20:32 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:20:33.827525404 +0000 UTC m=+281.397991026" watchObservedRunningTime="2026-10-30 19:20:33.828097717 +0000 UTC m=+281.398563314"
Oct 30 19:20:35.836913 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:20:35.836882    4892 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71x8lx7"]
Oct 30 19:20:35.839219 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:20:35.839203    4892 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71x8lx7"
Oct 30 19:20:35.863896 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:20:35.863872    4892 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2xtfk\" (UniqueName: \"kubernetes.io/projected/1b64221d-b6a4-4123-86a6-0fdfa035bf97-kube-api-access-2xtfk\") pod \"e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71x8lx7\" (UID: \"1b64221d-b6a4-4123-86a6-0fdfa035bf97\") " pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71x8lx7"
Oct 30 19:20:35.864036 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:20:35.863920    4892 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/1b64221d-b6a4-4123-86a6-0fdfa035bf97-util\") pod \"e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71x8lx7\" (UID: \"1b64221d-b6a4-4123-86a6-0fdfa035bf97\") " pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71x8lx7"
Oct 30 19:20:35.864115 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:20:35.864066    4892 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/1b64221d-b6a4-4123-86a6-0fdfa035bf97-bundle\") pod \"e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71x8lx7\" (UID: \"1b64221d-b6a4-4123-86a6-0fdfa035bf97\") " pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71x8lx7"
Oct 30 19:20:35.940797 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:20:35.940769    4892 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"kube-root-ca.crt\""
Oct 30 19:20:35.940925 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:20:35.940847    4892 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"openshift-service-ca.crt\""
Oct 30 19:20:35.940925 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:20:35.940887    4892 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-marketplace\"/\"e644421527c169d876fdb59235c927986d6bd2-9d2815fc-dockercfg-s5xz7\""
Oct 30 19:20:35.942108 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:20:35.942090    4892 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71x8lx7"]
Oct 30 19:20:35.965281 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:20:35.965261    4892 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/1b64221d-b6a4-4123-86a6-0fdfa035bf97-util\") pod \"e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71x8lx7\" (UID: \"1b64221d-b6a4-4123-86a6-0fdfa035bf97\") " pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71x8lx7"
Oct 30 19:20:35.965349 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:20:35.965301    4892 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/1b64221d-b6a4-4123-86a6-0fdfa035bf97-bundle\") pod \"e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71x8lx7\" (UID: \"1b64221d-b6a4-4123-86a6-0fdfa035bf97\") " pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71x8lx7"
Oct 30 19:20:35.965349 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:20:35.965322    4892 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2xtfk\" (UniqueName: \"kubernetes.io/projected/1b64221d-b6a4-4123-86a6-0fdfa035bf97-kube-api-access-2xtfk\") pod \"e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71x8lx7\" (UID: \"1b64221d-b6a4-4123-86a6-0fdfa035bf97\") " pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71x8lx7"
Oct 30 19:20:35.965642 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:20:35.965621    4892 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/1b64221d-b6a4-4123-86a6-0fdfa035bf97-util\") pod \"e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71x8lx7\" (UID: \"1b64221d-b6a4-4123-86a6-0fdfa035bf97\") " pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71x8lx7"
Oct 30 19:20:35.965714 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:20:35.965665    4892 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/1b64221d-b6a4-4123-86a6-0fdfa035bf97-bundle\") pod \"e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71x8lx7\" (UID: \"1b64221d-b6a4-4123-86a6-0fdfa035bf97\") " pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71x8lx7"
Oct 30 19:20:36.024412 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:20:36.024385    4892 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2xtfk\" (UniqueName: \"kubernetes.io/projected/1b64221d-b6a4-4123-86a6-0fdfa035bf97-kube-api-access-2xtfk\") pod \"e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71x8lx7\" (UID: \"1b64221d-b6a4-4123-86a6-0fdfa035bf97\") " pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71x8lx7"
Oct 30 19:20:36.151989 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:20:36.151951    4892 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71x8lx7"
Oct 30 19:20:36.525339 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:20:36.525318    4892 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71x8lx7"]
Oct 30 19:20:36.527946 ip-10-0-132-82 kubenswrapper[4892]: W1030 19:20:36.527911    4892 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1b64221d_b6a4_4123_86a6_0fdfa035bf97.slice/crio-c7f69493375a0692d9714e3f064eecb30cdae378563dfbb303c7f1657c9cc009 WatchSource:0}: Error finding container c7f69493375a0692d9714e3f064eecb30cdae378563dfbb303c7f1657c9cc009: Status 404 returned error can't find the container with id c7f69493375a0692d9714e3f064eecb30cdae378563dfbb303c7f1657c9cc009
Oct 30 19:20:36.728301 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:20:36.728257    4892 generic.go:356] "Generic (PLEG): container finished" podID="1b64221d-b6a4-4123-86a6-0fdfa035bf97" containerID="99dbfb7306789a0120afc437150bd902c27db97db68086ff1c485dd4aedcd924" exitCode=0
Oct 30 19:20:36.728440 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:20:36.728317    4892 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71x8lx7" event={"ID":"1b64221d-b6a4-4123-86a6-0fdfa035bf97","Type":"ContainerDied","Data":"99dbfb7306789a0120afc437150bd902c27db97db68086ff1c485dd4aedcd924"}
Oct 30 19:20:36.728440 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:20:36.728353    4892 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71x8lx7" event={"ID":"1b64221d-b6a4-4123-86a6-0fdfa035bf97","Type":"ContainerStarted","Data":"c7f69493375a0692d9714e3f064eecb30cdae378563dfbb303c7f1657c9cc009"}
Oct 30 19:20:37.734163 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:20:37.734131    4892 generic.go:356] "Generic (PLEG): container finished" podID="1b64221d-b6a4-4123-86a6-0fdfa035bf97" containerID="d942393f8865a3730ee6f14e5b28598dcd797437cf277e347528767544256fee" exitCode=0
Oct 30 19:20:37.734549 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:20:37.734219    4892 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71x8lx7" event={"ID":"1b64221d-b6a4-4123-86a6-0fdfa035bf97","Type":"ContainerDied","Data":"d942393f8865a3730ee6f14e5b28598dcd797437cf277e347528767544256fee"}
Oct 30 19:20:38.740273 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:20:38.740243    4892 generic.go:356] "Generic (PLEG): container finished" podID="1b64221d-b6a4-4123-86a6-0fdfa035bf97" containerID="3340e7c4e0ce2ee9b3f959f7a07197977976de70a2550b866bbe965794bd5dd4" exitCode=0
Oct 30 19:20:38.740732 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:20:38.740328    4892 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71x8lx7" event={"ID":"1b64221d-b6a4-4123-86a6-0fdfa035bf97","Type":"ContainerDied","Data":"3340e7c4e0ce2ee9b3f959f7a07197977976de70a2550b866bbe965794bd5dd4"}
Oct 30 19:20:39.866540 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:20:39.866515    4892 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71x8lx7"
Oct 30 19:20:39.893927 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:20:39.893902    4892 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1b64221d-b6a4-4123-86a6-0fdfa035bf97-kube-api-access-2xtfk\" (UniqueName: \"kubernetes.io/projected/1b64221d-b6a4-4123-86a6-0fdfa035bf97-kube-api-access-2xtfk\") pod \"1b64221d-b6a4-4123-86a6-0fdfa035bf97\" (UID: \"1b64221d-b6a4-4123-86a6-0fdfa035bf97\") "
Oct 30 19:20:39.894059 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:20:39.893997    4892 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1b64221d-b6a4-4123-86a6-0fdfa035bf97-util\" (UniqueName: \"kubernetes.io/empty-dir/1b64221d-b6a4-4123-86a6-0fdfa035bf97-util\") pod \"1b64221d-b6a4-4123-86a6-0fdfa035bf97\" (UID: \"1b64221d-b6a4-4123-86a6-0fdfa035bf97\") "
Oct 30 19:20:39.894137 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:20:39.894084    4892 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1b64221d-b6a4-4123-86a6-0fdfa035bf97-bundle\" (UniqueName: \"kubernetes.io/empty-dir/1b64221d-b6a4-4123-86a6-0fdfa035bf97-bundle\") pod \"1b64221d-b6a4-4123-86a6-0fdfa035bf97\" (UID: \"1b64221d-b6a4-4123-86a6-0fdfa035bf97\") "
Oct 30 19:20:39.894611 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:20:39.894539    4892 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1b64221d-b6a4-4123-86a6-0fdfa035bf97-bundle" pod "1b64221d-b6a4-4123-86a6-0fdfa035bf97" (UID: "1b64221d-b6a4-4123-86a6-0fdfa035bf97"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:20:39.896055 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:20:39.896031    4892 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1b64221d-b6a4-4123-86a6-0fdfa035bf97-kube-api-access-2xtfk" pod "1b64221d-b6a4-4123-86a6-0fdfa035bf97" (UID: "1b64221d-b6a4-4123-86a6-0fdfa035bf97"). InnerVolumeSpecName "kube-api-access-2xtfk". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:20:39.901630 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:20:39.901608    4892 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1b64221d-b6a4-4123-86a6-0fdfa035bf97-util" pod "1b64221d-b6a4-4123-86a6-0fdfa035bf97" (UID: "1b64221d-b6a4-4123-86a6-0fdfa035bf97"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:20:39.995342 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:20:39.995311    4892 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2xtfk\" (UniqueName: \"kubernetes.io/projected/1b64221d-b6a4-4123-86a6-0fdfa035bf97-kube-api-access-2xtfk\") on node \"ip-10-0-132-82.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 19:20:39.995342 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:20:39.995336    4892 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/1b64221d-b6a4-4123-86a6-0fdfa035bf97-util\") on node \"ip-10-0-132-82.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 19:20:39.995342 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:20:39.995349    4892 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/1b64221d-b6a4-4123-86a6-0fdfa035bf97-bundle\") on node \"ip-10-0-132-82.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 19:20:40.751839 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:20:40.751804    4892 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71x8lx7"
Oct 30 19:20:40.751992 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:20:40.751799    4892 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71x8lx7" event={"ID":"1b64221d-b6a4-4123-86a6-0fdfa035bf97","Type":"ContainerDied","Data":"c7f69493375a0692d9714e3f064eecb30cdae378563dfbb303c7f1657c9cc009"}
Oct 30 19:20:40.751992 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:20:40.751928    4892 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="c7f69493375a0692d9714e3f064eecb30cdae378563dfbb303c7f1657c9cc009"
Oct 30 19:20:49.885011 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:20:49.884981    4892 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/opendatahub-operator-controller-manager-69bfcc8945-w6764"]
Oct 30 19:20:49.885428 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:20:49.885301    4892 cpu_manager.go:414] "containerMap: removing container" podUID="1b64221d-b6a4-4123-86a6-0fdfa035bf97" containerName="extract"
Oct 30 19:20:49.885428 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:20:49.885314    4892 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1b64221d-b6a4-4123-86a6-0fdfa035bf97" containerName="extract"
Oct 30 19:20:49.885428 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:20:49.885333    4892 cpu_manager.go:414] "containerMap: removing container" podUID="1b64221d-b6a4-4123-86a6-0fdfa035bf97" containerName="pull"
Oct 30 19:20:49.885428 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:20:49.885338    4892 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1b64221d-b6a4-4123-86a6-0fdfa035bf97" containerName="pull"
Oct 30 19:20:49.885428 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:20:49.885362    4892 cpu_manager.go:414] "containerMap: removing container" podUID="1b64221d-b6a4-4123-86a6-0fdfa035bf97" containerName="util"
Oct 30 19:20:49.885428 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:20:49.885367    4892 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1b64221d-b6a4-4123-86a6-0fdfa035bf97" containerName="util"
Oct 30 19:20:49.885428 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:20:49.885415    4892 memory_manager.go:365] "RemoveStaleState removing state" podUID="1b64221d-b6a4-4123-86a6-0fdfa035bf97" containerName="extract"
Oct 30 19:20:49.889331 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:20:49.889299    4892 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/opendatahub-operator-controller-manager-69bfcc8945-w6764"
Oct 30 19:20:49.969353 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:20:49.969323    4892 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/aaef15fe-c1fe-44ed-88d9-ec1ac79f4896-apiservice-cert\") pod \"opendatahub-operator-controller-manager-69bfcc8945-w6764\" (UID: \"aaef15fe-c1fe-44ed-88d9-ec1ac79f4896\") " pod="opendatahub/opendatahub-operator-controller-manager-69bfcc8945-w6764"
Oct 30 19:20:49.969507 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:20:49.969359    4892 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gmqsh\" (UniqueName: \"kubernetes.io/projected/aaef15fe-c1fe-44ed-88d9-ec1ac79f4896-kube-api-access-gmqsh\") pod \"opendatahub-operator-controller-manager-69bfcc8945-w6764\" (UID: \"aaef15fe-c1fe-44ed-88d9-ec1ac79f4896\") " pod="opendatahub/opendatahub-operator-controller-manager-69bfcc8945-w6764"
Oct 30 19:20:49.969507 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:20:49.969387    4892 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/aaef15fe-c1fe-44ed-88d9-ec1ac79f4896-webhook-cert\") pod \"opendatahub-operator-controller-manager-69bfcc8945-w6764\" (UID: \"aaef15fe-c1fe-44ed-88d9-ec1ac79f4896\") " pod="opendatahub/opendatahub-operator-controller-manager-69bfcc8945-w6764"
Oct 30 19:20:49.969608 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:20:49.969506    4892 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/aaef15fe-c1fe-44ed-88d9-ec1ac79f4896-manifests\") pod \"opendatahub-operator-controller-manager-69bfcc8945-w6764\" (UID: \"aaef15fe-c1fe-44ed-88d9-ec1ac79f4896\") " pod="opendatahub/opendatahub-operator-controller-manager-69bfcc8945-w6764"
Oct 30 19:20:49.969608 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:20:49.969543    4892 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/aaef15fe-c1fe-44ed-88d9-ec1ac79f4896-tmp\") pod \"opendatahub-operator-controller-manager-69bfcc8945-w6764\" (UID: \"aaef15fe-c1fe-44ed-88d9-ec1ac79f4896\") " pod="opendatahub/opendatahub-operator-controller-manager-69bfcc8945-w6764"
Oct 30 19:20:49.969608 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:20:49.969562    4892 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/aaef15fe-c1fe-44ed-88d9-ec1ac79f4896-metrics-cert\") pod \"opendatahub-operator-controller-manager-69bfcc8945-w6764\" (UID: \"aaef15fe-c1fe-44ed-88d9-ec1ac79f4896\") " pod="opendatahub/opendatahub-operator-controller-manager-69bfcc8945-w6764"
Oct 30 19:20:50.013262 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:20:50.013228    4892 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"kube-root-ca.crt\""
Oct 30 19:20:50.013262 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:20:50.013240    4892 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-webhook-cert\""
Oct 30 19:20:50.013487 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:20:50.013278    4892 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-manager-service-cert\""
Oct 30 19:20:50.013487 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:20:50.013284    4892 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"openshift-service-ca.crt\""
Oct 30 19:20:50.013487 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:20:50.013312    4892 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-manager-dockercfg-2h29g\""
Oct 30 19:20:50.013487 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:20:50.013412    4892 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-metrics-tls\""
Oct 30 19:20:50.070249 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:20:50.070220    4892 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/aaef15fe-c1fe-44ed-88d9-ec1ac79f4896-manifests\") pod \"opendatahub-operator-controller-manager-69bfcc8945-w6764\" (UID: \"aaef15fe-c1fe-44ed-88d9-ec1ac79f4896\") " pod="opendatahub/opendatahub-operator-controller-manager-69bfcc8945-w6764"
Oct 30 19:20:50.070249 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:20:50.070256    4892 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/aaef15fe-c1fe-44ed-88d9-ec1ac79f4896-tmp\") pod \"opendatahub-operator-controller-manager-69bfcc8945-w6764\" (UID: \"aaef15fe-c1fe-44ed-88d9-ec1ac79f4896\") " pod="opendatahub/opendatahub-operator-controller-manager-69bfcc8945-w6764"
Oct 30 19:20:50.070507 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:20:50.070273    4892 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/aaef15fe-c1fe-44ed-88d9-ec1ac79f4896-metrics-cert\") pod \"opendatahub-operator-controller-manager-69bfcc8945-w6764\" (UID: \"aaef15fe-c1fe-44ed-88d9-ec1ac79f4896\") " pod="opendatahub/opendatahub-operator-controller-manager-69bfcc8945-w6764"
Oct 30 19:20:50.070507 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:20:50.070304    4892 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/aaef15fe-c1fe-44ed-88d9-ec1ac79f4896-apiservice-cert\") pod \"opendatahub-operator-controller-manager-69bfcc8945-w6764\" (UID: \"aaef15fe-c1fe-44ed-88d9-ec1ac79f4896\") " pod="opendatahub/opendatahub-operator-controller-manager-69bfcc8945-w6764"
Oct 30 19:20:50.070507 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:20:50.070419    4892 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gmqsh\" (UniqueName: \"kubernetes.io/projected/aaef15fe-c1fe-44ed-88d9-ec1ac79f4896-kube-api-access-gmqsh\") pod \"opendatahub-operator-controller-manager-69bfcc8945-w6764\" (UID: \"aaef15fe-c1fe-44ed-88d9-ec1ac79f4896\") " pod="opendatahub/opendatahub-operator-controller-manager-69bfcc8945-w6764"
Oct 30 19:20:50.070507 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:20:50.070486    4892 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/aaef15fe-c1fe-44ed-88d9-ec1ac79f4896-webhook-cert\") pod \"opendatahub-operator-controller-manager-69bfcc8945-w6764\" (UID: \"aaef15fe-c1fe-44ed-88d9-ec1ac79f4896\") " pod="opendatahub/opendatahub-operator-controller-manager-69bfcc8945-w6764"
Oct 30 19:20:50.070788 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:20:50.070760    4892 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/aaef15fe-c1fe-44ed-88d9-ec1ac79f4896-manifests\") pod \"opendatahub-operator-controller-manager-69bfcc8945-w6764\" (UID: \"aaef15fe-c1fe-44ed-88d9-ec1ac79f4896\") " pod="opendatahub/opendatahub-operator-controller-manager-69bfcc8945-w6764"
Oct 30 19:20:50.072913 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:20:50.072884    4892 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/aaef15fe-c1fe-44ed-88d9-ec1ac79f4896-tmp\") pod \"opendatahub-operator-controller-manager-69bfcc8945-w6764\" (UID: \"aaef15fe-c1fe-44ed-88d9-ec1ac79f4896\") " pod="opendatahub/opendatahub-operator-controller-manager-69bfcc8945-w6764"
Oct 30 19:20:50.073078 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:20:50.073058    4892 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/aaef15fe-c1fe-44ed-88d9-ec1ac79f4896-metrics-cert\") pod \"opendatahub-operator-controller-manager-69bfcc8945-w6764\" (UID: \"aaef15fe-c1fe-44ed-88d9-ec1ac79f4896\") " pod="opendatahub/opendatahub-operator-controller-manager-69bfcc8945-w6764"
Oct 30 19:20:50.073078 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:20:50.073058    4892 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/aaef15fe-c1fe-44ed-88d9-ec1ac79f4896-webhook-cert\") pod \"opendatahub-operator-controller-manager-69bfcc8945-w6764\" (UID: \"aaef15fe-c1fe-44ed-88d9-ec1ac79f4896\") " pod="opendatahub/opendatahub-operator-controller-manager-69bfcc8945-w6764"
Oct 30 19:20:50.073194 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:20:50.073164    4892 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/aaef15fe-c1fe-44ed-88d9-ec1ac79f4896-apiservice-cert\") pod \"opendatahub-operator-controller-manager-69bfcc8945-w6764\" (UID: \"aaef15fe-c1fe-44ed-88d9-ec1ac79f4896\") " pod="opendatahub/opendatahub-operator-controller-manager-69bfcc8945-w6764"
Oct 30 19:20:50.135351 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:20:50.135277    4892 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gmqsh\" (UniqueName: \"kubernetes.io/projected/aaef15fe-c1fe-44ed-88d9-ec1ac79f4896-kube-api-access-gmqsh\") pod \"opendatahub-operator-controller-manager-69bfcc8945-w6764\" (UID: \"aaef15fe-c1fe-44ed-88d9-ec1ac79f4896\") " pod="opendatahub/opendatahub-operator-controller-manager-69bfcc8945-w6764"
Oct 30 19:20:50.192189 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:20:50.192164    4892 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/opendatahub-operator-controller-manager-69bfcc8945-w6764"]
Oct 30 19:20:50.201809 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:20:50.201785    4892 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/opendatahub-operator-controller-manager-69bfcc8945-w6764"
Oct 30 19:20:50.687950 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:20:50.687926    4892 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/opendatahub-operator-controller-manager-69bfcc8945-w6764"]
Oct 30 19:20:50.689907 ip-10-0-132-82 kubenswrapper[4892]: W1030 19:20:50.689883    4892 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podaaef15fe_c1fe_44ed_88d9_ec1ac79f4896.slice/crio-47d8da99eb716fd2c758c1819d05e4635b242838bb99c1e69413629dad82b5a2 WatchSource:0}: Error finding container 47d8da99eb716fd2c758c1819d05e4635b242838bb99c1e69413629dad82b5a2: Status 404 returned error can't find the container with id 47d8da99eb716fd2c758c1819d05e4635b242838bb99c1e69413629dad82b5a2
Oct 30 19:20:50.797435 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:20:50.797406    4892 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-69bfcc8945-w6764" event={"ID":"aaef15fe-c1fe-44ed-88d9-ec1ac79f4896","Type":"ContainerStarted","Data":"47d8da99eb716fd2c758c1819d05e4635b242838bb99c1e69413629dad82b5a2"}
Oct 30 19:20:52.593079 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:20:52.593052    4892 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-tg5xj_11db9cec-653f-4aee-8937-0cf5b367efac/console-operator/3.log"
Oct 30 19:20:52.593812 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:20:52.593787    4892 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-tg5xj_11db9cec-653f-4aee-8937-0cf5b367efac/console-operator/3.log"
Oct 30 19:20:54.815649 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:20:54.815618    4892 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-69bfcc8945-w6764" event={"ID":"aaef15fe-c1fe-44ed-88d9-ec1ac79f4896","Type":"ContainerStarted","Data":"df5accc1d8c53b595fcd03af9254d131c40b7f446e4331b8d3128f5627a64db8"}
Oct 30 19:20:54.844217 ip-10-0-132-82 kubenswrapper[4892]: E1030 19:20:54.844188    4892 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-jrm87" containerName="extractor"
Oct 30 19:20:55.821660 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:20:55.821622    4892 generic.go:356] "Generic (PLEG): container finished" podID="aaef15fe-c1fe-44ed-88d9-ec1ac79f4896" containerID="df5accc1d8c53b595fcd03af9254d131c40b7f446e4331b8d3128f5627a64db8" exitCode=0
Oct 30 19:20:55.822093 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:20:55.821703    4892 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-69bfcc8945-w6764" event={"ID":"aaef15fe-c1fe-44ed-88d9-ec1ac79f4896","Type":"ContainerDied","Data":"df5accc1d8c53b595fcd03af9254d131c40b7f446e4331b8d3128f5627a64db8"}
Oct 30 19:20:56.828070 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:20:56.828032    4892 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-69bfcc8945-w6764" event={"ID":"aaef15fe-c1fe-44ed-88d9-ec1ac79f4896","Type":"ContainerStarted","Data":"01ca33bc7b7088211274d04fa2f2eea68dd750b824d80ecccc9d00f62e073454"}
Oct 30 19:20:56.828520 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:20:56.828207    4892 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/opendatahub-operator-controller-manager-69bfcc8945-w6764"
Oct 30 19:20:57.109109 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:20:57.109058    4892 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/opendatahub-operator-controller-manager-69bfcc8945-w6764" podStartSLOduration=3.234062423 podStartE2EDuration="8.109045484s" totalImagesPullingTime="4.467952362s" totalInitContainerRuntime="407.030699ms" isStatefulPod=false podCreationTimestamp="2026-10-30 19:20:49 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:20:57.059179747 +0000 UTC m=+304.629645345" watchObservedRunningTime="2026-10-30 19:20:57.109045484 +0000 UTC m=+304.679511082"
Oct 30 19:21:07.839904 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:21:07.839820    4892 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/opendatahub-operator-controller-manager-69bfcc8945-w6764"
Oct 30 19:21:23.298921 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:21:23.298887    4892 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/kube-auth-proxy-7f6998b7b5-wthwp"]
Oct 30 19:21:23.303321 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:21:23.303304    4892 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/kube-auth-proxy-7f6998b7b5-wthwp"
Oct 30 19:21:23.407405 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:21:23.407371    4892 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"kube-auth-proxy-creds\""
Oct 30 19:21:23.407592 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:21:23.407384    4892 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"kube-auth-proxy-tls\""
Oct 30 19:21:23.407592 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:21:23.407480    4892 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"kube-auth-proxy-dockercfg-zq789\""
Oct 30 19:21:23.408829 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:21:23.408808    4892 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/kube-auth-proxy-7f6998b7b5-wthwp"]
Oct 30 19:21:23.426925 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:21:23.426899    4892 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/6fb66959-7822-48c9-9f8b-8daf7fc7adde-tls-certs\") pod \"kube-auth-proxy-7f6998b7b5-wthwp\" (UID: \"6fb66959-7822-48c9-9f8b-8daf7fc7adde\") " pod="openshift-ingress/kube-auth-proxy-7f6998b7b5-wthwp"
Oct 30 19:21:23.427033 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:21:23.426951    4892 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6wkz2\" (UniqueName: \"kubernetes.io/projected/6fb66959-7822-48c9-9f8b-8daf7fc7adde-kube-api-access-6wkz2\") pod \"kube-auth-proxy-7f6998b7b5-wthwp\" (UID: \"6fb66959-7822-48c9-9f8b-8daf7fc7adde\") " pod="openshift-ingress/kube-auth-proxy-7f6998b7b5-wthwp"
Oct 30 19:21:23.427076 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:21:23.427047    4892 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/6fb66959-7822-48c9-9f8b-8daf7fc7adde-tmp\") pod \"kube-auth-proxy-7f6998b7b5-wthwp\" (UID: \"6fb66959-7822-48c9-9f8b-8daf7fc7adde\") " pod="openshift-ingress/kube-auth-proxy-7f6998b7b5-wthwp"
Oct 30 19:21:23.527947 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:21:23.527919    4892 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/6fb66959-7822-48c9-9f8b-8daf7fc7adde-tmp\") pod \"kube-auth-proxy-7f6998b7b5-wthwp\" (UID: \"6fb66959-7822-48c9-9f8b-8daf7fc7adde\") " pod="openshift-ingress/kube-auth-proxy-7f6998b7b5-wthwp"
Oct 30 19:21:23.528059 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:21:23.527981    4892 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/6fb66959-7822-48c9-9f8b-8daf7fc7adde-tls-certs\") pod \"kube-auth-proxy-7f6998b7b5-wthwp\" (UID: \"6fb66959-7822-48c9-9f8b-8daf7fc7adde\") " pod="openshift-ingress/kube-auth-proxy-7f6998b7b5-wthwp"
Oct 30 19:21:23.528059 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:21:23.528025    4892 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6wkz2\" (UniqueName: \"kubernetes.io/projected/6fb66959-7822-48c9-9f8b-8daf7fc7adde-kube-api-access-6wkz2\") pod \"kube-auth-proxy-7f6998b7b5-wthwp\" (UID: \"6fb66959-7822-48c9-9f8b-8daf7fc7adde\") " pod="openshift-ingress/kube-auth-proxy-7f6998b7b5-wthwp"
Oct 30 19:21:23.530276 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:21:23.530257    4892 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/6fb66959-7822-48c9-9f8b-8daf7fc7adde-tmp\") pod \"kube-auth-proxy-7f6998b7b5-wthwp\" (UID: \"6fb66959-7822-48c9-9f8b-8daf7fc7adde\") " pod="openshift-ingress/kube-auth-proxy-7f6998b7b5-wthwp"
Oct 30 19:21:23.530526 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:21:23.530510    4892 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/6fb66959-7822-48c9-9f8b-8daf7fc7adde-tls-certs\") pod \"kube-auth-proxy-7f6998b7b5-wthwp\" (UID: \"6fb66959-7822-48c9-9f8b-8daf7fc7adde\") " pod="openshift-ingress/kube-auth-proxy-7f6998b7b5-wthwp"
Oct 30 19:21:23.584427 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:21:23.584371    4892 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6wkz2\" (UniqueName: \"kubernetes.io/projected/6fb66959-7822-48c9-9f8b-8daf7fc7adde-kube-api-access-6wkz2\") pod \"kube-auth-proxy-7f6998b7b5-wthwp\" (UID: \"6fb66959-7822-48c9-9f8b-8daf7fc7adde\") " pod="openshift-ingress/kube-auth-proxy-7f6998b7b5-wthwp"
Oct 30 19:21:23.617466 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:21:23.617429    4892 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/kube-auth-proxy-7f6998b7b5-wthwp"
Oct 30 19:21:23.988070 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:21:23.988045    4892 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/kube-auth-proxy-7f6998b7b5-wthwp"]
Oct 30 19:21:23.990630 ip-10-0-132-82 kubenswrapper[4892]: W1030 19:21:23.990605    4892 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6fb66959_7822_48c9_9f8b_8daf7fc7adde.slice/crio-a27b07294c532f9798cf53ae028b05cb4f1a59c78d9f98ca0aee66cb84935a91 WatchSource:0}: Error finding container a27b07294c532f9798cf53ae028b05cb4f1a59c78d9f98ca0aee66cb84935a91: Status 404 returned error can't find the container with id a27b07294c532f9798cf53ae028b05cb4f1a59c78d9f98ca0aee66cb84935a91
Oct 30 19:21:23.992392 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:21:23.992373    4892 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Oct 30 19:21:24.677566 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:21:24.677527    4892 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7876bfcb74-cx4zq"]
Oct 30 19:21:24.682166 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:21:24.682142    4892 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-cx4zq"
Oct 30 19:21:24.734480 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:21:24.734450    4892 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"cacerts\""
Oct 30 19:21:24.734480 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:21:24.734470    4892 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istiod-tls\""
Oct 30 19:21:24.734638 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:21:24.734489    4892 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-gw-ca-root-cert\""
Oct 30 19:21:24.734695 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:21:24.734669    4892 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istiod-openshift-gateway-dockercfg-2vd8g\""
Oct 30 19:21:24.734695 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:21:24.734684    4892 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istio-kubeconfig\""
Oct 30 19:21:24.785069 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:21:24.785036    4892 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7876bfcb74-cx4zq"]
Oct 30 19:21:24.839943 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:21:24.839917    4892 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/68efc1e0-8b8f-4b21-b7c5-253f32b45dc2-istio-kubeconfig\") pod \"istiod-openshift-gateway-7876bfcb74-cx4zq\" (UID: \"68efc1e0-8b8f-4b21-b7c5-253f32b45dc2\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-cx4zq"
Oct 30 19:21:24.840087 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:21:24.839952    4892 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/68efc1e0-8b8f-4b21-b7c5-253f32b45dc2-local-certs\") pod \"istiod-openshift-gateway-7876bfcb74-cx4zq\" (UID: \"68efc1e0-8b8f-4b21-b7c5-253f32b45dc2\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-cx4zq"
Oct 30 19:21:24.840087 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:21:24.839994    4892 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/68efc1e0-8b8f-4b21-b7c5-253f32b45dc2-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-7876bfcb74-cx4zq\" (UID: \"68efc1e0-8b8f-4b21-b7c5-253f32b45dc2\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-cx4zq"
Oct 30 19:21:24.840184 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:21:24.840163    4892 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/68efc1e0-8b8f-4b21-b7c5-253f32b45dc2-istio-token\") pod \"istiod-openshift-gateway-7876bfcb74-cx4zq\" (UID: \"68efc1e0-8b8f-4b21-b7c5-253f32b45dc2\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-cx4zq"
Oct 30 19:21:24.840236 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:21:24.840208    4892 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/68efc1e0-8b8f-4b21-b7c5-253f32b45dc2-cacerts\") pod \"istiod-openshift-gateway-7876bfcb74-cx4zq\" (UID: \"68efc1e0-8b8f-4b21-b7c5-253f32b45dc2\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-cx4zq"
Oct 30 19:21:24.840295 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:21:24.840277    4892 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/68efc1e0-8b8f-4b21-b7c5-253f32b45dc2-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-7876bfcb74-cx4zq\" (UID: \"68efc1e0-8b8f-4b21-b7c5-253f32b45dc2\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-cx4zq"
Oct 30 19:21:24.840482 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:21:24.840368    4892 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jn87s\" (UniqueName: \"kubernetes.io/projected/68efc1e0-8b8f-4b21-b7c5-253f32b45dc2-kube-api-access-jn87s\") pod \"istiod-openshift-gateway-7876bfcb74-cx4zq\" (UID: \"68efc1e0-8b8f-4b21-b7c5-253f32b45dc2\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-cx4zq"
Oct 30 19:21:24.843875 ip-10-0-132-82 kubenswrapper[4892]: E1030 19:21:24.843847    4892 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-jrm87" containerName="extractor"
Oct 30 19:21:24.941734 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:21:24.941659    4892 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/68efc1e0-8b8f-4b21-b7c5-253f32b45dc2-istio-token\") pod \"istiod-openshift-gateway-7876bfcb74-cx4zq\" (UID: \"68efc1e0-8b8f-4b21-b7c5-253f32b45dc2\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-cx4zq"
Oct 30 19:21:24.941734 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:21:24.941702    4892 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/68efc1e0-8b8f-4b21-b7c5-253f32b45dc2-cacerts\") pod \"istiod-openshift-gateway-7876bfcb74-cx4zq\" (UID: \"68efc1e0-8b8f-4b21-b7c5-253f32b45dc2\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-cx4zq"
Oct 30 19:21:24.942004 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:21:24.941778    4892 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/68efc1e0-8b8f-4b21-b7c5-253f32b45dc2-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-7876bfcb74-cx4zq\" (UID: \"68efc1e0-8b8f-4b21-b7c5-253f32b45dc2\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-cx4zq"
Oct 30 19:21:24.942004 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:21:24.941815    4892 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jn87s\" (UniqueName: \"kubernetes.io/projected/68efc1e0-8b8f-4b21-b7c5-253f32b45dc2-kube-api-access-jn87s\") pod \"istiod-openshift-gateway-7876bfcb74-cx4zq\" (UID: \"68efc1e0-8b8f-4b21-b7c5-253f32b45dc2\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-cx4zq"
Oct 30 19:21:24.942004 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:21:24.941869    4892 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/68efc1e0-8b8f-4b21-b7c5-253f32b45dc2-istio-kubeconfig\") pod \"istiod-openshift-gateway-7876bfcb74-cx4zq\" (UID: \"68efc1e0-8b8f-4b21-b7c5-253f32b45dc2\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-cx4zq"
Oct 30 19:21:24.942004 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:21:24.941905    4892 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/68efc1e0-8b8f-4b21-b7c5-253f32b45dc2-local-certs\") pod \"istiod-openshift-gateway-7876bfcb74-cx4zq\" (UID: \"68efc1e0-8b8f-4b21-b7c5-253f32b45dc2\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-cx4zq"
Oct 30 19:21:24.942004 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:21:24.941936    4892 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/68efc1e0-8b8f-4b21-b7c5-253f32b45dc2-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-7876bfcb74-cx4zq\" (UID: \"68efc1e0-8b8f-4b21-b7c5-253f32b45dc2\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-cx4zq"
Oct 30 19:21:24.942527 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:21:24.942454    4892 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/68efc1e0-8b8f-4b21-b7c5-253f32b45dc2-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-7876bfcb74-cx4zq\" (UID: \"68efc1e0-8b8f-4b21-b7c5-253f32b45dc2\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-cx4zq"
Oct 30 19:21:24.944633 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:21:24.944606    4892 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/68efc1e0-8b8f-4b21-b7c5-253f32b45dc2-cacerts\") pod \"istiod-openshift-gateway-7876bfcb74-cx4zq\" (UID: \"68efc1e0-8b8f-4b21-b7c5-253f32b45dc2\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-cx4zq"
Oct 30 19:21:24.944735 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:21:24.944715    4892 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/68efc1e0-8b8f-4b21-b7c5-253f32b45dc2-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-7876bfcb74-cx4zq\" (UID: \"68efc1e0-8b8f-4b21-b7c5-253f32b45dc2\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-cx4zq"
Oct 30 19:21:24.944952 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:21:24.944929    4892 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/68efc1e0-8b8f-4b21-b7c5-253f32b45dc2-istio-kubeconfig\") pod \"istiod-openshift-gateway-7876bfcb74-cx4zq\" (UID: \"68efc1e0-8b8f-4b21-b7c5-253f32b45dc2\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-cx4zq"
Oct 30 19:21:24.945463 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:21:24.945441    4892 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/68efc1e0-8b8f-4b21-b7c5-253f32b45dc2-local-certs\") pod \"istiod-openshift-gateway-7876bfcb74-cx4zq\" (UID: \"68efc1e0-8b8f-4b21-b7c5-253f32b45dc2\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-cx4zq"
Oct 30 19:21:24.949462 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:21:24.949436    4892 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/kube-auth-proxy-7f6998b7b5-wthwp" event={"ID":"6fb66959-7822-48c9-9f8b-8daf7fc7adde","Type":"ContainerStarted","Data":"a27b07294c532f9798cf53ae028b05cb4f1a59c78d9f98ca0aee66cb84935a91"}
Oct 30 19:21:25.000495 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:21:25.000460    4892 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/68efc1e0-8b8f-4b21-b7c5-253f32b45dc2-istio-token\") pod \"istiod-openshift-gateway-7876bfcb74-cx4zq\" (UID: \"68efc1e0-8b8f-4b21-b7c5-253f32b45dc2\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-cx4zq"
Oct 30 19:21:25.000643 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:21:25.000538    4892 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jn87s\" (UniqueName: \"kubernetes.io/projected/68efc1e0-8b8f-4b21-b7c5-253f32b45dc2-kube-api-access-jn87s\") pod \"istiod-openshift-gateway-7876bfcb74-cx4zq\" (UID: \"68efc1e0-8b8f-4b21-b7c5-253f32b45dc2\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-cx4zq"
Oct 30 19:21:25.294548 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:21:25.294467    4892 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-cx4zq"
Oct 30 19:21:25.722597 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:21:25.722559    4892 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7876bfcb74-cx4zq"]
Oct 30 19:21:25.762595 ip-10-0-132-82 kubenswrapper[4892]: W1030 19:21:25.762570    4892 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod68efc1e0_8b8f_4b21_b7c5_253f32b45dc2.slice/crio-b5a2ac7a61a62497f76523d2ca6ffd1573df41b64f9ba9783d5645192a7d96ff WatchSource:0}: Error finding container b5a2ac7a61a62497f76523d2ca6ffd1573df41b64f9ba9783d5645192a7d96ff: Status 404 returned error can't find the container with id b5a2ac7a61a62497f76523d2ca6ffd1573df41b64f9ba9783d5645192a7d96ff
Oct 30 19:21:25.956328 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:21:25.956292    4892 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-cx4zq" event={"ID":"68efc1e0-8b8f-4b21-b7c5-253f32b45dc2","Type":"ContainerStarted","Data":"b5a2ac7a61a62497f76523d2ca6ffd1573df41b64f9ba9783d5645192a7d96ff"}
Oct 30 19:21:27.969220 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:21:27.969182    4892 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/kube-auth-proxy-7f6998b7b5-wthwp" event={"ID":"6fb66959-7822-48c9-9f8b-8daf7fc7adde","Type":"ContainerStarted","Data":"3e0191445bb40b0e105a52dcabb815f0c511f192a6bd7ada9353f921debf03fa"}
Oct 30 19:21:29.461830 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:21:29.461796    4892 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892000Ki","pods":"250"}
Oct 30 19:21:29.462172 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:21:29.461864    4892 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892000Ki","pods":"250"}
Oct 30 19:21:29.981050 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:21:29.981011    4892 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-cx4zq" event={"ID":"68efc1e0-8b8f-4b21-b7c5-253f32b45dc2","Type":"ContainerStarted","Data":"915d3662961637037793ebe56d67b11628cc03834ec14440ffcd24538487d530"}
Oct 30 19:21:29.981226 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:21:29.981087    4892 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-cx4zq"
Oct 30 19:21:30.211119 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:21:30.211064    4892 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-cx4zq" podStartSLOduration=2.51515633 podStartE2EDuration="6.211050027s" totalImagesPullingTime="3.695893697s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:21:24 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:21:30.160868852 +0000 UTC m=+337.731334451" watchObservedRunningTime="2026-10-30 19:21:30.211050027 +0000 UTC m=+337.781515625"
Oct 30 19:21:30.211284 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:21:30.211243    4892 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/kube-auth-proxy-7f6998b7b5-wthwp" podStartSLOduration=3.76321599 podStartE2EDuration="7.211239422s" totalImagesPullingTime="3.448023432s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:21:23 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:21:28.088789835 +0000 UTC m=+335.659255436" watchObservedRunningTime="2026-10-30 19:21:30.211239422 +0000 UTC m=+337.781705020"
Oct 30 19:21:30.986605 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:21:30.986571    4892 patch_prober.go:29] interesting pod/istiod-openshift-gateway-7876bfcb74-cx4zq container/discovery namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Oct 30 19:21:30.987004 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:21:30.986636    4892 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-cx4zq" podUID="68efc1e0-8b8f-4b21-b7c5-253f32b45dc2" containerName="discovery" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Oct 30 19:21:33.985715 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:21:33.985679    4892 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-cx4zq"
Oct 30 19:21:54.842399 ip-10-0-132-82 kubenswrapper[4892]: E1030 19:21:54.842362    4892 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-jrm87" containerName="extractor"
Oct 30 19:22:24.843953 ip-10-0-132-82 kubenswrapper[4892]: E1030 19:22:24.843913    4892 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-jrm87" containerName="extractor"
Oct 30 19:22:43.099777 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:22:43.099701    4892 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-rqhzf"]
Oct 30 19:22:43.103028 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:22:43.103011    4892 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-rqhzf"
Oct 30 19:22:43.208731 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:22:43.208704    4892 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-rqhzf"]
Oct 30 19:22:43.230213 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:22:43.230184    4892 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b1eed800-8bc3-4333-a929-dcb1477293fd-istio-token\") pod \"istiod-openshift-gateway-7ddd9cfdc8-rqhzf\" (UID: \"b1eed800-8bc3-4333-a929-dcb1477293fd\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-rqhzf"
Oct 30 19:22:43.230359 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:22:43.230230    4892 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/b1eed800-8bc3-4333-a929-dcb1477293fd-istio-kubeconfig\") pod \"istiod-openshift-gateway-7ddd9cfdc8-rqhzf\" (UID: \"b1eed800-8bc3-4333-a929-dcb1477293fd\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-rqhzf"
Oct 30 19:22:43.230359 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:22:43.230326    4892 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-29wcp\" (UniqueName: \"kubernetes.io/projected/b1eed800-8bc3-4333-a929-dcb1477293fd-kube-api-access-29wcp\") pod \"istiod-openshift-gateway-7ddd9cfdc8-rqhzf\" (UID: \"b1eed800-8bc3-4333-a929-dcb1477293fd\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-rqhzf"
Oct 30 19:22:43.230478 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:22:43.230367    4892 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/b1eed800-8bc3-4333-a929-dcb1477293fd-local-certs\") pod \"istiod-openshift-gateway-7ddd9cfdc8-rqhzf\" (UID: \"b1eed800-8bc3-4333-a929-dcb1477293fd\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-rqhzf"
Oct 30 19:22:43.230478 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:22:43.230426    4892 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/b1eed800-8bc3-4333-a929-dcb1477293fd-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-7ddd9cfdc8-rqhzf\" (UID: \"b1eed800-8bc3-4333-a929-dcb1477293fd\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-rqhzf"
Oct 30 19:22:43.230478 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:22:43.230470    4892 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/b1eed800-8bc3-4333-a929-dcb1477293fd-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-7ddd9cfdc8-rqhzf\" (UID: \"b1eed800-8bc3-4333-a929-dcb1477293fd\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-rqhzf"
Oct 30 19:22:43.230595 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:22:43.230530    4892 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/b1eed800-8bc3-4333-a929-dcb1477293fd-cacerts\") pod \"istiod-openshift-gateway-7ddd9cfdc8-rqhzf\" (UID: \"b1eed800-8bc3-4333-a929-dcb1477293fd\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-rqhzf"
Oct 30 19:22:43.331040 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:22:43.331012    4892 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-29wcp\" (UniqueName: \"kubernetes.io/projected/b1eed800-8bc3-4333-a929-dcb1477293fd-kube-api-access-29wcp\") pod \"istiod-openshift-gateway-7ddd9cfdc8-rqhzf\" (UID: \"b1eed800-8bc3-4333-a929-dcb1477293fd\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-rqhzf"
Oct 30 19:22:43.331182 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:22:43.331045    4892 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/b1eed800-8bc3-4333-a929-dcb1477293fd-local-certs\") pod \"istiod-openshift-gateway-7ddd9cfdc8-rqhzf\" (UID: \"b1eed800-8bc3-4333-a929-dcb1477293fd\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-rqhzf"
Oct 30 19:22:43.331182 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:22:43.331071    4892 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/b1eed800-8bc3-4333-a929-dcb1477293fd-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-7ddd9cfdc8-rqhzf\" (UID: \"b1eed800-8bc3-4333-a929-dcb1477293fd\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-rqhzf"
Oct 30 19:22:43.331182 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:22:43.331087    4892 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/b1eed800-8bc3-4333-a929-dcb1477293fd-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-7ddd9cfdc8-rqhzf\" (UID: \"b1eed800-8bc3-4333-a929-dcb1477293fd\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-rqhzf"
Oct 30 19:22:43.331182 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:22:43.331110    4892 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/b1eed800-8bc3-4333-a929-dcb1477293fd-cacerts\") pod \"istiod-openshift-gateway-7ddd9cfdc8-rqhzf\" (UID: \"b1eed800-8bc3-4333-a929-dcb1477293fd\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-rqhzf"
Oct 30 19:22:43.331380 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:22:43.331182    4892 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b1eed800-8bc3-4333-a929-dcb1477293fd-istio-token\") pod \"istiod-openshift-gateway-7ddd9cfdc8-rqhzf\" (UID: \"b1eed800-8bc3-4333-a929-dcb1477293fd\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-rqhzf"
Oct 30 19:22:43.331380 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:22:43.331208    4892 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/b1eed800-8bc3-4333-a929-dcb1477293fd-istio-kubeconfig\") pod \"istiod-openshift-gateway-7ddd9cfdc8-rqhzf\" (UID: \"b1eed800-8bc3-4333-a929-dcb1477293fd\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-rqhzf"
Oct 30 19:22:43.332194 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:22:43.332163    4892 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/b1eed800-8bc3-4333-a929-dcb1477293fd-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-7ddd9cfdc8-rqhzf\" (UID: \"b1eed800-8bc3-4333-a929-dcb1477293fd\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-rqhzf"
Oct 30 19:22:43.333702 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:22:43.333669    4892 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/b1eed800-8bc3-4333-a929-dcb1477293fd-local-certs\") pod \"istiod-openshift-gateway-7ddd9cfdc8-rqhzf\" (UID: \"b1eed800-8bc3-4333-a929-dcb1477293fd\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-rqhzf"
Oct 30 19:22:43.333801 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:22:43.333696    4892 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/b1eed800-8bc3-4333-a929-dcb1477293fd-cacerts\") pod \"istiod-openshift-gateway-7ddd9cfdc8-rqhzf\" (UID: \"b1eed800-8bc3-4333-a929-dcb1477293fd\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-rqhzf"
Oct 30 19:22:43.333843 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:22:43.333806    4892 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/b1eed800-8bc3-4333-a929-dcb1477293fd-istio-kubeconfig\") pod \"istiod-openshift-gateway-7ddd9cfdc8-rqhzf\" (UID: \"b1eed800-8bc3-4333-a929-dcb1477293fd\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-rqhzf"
Oct 30 19:22:43.333878 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:22:43.333838    4892 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/b1eed800-8bc3-4333-a929-dcb1477293fd-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-7ddd9cfdc8-rqhzf\" (UID: \"b1eed800-8bc3-4333-a929-dcb1477293fd\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-rqhzf"
Oct 30 19:22:43.387678 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:22:43.387652    4892 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b1eed800-8bc3-4333-a929-dcb1477293fd-istio-token\") pod \"istiod-openshift-gateway-7ddd9cfdc8-rqhzf\" (UID: \"b1eed800-8bc3-4333-a929-dcb1477293fd\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-rqhzf"
Oct 30 19:22:43.387843 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:22:43.387811    4892 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-29wcp\" (UniqueName: \"kubernetes.io/projected/b1eed800-8bc3-4333-a929-dcb1477293fd-kube-api-access-29wcp\") pod \"istiod-openshift-gateway-7ddd9cfdc8-rqhzf\" (UID: \"b1eed800-8bc3-4333-a929-dcb1477293fd\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-rqhzf"
Oct 30 19:22:43.419591 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:22:43.419548    4892 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-rqhzf"
Oct 30 19:22:43.840733 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:22:43.840649    4892 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-rqhzf"]
Oct 30 19:22:43.843823 ip-10-0-132-82 kubenswrapper[4892]: W1030 19:22:43.843800    4892 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb1eed800_8bc3_4333_a929_dcb1477293fd.slice/crio-7aba53f5ffcc398e9576462e5c4a15df808cd95bf3671f14784fc83b7f8aedcd WatchSource:0}: Error finding container 7aba53f5ffcc398e9576462e5c4a15df808cd95bf3671f14784fc83b7f8aedcd: Status 404 returned error can't find the container with id 7aba53f5ffcc398e9576462e5c4a15df808cd95bf3671f14784fc83b7f8aedcd
Oct 30 19:22:43.845856 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:22:43.845824    4892 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892000Ki","pods":"250"}
Oct 30 19:22:43.845947 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:22:43.845884    4892 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892000Ki","pods":"250"}
Oct 30 19:22:44.296375 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:22:44.296339    4892 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-rqhzf" event={"ID":"b1eed800-8bc3-4333-a929-dcb1477293fd","Type":"ContainerStarted","Data":"290e18ba70b26af6c5f1f6778bb793d84a85aceefc942fd1f9ecf267d6f59109"}
Oct 30 19:22:44.296760 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:22:44.296394    4892 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-rqhzf" event={"ID":"b1eed800-8bc3-4333-a929-dcb1477293fd","Type":"ContainerStarted","Data":"7aba53f5ffcc398e9576462e5c4a15df808cd95bf3671f14784fc83b7f8aedcd"}
Oct 30 19:22:44.296760 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:22:44.296531    4892 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-rqhzf"
Oct 30 19:22:44.298032 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:22:44.298007    4892 patch_prober.go:29] interesting pod/istiod-openshift-gateway-7ddd9cfdc8-rqhzf container/discovery namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Oct 30 19:22:44.298170 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:22:44.298062    4892 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-rqhzf" podUID="b1eed800-8bc3-4333-a929-dcb1477293fd" containerName="discovery" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Oct 30 19:22:44.412941 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:22:44.412898    4892 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-rqhzf" podStartSLOduration=1.412885762 podStartE2EDuration="1.412885762s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:22:43 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:22:44.411677584 +0000 UTC m=+411.982143182" watchObservedRunningTime="2026-10-30 19:22:44.412885762 +0000 UTC m=+411.983351360"
Oct 30 19:22:45.301230 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:22:45.301198    4892 patch_prober.go:29] interesting pod/istiod-openshift-gateway-7ddd9cfdc8-rqhzf container/discovery namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Oct 30 19:22:45.301609 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:22:45.301255    4892 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-rqhzf" podUID="b1eed800-8bc3-4333-a929-dcb1477293fd" containerName="discovery" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Oct 30 19:22:48.301512 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:22:48.301480    4892 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-rqhzf"
Oct 30 19:22:48.521193 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:22:48.521158    4892 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7876bfcb74-cx4zq"]
Oct 30 19:22:48.521431 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:22:48.521386    4892 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-cx4zq" podUID="68efc1e0-8b8f-4b21-b7c5-253f32b45dc2" containerName="discovery" containerID="cri-o://915d3662961637037793ebe56d67b11628cc03834ec14440ffcd24538487d530" gracePeriod=30
Oct 30 19:22:48.767894 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:22:48.767862    4892 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-cx4zq"
Oct 30 19:22:48.875757 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:22:48.875687    4892 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/68efc1e0-8b8f-4b21-b7c5-253f32b45dc2-istio-token\" (UniqueName: \"kubernetes.io/projected/68efc1e0-8b8f-4b21-b7c5-253f32b45dc2-istio-token\") pod \"68efc1e0-8b8f-4b21-b7c5-253f32b45dc2\" (UID: \"68efc1e0-8b8f-4b21-b7c5-253f32b45dc2\") "
Oct 30 19:22:48.875757 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:22:48.875740    4892 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/68efc1e0-8b8f-4b21-b7c5-253f32b45dc2-cacerts\" (UniqueName: \"kubernetes.io/secret/68efc1e0-8b8f-4b21-b7c5-253f32b45dc2-cacerts\") pod \"68efc1e0-8b8f-4b21-b7c5-253f32b45dc2\" (UID: \"68efc1e0-8b8f-4b21-b7c5-253f32b45dc2\") "
Oct 30 19:22:48.876012 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:22:48.875774    4892 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/68efc1e0-8b8f-4b21-b7c5-253f32b45dc2-istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/68efc1e0-8b8f-4b21-b7c5-253f32b45dc2-istio-csr-ca-configmap\") pod \"68efc1e0-8b8f-4b21-b7c5-253f32b45dc2\" (UID: \"68efc1e0-8b8f-4b21-b7c5-253f32b45dc2\") "
Oct 30 19:22:48.876012 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:22:48.875796    4892 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/68efc1e0-8b8f-4b21-b7c5-253f32b45dc2-istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/68efc1e0-8b8f-4b21-b7c5-253f32b45dc2-istio-kubeconfig\") pod \"68efc1e0-8b8f-4b21-b7c5-253f32b45dc2\" (UID: \"68efc1e0-8b8f-4b21-b7c5-253f32b45dc2\") "
Oct 30 19:22:48.876012 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:22:48.875853    4892 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/68efc1e0-8b8f-4b21-b7c5-253f32b45dc2-kube-api-access-jn87s\" (UniqueName: \"kubernetes.io/projected/68efc1e0-8b8f-4b21-b7c5-253f32b45dc2-kube-api-access-jn87s\") pod \"68efc1e0-8b8f-4b21-b7c5-253f32b45dc2\" (UID: \"68efc1e0-8b8f-4b21-b7c5-253f32b45dc2\") "
Oct 30 19:22:48.876012 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:22:48.875887    4892 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/68efc1e0-8b8f-4b21-b7c5-253f32b45dc2-local-certs\" (UniqueName: \"kubernetes.io/empty-dir/68efc1e0-8b8f-4b21-b7c5-253f32b45dc2-local-certs\") pod \"68efc1e0-8b8f-4b21-b7c5-253f32b45dc2\" (UID: \"68efc1e0-8b8f-4b21-b7c5-253f32b45dc2\") "
Oct 30 19:22:48.876012 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:22:48.875906    4892 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/68efc1e0-8b8f-4b21-b7c5-253f32b45dc2-istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/68efc1e0-8b8f-4b21-b7c5-253f32b45dc2-istio-csr-dns-cert\") pod \"68efc1e0-8b8f-4b21-b7c5-253f32b45dc2\" (UID: \"68efc1e0-8b8f-4b21-b7c5-253f32b45dc2\") "
Oct 30 19:22:48.876303 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:22:48.876269    4892 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/68efc1e0-8b8f-4b21-b7c5-253f32b45dc2-istio-csr-ca-configmap" pod "68efc1e0-8b8f-4b21-b7c5-253f32b45dc2" (UID: "68efc1e0-8b8f-4b21-b7c5-253f32b45dc2"). InnerVolumeSpecName "istio-csr-ca-configmap". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 19:22:48.878430 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:22:48.878400    4892 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/68efc1e0-8b8f-4b21-b7c5-253f32b45dc2-cacerts" pod "68efc1e0-8b8f-4b21-b7c5-253f32b45dc2" (UID: "68efc1e0-8b8f-4b21-b7c5-253f32b45dc2"). InnerVolumeSpecName "cacerts". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 19:22:48.878541 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:22:48.878444    4892 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/68efc1e0-8b8f-4b21-b7c5-253f32b45dc2-istio-token" pod "68efc1e0-8b8f-4b21-b7c5-253f32b45dc2" (UID: "68efc1e0-8b8f-4b21-b7c5-253f32b45dc2"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:22:48.878610 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:22:48.878570    4892 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/68efc1e0-8b8f-4b21-b7c5-253f32b45dc2-istio-kubeconfig" pod "68efc1e0-8b8f-4b21-b7c5-253f32b45dc2" (UID: "68efc1e0-8b8f-4b21-b7c5-253f32b45dc2"). InnerVolumeSpecName "istio-kubeconfig". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 19:22:48.878610 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:22:48.878568    4892 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/68efc1e0-8b8f-4b21-b7c5-253f32b45dc2-local-certs" pod "68efc1e0-8b8f-4b21-b7c5-253f32b45dc2" (UID: "68efc1e0-8b8f-4b21-b7c5-253f32b45dc2"). InnerVolumeSpecName "local-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:22:48.878719 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:22:48.878669    4892 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/68efc1e0-8b8f-4b21-b7c5-253f32b45dc2-kube-api-access-jn87s" pod "68efc1e0-8b8f-4b21-b7c5-253f32b45dc2" (UID: "68efc1e0-8b8f-4b21-b7c5-253f32b45dc2"). InnerVolumeSpecName "kube-api-access-jn87s". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:22:48.878777 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:22:48.878746    4892 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/68efc1e0-8b8f-4b21-b7c5-253f32b45dc2-istio-csr-dns-cert" pod "68efc1e0-8b8f-4b21-b7c5-253f32b45dc2" (UID: "68efc1e0-8b8f-4b21-b7c5-253f32b45dc2"). InnerVolumeSpecName "istio-csr-dns-cert". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 19:22:48.976667 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:22:48.976636    4892 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jn87s\" (UniqueName: \"kubernetes.io/projected/68efc1e0-8b8f-4b21-b7c5-253f32b45dc2-kube-api-access-jn87s\") on node \"ip-10-0-132-82.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 19:22:48.976667 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:22:48.976661    4892 reconciler_common.go:299] "Volume detached for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/68efc1e0-8b8f-4b21-b7c5-253f32b45dc2-local-certs\") on node \"ip-10-0-132-82.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 19:22:48.976667 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:22:48.976672    4892 reconciler_common.go:299] "Volume detached for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/68efc1e0-8b8f-4b21-b7c5-253f32b45dc2-istio-csr-dns-cert\") on node \"ip-10-0-132-82.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 19:22:48.976857 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:22:48.976681    4892 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/68efc1e0-8b8f-4b21-b7c5-253f32b45dc2-istio-token\") on node \"ip-10-0-132-82.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 19:22:48.976857 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:22:48.976689    4892 reconciler_common.go:299] "Volume detached for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/68efc1e0-8b8f-4b21-b7c5-253f32b45dc2-cacerts\") on node \"ip-10-0-132-82.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 19:22:48.976857 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:22:48.976698    4892 reconciler_common.go:299] "Volume detached for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/68efc1e0-8b8f-4b21-b7c5-253f32b45dc2-istio-csr-ca-configmap\") on node \"ip-10-0-132-82.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 19:22:48.976857 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:22:48.976706    4892 reconciler_common.go:299] "Volume detached for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/68efc1e0-8b8f-4b21-b7c5-253f32b45dc2-istio-kubeconfig\") on node \"ip-10-0-132-82.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 19:22:48.980556 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:22:48.980533    4892 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/llmisvc-controller-manager-869d5dcc5d-sgzp8"]
Oct 30 19:22:48.980858 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:22:48.980846    4892 cpu_manager.go:414] "containerMap: removing container" podUID="68efc1e0-8b8f-4b21-b7c5-253f32b45dc2" containerName="discovery"
Oct 30 19:22:48.980898 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:22:48.980859    4892 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="68efc1e0-8b8f-4b21-b7c5-253f32b45dc2" containerName="discovery"
Oct 30 19:22:48.980933 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:22:48.980914    4892 memory_manager.go:365] "RemoveStaleState removing state" podUID="68efc1e0-8b8f-4b21-b7c5-253f32b45dc2" containerName="discovery"
Oct 30 19:22:48.983945 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:22:48.983930    4892 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/llmisvc-controller-manager-869d5dcc5d-sgzp8"
Oct 30 19:22:49.036386 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:22:49.036365    4892 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"llmisvc-controller-manager-dockercfg-ts4dk\""
Oct 30 19:22:49.036386 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:22:49.036364    4892 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"llmisvc-webhook-server-cert\""
Oct 30 19:22:49.041726 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:22:49.041702    4892 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/llmisvc-controller-manager-869d5dcc5d-sgzp8"]
Oct 30 19:22:49.094897 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:22:49.094869    4892 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/model-serving-api-75bd89fdfc-d6qpr"]
Oct 30 19:22:49.098134 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:22:49.098119    4892 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/model-serving-api-75bd89fdfc-d6qpr"
Oct 30 19:22:49.150272 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:22:49.150247    4892 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"model-serving-api-tls\""
Oct 30 19:22:49.150272 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:22:49.150261    4892 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"model-serving-api-dockercfg-tbdc8\""
Oct 30 19:22:49.155632 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:22:49.155611    4892 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/model-serving-api-75bd89fdfc-d6qpr"]
Oct 30 19:22:49.178419 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:22:49.178395    4892 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/aa023ebb-d28a-4872-af0b-8ce5ed545078-cert\") pod \"llmisvc-controller-manager-869d5dcc5d-sgzp8\" (UID: \"aa023ebb-d28a-4872-af0b-8ce5ed545078\") " pod="opendatahub/llmisvc-controller-manager-869d5dcc5d-sgzp8"
Oct 30 19:22:49.178502 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:22:49.178440    4892 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jmgmg\" (UniqueName: \"kubernetes.io/projected/aa023ebb-d28a-4872-af0b-8ce5ed545078-kube-api-access-jmgmg\") pod \"llmisvc-controller-manager-869d5dcc5d-sgzp8\" (UID: \"aa023ebb-d28a-4872-af0b-8ce5ed545078\") " pod="opendatahub/llmisvc-controller-manager-869d5dcc5d-sgzp8"
Oct 30 19:22:49.219549 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:22:49.219526    4892 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/odh-model-controller-f87fcf56c-x697s"]
Oct 30 19:22:49.222648 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:22:49.222633    4892 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/odh-model-controller-f87fcf56c-x697s"
Oct 30 19:22:49.274825 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:22:49.274796    4892 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"odh-model-controller-metrics-tls\""
Oct 30 19:22:49.274925 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:22:49.274852    4892 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"odh-model-controller-webhook-cert\""
Oct 30 19:22:49.275045 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:22:49.275029    4892 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"odh-model-controller-dockercfg-r9zhv\""
Oct 30 19:22:49.278854 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:22:49.278821    4892 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/aa023ebb-d28a-4872-af0b-8ce5ed545078-cert\") pod \"llmisvc-controller-manager-869d5dcc5d-sgzp8\" (UID: \"aa023ebb-d28a-4872-af0b-8ce5ed545078\") " pod="opendatahub/llmisvc-controller-manager-869d5dcc5d-sgzp8"
Oct 30 19:22:49.278985 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:22:49.278868    4892 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jcb5c\" (UniqueName: \"kubernetes.io/projected/8bb5f724-d122-454a-a0db-0f9f67ee8c6a-kube-api-access-jcb5c\") pod \"model-serving-api-75bd89fdfc-d6qpr\" (UID: \"8bb5f724-d122-454a-a0db-0f9f67ee8c6a\") " pod="opendatahub/model-serving-api-75bd89fdfc-d6qpr"
Oct 30 19:22:49.278985 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:22:49.278908    4892 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/8bb5f724-d122-454a-a0db-0f9f67ee8c6a-tls-certs\") pod \"model-serving-api-75bd89fdfc-d6qpr\" (UID: \"8bb5f724-d122-454a-a0db-0f9f67ee8c6a\") " pod="opendatahub/model-serving-api-75bd89fdfc-d6qpr"
Oct 30 19:22:49.279119 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:22:49.278995    4892 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jmgmg\" (UniqueName: \"kubernetes.io/projected/aa023ebb-d28a-4872-af0b-8ce5ed545078-kube-api-access-jmgmg\") pod \"llmisvc-controller-manager-869d5dcc5d-sgzp8\" (UID: \"aa023ebb-d28a-4872-af0b-8ce5ed545078\") " pod="opendatahub/llmisvc-controller-manager-869d5dcc5d-sgzp8"
Oct 30 19:22:49.281322 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:22:49.281300    4892 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/odh-model-controller-f87fcf56c-x697s"]
Oct 30 19:22:49.281815 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:22:49.281794    4892 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/aa023ebb-d28a-4872-af0b-8ce5ed545078-cert\") pod \"llmisvc-controller-manager-869d5dcc5d-sgzp8\" (UID: \"aa023ebb-d28a-4872-af0b-8ce5ed545078\") " pod="opendatahub/llmisvc-controller-manager-869d5dcc5d-sgzp8"
Oct 30 19:22:49.318669 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:22:49.318641    4892 generic.go:356] "Generic (PLEG): container finished" podID="68efc1e0-8b8f-4b21-b7c5-253f32b45dc2" containerID="915d3662961637037793ebe56d67b11628cc03834ec14440ffcd24538487d530" exitCode=0
Oct 30 19:22:49.319032 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:22:49.318690    4892 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-cx4zq"
Oct 30 19:22:49.319032 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:22:49.318723    4892 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-cx4zq" event={"ID":"68efc1e0-8b8f-4b21-b7c5-253f32b45dc2","Type":"ContainerDied","Data":"915d3662961637037793ebe56d67b11628cc03834ec14440ffcd24538487d530"}
Oct 30 19:22:49.319032 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:22:49.318768    4892 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-cx4zq" event={"ID":"68efc1e0-8b8f-4b21-b7c5-253f32b45dc2","Type":"ContainerDied","Data":"b5a2ac7a61a62497f76523d2ca6ffd1573df41b64f9ba9783d5645192a7d96ff"}
Oct 30 19:22:49.319032 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:22:49.318791    4892 scope.go:122] "RemoveContainer" containerID="915d3662961637037793ebe56d67b11628cc03834ec14440ffcd24538487d530"
Oct 30 19:22:49.327556 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:22:49.327542    4892 scope.go:122] "RemoveContainer" containerID="915d3662961637037793ebe56d67b11628cc03834ec14440ffcd24538487d530"
Oct 30 19:22:49.327816 ip-10-0-132-82 kubenswrapper[4892]: E1030 19:22:49.327796    4892 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"915d3662961637037793ebe56d67b11628cc03834ec14440ffcd24538487d530\": container with ID starting with 915d3662961637037793ebe56d67b11628cc03834ec14440ffcd24538487d530 not found: ID does not exist" containerID="915d3662961637037793ebe56d67b11628cc03834ec14440ffcd24538487d530"
Oct 30 19:22:49.327875 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:22:49.327824    4892 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"915d3662961637037793ebe56d67b11628cc03834ec14440ffcd24538487d530"} err="failed to get container status \"915d3662961637037793ebe56d67b11628cc03834ec14440ffcd24538487d530\": rpc error: code = NotFound desc = could not find container \"915d3662961637037793ebe56d67b11628cc03834ec14440ffcd24538487d530\": container with ID starting with 915d3662961637037793ebe56d67b11628cc03834ec14440ffcd24538487d530 not found: ID does not exist"
Oct 30 19:22:49.338785 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:22:49.338766    4892 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jmgmg\" (UniqueName: \"kubernetes.io/projected/aa023ebb-d28a-4872-af0b-8ce5ed545078-kube-api-access-jmgmg\") pod \"llmisvc-controller-manager-869d5dcc5d-sgzp8\" (UID: \"aa023ebb-d28a-4872-af0b-8ce5ed545078\") " pod="opendatahub/llmisvc-controller-manager-869d5dcc5d-sgzp8"
Oct 30 19:22:49.379807 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:22:49.379786    4892 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jcb5c\" (UniqueName: \"kubernetes.io/projected/8bb5f724-d122-454a-a0db-0f9f67ee8c6a-kube-api-access-jcb5c\") pod \"model-serving-api-75bd89fdfc-d6qpr\" (UID: \"8bb5f724-d122-454a-a0db-0f9f67ee8c6a\") " pod="opendatahub/model-serving-api-75bd89fdfc-d6qpr"
Oct 30 19:22:49.379917 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:22:49.379825    4892 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/65933d4a-6b0a-4280-8e3c-a16a0ce6c6c3-metrics-certs\") pod \"odh-model-controller-f87fcf56c-x697s\" (UID: \"65933d4a-6b0a-4280-8e3c-a16a0ce6c6c3\") " pod="opendatahub/odh-model-controller-f87fcf56c-x697s"
Oct 30 19:22:49.379917 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:22:49.379871    4892 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/8bb5f724-d122-454a-a0db-0f9f67ee8c6a-tls-certs\") pod \"model-serving-api-75bd89fdfc-d6qpr\" (UID: \"8bb5f724-d122-454a-a0db-0f9f67ee8c6a\") " pod="opendatahub/model-serving-api-75bd89fdfc-d6qpr"
Oct 30 19:22:49.380062 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:22:49.380010    4892 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2h9rx\" (UniqueName: \"kubernetes.io/projected/65933d4a-6b0a-4280-8e3c-a16a0ce6c6c3-kube-api-access-2h9rx\") pod \"odh-model-controller-f87fcf56c-x697s\" (UID: \"65933d4a-6b0a-4280-8e3c-a16a0ce6c6c3\") " pod="opendatahub/odh-model-controller-f87fcf56c-x697s"
Oct 30 19:22:49.380062 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:22:49.380042    4892 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/65933d4a-6b0a-4280-8e3c-a16a0ce6c6c3-cert\") pod \"odh-model-controller-f87fcf56c-x697s\" (UID: \"65933d4a-6b0a-4280-8e3c-a16a0ce6c6c3\") " pod="opendatahub/odh-model-controller-f87fcf56c-x697s"
Oct 30 19:22:49.382149 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:22:49.382132    4892 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/8bb5f724-d122-454a-a0db-0f9f67ee8c6a-tls-certs\") pod \"model-serving-api-75bd89fdfc-d6qpr\" (UID: \"8bb5f724-d122-454a-a0db-0f9f67ee8c6a\") " pod="opendatahub/model-serving-api-75bd89fdfc-d6qpr"
Oct 30 19:22:49.436849 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:22:49.436800    4892 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jcb5c\" (UniqueName: \"kubernetes.io/projected/8bb5f724-d122-454a-a0db-0f9f67ee8c6a-kube-api-access-jcb5c\") pod \"model-serving-api-75bd89fdfc-d6qpr\" (UID: \"8bb5f724-d122-454a-a0db-0f9f67ee8c6a\") " pod="opendatahub/model-serving-api-75bd89fdfc-d6qpr"
Oct 30 19:22:49.446439 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:22:49.446414    4892 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7876bfcb74-cx4zq"]
Oct 30 19:22:49.447958 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:22:49.447937    4892 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7876bfcb74-cx4zq"]
Oct 30 19:22:49.481253 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:22:49.481236    4892 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/65933d4a-6b0a-4280-8e3c-a16a0ce6c6c3-metrics-certs\") pod \"odh-model-controller-f87fcf56c-x697s\" (UID: \"65933d4a-6b0a-4280-8e3c-a16a0ce6c6c3\") " pod="opendatahub/odh-model-controller-f87fcf56c-x697s"
Oct 30 19:22:49.481333 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:22:49.481289    4892 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2h9rx\" (UniqueName: \"kubernetes.io/projected/65933d4a-6b0a-4280-8e3c-a16a0ce6c6c3-kube-api-access-2h9rx\") pod \"odh-model-controller-f87fcf56c-x697s\" (UID: \"65933d4a-6b0a-4280-8e3c-a16a0ce6c6c3\") " pod="opendatahub/odh-model-controller-f87fcf56c-x697s"
Oct 30 19:22:49.481333 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:22:49.481315    4892 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/65933d4a-6b0a-4280-8e3c-a16a0ce6c6c3-cert\") pod \"odh-model-controller-f87fcf56c-x697s\" (UID: \"65933d4a-6b0a-4280-8e3c-a16a0ce6c6c3\") " pod="opendatahub/odh-model-controller-f87fcf56c-x697s"
Oct 30 19:22:49.483475 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:22:49.483452    4892 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/65933d4a-6b0a-4280-8e3c-a16a0ce6c6c3-metrics-certs\") pod \"odh-model-controller-f87fcf56c-x697s\" (UID: \"65933d4a-6b0a-4280-8e3c-a16a0ce6c6c3\") " pod="opendatahub/odh-model-controller-f87fcf56c-x697s"
Oct 30 19:22:49.483550 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:22:49.483505    4892 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/65933d4a-6b0a-4280-8e3c-a16a0ce6c6c3-cert\") pod \"odh-model-controller-f87fcf56c-x697s\" (UID: \"65933d4a-6b0a-4280-8e3c-a16a0ce6c6c3\") " pod="opendatahub/odh-model-controller-f87fcf56c-x697s"
Oct 30 19:22:49.538148 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:22:49.538127    4892 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2h9rx\" (UniqueName: \"kubernetes.io/projected/65933d4a-6b0a-4280-8e3c-a16a0ce6c6c3-kube-api-access-2h9rx\") pod \"odh-model-controller-f87fcf56c-x697s\" (UID: \"65933d4a-6b0a-4280-8e3c-a16a0ce6c6c3\") " pod="opendatahub/odh-model-controller-f87fcf56c-x697s"
Oct 30 19:22:49.595342 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:22:49.595316    4892 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/llmisvc-controller-manager-869d5dcc5d-sgzp8"
Oct 30 19:22:49.710755 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:22:49.710680    4892 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/model-serving-api-75bd89fdfc-d6qpr"
Oct 30 19:22:49.834440 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:22:49.834413    4892 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/odh-model-controller-f87fcf56c-x697s"
Oct 30 19:22:49.966669 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:22:49.966643    4892 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/llmisvc-controller-manager-869d5dcc5d-sgzp8"]
Oct 30 19:22:49.968590 ip-10-0-132-82 kubenswrapper[4892]: W1030 19:22:49.968565    4892 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-podaa023ebb_d28a_4872_af0b_8ce5ed545078.slice/crio-cad7e95b6a3ae6bb3ef1fb55928c448caf0010d50197d198d5bceb86918acfc6 WatchSource:0}: Error finding container cad7e95b6a3ae6bb3ef1fb55928c448caf0010d50197d198d5bceb86918acfc6: Status 404 returned error can't find the container with id cad7e95b6a3ae6bb3ef1fb55928c448caf0010d50197d198d5bceb86918acfc6
Oct 30 19:22:50.080236 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:22:50.080208    4892 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/model-serving-api-75bd89fdfc-d6qpr"]
Oct 30 19:22:50.082771 ip-10-0-132-82 kubenswrapper[4892]: W1030 19:22:50.082750    4892 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8bb5f724_d122_454a_a0db_0f9f67ee8c6a.slice/crio-e5e666dcc1b32658906df430c388438f1ed0621cd46316f36d2dd6e9a4a95125 WatchSource:0}: Error finding container e5e666dcc1b32658906df430c388438f1ed0621cd46316f36d2dd6e9a4a95125: Status 404 returned error can't find the container with id e5e666dcc1b32658906df430c388438f1ed0621cd46316f36d2dd6e9a4a95125
Oct 30 19:22:50.205731 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:22:50.205710    4892 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/odh-model-controller-f87fcf56c-x697s"]
Oct 30 19:22:50.208224 ip-10-0-132-82 kubenswrapper[4892]: W1030 19:22:50.208199    4892 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod65933d4a_6b0a_4280_8e3c_a16a0ce6c6c3.slice/crio-8dd485832afabb20555c84dad9533de7ecafba7e030ffc22613f23c2e9aa4d39 WatchSource:0}: Error finding container 8dd485832afabb20555c84dad9533de7ecafba7e030ffc22613f23c2e9aa4d39: Status 404 returned error can't find the container with id 8dd485832afabb20555c84dad9533de7ecafba7e030ffc22613f23c2e9aa4d39
Oct 30 19:22:50.328299 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:22:50.328221    4892 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/model-serving-api-75bd89fdfc-d6qpr" event={"ID":"8bb5f724-d122-454a-a0db-0f9f67ee8c6a","Type":"ContainerStarted","Data":"e5e666dcc1b32658906df430c388438f1ed0621cd46316f36d2dd6e9a4a95125"}
Oct 30 19:22:50.329420 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:22:50.329397    4892 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/odh-model-controller-f87fcf56c-x697s" event={"ID":"65933d4a-6b0a-4280-8e3c-a16a0ce6c6c3","Type":"ContainerStarted","Data":"8dd485832afabb20555c84dad9533de7ecafba7e030ffc22613f23c2e9aa4d39"}
Oct 30 19:22:50.331303 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:22:50.331282    4892 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/llmisvc-controller-manager-869d5dcc5d-sgzp8" event={"ID":"aa023ebb-d28a-4872-af0b-8ce5ed545078","Type":"ContainerStarted","Data":"cad7e95b6a3ae6bb3ef1fb55928c448caf0010d50197d198d5bceb86918acfc6"}
Oct 30 19:22:50.568007 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:22:50.567955    4892 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="68efc1e0-8b8f-4b21-b7c5-253f32b45dc2" path="/var/lib/kubelet/pods/68efc1e0-8b8f-4b21-b7c5-253f32b45dc2/volumes"
Oct 30 19:22:53.351380 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:22:53.351291    4892 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/model-serving-api-75bd89fdfc-d6qpr" event={"ID":"8bb5f724-d122-454a-a0db-0f9f67ee8c6a","Type":"ContainerStarted","Data":"92f5a4d80219f64e06c9255930f392e5e6105b116e98f45483c1449aa7ab3e7f"}
Oct 30 19:22:53.351788 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:22:53.351400    4892 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/model-serving-api-75bd89fdfc-d6qpr"
Oct 30 19:22:53.464647 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:22:53.464593    4892 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/model-serving-api-75bd89fdfc-d6qpr" podStartSLOduration=1.572714372 podStartE2EDuration="4.464579026s" totalImagesPullingTime="2.891864654s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:22:49 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:22:53.463510836 +0000 UTC m=+421.033976438" watchObservedRunningTime="2026-10-30 19:22:53.464579026 +0000 UTC m=+421.035044625"
Oct 30 19:22:54.356705 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:22:54.356664    4892 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/odh-model-controller-f87fcf56c-x697s" event={"ID":"65933d4a-6b0a-4280-8e3c-a16a0ce6c6c3","Type":"ContainerStarted","Data":"b608dedbeadad8ef8343c7324402880bc17bcf82f30f604944449966c13f0bdc"}
Oct 30 19:22:54.357193 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:22:54.356768    4892 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/odh-model-controller-f87fcf56c-x697s"
Oct 30 19:22:54.358170 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:22:54.358144    4892 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/llmisvc-controller-manager-869d5dcc5d-sgzp8" event={"ID":"aa023ebb-d28a-4872-af0b-8ce5ed545078","Type":"ContainerStarted","Data":"a9ecc7d98760a5e781be8ff8477c1bf8d60104b2ef2e27580ea30ca25b718787"}
Oct 30 19:22:54.358291 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:22:54.358232    4892 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/llmisvc-controller-manager-869d5dcc5d-sgzp8"
Oct 30 19:22:54.469397 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:22:54.469324    4892 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/odh-model-controller-f87fcf56c-x697s" podStartSLOduration=1.963360164 podStartE2EDuration="5.469307877s" totalImagesPullingTime="3.505947713s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:22:49 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:22:54.468241641 +0000 UTC m=+422.038707241" watchObservedRunningTime="2026-10-30 19:22:54.469307877 +0000 UTC m=+422.039773478"
Oct 30 19:22:54.584140 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:22:54.584089    4892 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/llmisvc-controller-manager-869d5dcc5d-sgzp8" podStartSLOduration=3.040008672 podStartE2EDuration="6.58407565s" totalImagesPullingTime="3.544066978s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:22:48 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:22:54.582735788 +0000 UTC m=+422.153201388" watchObservedRunningTime="2026-10-30 19:22:54.58407565 +0000 UTC m=+422.154541242"
Oct 30 19:22:54.843025 ip-10-0-132-82 kubenswrapper[4892]: E1030 19:22:54.842990    4892 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-jrm87" containerName="extractor"
Oct 30 19:23:04.362172 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:23:04.362139    4892 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/model-serving-api-75bd89fdfc-d6qpr"
Oct 30 19:23:05.364501 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:23:05.364474    4892 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/odh-model-controller-f87fcf56c-x697s"
Oct 30 19:23:24.842133 ip-10-0-132-82 kubenswrapper[4892]: E1030 19:23:24.842097    4892 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-jrm87" containerName="extractor"
Oct 30 19:23:25.364505 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:23:25.364477    4892 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/llmisvc-controller-manager-869d5dcc5d-sgzp8"
Oct 30 19:23:54.842835 ip-10-0-132-82 kubenswrapper[4892]: E1030 19:23:54.842799    4892 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-jrm87" containerName="extractor"
Oct 30 19:24:10.791663 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:24:10.791581    4892 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847a9v8hq"]
Oct 30 19:24:10.794097 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:24:10.794075    4892 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847a9v8hq"
Oct 30 19:24:10.803514 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:24:10.803491    4892 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-22ccc\" (UniqueName: \"kubernetes.io/projected/28eaf4e0-609f-41dd-8d6a-ebf328b2c6b8-kube-api-access-22ccc\") pod \"6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847a9v8hq\" (UID: \"28eaf4e0-609f-41dd-8d6a-ebf328b2c6b8\") " pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847a9v8hq"
Oct 30 19:24:10.803646 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:24:10.803612    4892 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/28eaf4e0-609f-41dd-8d6a-ebf328b2c6b8-bundle\") pod \"6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847a9v8hq\" (UID: \"28eaf4e0-609f-41dd-8d6a-ebf328b2c6b8\") " pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847a9v8hq"
Oct 30 19:24:10.803710 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:24:10.803685    4892 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/28eaf4e0-609f-41dd-8d6a-ebf328b2c6b8-util\") pod \"6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847a9v8hq\" (UID: \"28eaf4e0-609f-41dd-8d6a-ebf328b2c6b8\") " pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847a9v8hq"
Oct 30 19:24:10.847540 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:24:10.847509    4892 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d78czc9"]
Oct 30 19:24:10.849762 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:24:10.849744    4892 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d78czc9"
Oct 30 19:24:10.895393 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:24:10.895371    4892 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"openshift-service-ca.crt\""
Oct 30 19:24:10.895512 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:24:10.895430    4892 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"kube-root-ca.crt\""
Oct 30 19:24:10.895512 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:24:10.895430    4892 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-marketplace\"/\"6541d93aaee335d6227a8778ab4e840bbe7f52-f98fbcf8-dockercfg-hb4tp\""
Oct 30 19:24:10.899916 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:24:10.899895    4892 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847a9v8hq"]
Oct 30 19:24:10.904683 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:24:10.904658    4892 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-c4dk8\" (UniqueName: \"kubernetes.io/projected/1ca4bc7d-1118-4f18-b687-fc4368ff4f22-kube-api-access-c4dk8\") pod \"da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d78czc9\" (UID: \"1ca4bc7d-1118-4f18-b687-fc4368ff4f22\") " pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d78czc9"
Oct 30 19:24:10.904775 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:24:10.904703    4892 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-22ccc\" (UniqueName: \"kubernetes.io/projected/28eaf4e0-609f-41dd-8d6a-ebf328b2c6b8-kube-api-access-22ccc\") pod \"6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847a9v8hq\" (UID: \"28eaf4e0-609f-41dd-8d6a-ebf328b2c6b8\") " pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847a9v8hq"
Oct 30 19:24:10.904832 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:24:10.904814    4892 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/28eaf4e0-609f-41dd-8d6a-ebf328b2c6b8-bundle\") pod \"6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847a9v8hq\" (UID: \"28eaf4e0-609f-41dd-8d6a-ebf328b2c6b8\") " pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847a9v8hq"
Oct 30 19:24:10.904908 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:24:10.904892    4892 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/28eaf4e0-609f-41dd-8d6a-ebf328b2c6b8-util\") pod \"6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847a9v8hq\" (UID: \"28eaf4e0-609f-41dd-8d6a-ebf328b2c6b8\") " pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847a9v8hq"
Oct 30 19:24:10.904949 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:24:10.904932    4892 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/1ca4bc7d-1118-4f18-b687-fc4368ff4f22-util\") pod \"da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d78czc9\" (UID: \"1ca4bc7d-1118-4f18-b687-fc4368ff4f22\") " pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d78czc9"
Oct 30 19:24:10.905011 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:24:10.904987    4892 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/1ca4bc7d-1118-4f18-b687-fc4368ff4f22-bundle\") pod \"da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d78czc9\" (UID: \"1ca4bc7d-1118-4f18-b687-fc4368ff4f22\") " pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d78czc9"
Oct 30 19:24:10.905206 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:24:10.905189    4892 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/28eaf4e0-609f-41dd-8d6a-ebf328b2c6b8-bundle\") pod \"6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847a9v8hq\" (UID: \"28eaf4e0-609f-41dd-8d6a-ebf328b2c6b8\") " pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847a9v8hq"
Oct 30 19:24:10.905257 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:24:10.905222    4892 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/28eaf4e0-609f-41dd-8d6a-ebf328b2c6b8-util\") pod \"6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847a9v8hq\" (UID: \"28eaf4e0-609f-41dd-8d6a-ebf328b2c6b8\") " pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847a9v8hq"
Oct 30 19:24:10.909862 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:24:10.909839    4892 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d78czc9"]
Oct 30 19:24:10.946670 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:24:10.946647    4892 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c66hlhx"]
Oct 30 19:24:10.949088 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:24:10.949066    4892 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c66hlhx"
Oct 30 19:24:10.950531 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:24:10.950511    4892 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-marketplace\"/\"da112bd609bff87f7bacd2ba941225d7cff15c-bd73b2fc-dockercfg-dltwf\""
Oct 30 19:24:10.960954 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:24:10.960929    4892 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-22ccc\" (UniqueName: \"kubernetes.io/projected/28eaf4e0-609f-41dd-8d6a-ebf328b2c6b8-kube-api-access-22ccc\") pod \"6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847a9v8hq\" (UID: \"28eaf4e0-609f-41dd-8d6a-ebf328b2c6b8\") " pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847a9v8hq"
Oct 30 19:24:11.006144 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:24:11.006120    4892 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/e5a09d53-32da-4af8-b4ba-a3c092983230-util\") pod \"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c66hlhx\" (UID: \"e5a09d53-32da-4af8-b4ba-a3c092983230\") " pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c66hlhx"
Oct 30 19:24:11.006281 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:24:11.006259    4892 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tx9gc\" (UniqueName: \"kubernetes.io/projected/e5a09d53-32da-4af8-b4ba-a3c092983230-kube-api-access-tx9gc\") pod \"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c66hlhx\" (UID: \"e5a09d53-32da-4af8-b4ba-a3c092983230\") " pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c66hlhx"
Oct 30 19:24:11.006387 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:24:11.006359    4892 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/1ca4bc7d-1118-4f18-b687-fc4368ff4f22-util\") pod \"da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d78czc9\" (UID: \"1ca4bc7d-1118-4f18-b687-fc4368ff4f22\") " pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d78czc9"
Oct 30 19:24:11.006506 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:24:11.006457    4892 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/1ca4bc7d-1118-4f18-b687-fc4368ff4f22-bundle\") pod \"da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d78czc9\" (UID: \"1ca4bc7d-1118-4f18-b687-fc4368ff4f22\") " pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d78czc9"
Oct 30 19:24:11.006506 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:24:11.006501    4892 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/e5a09d53-32da-4af8-b4ba-a3c092983230-bundle\") pod \"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c66hlhx\" (UID: \"e5a09d53-32da-4af8-b4ba-a3c092983230\") " pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c66hlhx"
Oct 30 19:24:11.006624 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:24:11.006534    4892 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-c4dk8\" (UniqueName: \"kubernetes.io/projected/1ca4bc7d-1118-4f18-b687-fc4368ff4f22-kube-api-access-c4dk8\") pod \"da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d78czc9\" (UID: \"1ca4bc7d-1118-4f18-b687-fc4368ff4f22\") " pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d78czc9"
Oct 30 19:24:11.006742 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:24:11.006719    4892 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/1ca4bc7d-1118-4f18-b687-fc4368ff4f22-util\") pod \"da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d78czc9\" (UID: \"1ca4bc7d-1118-4f18-b687-fc4368ff4f22\") " pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d78czc9"
Oct 30 19:24:11.006812 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:24:11.006767    4892 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/1ca4bc7d-1118-4f18-b687-fc4368ff4f22-bundle\") pod \"da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d78czc9\" (UID: \"1ca4bc7d-1118-4f18-b687-fc4368ff4f22\") " pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d78czc9"
Oct 30 19:24:11.008252 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:24:11.008228    4892 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c66hlhx"]
Oct 30 19:24:11.037164 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:24:11.037143    4892 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cw85wp"]
Oct 30 19:24:11.039455 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:24:11.039442    4892 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cw85wp"
Oct 30 19:24:11.051104 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:24:11.051086    4892 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-marketplace\"/\"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3-b83c8cba-dockercfg-dwkkd\""
Oct 30 19:24:11.063312 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:24:11.063283    4892 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-c4dk8\" (UniqueName: \"kubernetes.io/projected/1ca4bc7d-1118-4f18-b687-fc4368ff4f22-kube-api-access-c4dk8\") pod \"da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d78czc9\" (UID: \"1ca4bc7d-1118-4f18-b687-fc4368ff4f22\") " pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d78czc9"
Oct 30 19:24:11.099223 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:24:11.099201    4892 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cw85wp"]
Oct 30 19:24:11.106463 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:24:11.106444    4892 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847a9v8hq"
Oct 30 19:24:11.107798 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:24:11.107763    4892 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/e5a09d53-32da-4af8-b4ba-a3c092983230-util\") pod \"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c66hlhx\" (UID: \"e5a09d53-32da-4af8-b4ba-a3c092983230\") " pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c66hlhx"
Oct 30 19:24:11.107912 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:24:11.107823    4892 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sktmd\" (UniqueName: \"kubernetes.io/projected/964169ca-3813-4787-96d1-4a8eed26f023-kube-api-access-sktmd\") pod \"e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cw85wp\" (UID: \"964169ca-3813-4787-96d1-4a8eed26f023\") " pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cw85wp"
Oct 30 19:24:11.107912 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:24:11.107870    4892 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tx9gc\" (UniqueName: \"kubernetes.io/projected/e5a09d53-32da-4af8-b4ba-a3c092983230-kube-api-access-tx9gc\") pod \"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c66hlhx\" (UID: \"e5a09d53-32da-4af8-b4ba-a3c092983230\") " pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c66hlhx"
Oct 30 19:24:11.108067 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:24:11.107989    4892 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/e5a09d53-32da-4af8-b4ba-a3c092983230-bundle\") pod \"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c66hlhx\" (UID: \"e5a09d53-32da-4af8-b4ba-a3c092983230\") " pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c66hlhx"
Oct 30 19:24:11.108067 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:24:11.108028    4892 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/964169ca-3813-4787-96d1-4a8eed26f023-util\") pod \"e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cw85wp\" (UID: \"964169ca-3813-4787-96d1-4a8eed26f023\") " pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cw85wp"
Oct 30 19:24:11.108488 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:24:11.108066    4892 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/964169ca-3813-4787-96d1-4a8eed26f023-bundle\") pod \"e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cw85wp\" (UID: \"964169ca-3813-4787-96d1-4a8eed26f023\") " pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cw85wp"
Oct 30 19:24:11.108488 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:24:11.108175    4892 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/e5a09d53-32da-4af8-b4ba-a3c092983230-util\") pod \"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c66hlhx\" (UID: \"e5a09d53-32da-4af8-b4ba-a3c092983230\") " pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c66hlhx"
Oct 30 19:24:11.108488 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:24:11.108322    4892 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/e5a09d53-32da-4af8-b4ba-a3c092983230-bundle\") pod \"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c66hlhx\" (UID: \"e5a09d53-32da-4af8-b4ba-a3c092983230\") " pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c66hlhx"
Oct 30 19:24:11.152786 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:24:11.152137    4892 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-marketplace\"/\"e3d011e69d9573cabb8c359e5ac709eb5ac240-be78d160-dockercfg-lq2zg\""
Oct 30 19:24:11.161287 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:24:11.161265    4892 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d78czc9"
Oct 30 19:24:11.165667 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:24:11.165640    4892 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tx9gc\" (UniqueName: \"kubernetes.io/projected/e5a09d53-32da-4af8-b4ba-a3c092983230-kube-api-access-tx9gc\") pod \"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c66hlhx\" (UID: \"e5a09d53-32da-4af8-b4ba-a3c092983230\") " pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c66hlhx"
Oct 30 19:24:11.209148 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:24:11.209118    4892 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/964169ca-3813-4787-96d1-4a8eed26f023-util\") pod \"e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cw85wp\" (UID: \"964169ca-3813-4787-96d1-4a8eed26f023\") " pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cw85wp"
Oct 30 19:24:11.209312 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:24:11.209158    4892 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/964169ca-3813-4787-96d1-4a8eed26f023-bundle\") pod \"e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cw85wp\" (UID: \"964169ca-3813-4787-96d1-4a8eed26f023\") " pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cw85wp"
Oct 30 19:24:11.209312 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:24:11.209194    4892 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sktmd\" (UniqueName: \"kubernetes.io/projected/964169ca-3813-4787-96d1-4a8eed26f023-kube-api-access-sktmd\") pod \"e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cw85wp\" (UID: \"964169ca-3813-4787-96d1-4a8eed26f023\") " pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cw85wp"
Oct 30 19:24:11.209679 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:24:11.209656    4892 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/964169ca-3813-4787-96d1-4a8eed26f023-util\") pod \"e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cw85wp\" (UID: \"964169ca-3813-4787-96d1-4a8eed26f023\") " pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cw85wp"
Oct 30 19:24:11.209765 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:24:11.209688    4892 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/964169ca-3813-4787-96d1-4a8eed26f023-bundle\") pod \"e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cw85wp\" (UID: \"964169ca-3813-4787-96d1-4a8eed26f023\") " pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cw85wp"
Oct 30 19:24:11.262142 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:24:11.262115    4892 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c66hlhx"
Oct 30 19:24:11.267065 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:24:11.267037    4892 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-sktmd\" (UniqueName: \"kubernetes.io/projected/964169ca-3813-4787-96d1-4a8eed26f023-kube-api-access-sktmd\") pod \"e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cw85wp\" (UID: \"964169ca-3813-4787-96d1-4a8eed26f023\") " pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cw85wp"
Oct 30 19:24:11.352294 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:24:11.352269    4892 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cw85wp"
Oct 30 19:24:11.478213 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:24:11.478185    4892 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847a9v8hq"]
Oct 30 19:24:11.480018 ip-10-0-132-82 kubenswrapper[4892]: W1030 19:24:11.479991    4892 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod28eaf4e0_609f_41dd_8d6a_ebf328b2c6b8.slice/crio-ea9614f06e2f1372265eeddff71e411dcd97a88b6f7bbdc5f84a9ac82c6787ca WatchSource:0}: Error finding container ea9614f06e2f1372265eeddff71e411dcd97a88b6f7bbdc5f84a9ac82c6787ca: Status 404 returned error can't find the container with id ea9614f06e2f1372265eeddff71e411dcd97a88b6f7bbdc5f84a9ac82c6787ca
Oct 30 19:24:11.580277 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:24:11.580253    4892 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d78czc9"]
Oct 30 19:24:11.626681 ip-10-0-132-82 kubenswrapper[4892]: W1030 19:24:11.626659    4892 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1ca4bc7d_1118_4f18_b687_fc4368ff4f22.slice/crio-991f077968226fa35c34bbbba27a1f56630ab450e0f9f289747cf16454da4c6b WatchSource:0}: Error finding container 991f077968226fa35c34bbbba27a1f56630ab450e0f9f289747cf16454da4c6b: Status 404 returned error can't find the container with id 991f077968226fa35c34bbbba27a1f56630ab450e0f9f289747cf16454da4c6b
Oct 30 19:24:11.633757 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:24:11.633733    4892 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c66hlhx"]
Oct 30 19:24:11.695198 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:24:11.695167    4892 generic.go:356] "Generic (PLEG): container finished" podID="28eaf4e0-609f-41dd-8d6a-ebf328b2c6b8" containerID="127ba6f451cba789d79f29b11cc7d7ca6d2c27653938355e8234ae048bc7fc0a" exitCode=0
Oct 30 19:24:11.695293 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:24:11.695213    4892 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847a9v8hq" event={"ID":"28eaf4e0-609f-41dd-8d6a-ebf328b2c6b8","Type":"ContainerDied","Data":"127ba6f451cba789d79f29b11cc7d7ca6d2c27653938355e8234ae048bc7fc0a"}
Oct 30 19:24:11.695293 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:24:11.695271    4892 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847a9v8hq" event={"ID":"28eaf4e0-609f-41dd-8d6a-ebf328b2c6b8","Type":"ContainerStarted","Data":"ea9614f06e2f1372265eeddff71e411dcd97a88b6f7bbdc5f84a9ac82c6787ca"}
Oct 30 19:24:11.697240 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:24:11.697019    4892 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d78czc9" event={"ID":"1ca4bc7d-1118-4f18-b687-fc4368ff4f22","Type":"ContainerStarted","Data":"6db2ece2a0785ff3be90e7ce6c99289ce89524decd20fc886cec50fef57d5f3b"}
Oct 30 19:24:11.697240 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:24:11.697074    4892 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d78czc9" event={"ID":"1ca4bc7d-1118-4f18-b687-fc4368ff4f22","Type":"ContainerStarted","Data":"991f077968226fa35c34bbbba27a1f56630ab450e0f9f289747cf16454da4c6b"}
Oct 30 19:24:11.698772 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:24:11.698753    4892 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c66hlhx" event={"ID":"e5a09d53-32da-4af8-b4ba-a3c092983230","Type":"ContainerStarted","Data":"5fa1091f7ebd7e44880cfe4d0ee495d1fb4e019eed0d369b22b3b16eeac84b0b"}
Oct 30 19:24:11.698861 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:24:11.698788    4892 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c66hlhx" event={"ID":"e5a09d53-32da-4af8-b4ba-a3c092983230","Type":"ContainerStarted","Data":"29b57f8ddd3d2759d6c33dbedf5ad2fc4b0619cf1a5532d2099bb32ddc05be53"}
Oct 30 19:24:11.720043 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:24:11.720012    4892 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cw85wp"]
Oct 30 19:24:11.724310 ip-10-0-132-82 kubenswrapper[4892]: W1030 19:24:11.724282    4892 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod964169ca_3813_4787_96d1_4a8eed26f023.slice/crio-8b7bd10e3e29efc8d6e8d4035868a80df0c92f2332aed6aca6791c8109bdbd02 WatchSource:0}: Error finding container 8b7bd10e3e29efc8d6e8d4035868a80df0c92f2332aed6aca6791c8109bdbd02: Status 404 returned error can't find the container with id 8b7bd10e3e29efc8d6e8d4035868a80df0c92f2332aed6aca6791c8109bdbd02
Oct 30 19:24:12.704885 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:24:12.704851    4892 generic.go:356] "Generic (PLEG): container finished" podID="1ca4bc7d-1118-4f18-b687-fc4368ff4f22" containerID="6db2ece2a0785ff3be90e7ce6c99289ce89524decd20fc886cec50fef57d5f3b" exitCode=0
Oct 30 19:24:12.705353 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:24:12.704920    4892 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d78czc9" event={"ID":"1ca4bc7d-1118-4f18-b687-fc4368ff4f22","Type":"ContainerDied","Data":"6db2ece2a0785ff3be90e7ce6c99289ce89524decd20fc886cec50fef57d5f3b"}
Oct 30 19:24:12.706649 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:24:12.706624    4892 generic.go:356] "Generic (PLEG): container finished" podID="e5a09d53-32da-4af8-b4ba-a3c092983230" containerID="5fa1091f7ebd7e44880cfe4d0ee495d1fb4e019eed0d369b22b3b16eeac84b0b" exitCode=0
Oct 30 19:24:12.706765 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:24:12.706714    4892 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c66hlhx" event={"ID":"e5a09d53-32da-4af8-b4ba-a3c092983230","Type":"ContainerDied","Data":"5fa1091f7ebd7e44880cfe4d0ee495d1fb4e019eed0d369b22b3b16eeac84b0b"}
Oct 30 19:24:12.708499 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:24:12.708480    4892 generic.go:356] "Generic (PLEG): container finished" podID="964169ca-3813-4787-96d1-4a8eed26f023" containerID="9180d1952ded8f1912c8a4110963055b470e02cc64fa1b5f2892396ecb60fe7c" exitCode=0
Oct 30 19:24:12.708574 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:24:12.708511    4892 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cw85wp" event={"ID":"964169ca-3813-4787-96d1-4a8eed26f023","Type":"ContainerDied","Data":"9180d1952ded8f1912c8a4110963055b470e02cc64fa1b5f2892396ecb60fe7c"}
Oct 30 19:24:12.708574 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:24:12.708544    4892 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cw85wp" event={"ID":"964169ca-3813-4787-96d1-4a8eed26f023","Type":"ContainerStarted","Data":"8b7bd10e3e29efc8d6e8d4035868a80df0c92f2332aed6aca6791c8109bdbd02"}
Oct 30 19:24:14.726761 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:24:14.726682    4892 generic.go:356] "Generic (PLEG): container finished" podID="28eaf4e0-609f-41dd-8d6a-ebf328b2c6b8" containerID="2e85a39a8edd36e07a6dbca45034ff11e77960aecdba3eca1af074412c04b5b3" exitCode=0
Oct 30 19:24:14.727141 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:24:14.726761    4892 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847a9v8hq" event={"ID":"28eaf4e0-609f-41dd-8d6a-ebf328b2c6b8","Type":"ContainerDied","Data":"2e85a39a8edd36e07a6dbca45034ff11e77960aecdba3eca1af074412c04b5b3"}
Oct 30 19:24:14.730157 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:24:14.730122    4892 generic.go:356] "Generic (PLEG): container finished" podID="964169ca-3813-4787-96d1-4a8eed26f023" containerID="017423fbfd04320a2f010007d020d5018f300020448bf2ad92b11dbcfb184910" exitCode=0
Oct 30 19:24:14.732417 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:24:14.730173    4892 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cw85wp" event={"ID":"964169ca-3813-4787-96d1-4a8eed26f023","Type":"ContainerDied","Data":"017423fbfd04320a2f010007d020d5018f300020448bf2ad92b11dbcfb184910"}
Oct 30 19:24:14.733689 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:24:14.733669    4892 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c66hlhx" event={"ID":"e5a09d53-32da-4af8-b4ba-a3c092983230","Type":"ContainerStarted","Data":"1ac3e0c47b953f9a586378cb7d74ca304a0844b681a0a6cf487256f598206308"}
Oct 30 19:24:15.740615 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:24:15.740587    4892 generic.go:356] "Generic (PLEG): container finished" podID="28eaf4e0-609f-41dd-8d6a-ebf328b2c6b8" containerID="b12a692ea43b0b5fa76b7d58f397efb3174da1bc3a83f0df8b6116b7e21c52aa" exitCode=0
Oct 30 19:24:15.741019 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:24:15.740661    4892 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847a9v8hq" event={"ID":"28eaf4e0-609f-41dd-8d6a-ebf328b2c6b8","Type":"ContainerDied","Data":"b12a692ea43b0b5fa76b7d58f397efb3174da1bc3a83f0df8b6116b7e21c52aa"}
Oct 30 19:24:15.742651 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:24:15.742629    4892 generic.go:356] "Generic (PLEG): container finished" podID="964169ca-3813-4787-96d1-4a8eed26f023" containerID="200417d3e22e0f375d698bf8d0f30993b8986579e26a90c6071ced35d2d646f6" exitCode=0
Oct 30 19:24:15.742733 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:24:15.742703    4892 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cw85wp" event={"ID":"964169ca-3813-4787-96d1-4a8eed26f023","Type":"ContainerDied","Data":"200417d3e22e0f375d698bf8d0f30993b8986579e26a90c6071ced35d2d646f6"}
Oct 30 19:24:15.744357 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:24:15.744333    4892 generic.go:356] "Generic (PLEG): container finished" podID="1ca4bc7d-1118-4f18-b687-fc4368ff4f22" containerID="399ee9984e4cbed3720fdee470c9cfff67f90df27b7865acb525837174ccf2c2" exitCode=0
Oct 30 19:24:15.744432 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:24:15.744397    4892 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d78czc9" event={"ID":"1ca4bc7d-1118-4f18-b687-fc4368ff4f22","Type":"ContainerDied","Data":"399ee9984e4cbed3720fdee470c9cfff67f90df27b7865acb525837174ccf2c2"}
Oct 30 19:24:15.746741 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:24:15.746718    4892 generic.go:356] "Generic (PLEG): container finished" podID="e5a09d53-32da-4af8-b4ba-a3c092983230" containerID="1ac3e0c47b953f9a586378cb7d74ca304a0844b681a0a6cf487256f598206308" exitCode=0
Oct 30 19:24:15.746741 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:24:15.746739    4892 generic.go:356] "Generic (PLEG): container finished" podID="e5a09d53-32da-4af8-b4ba-a3c092983230" containerID="abf144ae8f1017bbea0e6dd180b17919152eff191020ac88246cc6d923101302" exitCode=0
Oct 30 19:24:15.746870 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:24:15.746754    4892 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c66hlhx" event={"ID":"e5a09d53-32da-4af8-b4ba-a3c092983230","Type":"ContainerDied","Data":"1ac3e0c47b953f9a586378cb7d74ca304a0844b681a0a6cf487256f598206308"}
Oct 30 19:24:15.746870 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:24:15.746791    4892 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c66hlhx" event={"ID":"e5a09d53-32da-4af8-b4ba-a3c092983230","Type":"ContainerDied","Data":"abf144ae8f1017bbea0e6dd180b17919152eff191020ac88246cc6d923101302"}
Oct 30 19:24:16.753300 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:24:16.753264    4892 generic.go:356] "Generic (PLEG): container finished" podID="1ca4bc7d-1118-4f18-b687-fc4368ff4f22" containerID="0089c38c2a69611501a742fc3da78ddcdf8fc6b96b0bbc14094f3049680d6848" exitCode=0
Oct 30 19:24:16.753786 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:24:16.753327    4892 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d78czc9" event={"ID":"1ca4bc7d-1118-4f18-b687-fc4368ff4f22","Type":"ContainerDied","Data":"0089c38c2a69611501a742fc3da78ddcdf8fc6b96b0bbc14094f3049680d6848"}
Oct 30 19:24:16.892957 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:24:16.892934    4892 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847a9v8hq"
Oct 30 19:24:16.942850 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:24:16.942829    4892 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cw85wp"
Oct 30 19:24:16.945790 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:24:16.945770    4892 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c66hlhx"
Oct 30 19:24:16.954493 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:24:16.954473    4892 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/28eaf4e0-609f-41dd-8d6a-ebf328b2c6b8-kube-api-access-22ccc\" (UniqueName: \"kubernetes.io/projected/28eaf4e0-609f-41dd-8d6a-ebf328b2c6b8-kube-api-access-22ccc\") pod \"28eaf4e0-609f-41dd-8d6a-ebf328b2c6b8\" (UID: \"28eaf4e0-609f-41dd-8d6a-ebf328b2c6b8\") "
Oct 30 19:24:16.954562 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:24:16.954516    4892 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/28eaf4e0-609f-41dd-8d6a-ebf328b2c6b8-bundle\" (UniqueName: \"kubernetes.io/empty-dir/28eaf4e0-609f-41dd-8d6a-ebf328b2c6b8-bundle\") pod \"28eaf4e0-609f-41dd-8d6a-ebf328b2c6b8\" (UID: \"28eaf4e0-609f-41dd-8d6a-ebf328b2c6b8\") "
Oct 30 19:24:16.954562 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:24:16.954540    4892 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/28eaf4e0-609f-41dd-8d6a-ebf328b2c6b8-util\" (UniqueName: \"kubernetes.io/empty-dir/28eaf4e0-609f-41dd-8d6a-ebf328b2c6b8-util\") pod \"28eaf4e0-609f-41dd-8d6a-ebf328b2c6b8\" (UID: \"28eaf4e0-609f-41dd-8d6a-ebf328b2c6b8\") "
Oct 30 19:24:16.955294 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:24:16.955265    4892 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/28eaf4e0-609f-41dd-8d6a-ebf328b2c6b8-bundle" pod "28eaf4e0-609f-41dd-8d6a-ebf328b2c6b8" (UID: "28eaf4e0-609f-41dd-8d6a-ebf328b2c6b8"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:24:16.956541 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:24:16.956518    4892 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/28eaf4e0-609f-41dd-8d6a-ebf328b2c6b8-kube-api-access-22ccc" pod "28eaf4e0-609f-41dd-8d6a-ebf328b2c6b8" (UID: "28eaf4e0-609f-41dd-8d6a-ebf328b2c6b8"). InnerVolumeSpecName "kube-api-access-22ccc". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:24:16.960101 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:24:16.960083    4892 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/28eaf4e0-609f-41dd-8d6a-ebf328b2c6b8-util" pod "28eaf4e0-609f-41dd-8d6a-ebf328b2c6b8" (UID: "28eaf4e0-609f-41dd-8d6a-ebf328b2c6b8"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:24:17.055670 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:24:17.055614    4892 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e5a09d53-32da-4af8-b4ba-a3c092983230-kube-api-access-tx9gc\" (UniqueName: \"kubernetes.io/projected/e5a09d53-32da-4af8-b4ba-a3c092983230-kube-api-access-tx9gc\") pod \"e5a09d53-32da-4af8-b4ba-a3c092983230\" (UID: \"e5a09d53-32da-4af8-b4ba-a3c092983230\") "
Oct 30 19:24:17.055768 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:24:17.055670    4892 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/964169ca-3813-4787-96d1-4a8eed26f023-kube-api-access-sktmd\" (UniqueName: \"kubernetes.io/projected/964169ca-3813-4787-96d1-4a8eed26f023-kube-api-access-sktmd\") pod \"964169ca-3813-4787-96d1-4a8eed26f023\" (UID: \"964169ca-3813-4787-96d1-4a8eed26f023\") "
Oct 30 19:24:17.055768 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:24:17.055699    4892 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/964169ca-3813-4787-96d1-4a8eed26f023-bundle\" (UniqueName: \"kubernetes.io/empty-dir/964169ca-3813-4787-96d1-4a8eed26f023-bundle\") pod \"964169ca-3813-4787-96d1-4a8eed26f023\" (UID: \"964169ca-3813-4787-96d1-4a8eed26f023\") "
Oct 30 19:24:17.055768 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:24:17.055729    4892 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e5a09d53-32da-4af8-b4ba-a3c092983230-util\" (UniqueName: \"kubernetes.io/empty-dir/e5a09d53-32da-4af8-b4ba-a3c092983230-util\") pod \"e5a09d53-32da-4af8-b4ba-a3c092983230\" (UID: \"e5a09d53-32da-4af8-b4ba-a3c092983230\") "
Oct 30 19:24:17.055945 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:24:17.055794    4892 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e5a09d53-32da-4af8-b4ba-a3c092983230-bundle\" (UniqueName: \"kubernetes.io/empty-dir/e5a09d53-32da-4af8-b4ba-a3c092983230-bundle\") pod \"e5a09d53-32da-4af8-b4ba-a3c092983230\" (UID: \"e5a09d53-32da-4af8-b4ba-a3c092983230\") "
Oct 30 19:24:17.055945 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:24:17.055856    4892 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/964169ca-3813-4787-96d1-4a8eed26f023-util\" (UniqueName: \"kubernetes.io/empty-dir/964169ca-3813-4787-96d1-4a8eed26f023-util\") pod \"964169ca-3813-4787-96d1-4a8eed26f023\" (UID: \"964169ca-3813-4787-96d1-4a8eed26f023\") "
Oct 30 19:24:17.056138 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:24:17.056116    4892 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/28eaf4e0-609f-41dd-8d6a-ebf328b2c6b8-bundle\") on node \"ip-10-0-132-82.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 19:24:17.056138 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:24:17.056139    4892 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/28eaf4e0-609f-41dd-8d6a-ebf328b2c6b8-util\") on node \"ip-10-0-132-82.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 19:24:17.056282 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:24:17.056152    4892 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-22ccc\" (UniqueName: \"kubernetes.io/projected/28eaf4e0-609f-41dd-8d6a-ebf328b2c6b8-kube-api-access-22ccc\") on node \"ip-10-0-132-82.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 19:24:17.056358 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:24:17.056333    4892 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/964169ca-3813-4787-96d1-4a8eed26f023-bundle" pod "964169ca-3813-4787-96d1-4a8eed26f023" (UID: "964169ca-3813-4787-96d1-4a8eed26f023"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:24:17.056471 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:24:17.056349    4892 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e5a09d53-32da-4af8-b4ba-a3c092983230-bundle" pod "e5a09d53-32da-4af8-b4ba-a3c092983230" (UID: "e5a09d53-32da-4af8-b4ba-a3c092983230"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:24:17.057904 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:24:17.057885    4892 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/964169ca-3813-4787-96d1-4a8eed26f023-kube-api-access-sktmd" pod "964169ca-3813-4787-96d1-4a8eed26f023" (UID: "964169ca-3813-4787-96d1-4a8eed26f023"). InnerVolumeSpecName "kube-api-access-sktmd". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:24:17.057904 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:24:17.057890    4892 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e5a09d53-32da-4af8-b4ba-a3c092983230-kube-api-access-tx9gc" pod "e5a09d53-32da-4af8-b4ba-a3c092983230" (UID: "e5a09d53-32da-4af8-b4ba-a3c092983230"). InnerVolumeSpecName "kube-api-access-tx9gc". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:24:17.156940 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:24:17.156915    4892 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/e5a09d53-32da-4af8-b4ba-a3c092983230-bundle\") on node \"ip-10-0-132-82.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 19:24:17.156940 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:24:17.156941    4892 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tx9gc\" (UniqueName: \"kubernetes.io/projected/e5a09d53-32da-4af8-b4ba-a3c092983230-kube-api-access-tx9gc\") on node \"ip-10-0-132-82.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 19:24:17.157149 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:24:17.156957    4892 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-sktmd\" (UniqueName: \"kubernetes.io/projected/964169ca-3813-4787-96d1-4a8eed26f023-kube-api-access-sktmd\") on node \"ip-10-0-132-82.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 19:24:17.157149 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:24:17.156994    4892 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/964169ca-3813-4787-96d1-4a8eed26f023-bundle\") on node \"ip-10-0-132-82.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 19:24:17.315602 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:24:17.315530    4892 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/964169ca-3813-4787-96d1-4a8eed26f023-util" pod "964169ca-3813-4787-96d1-4a8eed26f023" (UID: "964169ca-3813-4787-96d1-4a8eed26f023"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:24:17.315790 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:24:17.315773    4892 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e5a09d53-32da-4af8-b4ba-a3c092983230-util" pod "e5a09d53-32da-4af8-b4ba-a3c092983230" (UID: "e5a09d53-32da-4af8-b4ba-a3c092983230"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:24:17.358604 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:24:17.358580    4892 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/964169ca-3813-4787-96d1-4a8eed26f023-util\") on node \"ip-10-0-132-82.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 19:24:17.358604 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:24:17.358602    4892 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/e5a09d53-32da-4af8-b4ba-a3c092983230-util\") on node \"ip-10-0-132-82.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 19:24:17.760399 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:24:17.760369    4892 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c66hlhx"
Oct 30 19:24:17.760903 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:24:17.760393    4892 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c66hlhx" event={"ID":"e5a09d53-32da-4af8-b4ba-a3c092983230","Type":"ContainerDied","Data":"29b57f8ddd3d2759d6c33dbedf5ad2fc4b0619cf1a5532d2099bb32ddc05be53"}
Oct 30 19:24:17.760903 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:24:17.760438    4892 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="29b57f8ddd3d2759d6c33dbedf5ad2fc4b0619cf1a5532d2099bb32ddc05be53"
Oct 30 19:24:17.762369 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:24:17.762341    4892 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847a9v8hq" event={"ID":"28eaf4e0-609f-41dd-8d6a-ebf328b2c6b8","Type":"ContainerDied","Data":"ea9614f06e2f1372265eeddff71e411dcd97a88b6f7bbdc5f84a9ac82c6787ca"}
Oct 30 19:24:17.762500 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:24:17.762380    4892 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="ea9614f06e2f1372265eeddff71e411dcd97a88b6f7bbdc5f84a9ac82c6787ca"
Oct 30 19:24:17.762500 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:24:17.762350    4892 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847a9v8hq"
Oct 30 19:24:17.772315 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:24:17.772294    4892 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cw85wp"
Oct 30 19:24:17.772432 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:24:17.772316    4892 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cw85wp" event={"ID":"964169ca-3813-4787-96d1-4a8eed26f023","Type":"ContainerDied","Data":"8b7bd10e3e29efc8d6e8d4035868a80df0c92f2332aed6aca6791c8109bdbd02"}
Oct 30 19:24:17.772432 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:24:17.772388    4892 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="8b7bd10e3e29efc8d6e8d4035868a80df0c92f2332aed6aca6791c8109bdbd02"
Oct 30 19:24:17.900513 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:24:17.900492    4892 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d78czc9"
Oct 30 19:24:17.963548 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:24:17.963505    4892 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1ca4bc7d-1118-4f18-b687-fc4368ff4f22-kube-api-access-c4dk8\" (UniqueName: \"kubernetes.io/projected/1ca4bc7d-1118-4f18-b687-fc4368ff4f22-kube-api-access-c4dk8\") pod \"1ca4bc7d-1118-4f18-b687-fc4368ff4f22\" (UID: \"1ca4bc7d-1118-4f18-b687-fc4368ff4f22\") "
Oct 30 19:24:17.963676 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:24:17.963573    4892 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1ca4bc7d-1118-4f18-b687-fc4368ff4f22-util\" (UniqueName: \"kubernetes.io/empty-dir/1ca4bc7d-1118-4f18-b687-fc4368ff4f22-util\") pod \"1ca4bc7d-1118-4f18-b687-fc4368ff4f22\" (UID: \"1ca4bc7d-1118-4f18-b687-fc4368ff4f22\") "
Oct 30 19:24:17.963739 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:24:17.963676    4892 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1ca4bc7d-1118-4f18-b687-fc4368ff4f22-bundle\" (UniqueName: \"kubernetes.io/empty-dir/1ca4bc7d-1118-4f18-b687-fc4368ff4f22-bundle\") pod \"1ca4bc7d-1118-4f18-b687-fc4368ff4f22\" (UID: \"1ca4bc7d-1118-4f18-b687-fc4368ff4f22\") "
Oct 30 19:24:17.964132 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:24:17.964104    4892 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1ca4bc7d-1118-4f18-b687-fc4368ff4f22-bundle" pod "1ca4bc7d-1118-4f18-b687-fc4368ff4f22" (UID: "1ca4bc7d-1118-4f18-b687-fc4368ff4f22"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:24:17.965728 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:24:17.965706    4892 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1ca4bc7d-1118-4f18-b687-fc4368ff4f22-kube-api-access-c4dk8" pod "1ca4bc7d-1118-4f18-b687-fc4368ff4f22" (UID: "1ca4bc7d-1118-4f18-b687-fc4368ff4f22"). InnerVolumeSpecName "kube-api-access-c4dk8". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:24:17.968163 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:24:17.968141    4892 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1ca4bc7d-1118-4f18-b687-fc4368ff4f22-util" pod "1ca4bc7d-1118-4f18-b687-fc4368ff4f22" (UID: "1ca4bc7d-1118-4f18-b687-fc4368ff4f22"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:24:18.065285 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:24:18.065226    4892 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/1ca4bc7d-1118-4f18-b687-fc4368ff4f22-bundle\") on node \"ip-10-0-132-82.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 19:24:18.065285 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:24:18.065257    4892 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-c4dk8\" (UniqueName: \"kubernetes.io/projected/1ca4bc7d-1118-4f18-b687-fc4368ff4f22-kube-api-access-c4dk8\") on node \"ip-10-0-132-82.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 19:24:18.065285 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:24:18.065267    4892 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/1ca4bc7d-1118-4f18-b687-fc4368ff4f22-util\") on node \"ip-10-0-132-82.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 19:24:18.779190 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:24:18.779162    4892 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d78czc9"
Oct 30 19:24:18.779190 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:24:18.779181    4892 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d78czc9" event={"ID":"1ca4bc7d-1118-4f18-b687-fc4368ff4f22","Type":"ContainerDied","Data":"991f077968226fa35c34bbbba27a1f56630ab450e0f9f289747cf16454da4c6b"}
Oct 30 19:24:18.779625 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:24:18.779229    4892 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="991f077968226fa35c34bbbba27a1f56630ab450e0f9f289747cf16454da4c6b"
Oct 30 19:24:24.842146 ip-10-0-132-82 kubenswrapper[4892]: E1030 19:24:24.842104    4892 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-jrm87" containerName="extractor"
Oct 30 19:24:29.019222 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:24:29.019188    4892 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-6b6fb56846-krq52"]
Oct 30 19:24:29.019591 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:24:29.019491    4892 cpu_manager.go:414] "containerMap: removing container" podUID="964169ca-3813-4787-96d1-4a8eed26f023" containerName="util"
Oct 30 19:24:29.019591 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:24:29.019501    4892 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="964169ca-3813-4787-96d1-4a8eed26f023" containerName="util"
Oct 30 19:24:29.019591 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:24:29.019515    4892 cpu_manager.go:414] "containerMap: removing container" podUID="28eaf4e0-609f-41dd-8d6a-ebf328b2c6b8" containerName="extract"
Oct 30 19:24:29.019591 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:24:29.019521    4892 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="28eaf4e0-609f-41dd-8d6a-ebf328b2c6b8" containerName="extract"
Oct 30 19:24:29.019591 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:24:29.019530    4892 cpu_manager.go:414] "containerMap: removing container" podUID="e5a09d53-32da-4af8-b4ba-a3c092983230" containerName="extract"
Oct 30 19:24:29.019591 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:24:29.019535    4892 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e5a09d53-32da-4af8-b4ba-a3c092983230" containerName="extract"
Oct 30 19:24:29.019591 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:24:29.019547    4892 cpu_manager.go:414] "containerMap: removing container" podUID="964169ca-3813-4787-96d1-4a8eed26f023" containerName="extract"
Oct 30 19:24:29.019591 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:24:29.019551    4892 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="964169ca-3813-4787-96d1-4a8eed26f023" containerName="extract"
Oct 30 19:24:29.019591 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:24:29.019560    4892 cpu_manager.go:414] "containerMap: removing container" podUID="28eaf4e0-609f-41dd-8d6a-ebf328b2c6b8" containerName="util"
Oct 30 19:24:29.019591 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:24:29.019564    4892 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="28eaf4e0-609f-41dd-8d6a-ebf328b2c6b8" containerName="util"
Oct 30 19:24:29.019591 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:24:29.019571    4892 cpu_manager.go:414] "containerMap: removing container" podUID="1ca4bc7d-1118-4f18-b687-fc4368ff4f22" containerName="util"
Oct 30 19:24:29.019591 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:24:29.019576    4892 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1ca4bc7d-1118-4f18-b687-fc4368ff4f22" containerName="util"
Oct 30 19:24:29.019591 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:24:29.019581    4892 cpu_manager.go:414] "containerMap: removing container" podUID="1ca4bc7d-1118-4f18-b687-fc4368ff4f22" containerName="extract"
Oct 30 19:24:29.019591 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:24:29.019585    4892 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1ca4bc7d-1118-4f18-b687-fc4368ff4f22" containerName="extract"
Oct 30 19:24:29.019591 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:24:29.019595    4892 cpu_manager.go:414] "containerMap: removing container" podUID="28eaf4e0-609f-41dd-8d6a-ebf328b2c6b8" containerName="pull"
Oct 30 19:24:29.020070 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:24:29.019601    4892 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="28eaf4e0-609f-41dd-8d6a-ebf328b2c6b8" containerName="pull"
Oct 30 19:24:29.020070 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:24:29.019607    4892 cpu_manager.go:414] "containerMap: removing container" podUID="1ca4bc7d-1118-4f18-b687-fc4368ff4f22" containerName="pull"
Oct 30 19:24:29.020070 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:24:29.019611    4892 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1ca4bc7d-1118-4f18-b687-fc4368ff4f22" containerName="pull"
Oct 30 19:24:29.020070 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:24:29.019620    4892 cpu_manager.go:414] "containerMap: removing container" podUID="e5a09d53-32da-4af8-b4ba-a3c092983230" containerName="pull"
Oct 30 19:24:29.020070 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:24:29.019624    4892 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e5a09d53-32da-4af8-b4ba-a3c092983230" containerName="pull"
Oct 30 19:24:29.020070 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:24:29.019632    4892 cpu_manager.go:414] "containerMap: removing container" podUID="964169ca-3813-4787-96d1-4a8eed26f023" containerName="pull"
Oct 30 19:24:29.020070 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:24:29.019636    4892 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="964169ca-3813-4787-96d1-4a8eed26f023" containerName="pull"
Oct 30 19:24:29.020070 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:24:29.019644    4892 cpu_manager.go:414] "containerMap: removing container" podUID="e5a09d53-32da-4af8-b4ba-a3c092983230" containerName="util"
Oct 30 19:24:29.020070 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:24:29.019648    4892 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e5a09d53-32da-4af8-b4ba-a3c092983230" containerName="util"
Oct 30 19:24:29.020070 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:24:29.019697    4892 memory_manager.go:365] "RemoveStaleState removing state" podUID="28eaf4e0-609f-41dd-8d6a-ebf328b2c6b8" containerName="extract"
Oct 30 19:24:29.020070 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:24:29.019703    4892 memory_manager.go:365] "RemoveStaleState removing state" podUID="e5a09d53-32da-4af8-b4ba-a3c092983230" containerName="extract"
Oct 30 19:24:29.020070 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:24:29.019708    4892 memory_manager.go:365] "RemoveStaleState removing state" podUID="964169ca-3813-4787-96d1-4a8eed26f023" containerName="extract"
Oct 30 19:24:29.020070 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:24:29.019714    4892 memory_manager.go:365] "RemoveStaleState removing state" podUID="1ca4bc7d-1118-4f18-b687-fc4368ff4f22" containerName="extract"
Oct 30 19:24:29.023176 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:24:29.023154    4892 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-6b6fb56846-krq52"
Oct 30 19:24:29.077412 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:24:29.077380    4892 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-6b6fb56846-krq52"]
Oct 30 19:24:29.077871 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:24:29.077849    4892 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"openshift-service-ca.crt\""
Oct 30 19:24:29.078043 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:24:29.078023    4892 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"kube-root-ca.crt\""
Oct 30 19:24:29.124250 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:24:29.124222    4892 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"kuadrant-operator-controller-manager-dockercfg-82bmx\""
Oct 30 19:24:29.150127 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:24:29.150101    4892 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/6392742e-b112-4b31-8109-0c64cd373363-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-6b6fb56846-krq52\" (UID: \"6392742e-b112-4b31-8109-0c64cd373363\") " pod="kuadrant-system/kuadrant-operator-controller-manager-6b6fb56846-krq52"
Oct 30 19:24:29.150229 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:24:29.150154    4892 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hz5d4\" (UniqueName: \"kubernetes.io/projected/6392742e-b112-4b31-8109-0c64cd373363-kube-api-access-hz5d4\") pod \"kuadrant-operator-controller-manager-6b6fb56846-krq52\" (UID: \"6392742e-b112-4b31-8109-0c64cd373363\") " pod="kuadrant-system/kuadrant-operator-controller-manager-6b6fb56846-krq52"
Oct 30 19:24:29.250563 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:24:29.250530    4892 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hz5d4\" (UniqueName: \"kubernetes.io/projected/6392742e-b112-4b31-8109-0c64cd373363-kube-api-access-hz5d4\") pod \"kuadrant-operator-controller-manager-6b6fb56846-krq52\" (UID: \"6392742e-b112-4b31-8109-0c64cd373363\") " pod="kuadrant-system/kuadrant-operator-controller-manager-6b6fb56846-krq52"
Oct 30 19:24:29.250674 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:24:29.250593    4892 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/6392742e-b112-4b31-8109-0c64cd373363-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-6b6fb56846-krq52\" (UID: \"6392742e-b112-4b31-8109-0c64cd373363\") " pod="kuadrant-system/kuadrant-operator-controller-manager-6b6fb56846-krq52"
Oct 30 19:24:29.250940 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:24:29.250924    4892 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/6392742e-b112-4b31-8109-0c64cd373363-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-6b6fb56846-krq52\" (UID: \"6392742e-b112-4b31-8109-0c64cd373363\") " pod="kuadrant-system/kuadrant-operator-controller-manager-6b6fb56846-krq52"
Oct 30 19:24:29.310814 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:24:29.310767    4892 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hz5d4\" (UniqueName: \"kubernetes.io/projected/6392742e-b112-4b31-8109-0c64cd373363-kube-api-access-hz5d4\") pod \"kuadrant-operator-controller-manager-6b6fb56846-krq52\" (UID: \"6392742e-b112-4b31-8109-0c64cd373363\") " pod="kuadrant-system/kuadrant-operator-controller-manager-6b6fb56846-krq52"
Oct 30 19:24:29.336692 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:24:29.336669    4892 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-6b6fb56846-krq52"
Oct 30 19:24:29.818609 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:24:29.818585    4892 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-6b6fb56846-krq52"]
Oct 30 19:24:29.821881 ip-10-0-132-82 kubenswrapper[4892]: W1030 19:24:29.821853    4892 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6392742e_b112_4b31_8109_0c64cd373363.slice/crio-b7f6ae611a90774a9623e0445f1e6af0e13041bdfefb7f56a4ac4765fb55b5d9 WatchSource:0}: Error finding container b7f6ae611a90774a9623e0445f1e6af0e13041bdfefb7f56a4ac4765fb55b5d9: Status 404 returned error can't find the container with id b7f6ae611a90774a9623e0445f1e6af0e13041bdfefb7f56a4ac4765fb55b5d9
Oct 30 19:24:29.829672 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:24:29.829642    4892 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-6b6fb56846-krq52" event={"ID":"6392742e-b112-4b31-8109-0c64cd373363","Type":"ContainerStarted","Data":"b7f6ae611a90774a9623e0445f1e6af0e13041bdfefb7f56a4ac4765fb55b5d9"}
Oct 30 19:24:37.871259 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:24:37.871224    4892 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-6b6fb56846-krq52" event={"ID":"6392742e-b112-4b31-8109-0c64cd373363","Type":"ContainerStarted","Data":"68f17e9fe4afe6a5af9ae2163e37bf7b657388b23961740237449daa464017c1"}
Oct 30 19:24:37.871772 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:24:37.871367    4892 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/kuadrant-operator-controller-manager-6b6fb56846-krq52"
Oct 30 19:24:38.091121 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:24:38.091049    4892 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kuadrant-system/kuadrant-operator-controller-manager-6b6fb56846-krq52" podStartSLOduration=2.744090635 podStartE2EDuration="10.0910327s" totalImagesPullingTime="7.346942065s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:24:28 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:24:38.088976232 +0000 UTC m=+525.659441823" watchObservedRunningTime="2026-10-30 19:24:38.0910327 +0000 UTC m=+525.661498294"
Oct 30 19:24:48.877767 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:24:48.877738    4892 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/kuadrant-operator-controller-manager-6b6fb56846-krq52"
Oct 30 19:24:54.843811 ip-10-0-132-82 kubenswrapper[4892]: E1030 19:24:54.843775    4892 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-jrm87" containerName="extractor"
Oct 30 19:25:00.502371 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:25:00.502334    4892 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-6b76784d48-bcqxc"]
Oct 30 19:25:00.506741 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:25:00.506719    4892 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-6b76784d48-bcqxc"
Oct 30 19:25:00.557818 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:25:00.557789    4892 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-6b76784d48-bcqxc"]
Oct 30 19:25:00.603060 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:25:00.603033    4892 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/2c62e5e6-b713-4410-997b-99491b2135d4-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-6b76784d48-bcqxc\" (UID: \"2c62e5e6-b713-4410-997b-99491b2135d4\") " pod="kuadrant-system/kuadrant-operator-controller-manager-6b76784d48-bcqxc"
Oct 30 19:25:00.603175 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:25:00.603091    4892 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-d5sml\" (UniqueName: \"kubernetes.io/projected/2c62e5e6-b713-4410-997b-99491b2135d4-kube-api-access-d5sml\") pod \"kuadrant-operator-controller-manager-6b76784d48-bcqxc\" (UID: \"2c62e5e6-b713-4410-997b-99491b2135d4\") " pod="kuadrant-system/kuadrant-operator-controller-manager-6b76784d48-bcqxc"
Oct 30 19:25:00.703662 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:25:00.703637    4892 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/2c62e5e6-b713-4410-997b-99491b2135d4-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-6b76784d48-bcqxc\" (UID: \"2c62e5e6-b713-4410-997b-99491b2135d4\") " pod="kuadrant-system/kuadrant-operator-controller-manager-6b76784d48-bcqxc"
Oct 30 19:25:00.703813 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:25:00.703792    4892 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-d5sml\" (UniqueName: \"kubernetes.io/projected/2c62e5e6-b713-4410-997b-99491b2135d4-kube-api-access-d5sml\") pod \"kuadrant-operator-controller-manager-6b76784d48-bcqxc\" (UID: \"2c62e5e6-b713-4410-997b-99491b2135d4\") " pod="kuadrant-system/kuadrant-operator-controller-manager-6b76784d48-bcqxc"
Oct 30 19:25:00.704081 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:25:00.704065    4892 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/2c62e5e6-b713-4410-997b-99491b2135d4-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-6b76784d48-bcqxc\" (UID: \"2c62e5e6-b713-4410-997b-99491b2135d4\") " pod="kuadrant-system/kuadrant-operator-controller-manager-6b76784d48-bcqxc"
Oct 30 19:25:00.762280 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:25:00.762223    4892 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-d5sml\" (UniqueName: \"kubernetes.io/projected/2c62e5e6-b713-4410-997b-99491b2135d4-kube-api-access-d5sml\") pod \"kuadrant-operator-controller-manager-6b76784d48-bcqxc\" (UID: \"2c62e5e6-b713-4410-997b-99491b2135d4\") " pod="kuadrant-system/kuadrant-operator-controller-manager-6b76784d48-bcqxc"
Oct 30 19:25:00.818022 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:25:00.818004    4892 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-6b76784d48-bcqxc"
Oct 30 19:25:01.307808 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:25:01.307775    4892 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-6b76784d48-bcqxc"]
Oct 30 19:25:01.343737 ip-10-0-132-82 kubenswrapper[4892]: W1030 19:25:01.343706    4892 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2c62e5e6_b713_4410_997b_99491b2135d4.slice/crio-6db1b0763ca1a8db2c456a7c4c00c332abc271b4692ae4cfd7e567754434f96b WatchSource:0}: Error finding container 6db1b0763ca1a8db2c456a7c4c00c332abc271b4692ae4cfd7e567754434f96b: Status 404 returned error can't find the container with id 6db1b0763ca1a8db2c456a7c4c00c332abc271b4692ae4cfd7e567754434f96b
Oct 30 19:25:01.982896 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:25:01.982857    4892 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-6b76784d48-bcqxc" event={"ID":"2c62e5e6-b713-4410-997b-99491b2135d4","Type":"ContainerStarted","Data":"20595a1aac4116f47c992b6c127fff1b91b58a90f6c9fa611b06a2a6cd0613f1"}
Oct 30 19:25:01.983384 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:25:01.982915    4892 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-6b76784d48-bcqxc" event={"ID":"2c62e5e6-b713-4410-997b-99491b2135d4","Type":"ContainerStarted","Data":"6db1b0763ca1a8db2c456a7c4c00c332abc271b4692ae4cfd7e567754434f96b"}
Oct 30 19:25:01.983384 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:25:01.983005    4892 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/kuadrant-operator-controller-manager-6b76784d48-bcqxc"
Oct 30 19:25:02.158026 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:25:02.157948    4892 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kuadrant-system/kuadrant-operator-controller-manager-6b76784d48-bcqxc" podStartSLOduration=2.157927974 podStartE2EDuration="2.157927974s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:25:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:25:02.156405939 +0000 UTC m=+549.726871538" watchObservedRunningTime="2026-10-30 19:25:02.157927974 +0000 UTC m=+549.728393577"
Oct 30 19:25:02.812005 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:25:02.811959    4892 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-6b6fb56846-krq52"]
Oct 30 19:25:02.812184 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:25:02.812160    4892 kuberuntime_container.go:898] "Killing container with a grace period" pod="kuadrant-system/kuadrant-operator-controller-manager-6b6fb56846-krq52" podUID="6392742e-b112-4b31-8109-0c64cd373363" containerName="manager" containerID="cri-o://68f17e9fe4afe6a5af9ae2163e37bf7b657388b23961740237449daa464017c1" gracePeriod=2
Oct 30 19:25:02.862274 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:25:02.862246    4892 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-6b6fb56846-krq52"]
Oct 30 19:25:02.864427 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:25:02.864403    4892 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-6b6fb56846-jx7ft"]
Oct 30 19:25:02.864800 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:25:02.864784    4892 cpu_manager.go:414] "containerMap: removing container" podUID="6392742e-b112-4b31-8109-0c64cd373363" containerName="manager"
Oct 30 19:25:02.864865 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:25:02.864804    4892 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6392742e-b112-4b31-8109-0c64cd373363" containerName="manager"
Oct 30 19:25:02.864929 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:25:02.864918    4892 memory_manager.go:365] "RemoveStaleState removing state" podUID="6392742e-b112-4b31-8109-0c64cd373363" containerName="manager"
Oct 30 19:25:02.866996 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:25:02.866979    4892 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-6b6fb56846-jx7ft"
Oct 30 19:25:02.908922 ip-10-0-132-82 kubenswrapper[4892]: E1030 19:25:02.908889    4892 status_manager.go:1045] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-6b6fb56846-krq52\" is forbidden: User \"system:node:ip-10-0-132-82.us-west-1.compute.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-132-82.us-west-1.compute.internal' and this object" podUID="6392742e-b112-4b31-8109-0c64cd373363" pod="kuadrant-system/kuadrant-operator-controller-manager-6b6fb56846-krq52"
Oct 30 19:25:02.911062 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:25:02.911037    4892 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-6b76784d48-bcqxc"]
Oct 30 19:25:02.914047 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:25:02.914022    4892 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-6b76784d48-bcqxc"]
Oct 30 19:25:02.914684 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:25:02.914661    4892 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-6b76784d48-dzb5z"]
Oct 30 19:25:02.915117 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:25:02.915098    4892 cpu_manager.go:414] "containerMap: removing container" podUID="2c62e5e6-b713-4410-997b-99491b2135d4" containerName="manager"
Oct 30 19:25:02.915117 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:25:02.915115    4892 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2c62e5e6-b713-4410-997b-99491b2135d4" containerName="manager"
Oct 30 19:25:02.915317 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:25:02.915207    4892 memory_manager.go:365] "RemoveStaleState removing state" podUID="2c62e5e6-b713-4410-997b-99491b2135d4" containerName="manager"
Oct 30 19:25:02.917196 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:25:02.917176    4892 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-6b76784d48-dzb5z"
Oct 30 19:25:02.961454 ip-10-0-132-82 kubenswrapper[4892]: E1030 19:25:02.961398    4892 status_manager.go:1045] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-6b6fb56846-krq52\" is forbidden: User \"system:node:ip-10-0-132-82.us-west-1.compute.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-132-82.us-west-1.compute.internal' and this object" podUID="6392742e-b112-4b31-8109-0c64cd373363" pod="kuadrant-system/kuadrant-operator-controller-manager-6b6fb56846-krq52"
Oct 30 19:25:02.986413 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:25:02.986381    4892 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-6b6fb56846-jx7ft"]
Oct 30 19:25:02.990418 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:25:02.990333    4892 generic.go:356] "Generic (PLEG): container finished" podID="6392742e-b112-4b31-8109-0c64cd373363" containerID="68f17e9fe4afe6a5af9ae2163e37bf7b657388b23961740237449daa464017c1" exitCode=0
Oct 30 19:25:03.024026 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:25:03.023996    4892 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/b0a7ae76-0cab-4081-9e2d-0fbc0332a8c2-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-6b6fb56846-jx7ft\" (UID: \"b0a7ae76-0cab-4081-9e2d-0fbc0332a8c2\") " pod="kuadrant-system/kuadrant-operator-controller-manager-6b6fb56846-jx7ft"
Oct 30 19:25:03.024166 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:25:03.024050    4892 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/4f6ef7c9-be20-498f-861d-26247190eed8-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-6b76784d48-dzb5z\" (UID: \"4f6ef7c9-be20-498f-861d-26247190eed8\") " pod="kuadrant-system/kuadrant-operator-controller-manager-6b76784d48-dzb5z"
Oct 30 19:25:03.024166 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:25:03.024128    4892 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-h7bnf\" (UniqueName: \"kubernetes.io/projected/4f6ef7c9-be20-498f-861d-26247190eed8-kube-api-access-h7bnf\") pod \"kuadrant-operator-controller-manager-6b76784d48-dzb5z\" (UID: \"4f6ef7c9-be20-498f-861d-26247190eed8\") " pod="kuadrant-system/kuadrant-operator-controller-manager-6b76784d48-dzb5z"
Oct 30 19:25:03.024246 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:25:03.024179    4892 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bfgps\" (UniqueName: \"kubernetes.io/projected/b0a7ae76-0cab-4081-9e2d-0fbc0332a8c2-kube-api-access-bfgps\") pod \"kuadrant-operator-controller-manager-6b6fb56846-jx7ft\" (UID: \"b0a7ae76-0cab-4081-9e2d-0fbc0332a8c2\") " pod="kuadrant-system/kuadrant-operator-controller-manager-6b6fb56846-jx7ft"
Oct 30 19:25:03.039883 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:25:03.039838    4892 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-6b76784d48-dzb5z"]
Oct 30 19:25:03.096998 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:25:03.096974    4892 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-6b6fb56846-krq52"
Oct 30 19:25:03.125343 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:25:03.125318    4892 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/b0a7ae76-0cab-4081-9e2d-0fbc0332a8c2-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-6b6fb56846-jx7ft\" (UID: \"b0a7ae76-0cab-4081-9e2d-0fbc0332a8c2\") " pod="kuadrant-system/kuadrant-operator-controller-manager-6b6fb56846-jx7ft"
Oct 30 19:25:03.125439 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:25:03.125365    4892 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/4f6ef7c9-be20-498f-861d-26247190eed8-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-6b76784d48-dzb5z\" (UID: \"4f6ef7c9-be20-498f-861d-26247190eed8\") " pod="kuadrant-system/kuadrant-operator-controller-manager-6b76784d48-dzb5z"
Oct 30 19:25:03.125439 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:25:03.125402    4892 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-h7bnf\" (UniqueName: \"kubernetes.io/projected/4f6ef7c9-be20-498f-861d-26247190eed8-kube-api-access-h7bnf\") pod \"kuadrant-operator-controller-manager-6b76784d48-dzb5z\" (UID: \"4f6ef7c9-be20-498f-861d-26247190eed8\") " pod="kuadrant-system/kuadrant-operator-controller-manager-6b76784d48-dzb5z"
Oct 30 19:25:03.125439 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:25:03.125434    4892 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bfgps\" (UniqueName: \"kubernetes.io/projected/b0a7ae76-0cab-4081-9e2d-0fbc0332a8c2-kube-api-access-bfgps\") pod \"kuadrant-operator-controller-manager-6b6fb56846-jx7ft\" (UID: \"b0a7ae76-0cab-4081-9e2d-0fbc0332a8c2\") " pod="kuadrant-system/kuadrant-operator-controller-manager-6b6fb56846-jx7ft"
Oct 30 19:25:03.125748 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:25:03.125728    4892 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/b0a7ae76-0cab-4081-9e2d-0fbc0332a8c2-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-6b6fb56846-jx7ft\" (UID: \"b0a7ae76-0cab-4081-9e2d-0fbc0332a8c2\") " pod="kuadrant-system/kuadrant-operator-controller-manager-6b6fb56846-jx7ft"
Oct 30 19:25:03.125748 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:25:03.125742    4892 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/4f6ef7c9-be20-498f-861d-26247190eed8-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-6b76784d48-dzb5z\" (UID: \"4f6ef7c9-be20-498f-861d-26247190eed8\") " pod="kuadrant-system/kuadrant-operator-controller-manager-6b76784d48-dzb5z"
Oct 30 19:25:03.133193 ip-10-0-132-82 kubenswrapper[4892]: E1030 19:25:03.133167    4892 status_manager.go:1045] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-6b6fb56846-krq52\" is forbidden: User \"system:node:ip-10-0-132-82.us-west-1.compute.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-132-82.us-west-1.compute.internal' and this object" podUID="6392742e-b112-4b31-8109-0c64cd373363" pod="kuadrant-system/kuadrant-operator-controller-manager-6b6fb56846-krq52"
Oct 30 19:25:03.195301 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:25:03.195275    4892 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bfgps\" (UniqueName: \"kubernetes.io/projected/b0a7ae76-0cab-4081-9e2d-0fbc0332a8c2-kube-api-access-bfgps\") pod \"kuadrant-operator-controller-manager-6b6fb56846-jx7ft\" (UID: \"b0a7ae76-0cab-4081-9e2d-0fbc0332a8c2\") " pod="kuadrant-system/kuadrant-operator-controller-manager-6b6fb56846-jx7ft"
Oct 30 19:25:03.195385 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:25:03.195364    4892 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-h7bnf\" (UniqueName: \"kubernetes.io/projected/4f6ef7c9-be20-498f-861d-26247190eed8-kube-api-access-h7bnf\") pod \"kuadrant-operator-controller-manager-6b76784d48-dzb5z\" (UID: \"4f6ef7c9-be20-498f-861d-26247190eed8\") " pod="kuadrant-system/kuadrant-operator-controller-manager-6b76784d48-dzb5z"
Oct 30 19:25:03.199108 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:25:03.199089    4892 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-6b6fb56846-jx7ft"
Oct 30 19:25:03.225988 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:25:03.225945    4892 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6392742e-b112-4b31-8109-0c64cd373363-kube-api-access-hz5d4\" (UniqueName: \"kubernetes.io/projected/6392742e-b112-4b31-8109-0c64cd373363-kube-api-access-hz5d4\") pod \"6392742e-b112-4b31-8109-0c64cd373363\" (UID: \"6392742e-b112-4b31-8109-0c64cd373363\") "
Oct 30 19:25:03.226102 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:25:03.226048    4892 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6392742e-b112-4b31-8109-0c64cd373363-extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/6392742e-b112-4b31-8109-0c64cd373363-extensions-socket-volume\") pod \"6392742e-b112-4b31-8109-0c64cd373363\" (UID: \"6392742e-b112-4b31-8109-0c64cd373363\") "
Oct 30 19:25:03.226418 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:25:03.226391    4892 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6392742e-b112-4b31-8109-0c64cd373363-extensions-socket-volume" pod "6392742e-b112-4b31-8109-0c64cd373363" (UID: "6392742e-b112-4b31-8109-0c64cd373363"). InnerVolumeSpecName "extensions-socket-volume". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:25:03.228045 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:25:03.228021    4892 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6392742e-b112-4b31-8109-0c64cd373363-kube-api-access-hz5d4" pod "6392742e-b112-4b31-8109-0c64cd373363" (UID: "6392742e-b112-4b31-8109-0c64cd373363"). InnerVolumeSpecName "kube-api-access-hz5d4". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:25:03.231207 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:25:03.231189    4892 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-6b76784d48-dzb5z"
Oct 30 19:25:03.327221 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:25:03.327197    4892 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hz5d4\" (UniqueName: \"kubernetes.io/projected/6392742e-b112-4b31-8109-0c64cd373363-kube-api-access-hz5d4\") on node \"ip-10-0-132-82.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 19:25:03.327221 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:25:03.327222    4892 reconciler_common.go:299] "Volume detached for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/6392742e-b112-4b31-8109-0c64cd373363-extensions-socket-volume\") on node \"ip-10-0-132-82.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 19:25:03.328009 ip-10-0-132-82 kubenswrapper[4892]: E1030 19:25:03.327932    4892 status_manager.go:1045] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-6b6fb56846-krq52\" is forbidden: User \"system:node:ip-10-0-132-82.us-west-1.compute.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-132-82.us-west-1.compute.internal' and this object" podUID="6392742e-b112-4b31-8109-0c64cd373363" pod="kuadrant-system/kuadrant-operator-controller-manager-6b6fb56846-krq52"
Oct 30 19:25:03.637437 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:25:03.637413    4892 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-6b6fb56846-jx7ft"]
Oct 30 19:25:03.639613 ip-10-0-132-82 kubenswrapper[4892]: W1030 19:25:03.639587    4892 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb0a7ae76_0cab_4081_9e2d_0fbc0332a8c2.slice/crio-24d122d8e08d8a3940ca770839b2dddccedede21207683aaf116538ced9cdf74 WatchSource:0}: Error finding container 24d122d8e08d8a3940ca770839b2dddccedede21207683aaf116538ced9cdf74: Status 404 returned error can't find the container with id 24d122d8e08d8a3940ca770839b2dddccedede21207683aaf116538ced9cdf74
Oct 30 19:25:03.664928 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:25:03.664906    4892 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-6b76784d48-dzb5z"]
Oct 30 19:25:03.671037 ip-10-0-132-82 kubenswrapper[4892]: W1030 19:25:03.671010    4892 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4f6ef7c9_be20_498f_861d_26247190eed8.slice/crio-b327795191482f9d0892930943880fec9dbd1fa4dd3c2bef40d9a447ead924ed WatchSource:0}: Error finding container b327795191482f9d0892930943880fec9dbd1fa4dd3c2bef40d9a447ead924ed: Status 404 returned error can't find the container with id b327795191482f9d0892930943880fec9dbd1fa4dd3c2bef40d9a447ead924ed
Oct 30 19:25:03.997593 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:25:03.997402    4892 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-6b76784d48-dzb5z" event={"ID":"4f6ef7c9-be20-498f-861d-26247190eed8","Type":"ContainerStarted","Data":"5e4cd12f695f940d8882a0aa1544523e1b507fdba375a926fb9cdfd22d7c564b"}
Oct 30 19:25:03.997593 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:25:03.997458    4892 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-6b76784d48-dzb5z" event={"ID":"4f6ef7c9-be20-498f-861d-26247190eed8","Type":"ContainerStarted","Data":"b327795191482f9d0892930943880fec9dbd1fa4dd3c2bef40d9a447ead924ed"}
Oct 30 19:25:03.997593 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:25:03.997484    4892 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/kuadrant-operator-controller-manager-6b76784d48-dzb5z"
Oct 30 19:25:03.999281 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:25:03.999250    4892 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-6b6fb56846-jx7ft" event={"ID":"b0a7ae76-0cab-4081-9e2d-0fbc0332a8c2","Type":"ContainerStarted","Data":"34f5a35289a2723abde5bd02fe9ce9c64be2bbc60ac9e453f6b40478dbab69a8"}
Oct 30 19:25:03.999420 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:25:03.999306    4892 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-6b6fb56846-jx7ft" event={"ID":"b0a7ae76-0cab-4081-9e2d-0fbc0332a8c2","Type":"ContainerStarted","Data":"24d122d8e08d8a3940ca770839b2dddccedede21207683aaf116538ced9cdf74"}
Oct 30 19:25:03.999501 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:25:03.999485    4892 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/kuadrant-operator-controller-manager-6b6fb56846-jx7ft"
Oct 30 19:25:04.000713 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:25:04.000696    4892 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-6b6fb56846-krq52"
Oct 30 19:25:04.000805 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:25:04.000737    4892 scope.go:122] "RemoveContainer" containerID="68f17e9fe4afe6a5af9ae2163e37bf7b657388b23961740237449daa464017c1"
Oct 30 19:25:04.000805 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:25:04.000775    4892 kuberuntime_container.go:898] "Killing container with a grace period" pod="kuadrant-system/kuadrant-operator-controller-manager-6b76784d48-bcqxc" podUID="2c62e5e6-b713-4410-997b-99491b2135d4" containerName="manager" containerID="cri-o://20595a1aac4116f47c992b6c127fff1b91b58a90f6c9fa611b06a2a6cd0613f1" gracePeriod=2
Oct 30 19:25:04.049663 ip-10-0-132-82 kubenswrapper[4892]: E1030 19:25:04.049637    4892 status_manager.go:1045] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-6b6fb56846-krq52\" is forbidden: User \"system:node:ip-10-0-132-82.us-west-1.compute.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-132-82.us-west-1.compute.internal' and this object" podUID="6392742e-b112-4b31-8109-0c64cd373363" pod="kuadrant-system/kuadrant-operator-controller-manager-6b6fb56846-krq52"
Oct 30 19:25:04.178897 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:25:04.178843    4892 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kuadrant-system/kuadrant-operator-controller-manager-6b76784d48-dzb5z" podStartSLOduration=2.178830177 podStartE2EDuration="2.178830177s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:25:02 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:25:04.176357525 +0000 UTC m=+551.746823124" watchObservedRunningTime="2026-10-30 19:25:04.178830177 +0000 UTC m=+551.749295776"
Oct 30 19:25:04.226953 ip-10-0-132-82 kubenswrapper[4892]: E1030 19:25:04.226911    4892 status_manager.go:1045] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-6b6fb56846-krq52\" is forbidden: User \"system:node:ip-10-0-132-82.us-west-1.compute.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-132-82.us-west-1.compute.internal' and this object" podUID="6392742e-b112-4b31-8109-0c64cd373363" pod="kuadrant-system/kuadrant-operator-controller-manager-6b6fb56846-krq52"
Oct 30 19:25:04.277797 ip-10-0-132-82 kubenswrapper[4892]: E1030 19:25:04.277720    4892 status_manager.go:1045] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-6b76784d48-bcqxc\" is forbidden: User \"system:node:ip-10-0-132-82.us-west-1.compute.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-132-82.us-west-1.compute.internal' and this object" podUID="2c62e5e6-b713-4410-997b-99491b2135d4" pod="kuadrant-system/kuadrant-operator-controller-manager-6b76784d48-bcqxc"
Oct 30 19:25:04.319875 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:25:04.319845    4892 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-6b76784d48-bcqxc"
Oct 30 19:25:04.335897 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:25:04.335870    4892 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2c62e5e6-b713-4410-997b-99491b2135d4-extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/2c62e5e6-b713-4410-997b-99491b2135d4-extensions-socket-volume\") pod \"2c62e5e6-b713-4410-997b-99491b2135d4\" (UID: \"2c62e5e6-b713-4410-997b-99491b2135d4\") "
Oct 30 19:25:04.339991 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:25:04.336537    4892 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2c62e5e6-b713-4410-997b-99491b2135d4-extensions-socket-volume" pod "2c62e5e6-b713-4410-997b-99491b2135d4" (UID: "2c62e5e6-b713-4410-997b-99491b2135d4"). InnerVolumeSpecName "extensions-socket-volume". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:25:04.402620 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:25:04.402577    4892 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kuadrant-system/kuadrant-operator-controller-manager-6b6fb56846-jx7ft" podStartSLOduration=2.402565647 podStartE2EDuration="2.402565647s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:25:02 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:25:04.400775063 +0000 UTC m=+551.971240661" watchObservedRunningTime="2026-10-30 19:25:04.402565647 +0000 UTC m=+551.973031236"
Oct 30 19:25:04.437015 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:25:04.436989    4892 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2c62e5e6-b713-4410-997b-99491b2135d4-kube-api-access-d5sml\" (UniqueName: \"kubernetes.io/projected/2c62e5e6-b713-4410-997b-99491b2135d4-kube-api-access-d5sml\") pod \"2c62e5e6-b713-4410-997b-99491b2135d4\" (UID: \"2c62e5e6-b713-4410-997b-99491b2135d4\") "
Oct 30 19:25:04.437212 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:25:04.437200    4892 reconciler_common.go:299] "Volume detached for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/2c62e5e6-b713-4410-997b-99491b2135d4-extensions-socket-volume\") on node \"ip-10-0-132-82.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 19:25:04.439183 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:25:04.439160    4892 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2c62e5e6-b713-4410-997b-99491b2135d4-kube-api-access-d5sml" pod "2c62e5e6-b713-4410-997b-99491b2135d4" (UID: "2c62e5e6-b713-4410-997b-99491b2135d4"). InnerVolumeSpecName "kube-api-access-d5sml". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:25:04.451469 ip-10-0-132-82 kubenswrapper[4892]: E1030 19:25:04.451440    4892 status_manager.go:1045] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-6b6fb56846-krq52\" is forbidden: User \"system:node:ip-10-0-132-82.us-west-1.compute.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-132-82.us-west-1.compute.internal' and this object" podUID="6392742e-b112-4b31-8109-0c64cd373363" pod="kuadrant-system/kuadrant-operator-controller-manager-6b6fb56846-krq52"
Oct 30 19:25:04.502145 ip-10-0-132-82 kubenswrapper[4892]: E1030 19:25:04.502116    4892 status_manager.go:1045] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-6b76784d48-bcqxc\" is forbidden: User \"system:node:ip-10-0-132-82.us-west-1.compute.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-132-82.us-west-1.compute.internal' and this object" podUID="2c62e5e6-b713-4410-997b-99491b2135d4" pod="kuadrant-system/kuadrant-operator-controller-manager-6b76784d48-bcqxc"
Oct 30 19:25:04.537990 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:25:04.537914    4892 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-d5sml\" (UniqueName: \"kubernetes.io/projected/2c62e5e6-b713-4410-997b-99491b2135d4-kube-api-access-d5sml\") on node \"ip-10-0-132-82.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 19:25:04.567727 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:25:04.567699    4892 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="2c62e5e6-b713-4410-997b-99491b2135d4" path="/var/lib/kubelet/pods/2c62e5e6-b713-4410-997b-99491b2135d4/volumes"
Oct 30 19:25:04.568077 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:25:04.568063    4892 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="6392742e-b112-4b31-8109-0c64cd373363" path="/var/lib/kubelet/pods/6392742e-b112-4b31-8109-0c64cd373363/volumes"
Oct 30 19:25:05.007221 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:25:05.007192    4892 generic.go:356] "Generic (PLEG): container finished" podID="2c62e5e6-b713-4410-997b-99491b2135d4" containerID="20595a1aac4116f47c992b6c127fff1b91b58a90f6c9fa611b06a2a6cd0613f1" exitCode=0
Oct 30 19:25:05.007687 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:25:05.007252    4892 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-6b76784d48-bcqxc"
Oct 30 19:25:05.007687 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:25:05.007304    4892 scope.go:122] "RemoveContainer" containerID="20595a1aac4116f47c992b6c127fff1b91b58a90f6c9fa611b06a2a6cd0613f1"
Oct 30 19:25:05.016123 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:25:05.016102    4892 scope.go:122] "RemoveContainer" containerID="20595a1aac4116f47c992b6c127fff1b91b58a90f6c9fa611b06a2a6cd0613f1"
Oct 30 19:25:05.016385 ip-10-0-132-82 kubenswrapper[4892]: E1030 19:25:05.016366    4892 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"20595a1aac4116f47c992b6c127fff1b91b58a90f6c9fa611b06a2a6cd0613f1\": container with ID starting with 20595a1aac4116f47c992b6c127fff1b91b58a90f6c9fa611b06a2a6cd0613f1 not found: ID does not exist" containerID="20595a1aac4116f47c992b6c127fff1b91b58a90f6c9fa611b06a2a6cd0613f1"
Oct 30 19:25:05.016450 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:25:05.016398    4892 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"20595a1aac4116f47c992b6c127fff1b91b58a90f6c9fa611b06a2a6cd0613f1"} err="failed to get container status \"20595a1aac4116f47c992b6c127fff1b91b58a90f6c9fa611b06a2a6cd0613f1\": rpc error: code = NotFound desc = could not find container \"20595a1aac4116f47c992b6c127fff1b91b58a90f6c9fa611b06a2a6cd0613f1\": container with ID starting with 20595a1aac4116f47c992b6c127fff1b91b58a90f6c9fa611b06a2a6cd0613f1 not found: ID does not exist"
Oct 30 19:25:14.055779 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:25:14.055731    4892 generic.go:356] "Generic (PLEG): container finished" podID="4f6ef7c9-be20-498f-861d-26247190eed8" containerID="5e4cd12f695f940d8882a0aa1544523e1b507fdba375a926fb9cdfd22d7c564b" exitCode=1
Oct 30 19:25:14.056200 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:25:14.055805    4892 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-6b76784d48-dzb5z" event={"ID":"4f6ef7c9-be20-498f-861d-26247190eed8","Type":"ContainerDied","Data":"5e4cd12f695f940d8882a0aa1544523e1b507fdba375a926fb9cdfd22d7c564b"}
Oct 30 19:25:14.056200 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:25:14.056157    4892 scope.go:122] "RemoveContainer" containerID="5e4cd12f695f940d8882a0aa1544523e1b507fdba375a926fb9cdfd22d7c564b"
Oct 30 19:25:15.011210 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:25:15.011182    4892 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/kuadrant-operator-controller-manager-6b6fb56846-jx7ft"
Oct 30 19:25:15.062979 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:25:15.062936    4892 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-6b76784d48-dzb5z" event={"ID":"4f6ef7c9-be20-498f-861d-26247190eed8","Type":"ContainerStarted","Data":"10ca84983aa60bf70a5deb02cae5f50ff282159708bd543202c4e58362ae1512"}
Oct 30 19:25:15.063351 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:25:15.063177    4892 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/kuadrant-operator-controller-manager-6b76784d48-dzb5z"
Oct 30 19:25:24.845155 ip-10-0-132-82 kubenswrapper[4892]: E1030 19:25:24.845115    4892 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-jrm87" containerName="extractor"
Oct 30 19:25:26.071548 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:25:26.071518    4892 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/kuadrant-operator-controller-manager-6b76784d48-dzb5z"
Oct 30 19:25:26.291935 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:25:26.291907    4892 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-6b6fb56846-jx7ft"]
Oct 30 19:25:26.292161 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:25:26.292141    4892 kuberuntime_container.go:898] "Killing container with a grace period" pod="kuadrant-system/kuadrant-operator-controller-manager-6b6fb56846-jx7ft" podUID="b0a7ae76-0cab-4081-9e2d-0fbc0332a8c2" containerName="manager" containerID="cri-o://34f5a35289a2723abde5bd02fe9ce9c64be2bbc60ac9e453f6b40478dbab69a8" gracePeriod=10
Oct 30 19:25:26.523473 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:25:26.523445    4892 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-57bb9bfff-qflpx"]
Oct 30 19:25:26.526840 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:25:26.526821    4892 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-57bb9bfff-qflpx"
Oct 30 19:25:26.537367 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:25:26.537349    4892 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-6b6fb56846-jx7ft"
Oct 30 19:25:26.588284 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:25:26.588254    4892 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-57bb9bfff-qflpx"]
Oct 30 19:25:26.598297 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:25:26.598276    4892 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b0a7ae76-0cab-4081-9e2d-0fbc0332a8c2-extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/b0a7ae76-0cab-4081-9e2d-0fbc0332a8c2-extensions-socket-volume\") pod \"b0a7ae76-0cab-4081-9e2d-0fbc0332a8c2\" (UID: \"b0a7ae76-0cab-4081-9e2d-0fbc0332a8c2\") "
Oct 30 19:25:26.598422 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:25:26.598404    4892 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b0a7ae76-0cab-4081-9e2d-0fbc0332a8c2-kube-api-access-bfgps\" (UniqueName: \"kubernetes.io/projected/b0a7ae76-0cab-4081-9e2d-0fbc0332a8c2-kube-api-access-bfgps\") pod \"b0a7ae76-0cab-4081-9e2d-0fbc0332a8c2\" (UID: \"b0a7ae76-0cab-4081-9e2d-0fbc0332a8c2\") "
Oct 30 19:25:26.598548 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:25:26.598531    4892 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7b4sb\" (UniqueName: \"kubernetes.io/projected/55c71a1d-feca-47a1-b100-738a982b6d47-kube-api-access-7b4sb\") pod \"kuadrant-operator-controller-manager-57bb9bfff-qflpx\" (UID: \"55c71a1d-feca-47a1-b100-738a982b6d47\") " pod="kuadrant-system/kuadrant-operator-controller-manager-57bb9bfff-qflpx"
Oct 30 19:25:26.598626 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:25:26.598597    4892 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/55c71a1d-feca-47a1-b100-738a982b6d47-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-57bb9bfff-qflpx\" (UID: \"55c71a1d-feca-47a1-b100-738a982b6d47\") " pod="kuadrant-system/kuadrant-operator-controller-manager-57bb9bfff-qflpx"
Oct 30 19:25:26.598666 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:25:26.598621    4892 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b0a7ae76-0cab-4081-9e2d-0fbc0332a8c2-extensions-socket-volume" pod "b0a7ae76-0cab-4081-9e2d-0fbc0332a8c2" (UID: "b0a7ae76-0cab-4081-9e2d-0fbc0332a8c2"). InnerVolumeSpecName "extensions-socket-volume". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:25:26.598762 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:25:26.598747    4892 reconciler_common.go:299] "Volume detached for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/b0a7ae76-0cab-4081-9e2d-0fbc0332a8c2-extensions-socket-volume\") on node \"ip-10-0-132-82.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 19:25:26.600578 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:25:26.600557    4892 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b0a7ae76-0cab-4081-9e2d-0fbc0332a8c2-kube-api-access-bfgps" pod "b0a7ae76-0cab-4081-9e2d-0fbc0332a8c2" (UID: "b0a7ae76-0cab-4081-9e2d-0fbc0332a8c2"). InnerVolumeSpecName "kube-api-access-bfgps". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:25:26.699739 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:25:26.699678    4892 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7b4sb\" (UniqueName: \"kubernetes.io/projected/55c71a1d-feca-47a1-b100-738a982b6d47-kube-api-access-7b4sb\") pod \"kuadrant-operator-controller-manager-57bb9bfff-qflpx\" (UID: \"55c71a1d-feca-47a1-b100-738a982b6d47\") " pod="kuadrant-system/kuadrant-operator-controller-manager-57bb9bfff-qflpx"
Oct 30 19:25:26.699847 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:25:26.699734    4892 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/55c71a1d-feca-47a1-b100-738a982b6d47-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-57bb9bfff-qflpx\" (UID: \"55c71a1d-feca-47a1-b100-738a982b6d47\") " pod="kuadrant-system/kuadrant-operator-controller-manager-57bb9bfff-qflpx"
Oct 30 19:25:26.699847 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:25:26.699794    4892 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bfgps\" (UniqueName: \"kubernetes.io/projected/b0a7ae76-0cab-4081-9e2d-0fbc0332a8c2-kube-api-access-bfgps\") on node \"ip-10-0-132-82.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 19:25:26.700139 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:25:26.700120    4892 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/55c71a1d-feca-47a1-b100-738a982b6d47-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-57bb9bfff-qflpx\" (UID: \"55c71a1d-feca-47a1-b100-738a982b6d47\") " pod="kuadrant-system/kuadrant-operator-controller-manager-57bb9bfff-qflpx"
Oct 30 19:25:26.760128 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:25:26.760100    4892 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7b4sb\" (UniqueName: \"kubernetes.io/projected/55c71a1d-feca-47a1-b100-738a982b6d47-kube-api-access-7b4sb\") pod \"kuadrant-operator-controller-manager-57bb9bfff-qflpx\" (UID: \"55c71a1d-feca-47a1-b100-738a982b6d47\") " pod="kuadrant-system/kuadrant-operator-controller-manager-57bb9bfff-qflpx"
Oct 30 19:25:26.845388 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:25:26.845363    4892 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-57bb9bfff-qflpx"
Oct 30 19:25:27.122897 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:25:27.122825    4892 generic.go:356] "Generic (PLEG): container finished" podID="b0a7ae76-0cab-4081-9e2d-0fbc0332a8c2" containerID="34f5a35289a2723abde5bd02fe9ce9c64be2bbc60ac9e453f6b40478dbab69a8" exitCode=0
Oct 30 19:25:27.122897 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:25:27.122874    4892 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-6b6fb56846-jx7ft"
Oct 30 19:25:27.122897 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:25:27.122885    4892 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-6b6fb56846-jx7ft" event={"ID":"b0a7ae76-0cab-4081-9e2d-0fbc0332a8c2","Type":"ContainerDied","Data":"34f5a35289a2723abde5bd02fe9ce9c64be2bbc60ac9e453f6b40478dbab69a8"}
Oct 30 19:25:27.123385 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:25:27.122936    4892 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-6b6fb56846-jx7ft" event={"ID":"b0a7ae76-0cab-4081-9e2d-0fbc0332a8c2","Type":"ContainerDied","Data":"24d122d8e08d8a3940ca770839b2dddccedede21207683aaf116538ced9cdf74"}
Oct 30 19:25:27.123385 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:25:27.122987    4892 scope.go:122] "RemoveContainer" containerID="34f5a35289a2723abde5bd02fe9ce9c64be2bbc60ac9e453f6b40478dbab69a8"
Oct 30 19:25:27.131742 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:25:27.131719    4892 scope.go:122] "RemoveContainer" containerID="34f5a35289a2723abde5bd02fe9ce9c64be2bbc60ac9e453f6b40478dbab69a8"
Oct 30 19:25:27.132122 ip-10-0-132-82 kubenswrapper[4892]: E1030 19:25:27.132100    4892 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"34f5a35289a2723abde5bd02fe9ce9c64be2bbc60ac9e453f6b40478dbab69a8\": container with ID starting with 34f5a35289a2723abde5bd02fe9ce9c64be2bbc60ac9e453f6b40478dbab69a8 not found: ID does not exist" containerID="34f5a35289a2723abde5bd02fe9ce9c64be2bbc60ac9e453f6b40478dbab69a8"
Oct 30 19:25:27.132188 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:25:27.132133    4892 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"34f5a35289a2723abde5bd02fe9ce9c64be2bbc60ac9e453f6b40478dbab69a8"} err="failed to get container status \"34f5a35289a2723abde5bd02fe9ce9c64be2bbc60ac9e453f6b40478dbab69a8\": rpc error: code = NotFound desc = could not find container \"34f5a35289a2723abde5bd02fe9ce9c64be2bbc60ac9e453f6b40478dbab69a8\": container with ID starting with 34f5a35289a2723abde5bd02fe9ce9c64be2bbc60ac9e453f6b40478dbab69a8 not found: ID does not exist"
Oct 30 19:25:27.268870 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:25:27.268850    4892 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-57bb9bfff-qflpx"]
Oct 30 19:25:27.271418 ip-10-0-132-82 kubenswrapper[4892]: W1030 19:25:27.271385    4892 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod55c71a1d_feca_47a1_b100_738a982b6d47.slice/crio-fc44631b285988f7f2f54a4da2d22d6fa9563574f02ae91669b5b027b0ea56f1 WatchSource:0}: Error finding container fc44631b285988f7f2f54a4da2d22d6fa9563574f02ae91669b5b027b0ea56f1: Status 404 returned error can't find the container with id fc44631b285988f7f2f54a4da2d22d6fa9563574f02ae91669b5b027b0ea56f1
Oct 30 19:25:27.288411 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:25:27.288389    4892 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-6b6fb56846-jx7ft"]
Oct 30 19:25:27.295128 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:25:27.295107    4892 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-6b6fb56846-jx7ft"]
Oct 30 19:25:28.130396 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:25:28.130349    4892 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-57bb9bfff-qflpx" event={"ID":"55c71a1d-feca-47a1-b100-738a982b6d47","Type":"ContainerStarted","Data":"9ec83f08186618bf35d3b4f36f878125a5c7e34ed6992d613bd37006c90e0a55"}
Oct 30 19:25:28.130815 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:25:28.130454    4892 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/kuadrant-operator-controller-manager-57bb9bfff-qflpx"
Oct 30 19:25:28.130815 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:25:28.130486    4892 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-57bb9bfff-qflpx" event={"ID":"55c71a1d-feca-47a1-b100-738a982b6d47","Type":"ContainerStarted","Data":"fc44631b285988f7f2f54a4da2d22d6fa9563574f02ae91669b5b027b0ea56f1"}
Oct 30 19:25:28.260162 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:25:28.260111    4892 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kuadrant-system/kuadrant-operator-controller-manager-57bb9bfff-qflpx" podStartSLOduration=2.260098354 podStartE2EDuration="2.260098354s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:25:26 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:25:28.259565006 +0000 UTC m=+575.830030608" watchObservedRunningTime="2026-10-30 19:25:28.260098354 +0000 UTC m=+575.830563953"
Oct 30 19:25:28.568598 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:25:28.568567    4892 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b0a7ae76-0cab-4081-9e2d-0fbc0332a8c2" path="/var/lib/kubelet/pods/b0a7ae76-0cab-4081-9e2d-0fbc0332a8c2/volumes"
Oct 30 19:25:39.138731 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:25:39.138652    4892 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/kuadrant-operator-controller-manager-57bb9bfff-qflpx"
Oct 30 19:25:39.356375 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:25:39.356341    4892 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-6b76784d48-dzb5z"]
Oct 30 19:25:39.356586 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:25:39.356555    4892 kuberuntime_container.go:898] "Killing container with a grace period" pod="kuadrant-system/kuadrant-operator-controller-manager-6b76784d48-dzb5z" podUID="4f6ef7c9-be20-498f-861d-26247190eed8" containerName="manager" containerID="cri-o://10ca84983aa60bf70a5deb02cae5f50ff282159708bd543202c4e58362ae1512" gracePeriod=10
Oct 30 19:25:39.605386 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:25:39.605359    4892 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-6b76784d48-dzb5z"
Oct 30 19:25:39.695621 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:25:39.695594    4892 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4f6ef7c9-be20-498f-861d-26247190eed8-extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/4f6ef7c9-be20-498f-861d-26247190eed8-extensions-socket-volume\") pod \"4f6ef7c9-be20-498f-861d-26247190eed8\" (UID: \"4f6ef7c9-be20-498f-861d-26247190eed8\") "
Oct 30 19:25:39.695755 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:25:39.695655    4892 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4f6ef7c9-be20-498f-861d-26247190eed8-kube-api-access-h7bnf\" (UniqueName: \"kubernetes.io/projected/4f6ef7c9-be20-498f-861d-26247190eed8-kube-api-access-h7bnf\") pod \"4f6ef7c9-be20-498f-861d-26247190eed8\" (UID: \"4f6ef7c9-be20-498f-861d-26247190eed8\") "
Oct 30 19:25:39.696011 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:25:39.695990    4892 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4f6ef7c9-be20-498f-861d-26247190eed8-extensions-socket-volume" pod "4f6ef7c9-be20-498f-861d-26247190eed8" (UID: "4f6ef7c9-be20-498f-861d-26247190eed8"). InnerVolumeSpecName "extensions-socket-volume". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:25:39.697824 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:25:39.697800    4892 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4f6ef7c9-be20-498f-861d-26247190eed8-kube-api-access-h7bnf" pod "4f6ef7c9-be20-498f-861d-26247190eed8" (UID: "4f6ef7c9-be20-498f-861d-26247190eed8"). InnerVolumeSpecName "kube-api-access-h7bnf". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:25:39.796862 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:25:39.796793    4892 reconciler_common.go:299] "Volume detached for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/4f6ef7c9-be20-498f-861d-26247190eed8-extensions-socket-volume\") on node \"ip-10-0-132-82.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 19:25:39.796862 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:25:39.796821    4892 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-h7bnf\" (UniqueName: \"kubernetes.io/projected/4f6ef7c9-be20-498f-861d-26247190eed8-kube-api-access-h7bnf\") on node \"ip-10-0-132-82.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 19:25:40.193937 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:25:40.193899    4892 generic.go:356] "Generic (PLEG): container finished" podID="4f6ef7c9-be20-498f-861d-26247190eed8" containerID="10ca84983aa60bf70a5deb02cae5f50ff282159708bd543202c4e58362ae1512" exitCode=0
Oct 30 19:25:40.194419 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:25:40.193996    4892 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-6b76784d48-dzb5z"
Oct 30 19:25:40.194419 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:25:40.193995    4892 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-6b76784d48-dzb5z" event={"ID":"4f6ef7c9-be20-498f-861d-26247190eed8","Type":"ContainerDied","Data":"10ca84983aa60bf70a5deb02cae5f50ff282159708bd543202c4e58362ae1512"}
Oct 30 19:25:40.194419 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:25:40.194138    4892 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-6b76784d48-dzb5z" event={"ID":"4f6ef7c9-be20-498f-861d-26247190eed8","Type":"ContainerDied","Data":"b327795191482f9d0892930943880fec9dbd1fa4dd3c2bef40d9a447ead924ed"}
Oct 30 19:25:40.194419 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:25:40.194161    4892 scope.go:122] "RemoveContainer" containerID="10ca84983aa60bf70a5deb02cae5f50ff282159708bd543202c4e58362ae1512"
Oct 30 19:25:40.203621 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:25:40.203605    4892 scope.go:122] "RemoveContainer" containerID="5e4cd12f695f940d8882a0aa1544523e1b507fdba375a926fb9cdfd22d7c564b"
Oct 30 19:25:40.211286 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:25:40.211268    4892 scope.go:122] "RemoveContainer" containerID="10ca84983aa60bf70a5deb02cae5f50ff282159708bd543202c4e58362ae1512"
Oct 30 19:25:40.211575 ip-10-0-132-82 kubenswrapper[4892]: E1030 19:25:40.211556    4892 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"10ca84983aa60bf70a5deb02cae5f50ff282159708bd543202c4e58362ae1512\": container with ID starting with 10ca84983aa60bf70a5deb02cae5f50ff282159708bd543202c4e58362ae1512 not found: ID does not exist" containerID="10ca84983aa60bf70a5deb02cae5f50ff282159708bd543202c4e58362ae1512"
Oct 30 19:25:40.211660 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:25:40.211589    4892 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"10ca84983aa60bf70a5deb02cae5f50ff282159708bd543202c4e58362ae1512"} err="failed to get container status \"10ca84983aa60bf70a5deb02cae5f50ff282159708bd543202c4e58362ae1512\": rpc error: code = NotFound desc = could not find container \"10ca84983aa60bf70a5deb02cae5f50ff282159708bd543202c4e58362ae1512\": container with ID starting with 10ca84983aa60bf70a5deb02cae5f50ff282159708bd543202c4e58362ae1512 not found: ID does not exist"
Oct 30 19:25:40.211660 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:25:40.211614    4892 scope.go:122] "RemoveContainer" containerID="5e4cd12f695f940d8882a0aa1544523e1b507fdba375a926fb9cdfd22d7c564b"
Oct 30 19:25:40.211883 ip-10-0-132-82 kubenswrapper[4892]: E1030 19:25:40.211866    4892 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5e4cd12f695f940d8882a0aa1544523e1b507fdba375a926fb9cdfd22d7c564b\": container with ID starting with 5e4cd12f695f940d8882a0aa1544523e1b507fdba375a926fb9cdfd22d7c564b not found: ID does not exist" containerID="5e4cd12f695f940d8882a0aa1544523e1b507fdba375a926fb9cdfd22d7c564b"
Oct 30 19:25:40.211924 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:25:40.211890    4892 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5e4cd12f695f940d8882a0aa1544523e1b507fdba375a926fb9cdfd22d7c564b"} err="failed to get container status \"5e4cd12f695f940d8882a0aa1544523e1b507fdba375a926fb9cdfd22d7c564b\": rpc error: code = NotFound desc = could not find container \"5e4cd12f695f940d8882a0aa1544523e1b507fdba375a926fb9cdfd22d7c564b\": container with ID starting with 5e4cd12f695f940d8882a0aa1544523e1b507fdba375a926fb9cdfd22d7c564b not found: ID does not exist"
Oct 30 19:25:40.358865 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:25:40.358832    4892 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-6b76784d48-dzb5z"]
Oct 30 19:25:40.408291 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:25:40.408260    4892 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-6b76784d48-dzb5z"]
Oct 30 19:25:40.567093 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:25:40.567021    4892 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="4f6ef7c9-be20-498f-861d-26247190eed8" path="/var/lib/kubelet/pods/4f6ef7c9-be20-498f-861d-26247190eed8/volumes"
Oct 30 19:25:52.632577 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:25:52.632556    4892 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-tg5xj_11db9cec-653f-4aee-8937-0cf5b367efac/console-operator/3.log"
Oct 30 19:25:52.635660 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:25:52.635634    4892 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-tg5xj_11db9cec-653f-4aee-8937-0cf5b367efac/console-operator/3.log"
Oct 30 19:25:53.869873 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:25:53.869839    4892 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-vvbng"]
Oct 30 19:25:53.870278 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:25:53.870177    4892 cpu_manager.go:414] "containerMap: removing container" podUID="4f6ef7c9-be20-498f-861d-26247190eed8" containerName="manager"
Oct 30 19:25:53.870278 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:25:53.870188    4892 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4f6ef7c9-be20-498f-861d-26247190eed8" containerName="manager"
Oct 30 19:25:53.870278 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:25:53.870203    4892 cpu_manager.go:414] "containerMap: removing container" podUID="4f6ef7c9-be20-498f-861d-26247190eed8" containerName="manager"
Oct 30 19:25:53.870278 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:25:53.870208    4892 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4f6ef7c9-be20-498f-861d-26247190eed8" containerName="manager"
Oct 30 19:25:53.870278 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:25:53.870219    4892 cpu_manager.go:414] "containerMap: removing container" podUID="b0a7ae76-0cab-4081-9e2d-0fbc0332a8c2" containerName="manager"
Oct 30 19:25:53.870278 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:25:53.870224    4892 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b0a7ae76-0cab-4081-9e2d-0fbc0332a8c2" containerName="manager"
Oct 30 19:25:53.870278 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:25:53.870283    4892 memory_manager.go:365] "RemoveStaleState removing state" podUID="4f6ef7c9-be20-498f-861d-26247190eed8" containerName="manager"
Oct 30 19:25:53.870491 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:25:53.870293    4892 memory_manager.go:365] "RemoveStaleState removing state" podUID="4f6ef7c9-be20-498f-861d-26247190eed8" containerName="manager"
Oct 30 19:25:53.870491 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:25:53.870300    4892 memory_manager.go:365] "RemoveStaleState removing state" podUID="b0a7ae76-0cab-4081-9e2d-0fbc0332a8c2" containerName="manager"
Oct 30 19:25:53.874771 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:25:53.874753    4892 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-vvbng"
Oct 30 19:25:53.926563 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:25:53.926540    4892 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"maas-default-gateway-openshift-default-dockercfg-f2npg\""
Oct 30 19:25:53.978098 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:25:53.978072    4892 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-vvbng"]
Oct 30 19:25:54.002823 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:25:54.002790    4892 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/52ecec31-2002-4855-a815-74843a6c459a-istio-podinfo\") pod \"maas-default-gateway-openshift-default-6db586dd6c-vvbng\" (UID: \"52ecec31-2002-4855-a815-74843a6c459a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-vvbng"
Oct 30 19:25:54.002953 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:25:54.002827    4892 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-t95n7\" (UniqueName: \"kubernetes.io/projected/52ecec31-2002-4855-a815-74843a6c459a-kube-api-access-t95n7\") pod \"maas-default-gateway-openshift-default-6db586dd6c-vvbng\" (UID: \"52ecec31-2002-4855-a815-74843a6c459a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-vvbng"
Oct 30 19:25:54.002953 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:25:54.002918    4892 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/52ecec31-2002-4855-a815-74843a6c459a-istio-envoy\") pod \"maas-default-gateway-openshift-default-6db586dd6c-vvbng\" (UID: \"52ecec31-2002-4855-a815-74843a6c459a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-vvbng"
Oct 30 19:25:54.002953 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:25:54.002946    4892 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/52ecec31-2002-4855-a815-74843a6c459a-istio-token\") pod \"maas-default-gateway-openshift-default-6db586dd6c-vvbng\" (UID: \"52ecec31-2002-4855-a815-74843a6c459a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-vvbng"
Oct 30 19:25:54.003098 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:25:54.003032    4892 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/52ecec31-2002-4855-a815-74843a6c459a-credential-socket\") pod \"maas-default-gateway-openshift-default-6db586dd6c-vvbng\" (UID: \"52ecec31-2002-4855-a815-74843a6c459a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-vvbng"
Oct 30 19:25:54.003098 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:25:54.003061    4892 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/52ecec31-2002-4855-a815-74843a6c459a-workload-certs\") pod \"maas-default-gateway-openshift-default-6db586dd6c-vvbng\" (UID: \"52ecec31-2002-4855-a815-74843a6c459a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-vvbng"
Oct 30 19:25:54.003183 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:25:54.003145    4892 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/52ecec31-2002-4855-a815-74843a6c459a-istio-data\") pod \"maas-default-gateway-openshift-default-6db586dd6c-vvbng\" (UID: \"52ecec31-2002-4855-a815-74843a6c459a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-vvbng"
Oct 30 19:25:54.003229 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:25:54.003191    4892 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/52ecec31-2002-4855-a815-74843a6c459a-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-6db586dd6c-vvbng\" (UID: \"52ecec31-2002-4855-a815-74843a6c459a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-vvbng"
Oct 30 19:25:54.003229 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:25:54.003216    4892 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/52ecec31-2002-4855-a815-74843a6c459a-workload-socket\") pod \"maas-default-gateway-openshift-default-6db586dd6c-vvbng\" (UID: \"52ecec31-2002-4855-a815-74843a6c459a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-vvbng"
Oct 30 19:25:54.104379 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:25:54.104342    4892 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/52ecec31-2002-4855-a815-74843a6c459a-workload-certs\") pod \"maas-default-gateway-openshift-default-6db586dd6c-vvbng\" (UID: \"52ecec31-2002-4855-a815-74843a6c459a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-vvbng"
Oct 30 19:25:54.104577 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:25:54.104387    4892 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/52ecec31-2002-4855-a815-74843a6c459a-istio-data\") pod \"maas-default-gateway-openshift-default-6db586dd6c-vvbng\" (UID: \"52ecec31-2002-4855-a815-74843a6c459a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-vvbng"
Oct 30 19:25:54.104577 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:25:54.104433    4892 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/52ecec31-2002-4855-a815-74843a6c459a-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-6db586dd6c-vvbng\" (UID: \"52ecec31-2002-4855-a815-74843a6c459a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-vvbng"
Oct 30 19:25:54.104577 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:25:54.104457    4892 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/52ecec31-2002-4855-a815-74843a6c459a-workload-socket\") pod \"maas-default-gateway-openshift-default-6db586dd6c-vvbng\" (UID: \"52ecec31-2002-4855-a815-74843a6c459a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-vvbng"
Oct 30 19:25:54.104577 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:25:54.104485    4892 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/52ecec31-2002-4855-a815-74843a6c459a-istio-podinfo\") pod \"maas-default-gateway-openshift-default-6db586dd6c-vvbng\" (UID: \"52ecec31-2002-4855-a815-74843a6c459a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-vvbng"
Oct 30 19:25:54.104577 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:25:54.104513    4892 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-t95n7\" (UniqueName: \"kubernetes.io/projected/52ecec31-2002-4855-a815-74843a6c459a-kube-api-access-t95n7\") pod \"maas-default-gateway-openshift-default-6db586dd6c-vvbng\" (UID: \"52ecec31-2002-4855-a815-74843a6c459a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-vvbng"
Oct 30 19:25:54.104577 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:25:54.104558    4892 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/52ecec31-2002-4855-a815-74843a6c459a-istio-envoy\") pod \"maas-default-gateway-openshift-default-6db586dd6c-vvbng\" (UID: \"52ecec31-2002-4855-a815-74843a6c459a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-vvbng"
Oct 30 19:25:54.104887 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:25:54.104584    4892 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/52ecec31-2002-4855-a815-74843a6c459a-istio-token\") pod \"maas-default-gateway-openshift-default-6db586dd6c-vvbng\" (UID: \"52ecec31-2002-4855-a815-74843a6c459a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-vvbng"
Oct 30 19:25:54.104887 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:25:54.104606    4892 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/52ecec31-2002-4855-a815-74843a6c459a-credential-socket\") pod \"maas-default-gateway-openshift-default-6db586dd6c-vvbng\" (UID: \"52ecec31-2002-4855-a815-74843a6c459a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-vvbng"
Oct 30 19:25:54.105295 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:25:54.105265    4892 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/52ecec31-2002-4855-a815-74843a6c459a-workload-certs\") pod \"maas-default-gateway-openshift-default-6db586dd6c-vvbng\" (UID: \"52ecec31-2002-4855-a815-74843a6c459a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-vvbng"
Oct 30 19:25:54.105486 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:25:54.105372    4892 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/52ecec31-2002-4855-a815-74843a6c459a-workload-socket\") pod \"maas-default-gateway-openshift-default-6db586dd6c-vvbng\" (UID: \"52ecec31-2002-4855-a815-74843a6c459a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-vvbng"
Oct 30 19:25:54.105486 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:25:54.105377    4892 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/52ecec31-2002-4855-a815-74843a6c459a-istio-data\") pod \"maas-default-gateway-openshift-default-6db586dd6c-vvbng\" (UID: \"52ecec31-2002-4855-a815-74843a6c459a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-vvbng"
Oct 30 19:25:54.105728 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:25:54.105410    4892 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/52ecec31-2002-4855-a815-74843a6c459a-credential-socket\") pod \"maas-default-gateway-openshift-default-6db586dd6c-vvbng\" (UID: \"52ecec31-2002-4855-a815-74843a6c459a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-vvbng"
Oct 30 19:25:54.106136 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:25:54.106111    4892 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/52ecec31-2002-4855-a815-74843a6c459a-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-6db586dd6c-vvbng\" (UID: \"52ecec31-2002-4855-a815-74843a6c459a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-vvbng"
Oct 30 19:25:54.107421 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:25:54.107392    4892 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/52ecec31-2002-4855-a815-74843a6c459a-istio-envoy\") pod \"maas-default-gateway-openshift-default-6db586dd6c-vvbng\" (UID: \"52ecec31-2002-4855-a815-74843a6c459a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-vvbng"
Oct 30 19:25:54.107511 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:25:54.107447    4892 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/52ecec31-2002-4855-a815-74843a6c459a-istio-podinfo\") pod \"maas-default-gateway-openshift-default-6db586dd6c-vvbng\" (UID: \"52ecec31-2002-4855-a815-74843a6c459a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-vvbng"
Oct 30 19:25:54.162131 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:25:54.162105    4892 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/52ecec31-2002-4855-a815-74843a6c459a-istio-token\") pod \"maas-default-gateway-openshift-default-6db586dd6c-vvbng\" (UID: \"52ecec31-2002-4855-a815-74843a6c459a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-vvbng"
Oct 30 19:25:54.162222 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:25:54.162194    4892 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-t95n7\" (UniqueName: \"kubernetes.io/projected/52ecec31-2002-4855-a815-74843a6c459a-kube-api-access-t95n7\") pod \"maas-default-gateway-openshift-default-6db586dd6c-vvbng\" (UID: \"52ecec31-2002-4855-a815-74843a6c459a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-vvbng"
Oct 30 19:25:54.186974 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:25:54.186950    4892 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-vvbng"
Oct 30 19:25:54.615211 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:25:54.614195    4892 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-vvbng"]
Oct 30 19:25:54.617991 ip-10-0-132-82 kubenswrapper[4892]: W1030 19:25:54.617946    4892 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod52ecec31_2002_4855_a815_74843a6c459a.slice/crio-67cd7f02907b95733b592d528871c7be106356f54c1408fb286e0b91a2dbb581 WatchSource:0}: Error finding container 67cd7f02907b95733b592d528871c7be106356f54c1408fb286e0b91a2dbb581: Status 404 returned error can't find the container with id 67cd7f02907b95733b592d528871c7be106356f54c1408fb286e0b91a2dbb581
Oct 30 19:25:54.843151 ip-10-0-132-82 kubenswrapper[4892]: E1030 19:25:54.843117    4892 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-jrm87" containerName="extractor"
Oct 30 19:25:55.271056 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:25:55.271024    4892 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-vvbng" event={"ID":"52ecec31-2002-4855-a815-74843a6c459a","Type":"ContainerStarted","Data":"67cd7f02907b95733b592d528871c7be106356f54c1408fb286e0b91a2dbb581"}
Oct 30 19:25:58.722860 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:25:58.722827    4892 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892000Ki","pods":"250"}
Oct 30 19:25:58.723180 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:25:58.722895    4892 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892000Ki","pods":"250"}
Oct 30 19:25:58.723180 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:25:58.722927    4892 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892000Ki","pods":"250"}
Oct 30 19:25:59.300045 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:25:59.300005    4892 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-vvbng" event={"ID":"52ecec31-2002-4855-a815-74843a6c459a","Type":"ContainerStarted","Data":"d551544ddfb7049fef6df2b158d82d907308a9e655c11de82a56c8d500a691b7"}
Oct 30 19:25:59.466319 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:25:59.466272    4892 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-vvbng" podStartSLOduration=2.363717747 podStartE2EDuration="6.466256896s" totalImagesPullingTime="4.102539149s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:25:53 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:25:59.465142339 +0000 UTC m=+607.035607942" watchObservedRunningTime="2026-10-30 19:25:59.466256896 +0000 UTC m=+607.036722506"
Oct 30 19:26:00.188085 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:26:00.188048    4892 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-vvbng"
Oct 30 19:26:00.193202 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:26:00.193177    4892 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-vvbng"
Oct 30 19:26:00.304959 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:26:00.304928    4892 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-vvbng"
Oct 30 19:26:00.305694 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:26:00.305677    4892 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-vvbng"
Oct 30 19:26:06.938648 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:26:06.938615    4892 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-fsfvk"]
Oct 30 19:26:06.941995 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:26:06.941957    4892 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-7c9b87d574-fsfvk"
Oct 30 19:26:06.996352 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:26:06.996327    4892 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"limitador-limits-config-limitador\""
Oct 30 19:26:06.998543 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:26:06.998521    4892 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-fsfvk"]
Oct 30 19:26:07.045733 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:26:07.045712    4892 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"default-dockercfg-8l6kf\""
Oct 30 19:26:07.110539 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:26:07.110511    4892 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mld8r\" (UniqueName: \"kubernetes.io/projected/a2799daf-05de-4827-90df-db8ac57d645a-kube-api-access-mld8r\") pod \"limitador-limitador-7c9b87d574-fsfvk\" (UID: \"a2799daf-05de-4827-90df-db8ac57d645a\") " pod="kuadrant-system/limitador-limitador-7c9b87d574-fsfvk"
Oct 30 19:26:07.110648 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:26:07.110567    4892 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/a2799daf-05de-4827-90df-db8ac57d645a-config-file\") pod \"limitador-limitador-7c9b87d574-fsfvk\" (UID: \"a2799daf-05de-4827-90df-db8ac57d645a\") " pod="kuadrant-system/limitador-limitador-7c9b87d574-fsfvk"
Oct 30 19:26:07.211902 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:26:07.211848    4892 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mld8r\" (UniqueName: \"kubernetes.io/projected/a2799daf-05de-4827-90df-db8ac57d645a-kube-api-access-mld8r\") pod \"limitador-limitador-7c9b87d574-fsfvk\" (UID: \"a2799daf-05de-4827-90df-db8ac57d645a\") " pod="kuadrant-system/limitador-limitador-7c9b87d574-fsfvk"
Oct 30 19:26:07.212021 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:26:07.211902    4892 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/a2799daf-05de-4827-90df-db8ac57d645a-config-file\") pod \"limitador-limitador-7c9b87d574-fsfvk\" (UID: \"a2799daf-05de-4827-90df-db8ac57d645a\") " pod="kuadrant-system/limitador-limitador-7c9b87d574-fsfvk"
Oct 30 19:26:07.212463 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:26:07.212446    4892 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/a2799daf-05de-4827-90df-db8ac57d645a-config-file\") pod \"limitador-limitador-7c9b87d574-fsfvk\" (UID: \"a2799daf-05de-4827-90df-db8ac57d645a\") " pod="kuadrant-system/limitador-limitador-7c9b87d574-fsfvk"
Oct 30 19:26:07.265988 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:26:07.264513    4892 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-fsfvk"]
Oct 30 19:26:07.270886 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:26:07.270863    4892 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mld8r\" (UniqueName: \"kubernetes.io/projected/a2799daf-05de-4827-90df-db8ac57d645a-kube-api-access-mld8r\") pod \"limitador-limitador-7c9b87d574-fsfvk\" (UID: \"a2799daf-05de-4827-90df-db8ac57d645a\") " pod="kuadrant-system/limitador-limitador-7c9b87d574-fsfvk"
Oct 30 19:26:07.554694 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:26:07.554625    4892 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-7c9b87d574-fsfvk"
Oct 30 19:26:07.926978 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:26:07.926936    4892 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-fsfvk"]
Oct 30 19:26:07.929080 ip-10-0-132-82 kubenswrapper[4892]: W1030 19:26:07.929048    4892 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda2799daf_05de_4827_90df_db8ac57d645a.slice/crio-c8e2d818bd6fc721e94431c89820a6cb88220139169d4a3983a6bdeabb3cc1e8 WatchSource:0}: Error finding container c8e2d818bd6fc721e94431c89820a6cb88220139169d4a3983a6bdeabb3cc1e8: Status 404 returned error can't find the container with id c8e2d818bd6fc721e94431c89820a6cb88220139169d4a3983a6bdeabb3cc1e8
Oct 30 19:26:08.345037 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:26:08.344941    4892 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-7c9b87d574-fsfvk" event={"ID":"a2799daf-05de-4827-90df-db8ac57d645a","Type":"ContainerStarted","Data":"c8e2d818bd6fc721e94431c89820a6cb88220139169d4a3983a6bdeabb3cc1e8"}
Oct 30 19:26:10.356839 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:26:10.356804    4892 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-7c9b87d574-fsfvk" event={"ID":"a2799daf-05de-4827-90df-db8ac57d645a","Type":"ContainerStarted","Data":"36b4d68f807a9985a05a678a66bae773052865c1f855c57e216c85cb527acb7a"}
Oct 30 19:26:10.357290 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:26:10.356902    4892 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/limitador-limitador-7c9b87d574-fsfvk"
Oct 30 19:26:10.470805 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:26:10.470759    4892 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kuadrant-system/limitador-limitador-7c9b87d574-fsfvk" podStartSLOduration=2.673477979 podStartE2EDuration="4.470746009s" totalImagesPullingTime="1.79726803s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:26:06 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:26:10.469054444 +0000 UTC m=+618.039520044" watchObservedRunningTime="2026-10-30 19:26:10.470746009 +0000 UTC m=+618.041211608"
Oct 30 19:26:21.362569 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:26:21.362535    4892 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/limitador-limitador-7c9b87d574-fsfvk"
Oct 30 19:26:24.843567 ip-10-0-132-82 kubenswrapper[4892]: E1030 19:26:24.843534    4892 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-jrm87" containerName="extractor"
Oct 30 19:26:54.842947 ip-10-0-132-82 kubenswrapper[4892]: E1030 19:26:54.842914    4892 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-jrm87" containerName="extractor"
Oct 30 19:27:24.843684 ip-10-0-132-82 kubenswrapper[4892]: E1030 19:27:24.843646    4892 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-jrm87" containerName="extractor"
Oct 30 19:27:54.843186 ip-10-0-132-82 kubenswrapper[4892]: E1030 19:27:54.843145    4892 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-jrm87" containerName="extractor"
Oct 30 19:28:01.037226 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:28:01.037196    4892 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-fsfvk"]
Oct 30 19:28:10.939585 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:28:10.939552    4892 generic.go:356] "Generic (PLEG): container finished" podID="52ecec31-2002-4855-a815-74843a6c459a" containerID="d551544ddfb7049fef6df2b158d82d907308a9e655c11de82a56c8d500a691b7" exitCode=0
Oct 30 19:28:10.940045 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:28:10.939614    4892 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-vvbng" event={"ID":"52ecec31-2002-4855-a815-74843a6c459a","Type":"ContainerDied","Data":"d551544ddfb7049fef6df2b158d82d907308a9e655c11de82a56c8d500a691b7"}
Oct 30 19:28:10.940045 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:28:10.939989    4892 scope.go:122] "RemoveContainer" containerID="d551544ddfb7049fef6df2b158d82d907308a9e655c11de82a56c8d500a691b7"
Oct 30 19:28:10.941068 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:28:10.941031    4892 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892000Ki","pods":"250"}
Oct 30 19:28:10.941157 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:28:10.941119    4892 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892000Ki","pods":"250"}
Oct 30 19:28:10.941203 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:28:10.941170    4892 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892000Ki","pods":"250"}
Oct 30 19:28:11.958416 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:28:11.958379    4892 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-vvbng" event={"ID":"52ecec31-2002-4855-a815-74843a6c459a","Type":"ContainerStarted","Data":"68b7c92a120af3bacf8a9de93fd54371cd68d3bb70405545e367fd5bcfed4d80"}
Oct 30 19:28:12.187142 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:28:12.187113    4892 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-vvbng"
Oct 30 19:28:12.188214 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:28:12.188187    4892 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6db586dd6c-vvbng container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.46:15021/healthz/ready\": dial tcp 10.132.0.46:15021: connect: connection refused" start-of-body=
Oct 30 19:28:12.188338 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:28:12.188235    4892 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-vvbng" podUID="52ecec31-2002-4855-a815-74843a6c459a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.46:15021/healthz/ready\": dial tcp 10.132.0.46:15021: connect: connection refused"
Oct 30 19:28:13.187627 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:28:13.187592    4892 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6db586dd6c-vvbng container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.46:15021/healthz/ready\": dial tcp 10.132.0.46:15021: connect: connection refused" start-of-body=
Oct 30 19:28:13.188073 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:28:13.187643    4892 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-vvbng" podUID="52ecec31-2002-4855-a815-74843a6c459a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.46:15021/healthz/ready\": dial tcp 10.132.0.46:15021: connect: connection refused"
Oct 30 19:28:14.187734 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:28:14.187694    4892 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6db586dd6c-vvbng container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.46:15021/healthz/ready\": dial tcp 10.132.0.46:15021: connect: connection refused" start-of-body=
Oct 30 19:28:14.188179 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:28:14.187756    4892 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-vvbng" podUID="52ecec31-2002-4855-a815-74843a6c459a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.46:15021/healthz/ready\": dial tcp 10.132.0.46:15021: connect: connection refused"
Oct 30 19:28:15.188318 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:28:15.188275    4892 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6db586dd6c-vvbng container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.46:15021/healthz/ready\": dial tcp 10.132.0.46:15021: connect: connection refused" start-of-body=
Oct 30 19:28:15.188708 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:28:15.188347    4892 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-vvbng" podUID="52ecec31-2002-4855-a815-74843a6c459a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.46:15021/healthz/ready\": dial tcp 10.132.0.46:15021: connect: connection refused"
Oct 30 19:28:15.305228 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:28:15.305200    4892 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-vvbng"
Oct 30 19:28:16.187828 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:28:16.187791    4892 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6db586dd6c-vvbng container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.46:15021/healthz/ready\": dial tcp 10.132.0.46:15021: connect: connection refused" start-of-body=
Oct 30 19:28:16.188050 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:28:16.187865    4892 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-vvbng" podUID="52ecec31-2002-4855-a815-74843a6c459a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.46:15021/healthz/ready\": dial tcp 10.132.0.46:15021: connect: connection refused"
Oct 30 19:28:17.193224 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:28:17.193181    4892 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-vvbng"
Oct 30 19:28:17.194550 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:28:17.194529    4892 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-vvbng"
Oct 30 19:28:24.843264 ip-10-0-132-82 kubenswrapper[4892]: E1030 19:28:24.843221    4892 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-jrm87" containerName="extractor"
Oct 30 19:28:28.334058 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:28:28.334017    4892 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/llmisvc-controller-manager-6466c554d8-q6d8c"]
Oct 30 19:28:28.336669 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:28:28.336643    4892 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/llmisvc-controller-manager-6466c554d8-q6d8c"
Oct 30 19:28:28.381618 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:28:28.381593    4892 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/f826b6a0-802b-4275-9bad-f437ed79959a-cert\") pod \"llmisvc-controller-manager-6466c554d8-q6d8c\" (UID: \"f826b6a0-802b-4275-9bad-f437ed79959a\") " pod="opendatahub/llmisvc-controller-manager-6466c554d8-q6d8c"
Oct 30 19:28:28.381716 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:28:28.381621    4892 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-frbws\" (UniqueName: \"kubernetes.io/projected/f826b6a0-802b-4275-9bad-f437ed79959a-kube-api-access-frbws\") pod \"llmisvc-controller-manager-6466c554d8-q6d8c\" (UID: \"f826b6a0-802b-4275-9bad-f437ed79959a\") " pod="opendatahub/llmisvc-controller-manager-6466c554d8-q6d8c"
Oct 30 19:28:28.396215 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:28:28.396186    4892 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/llmisvc-controller-manager-6466c554d8-q6d8c"]
Oct 30 19:28:28.482943 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:28:28.482913    4892 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/f826b6a0-802b-4275-9bad-f437ed79959a-cert\") pod \"llmisvc-controller-manager-6466c554d8-q6d8c\" (UID: \"f826b6a0-802b-4275-9bad-f437ed79959a\") " pod="opendatahub/llmisvc-controller-manager-6466c554d8-q6d8c"
Oct 30 19:28:28.482943 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:28:28.482943    4892 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-frbws\" (UniqueName: \"kubernetes.io/projected/f826b6a0-802b-4275-9bad-f437ed79959a-kube-api-access-frbws\") pod \"llmisvc-controller-manager-6466c554d8-q6d8c\" (UID: \"f826b6a0-802b-4275-9bad-f437ed79959a\") " pod="opendatahub/llmisvc-controller-manager-6466c554d8-q6d8c"
Oct 30 19:28:28.485625 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:28:28.485600    4892 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/f826b6a0-802b-4275-9bad-f437ed79959a-cert\") pod \"llmisvc-controller-manager-6466c554d8-q6d8c\" (UID: \"f826b6a0-802b-4275-9bad-f437ed79959a\") " pod="opendatahub/llmisvc-controller-manager-6466c554d8-q6d8c"
Oct 30 19:28:28.539745 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:28:28.539714    4892 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-frbws\" (UniqueName: \"kubernetes.io/projected/f826b6a0-802b-4275-9bad-f437ed79959a-kube-api-access-frbws\") pod \"llmisvc-controller-manager-6466c554d8-q6d8c\" (UID: \"f826b6a0-802b-4275-9bad-f437ed79959a\") " pod="opendatahub/llmisvc-controller-manager-6466c554d8-q6d8c"
Oct 30 19:28:28.648306 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:28:28.648278    4892 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/llmisvc-controller-manager-6466c554d8-q6d8c"
Oct 30 19:28:29.016823 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:28:29.016798    4892 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/llmisvc-controller-manager-6466c554d8-q6d8c"]
Oct 30 19:28:29.019109 ip-10-0-132-82 kubenswrapper[4892]: W1030 19:28:29.019075    4892 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf826b6a0_802b_4275_9bad_f437ed79959a.slice/crio-50afac8c1c09017c6682850017c3f824565be60bd5926d4a61dd40ad8f86248d WatchSource:0}: Error finding container 50afac8c1c09017c6682850017c3f824565be60bd5926d4a61dd40ad8f86248d: Status 404 returned error can't find the container with id 50afac8c1c09017c6682850017c3f824565be60bd5926d4a61dd40ad8f86248d
Oct 30 19:28:29.020781 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:28:29.020762    4892 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Oct 30 19:28:29.040341 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:28:29.040316    4892 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/llmisvc-controller-manager-6466c554d8-q6d8c" event={"ID":"f826b6a0-802b-4275-9bad-f437ed79959a","Type":"ContainerStarted","Data":"50afac8c1c09017c6682850017c3f824565be60bd5926d4a61dd40ad8f86248d"}
Oct 30 19:28:30.047035 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:28:30.047003    4892 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/llmisvc-controller-manager-6466c554d8-q6d8c" event={"ID":"f826b6a0-802b-4275-9bad-f437ed79959a","Type":"ContainerStarted","Data":"aff2d5dd8d4af4d6634ba3be5ce5bfcf6921a3377363d97e67d6e0bfeaac2a44"}
Oct 30 19:28:30.047425 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:28:30.047097    4892 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/llmisvc-controller-manager-6466c554d8-q6d8c"
Oct 30 19:28:30.162233 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:28:30.162171    4892 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/llmisvc-controller-manager-6466c554d8-q6d8c" podStartSLOduration=1.364937875 podStartE2EDuration="2.16215462s" totalImagesPullingTime="797.216745ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:28:28 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:28:30.15958091 +0000 UTC m=+757.730046491" watchObservedRunningTime="2026-10-30 19:28:30.16215462 +0000 UTC m=+757.732620218"
Oct 30 19:28:54.842887 ip-10-0-132-82 kubenswrapper[4892]: E1030 19:28:54.842845    4892 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-jrm87" containerName="extractor"
Oct 30 19:29:01.054371 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:29:01.054333    4892 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/llmisvc-controller-manager-6466c554d8-q6d8c"
Oct 30 19:29:01.220111 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:29:01.220080    4892 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["opendatahub/llmisvc-controller-manager-869d5dcc5d-sgzp8"]
Oct 30 19:29:01.220314 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:29:01.220275    4892 kuberuntime_container.go:898] "Killing container with a grace period" pod="opendatahub/llmisvc-controller-manager-869d5dcc5d-sgzp8" podUID="aa023ebb-d28a-4872-af0b-8ce5ed545078" containerName="manager" containerID="cri-o://a9ecc7d98760a5e781be8ff8477c1bf8d60104b2ef2e27580ea30ca25b718787" gracePeriod=30
Oct 30 19:29:01.470899 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:29:01.470878    4892 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/llmisvc-controller-manager-869d5dcc5d-sgzp8"
Oct 30 19:29:01.635561 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:29:01.635541    4892 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/aa023ebb-d28a-4872-af0b-8ce5ed545078-kube-api-access-jmgmg\" (UniqueName: \"kubernetes.io/projected/aa023ebb-d28a-4872-af0b-8ce5ed545078-kube-api-access-jmgmg\") pod \"aa023ebb-d28a-4872-af0b-8ce5ed545078\" (UID: \"aa023ebb-d28a-4872-af0b-8ce5ed545078\") "
Oct 30 19:29:01.635725 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:29:01.635598    4892 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/aa023ebb-d28a-4872-af0b-8ce5ed545078-cert\" (UniqueName: \"kubernetes.io/secret/aa023ebb-d28a-4872-af0b-8ce5ed545078-cert\") pod \"aa023ebb-d28a-4872-af0b-8ce5ed545078\" (UID: \"aa023ebb-d28a-4872-af0b-8ce5ed545078\") "
Oct 30 19:29:01.637776 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:29:01.637752    4892 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/aa023ebb-d28a-4872-af0b-8ce5ed545078-cert" pod "aa023ebb-d28a-4872-af0b-8ce5ed545078" (UID: "aa023ebb-d28a-4872-af0b-8ce5ed545078"). InnerVolumeSpecName "cert". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 19:29:01.637872 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:29:01.637845    4892 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/aa023ebb-d28a-4872-af0b-8ce5ed545078-kube-api-access-jmgmg" pod "aa023ebb-d28a-4872-af0b-8ce5ed545078" (UID: "aa023ebb-d28a-4872-af0b-8ce5ed545078"). InnerVolumeSpecName "kube-api-access-jmgmg". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:29:01.736175 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:29:01.736148    4892 reconciler_common.go:299] "Volume detached for volume \"cert\" (UniqueName: \"kubernetes.io/secret/aa023ebb-d28a-4872-af0b-8ce5ed545078-cert\") on node \"ip-10-0-132-82.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 19:29:01.736175 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:29:01.736172    4892 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jmgmg\" (UniqueName: \"kubernetes.io/projected/aa023ebb-d28a-4872-af0b-8ce5ed545078-kube-api-access-jmgmg\") on node \"ip-10-0-132-82.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 19:29:02.202788 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:29:02.202748    4892 generic.go:356] "Generic (PLEG): container finished" podID="aa023ebb-d28a-4872-af0b-8ce5ed545078" containerID="a9ecc7d98760a5e781be8ff8477c1bf8d60104b2ef2e27580ea30ca25b718787" exitCode=0
Oct 30 19:29:02.202788 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:29:02.202786    4892 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/llmisvc-controller-manager-869d5dcc5d-sgzp8"
Oct 30 19:29:02.203300 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:29:02.202850    4892 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/llmisvc-controller-manager-869d5dcc5d-sgzp8" event={"ID":"aa023ebb-d28a-4872-af0b-8ce5ed545078","Type":"ContainerDied","Data":"a9ecc7d98760a5e781be8ff8477c1bf8d60104b2ef2e27580ea30ca25b718787"}
Oct 30 19:29:02.203300 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:29:02.202900    4892 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/llmisvc-controller-manager-869d5dcc5d-sgzp8" event={"ID":"aa023ebb-d28a-4872-af0b-8ce5ed545078","Type":"ContainerDied","Data":"cad7e95b6a3ae6bb3ef1fb55928c448caf0010d50197d198d5bceb86918acfc6"}
Oct 30 19:29:02.203300 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:29:02.202932    4892 scope.go:122] "RemoveContainer" containerID="a9ecc7d98760a5e781be8ff8477c1bf8d60104b2ef2e27580ea30ca25b718787"
Oct 30 19:29:02.212487 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:29:02.212467    4892 scope.go:122] "RemoveContainer" containerID="a9ecc7d98760a5e781be8ff8477c1bf8d60104b2ef2e27580ea30ca25b718787"
Oct 30 19:29:02.212737 ip-10-0-132-82 kubenswrapper[4892]: E1030 19:29:02.212719    4892 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a9ecc7d98760a5e781be8ff8477c1bf8d60104b2ef2e27580ea30ca25b718787\": container with ID starting with a9ecc7d98760a5e781be8ff8477c1bf8d60104b2ef2e27580ea30ca25b718787 not found: ID does not exist" containerID="a9ecc7d98760a5e781be8ff8477c1bf8d60104b2ef2e27580ea30ca25b718787"
Oct 30 19:29:02.212790 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:29:02.212746    4892 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a9ecc7d98760a5e781be8ff8477c1bf8d60104b2ef2e27580ea30ca25b718787"} err="failed to get container status \"a9ecc7d98760a5e781be8ff8477c1bf8d60104b2ef2e27580ea30ca25b718787\": rpc error: code = NotFound desc = could not find container \"a9ecc7d98760a5e781be8ff8477c1bf8d60104b2ef2e27580ea30ca25b718787\": container with ID starting with a9ecc7d98760a5e781be8ff8477c1bf8d60104b2ef2e27580ea30ca25b718787 not found: ID does not exist"
Oct 30 19:29:02.233289 ip-10-0-132-82 kubenswrapper[4892]: E1030 19:29:02.233263    4892 cadvisor_stats_provider.go:569] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-podaa023ebb_d28a_4872_af0b_8ce5ed545078.slice/crio-cad7e95b6a3ae6bb3ef1fb55928c448caf0010d50197d198d5bceb86918acfc6\": RecentStats: unable to find data in memory cache], [\"/kubepods.slice/kubepods-podaa023ebb_d28a_4872_af0b_8ce5ed545078.slice\": RecentStats: unable to find data in memory cache]"
Oct 30 19:29:02.365376 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:29:02.365328    4892 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["opendatahub/llmisvc-controller-manager-869d5dcc5d-sgzp8"]
Oct 30 19:29:02.414540 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:29:02.414504    4892 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["opendatahub/llmisvc-controller-manager-869d5dcc5d-sgzp8"]
Oct 30 19:29:02.568691 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:29:02.568617    4892 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="aa023ebb-d28a-4872-af0b-8ce5ed545078" path="/var/lib/kubelet/pods/aa023ebb-d28a-4872-af0b-8ce5ed545078/volumes"
Oct 30 19:29:24.770233 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:29:24.770191    4892 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-fsfvk"]
Oct 30 19:29:24.843801 ip-10-0-132-82 kubenswrapper[4892]: E1030 19:29:24.843761    4892 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-jrm87" containerName="extractor"
Oct 30 19:29:34.740947 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:29:34.740910    4892 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-fsfvk"]
Oct 30 19:29:46.354524 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:29:46.354481    4892 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-fsfvk"]
Oct 30 19:29:54.843313 ip-10-0-132-82 kubenswrapper[4892]: E1030 19:29:54.843258    4892 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-jrm87" containerName="extractor"
Oct 30 19:29:56.965707 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:29:56.965673    4892 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-fsfvk"]
Oct 30 19:30:10.878851 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:30:10.878774    4892 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-xqnps"]
Oct 30 19:30:10.879338 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:30:10.879120    4892 cpu_manager.go:414] "containerMap: removing container" podUID="aa023ebb-d28a-4872-af0b-8ce5ed545078" containerName="manager"
Oct 30 19:30:10.879338 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:30:10.879133    4892 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="aa023ebb-d28a-4872-af0b-8ce5ed545078" containerName="manager"
Oct 30 19:30:10.879338 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:30:10.879257    4892 memory_manager.go:365] "RemoveStaleState removing state" podUID="aa023ebb-d28a-4872-af0b-8ce5ed545078" containerName="manager"
Oct 30 19:30:10.882298 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:30:10.882278    4892 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-xqnps"
Oct 30 19:30:10.985380 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:30:10.985350    4892 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-xqnps"]
Oct 30 19:30:10.993841 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:30:10.993812    4892 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4ac820b6-1c82-4e99-9a96-59ed74259ae7-istio-token\") pod \"maas-default-gateway-openshift-default-6db586dd6c-xqnps\" (UID: \"4ac820b6-1c82-4e99-9a96-59ed74259ae7\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-xqnps"
Oct 30 19:30:10.993977 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:30:10.993861    4892 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4ac820b6-1c82-4e99-9a96-59ed74259ae7-istio-data\") pod \"maas-default-gateway-openshift-default-6db586dd6c-xqnps\" (UID: \"4ac820b6-1c82-4e99-9a96-59ed74259ae7\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-xqnps"
Oct 30 19:30:10.993977 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:30:10.993918    4892 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4ac820b6-1c82-4e99-9a96-59ed74259ae7-workload-socket\") pod \"maas-default-gateway-openshift-default-6db586dd6c-xqnps\" (UID: \"4ac820b6-1c82-4e99-9a96-59ed74259ae7\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-xqnps"
Oct 30 19:30:10.994177 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:30:10.994151    4892 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4ac820b6-1c82-4e99-9a96-59ed74259ae7-istio-podinfo\") pod \"maas-default-gateway-openshift-default-6db586dd6c-xqnps\" (UID: \"4ac820b6-1c82-4e99-9a96-59ed74259ae7\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-xqnps"
Oct 30 19:30:10.994464 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:30:10.994203    4892 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jfrhb\" (UniqueName: \"kubernetes.io/projected/4ac820b6-1c82-4e99-9a96-59ed74259ae7-kube-api-access-jfrhb\") pod \"maas-default-gateway-openshift-default-6db586dd6c-xqnps\" (UID: \"4ac820b6-1c82-4e99-9a96-59ed74259ae7\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-xqnps"
Oct 30 19:30:10.994464 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:30:10.994236    4892 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4ac820b6-1c82-4e99-9a96-59ed74259ae7-credential-socket\") pod \"maas-default-gateway-openshift-default-6db586dd6c-xqnps\" (UID: \"4ac820b6-1c82-4e99-9a96-59ed74259ae7\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-xqnps"
Oct 30 19:30:10.994464 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:30:10.994294    4892 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4ac820b6-1c82-4e99-9a96-59ed74259ae7-istio-envoy\") pod \"maas-default-gateway-openshift-default-6db586dd6c-xqnps\" (UID: \"4ac820b6-1c82-4e99-9a96-59ed74259ae7\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-xqnps"
Oct 30 19:30:10.994464 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:30:10.994363    4892 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4ac820b6-1c82-4e99-9a96-59ed74259ae7-workload-certs\") pod \"maas-default-gateway-openshift-default-6db586dd6c-xqnps\" (UID: \"4ac820b6-1c82-4e99-9a96-59ed74259ae7\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-xqnps"
Oct 30 19:30:10.994464 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:30:10.994410    4892 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4ac820b6-1c82-4e99-9a96-59ed74259ae7-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-6db586dd6c-xqnps\" (UID: \"4ac820b6-1c82-4e99-9a96-59ed74259ae7\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-xqnps"
Oct 30 19:30:11.095583 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:30:11.095549    4892 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4ac820b6-1c82-4e99-9a96-59ed74259ae7-istio-token\") pod \"maas-default-gateway-openshift-default-6db586dd6c-xqnps\" (UID: \"4ac820b6-1c82-4e99-9a96-59ed74259ae7\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-xqnps"
Oct 30 19:30:11.095743 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:30:11.095590    4892 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4ac820b6-1c82-4e99-9a96-59ed74259ae7-istio-data\") pod \"maas-default-gateway-openshift-default-6db586dd6c-xqnps\" (UID: \"4ac820b6-1c82-4e99-9a96-59ed74259ae7\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-xqnps"
Oct 30 19:30:11.095743 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:30:11.095712    4892 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4ac820b6-1c82-4e99-9a96-59ed74259ae7-workload-socket\") pod \"maas-default-gateway-openshift-default-6db586dd6c-xqnps\" (UID: \"4ac820b6-1c82-4e99-9a96-59ed74259ae7\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-xqnps"
Oct 30 19:30:11.095899 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:30:11.095757    4892 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4ac820b6-1c82-4e99-9a96-59ed74259ae7-istio-podinfo\") pod \"maas-default-gateway-openshift-default-6db586dd6c-xqnps\" (UID: \"4ac820b6-1c82-4e99-9a96-59ed74259ae7\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-xqnps"
Oct 30 19:30:11.095899 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:30:11.095781    4892 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jfrhb\" (UniqueName: \"kubernetes.io/projected/4ac820b6-1c82-4e99-9a96-59ed74259ae7-kube-api-access-jfrhb\") pod \"maas-default-gateway-openshift-default-6db586dd6c-xqnps\" (UID: \"4ac820b6-1c82-4e99-9a96-59ed74259ae7\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-xqnps"
Oct 30 19:30:11.095899 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:30:11.095799    4892 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4ac820b6-1c82-4e99-9a96-59ed74259ae7-credential-socket\") pod \"maas-default-gateway-openshift-default-6db586dd6c-xqnps\" (UID: \"4ac820b6-1c82-4e99-9a96-59ed74259ae7\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-xqnps"
Oct 30 19:30:11.095899 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:30:11.095826    4892 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4ac820b6-1c82-4e99-9a96-59ed74259ae7-istio-envoy\") pod \"maas-default-gateway-openshift-default-6db586dd6c-xqnps\" (UID: \"4ac820b6-1c82-4e99-9a96-59ed74259ae7\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-xqnps"
Oct 30 19:30:11.095899 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:30:11.095848    4892 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4ac820b6-1c82-4e99-9a96-59ed74259ae7-workload-certs\") pod \"maas-default-gateway-openshift-default-6db586dd6c-xqnps\" (UID: \"4ac820b6-1c82-4e99-9a96-59ed74259ae7\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-xqnps"
Oct 30 19:30:11.096187 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:30:11.095902    4892 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4ac820b6-1c82-4e99-9a96-59ed74259ae7-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-6db586dd6c-xqnps\" (UID: \"4ac820b6-1c82-4e99-9a96-59ed74259ae7\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-xqnps"
Oct 30 19:30:11.096187 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:30:11.095993    4892 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4ac820b6-1c82-4e99-9a96-59ed74259ae7-istio-data\") pod \"maas-default-gateway-openshift-default-6db586dd6c-xqnps\" (UID: \"4ac820b6-1c82-4e99-9a96-59ed74259ae7\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-xqnps"
Oct 30 19:30:11.096187 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:30:11.096096    4892 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4ac820b6-1c82-4e99-9a96-59ed74259ae7-workload-socket\") pod \"maas-default-gateway-openshift-default-6db586dd6c-xqnps\" (UID: \"4ac820b6-1c82-4e99-9a96-59ed74259ae7\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-xqnps"
Oct 30 19:30:11.096187 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:30:11.096155    4892 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4ac820b6-1c82-4e99-9a96-59ed74259ae7-credential-socket\") pod \"maas-default-gateway-openshift-default-6db586dd6c-xqnps\" (UID: \"4ac820b6-1c82-4e99-9a96-59ed74259ae7\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-xqnps"
Oct 30 19:30:11.096401 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:30:11.096291    4892 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4ac820b6-1c82-4e99-9a96-59ed74259ae7-workload-certs\") pod \"maas-default-gateway-openshift-default-6db586dd6c-xqnps\" (UID: \"4ac820b6-1c82-4e99-9a96-59ed74259ae7\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-xqnps"
Oct 30 19:30:11.096591 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:30:11.096572    4892 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4ac820b6-1c82-4e99-9a96-59ed74259ae7-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-6db586dd6c-xqnps\" (UID: \"4ac820b6-1c82-4e99-9a96-59ed74259ae7\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-xqnps"
Oct 30 19:30:11.098164 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:30:11.098142    4892 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4ac820b6-1c82-4e99-9a96-59ed74259ae7-istio-envoy\") pod \"maas-default-gateway-openshift-default-6db586dd6c-xqnps\" (UID: \"4ac820b6-1c82-4e99-9a96-59ed74259ae7\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-xqnps"
Oct 30 19:30:11.098356 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:30:11.098340    4892 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4ac820b6-1c82-4e99-9a96-59ed74259ae7-istio-podinfo\") pod \"maas-default-gateway-openshift-default-6db586dd6c-xqnps\" (UID: \"4ac820b6-1c82-4e99-9a96-59ed74259ae7\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-xqnps"
Oct 30 19:30:11.152413 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:30:11.152394    4892 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jfrhb\" (UniqueName: \"kubernetes.io/projected/4ac820b6-1c82-4e99-9a96-59ed74259ae7-kube-api-access-jfrhb\") pod \"maas-default-gateway-openshift-default-6db586dd6c-xqnps\" (UID: \"4ac820b6-1c82-4e99-9a96-59ed74259ae7\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-xqnps"
Oct 30 19:30:11.152645 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:30:11.152627    4892 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4ac820b6-1c82-4e99-9a96-59ed74259ae7-istio-token\") pod \"maas-default-gateway-openshift-default-6db586dd6c-xqnps\" (UID: \"4ac820b6-1c82-4e99-9a96-59ed74259ae7\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-xqnps"
Oct 30 19:30:11.212819 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:30:11.212797    4892 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-xqnps"
Oct 30 19:30:11.636474 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:30:11.636446    4892 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-xqnps"]
Oct 30 19:30:11.639580 ip-10-0-132-82 kubenswrapper[4892]: W1030 19:30:11.639554    4892 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4ac820b6_1c82_4e99_9a96_59ed74259ae7.slice/crio-9ab21d3855dae5b7859fb614f4f3a92d864d4214a0eca491e894c343eb11bd24 WatchSource:0}: Error finding container 9ab21d3855dae5b7859fb614f4f3a92d864d4214a0eca491e894c343eb11bd24: Status 404 returned error can't find the container with id 9ab21d3855dae5b7859fb614f4f3a92d864d4214a0eca491e894c343eb11bd24
Oct 30 19:30:11.641606 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:30:11.641572    4892 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892000Ki","pods":"250"}
Oct 30 19:30:11.641689 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:30:11.641658    4892 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892000Ki","pods":"250"}
Oct 30 19:30:11.641733 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:30:11.641689    4892 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892000Ki","pods":"250"}
Oct 30 19:30:12.574827 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:30:12.574791    4892 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-xqnps" event={"ID":"4ac820b6-1c82-4e99-9a96-59ed74259ae7","Type":"ContainerStarted","Data":"0faea4937790e22d09d64392601c213319461aa958c6b746d3c018482a657cb8"}
Oct 30 19:30:12.575252 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:30:12.574839    4892 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-xqnps" event={"ID":"4ac820b6-1c82-4e99-9a96-59ed74259ae7","Type":"ContainerStarted","Data":"9ab21d3855dae5b7859fb614f4f3a92d864d4214a0eca491e894c343eb11bd24"}
Oct 30 19:30:12.766207 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:30:12.766160    4892 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-xqnps" podStartSLOduration=2.766145895 podStartE2EDuration="2.766145895s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:30:10 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:30:12.7653333 +0000 UTC m=+860.335798933" watchObservedRunningTime="2026-10-30 19:30:12.766145895 +0000 UTC m=+860.336611490"
Oct 30 19:30:13.213558 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:30:13.213520    4892 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-xqnps"
Oct 30 19:30:13.215014 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:30:13.214988    4892 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6db586dd6c-xqnps container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.49:15021/healthz/ready\": dial tcp 10.132.0.49:15021: connect: connection refused" start-of-body=
Oct 30 19:30:13.215141 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:30:13.215033    4892 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-xqnps" podUID="4ac820b6-1c82-4e99-9a96-59ed74259ae7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.49:15021/healthz/ready\": dial tcp 10.132.0.49:15021: connect: connection refused"
Oct 30 19:30:14.213795 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:30:14.213760    4892 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6db586dd6c-xqnps container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.49:15021/healthz/ready\": dial tcp 10.132.0.49:15021: connect: connection refused" start-of-body=
Oct 30 19:30:14.214228 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:30:14.213824    4892 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-xqnps" podUID="4ac820b6-1c82-4e99-9a96-59ed74259ae7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.49:15021/healthz/ready\": dial tcp 10.132.0.49:15021: connect: connection refused"
Oct 30 19:30:15.214183 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:30:15.214149    4892 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6db586dd6c-xqnps container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.49:15021/healthz/ready\": dial tcp 10.132.0.49:15021: connect: connection refused" start-of-body=
Oct 30 19:30:15.214607 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:30:15.214208    4892 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-xqnps" podUID="4ac820b6-1c82-4e99-9a96-59ed74259ae7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.49:15021/healthz/ready\": dial tcp 10.132.0.49:15021: connect: connection refused"
Oct 30 19:30:16.214130 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:30:16.214096    4892 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6db586dd6c-xqnps container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.49:15021/healthz/ready\": dial tcp 10.132.0.49:15021: connect: connection refused" start-of-body=
Oct 30 19:30:16.214389 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:30:16.214151    4892 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-xqnps" podUID="4ac820b6-1c82-4e99-9a96-59ed74259ae7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.49:15021/healthz/ready\": dial tcp 10.132.0.49:15021: connect: connection refused"
Oct 30 19:30:17.214134 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:30:17.214099    4892 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6db586dd6c-xqnps container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.49:15021/healthz/ready\": dial tcp 10.132.0.49:15021: connect: connection refused" start-of-body=
Oct 30 19:30:17.214302 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:30:17.214148    4892 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-xqnps" podUID="4ac820b6-1c82-4e99-9a96-59ed74259ae7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.49:15021/healthz/ready\": dial tcp 10.132.0.49:15021: connect: connection refused"
Oct 30 19:30:18.213525 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:30:18.213486    4892 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6db586dd6c-xqnps container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.49:15021/healthz/ready\": dial tcp 10.132.0.49:15021: connect: connection refused" start-of-body=
Oct 30 19:30:18.213916 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:30:18.213553    4892 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-xqnps" podUID="4ac820b6-1c82-4e99-9a96-59ed74259ae7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.49:15021/healthz/ready\": dial tcp 10.132.0.49:15021: connect: connection refused"
Oct 30 19:30:19.858888 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:30:19.858857    4892 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-xqnps"
Oct 30 19:30:19.859432 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:30:19.859265    4892 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-xqnps"
Oct 30 19:30:19.860121 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:30:19.860103    4892 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-xqnps"
Oct 30 19:30:24.844599 ip-10-0-132-82 kubenswrapper[4892]: E1030 19:30:24.844549    4892 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-jrm87" containerName="extractor"
Oct 30 19:30:52.673796 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:30:52.673764    4892 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-tg5xj_11db9cec-653f-4aee-8937-0cf5b367efac/console-operator/3.log"
Oct 30 19:30:52.676318 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:30:52.676287    4892 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-tg5xj_11db9cec-653f-4aee-8937-0cf5b367efac/console-operator/3.log"
Oct 30 19:30:54.843413 ip-10-0-132-82 kubenswrapper[4892]: E1030 19:30:54.843378    4892 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-jrm87" containerName="extractor"
Oct 30 19:30:56.012808 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:30:56.012768    4892 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-7xzjb"]
Oct 30 19:30:56.016361 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:30:56.016337    4892 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-7xzjb"
Oct 30 19:30:56.058253 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:30:56.058227    4892 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/cbfb2293-8265-4bf9-bcab-68351ec3b9ce-workload-socket\") pod \"maas-default-gateway-openshift-default-6db586dd6c-7xzjb\" (UID: \"cbfb2293-8265-4bf9-bcab-68351ec3b9ce\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-7xzjb"
Oct 30 19:30:56.058380 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:30:56.058261    4892 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/cbfb2293-8265-4bf9-bcab-68351ec3b9ce-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-6db586dd6c-7xzjb\" (UID: \"cbfb2293-8265-4bf9-bcab-68351ec3b9ce\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-7xzjb"
Oct 30 19:30:56.058380 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:30:56.058280    4892 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/cbfb2293-8265-4bf9-bcab-68351ec3b9ce-credential-socket\") pod \"maas-default-gateway-openshift-default-6db586dd6c-7xzjb\" (UID: \"cbfb2293-8265-4bf9-bcab-68351ec3b9ce\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-7xzjb"
Oct 30 19:30:56.058450 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:30:56.058384    4892 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/cbfb2293-8265-4bf9-bcab-68351ec3b9ce-istio-data\") pod \"maas-default-gateway-openshift-default-6db586dd6c-7xzjb\" (UID: \"cbfb2293-8265-4bf9-bcab-68351ec3b9ce\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-7xzjb"
Oct 30 19:30:56.058450 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:30:56.058407    4892 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/cbfb2293-8265-4bf9-bcab-68351ec3b9ce-istio-podinfo\") pod \"maas-default-gateway-openshift-default-6db586dd6c-7xzjb\" (UID: \"cbfb2293-8265-4bf9-bcab-68351ec3b9ce\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-7xzjb"
Oct 30 19:30:56.058450 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:30:56.058424    4892 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/cbfb2293-8265-4bf9-bcab-68351ec3b9ce-istio-envoy\") pod \"maas-default-gateway-openshift-default-6db586dd6c-7xzjb\" (UID: \"cbfb2293-8265-4bf9-bcab-68351ec3b9ce\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-7xzjb"
Oct 30 19:30:56.058450 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:30:56.058437    4892 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/cbfb2293-8265-4bf9-bcab-68351ec3b9ce-istio-token\") pod \"maas-default-gateway-openshift-default-6db586dd6c-7xzjb\" (UID: \"cbfb2293-8265-4bf9-bcab-68351ec3b9ce\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-7xzjb"
Oct 30 19:30:56.058595 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:30:56.058542    4892 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/cbfb2293-8265-4bf9-bcab-68351ec3b9ce-workload-certs\") pod \"maas-default-gateway-openshift-default-6db586dd6c-7xzjb\" (UID: \"cbfb2293-8265-4bf9-bcab-68351ec3b9ce\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-7xzjb"
Oct 30 19:30:56.058595 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:30:56.058562    4892 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rv7sf\" (UniqueName: \"kubernetes.io/projected/cbfb2293-8265-4bf9-bcab-68351ec3b9ce-kube-api-access-rv7sf\") pod \"maas-default-gateway-openshift-default-6db586dd6c-7xzjb\" (UID: \"cbfb2293-8265-4bf9-bcab-68351ec3b9ce\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-7xzjb"
Oct 30 19:30:56.159810 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:30:56.159778    4892 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/cbfb2293-8265-4bf9-bcab-68351ec3b9ce-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-6db586dd6c-7xzjb\" (UID: \"cbfb2293-8265-4bf9-bcab-68351ec3b9ce\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-7xzjb"
Oct 30 19:30:56.159993 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:30:56.159812    4892 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/cbfb2293-8265-4bf9-bcab-68351ec3b9ce-credential-socket\") pod \"maas-default-gateway-openshift-default-6db586dd6c-7xzjb\" (UID: \"cbfb2293-8265-4bf9-bcab-68351ec3b9ce\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-7xzjb"
Oct 30 19:30:56.159993 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:30:56.159859    4892 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/cbfb2293-8265-4bf9-bcab-68351ec3b9ce-istio-data\") pod \"maas-default-gateway-openshift-default-6db586dd6c-7xzjb\" (UID: \"cbfb2293-8265-4bf9-bcab-68351ec3b9ce\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-7xzjb"
Oct 30 19:30:56.159993 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:30:56.159878    4892 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/cbfb2293-8265-4bf9-bcab-68351ec3b9ce-istio-podinfo\") pod \"maas-default-gateway-openshift-default-6db586dd6c-7xzjb\" (UID: \"cbfb2293-8265-4bf9-bcab-68351ec3b9ce\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-7xzjb"
Oct 30 19:30:56.159993 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:30:56.159895    4892 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/cbfb2293-8265-4bf9-bcab-68351ec3b9ce-istio-envoy\") pod \"maas-default-gateway-openshift-default-6db586dd6c-7xzjb\" (UID: \"cbfb2293-8265-4bf9-bcab-68351ec3b9ce\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-7xzjb"
Oct 30 19:30:56.159993 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:30:56.159910    4892 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/cbfb2293-8265-4bf9-bcab-68351ec3b9ce-istio-token\") pod \"maas-default-gateway-openshift-default-6db586dd6c-7xzjb\" (UID: \"cbfb2293-8265-4bf9-bcab-68351ec3b9ce\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-7xzjb"
Oct 30 19:30:56.160211 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:30:56.160003    4892 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/cbfb2293-8265-4bf9-bcab-68351ec3b9ce-workload-certs\") pod \"maas-default-gateway-openshift-default-6db586dd6c-7xzjb\" (UID: \"cbfb2293-8265-4bf9-bcab-68351ec3b9ce\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-7xzjb"
Oct 30 19:30:56.160211 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:30:56.160126    4892 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rv7sf\" (UniqueName: \"kubernetes.io/projected/cbfb2293-8265-4bf9-bcab-68351ec3b9ce-kube-api-access-rv7sf\") pod \"maas-default-gateway-openshift-default-6db586dd6c-7xzjb\" (UID: \"cbfb2293-8265-4bf9-bcab-68351ec3b9ce\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-7xzjb"
Oct 30 19:30:56.160211 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:30:56.160198    4892 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/cbfb2293-8265-4bf9-bcab-68351ec3b9ce-workload-socket\") pod \"maas-default-gateway-openshift-default-6db586dd6c-7xzjb\" (UID: \"cbfb2293-8265-4bf9-bcab-68351ec3b9ce\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-7xzjb"
Oct 30 19:30:56.160669 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:30:56.160648    4892 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/cbfb2293-8265-4bf9-bcab-68351ec3b9ce-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-6db586dd6c-7xzjb\" (UID: \"cbfb2293-8265-4bf9-bcab-68351ec3b9ce\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-7xzjb"
Oct 30 19:30:56.160823 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:30:56.160805    4892 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/cbfb2293-8265-4bf9-bcab-68351ec3b9ce-credential-socket\") pod \"maas-default-gateway-openshift-default-6db586dd6c-7xzjb\" (UID: \"cbfb2293-8265-4bf9-bcab-68351ec3b9ce\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-7xzjb"
Oct 30 19:30:56.160897 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:30:56.160804    4892 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/cbfb2293-8265-4bf9-bcab-68351ec3b9ce-istio-data\") pod \"maas-default-gateway-openshift-default-6db586dd6c-7xzjb\" (UID: \"cbfb2293-8265-4bf9-bcab-68351ec3b9ce\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-7xzjb"
Oct 30 19:30:56.160941 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:30:56.160920    4892 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/cbfb2293-8265-4bf9-bcab-68351ec3b9ce-workload-certs\") pod \"maas-default-gateway-openshift-default-6db586dd6c-7xzjb\" (UID: \"cbfb2293-8265-4bf9-bcab-68351ec3b9ce\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-7xzjb"
Oct 30 19:30:56.160941 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:30:56.160928    4892 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/cbfb2293-8265-4bf9-bcab-68351ec3b9ce-workload-socket\") pod \"maas-default-gateway-openshift-default-6db586dd6c-7xzjb\" (UID: \"cbfb2293-8265-4bf9-bcab-68351ec3b9ce\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-7xzjb"
Oct 30 19:30:56.162862 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:30:56.162842    4892 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/cbfb2293-8265-4bf9-bcab-68351ec3b9ce-istio-envoy\") pod \"maas-default-gateway-openshift-default-6db586dd6c-7xzjb\" (UID: \"cbfb2293-8265-4bf9-bcab-68351ec3b9ce\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-7xzjb"
Oct 30 19:30:56.163026 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:30:56.163009    4892 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/cbfb2293-8265-4bf9-bcab-68351ec3b9ce-istio-podinfo\") pod \"maas-default-gateway-openshift-default-6db586dd6c-7xzjb\" (UID: \"cbfb2293-8265-4bf9-bcab-68351ec3b9ce\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-7xzjb"
Oct 30 19:30:56.213375 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:30:56.213353    4892 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-7xzjb"]
Oct 30 19:30:56.221953 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:30:56.221928    4892 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rv7sf\" (UniqueName: \"kubernetes.io/projected/cbfb2293-8265-4bf9-bcab-68351ec3b9ce-kube-api-access-rv7sf\") pod \"maas-default-gateway-openshift-default-6db586dd6c-7xzjb\" (UID: \"cbfb2293-8265-4bf9-bcab-68351ec3b9ce\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-7xzjb"
Oct 30 19:30:56.225769 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:30:56.225748    4892 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/cbfb2293-8265-4bf9-bcab-68351ec3b9ce-istio-token\") pod \"maas-default-gateway-openshift-default-6db586dd6c-7xzjb\" (UID: \"cbfb2293-8265-4bf9-bcab-68351ec3b9ce\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-7xzjb"
Oct 30 19:30:56.328850 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:30:56.328781    4892 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-7xzjb"
Oct 30 19:30:56.751514 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:30:56.751491    4892 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-7xzjb"]
Oct 30 19:30:56.753486 ip-10-0-132-82 kubenswrapper[4892]: W1030 19:30:56.753462    4892 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcbfb2293_8265_4bf9_bcab_68351ec3b9ce.slice/crio-a837a2f159ef3e2a3b7d2869fe808b2ab4403344e740e176dddaf49d17d7675c WatchSource:0}: Error finding container a837a2f159ef3e2a3b7d2869fe808b2ab4403344e740e176dddaf49d17d7675c: Status 404 returned error can't find the container with id a837a2f159ef3e2a3b7d2869fe808b2ab4403344e740e176dddaf49d17d7675c
Oct 30 19:30:56.759001 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:30:56.756145    4892 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892000Ki","pods":"250"}
Oct 30 19:30:56.759001 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:30:56.756293    4892 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892000Ki","pods":"250"}
Oct 30 19:30:56.759001 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:30:56.756381    4892 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892000Ki","pods":"250"}
Oct 30 19:30:56.792301 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:30:56.792276    4892 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-7xzjb" event={"ID":"cbfb2293-8265-4bf9-bcab-68351ec3b9ce","Type":"ContainerStarted","Data":"a837a2f159ef3e2a3b7d2869fe808b2ab4403344e740e176dddaf49d17d7675c"}
Oct 30 19:30:57.800690 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:30:57.800652    4892 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-7xzjb" event={"ID":"cbfb2293-8265-4bf9-bcab-68351ec3b9ce","Type":"ContainerStarted","Data":"1be086ce255b0acc339d50ed3248afe67be3c89237a38b74d3b4dd0fd96fc436"}
Oct 30 19:30:57.999030 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:30:57.998952    4892 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-7xzjb" podStartSLOduration=2.998937456 podStartE2EDuration="2.998937456s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:30:55 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:30:57.997265505 +0000 UTC m=+905.567731104" watchObservedRunningTime="2026-10-30 19:30:57.998937456 +0000 UTC m=+905.569403055"
Oct 30 19:30:58.335995 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:30:58.333760    4892 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-7xzjb"
Oct 30 19:30:58.335995 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:30:58.333878    4892 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6db586dd6c-7xzjb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.50:15021/healthz/ready\": dial tcp 10.132.0.50:15021: connect: connection refused" start-of-body=
Oct 30 19:30:58.335995 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:30:58.333934    4892 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-7xzjb" podUID="cbfb2293-8265-4bf9-bcab-68351ec3b9ce" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.50:15021/healthz/ready\": dial tcp 10.132.0.50:15021: connect: connection refused"
Oct 30 19:30:59.331369 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:30:59.331332    4892 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6db586dd6c-7xzjb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.50:15021/healthz/ready\": dial tcp 10.132.0.50:15021: connect: connection refused" start-of-body=
Oct 30 19:30:59.332006 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:30:59.331948    4892 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-7xzjb" podUID="cbfb2293-8265-4bf9-bcab-68351ec3b9ce" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.50:15021/healthz/ready\": dial tcp 10.132.0.50:15021: connect: connection refused"
Oct 30 19:31:00.345401 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:31:00.345367    4892 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6db586dd6c-7xzjb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.50:15021/healthz/ready\": dial tcp 10.132.0.50:15021: connect: connection refused" start-of-body=
Oct 30 19:31:00.346025 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:31:00.345991    4892 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-7xzjb" podUID="cbfb2293-8265-4bf9-bcab-68351ec3b9ce" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.50:15021/healthz/ready\": dial tcp 10.132.0.50:15021: connect: connection refused"
Oct 30 19:31:01.336990 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:31:01.331380    4892 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6db586dd6c-7xzjb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.50:15021/healthz/ready\": dial tcp 10.132.0.50:15021: connect: connection refused" start-of-body=
Oct 30 19:31:01.336990 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:31:01.331441    4892 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-7xzjb" podUID="cbfb2293-8265-4bf9-bcab-68351ec3b9ce" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.50:15021/healthz/ready\": dial tcp 10.132.0.50:15021: connect: connection refused"
Oct 30 19:31:02.346989 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:31:02.332347    4892 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6db586dd6c-7xzjb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.50:15021/healthz/ready\": dial tcp 10.132.0.50:15021: connect: connection refused" start-of-body=
Oct 30 19:31:02.346989 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:31:02.332406    4892 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-7xzjb" podUID="cbfb2293-8265-4bf9-bcab-68351ec3b9ce" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.50:15021/healthz/ready\": dial tcp 10.132.0.50:15021: connect: connection refused"
Oct 30 19:31:03.329840 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:31:03.329805    4892 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6db586dd6c-7xzjb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.50:15021/healthz/ready\": dial tcp 10.132.0.50:15021: connect: connection refused" start-of-body=
Oct 30 19:31:03.330028 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:31:03.329871    4892 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-7xzjb" podUID="cbfb2293-8265-4bf9-bcab-68351ec3b9ce" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.50:15021/healthz/ready\": dial tcp 10.132.0.50:15021: connect: connection refused"
Oct 30 19:31:05.071655 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:31:05.071625    4892 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-7xzjb"
Oct 30 19:31:05.072172 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:31:05.072035    4892 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-7xzjb"
Oct 30 19:31:05.072821 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:31:05.072799    4892 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-7xzjb"
Oct 30 19:31:24.843270 ip-10-0-132-82 kubenswrapper[4892]: E1030 19:31:24.843232    4892 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-jrm87" containerName="extractor"
Oct 30 19:31:54.843548 ip-10-0-132-82 kubenswrapper[4892]: E1030 19:31:54.843509    4892 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-jrm87" containerName="extractor"
Oct 30 19:32:21.416876 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:32:21.416841    4892 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w6-5f55cd-85cb4d6b67-9qrfs"]
Oct 30 19:32:21.424025 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:32:21.421416    4892 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w6-5f55cd-85cb4d6b67-9qrfs"
Oct 30 19:32:21.476645 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:32:21.476613    4892 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-worker-w6-5f55cd\""
Oct 30 19:32:21.512543 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:32:21.512492    4892 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4286dc2a-9c31-454f-891d-31d896d7a66b-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w6-5f55cd-85cb4d6b67-9qrfs\" (UID: \"4286dc2a-9c31-454f-891d-31d896d7a66b\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w6-5f55cd-85cb4d6b67-9qrfs"
Oct 30 19:32:21.512735 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:32:21.512547    4892 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tf5gc\" (UniqueName: \"kubernetes.io/projected/4286dc2a-9c31-454f-891d-31d896d7a66b-kube-api-access-tf5gc\") pod \"payload-pre-processing-e2e-worker-w6-5f55cd-85cb4d6b67-9qrfs\" (UID: \"4286dc2a-9c31-454f-891d-31d896d7a66b\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w6-5f55cd-85cb4d6b67-9qrfs"
Oct 30 19:32:21.529399 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:32:21.529036    4892 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w6-5f55cd-85cb4d6b67-9qrfs"]
Oct 30 19:32:21.530297 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:32:21.530272    4892 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-worker-w6-5f55cd-dockercfg-7hqqw\""
Oct 30 19:32:21.579085 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:32:21.579060    4892 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w6-5f55cd-688df7c466-t26tq"]
Oct 30 19:32:21.582539 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:32:21.582521    4892 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w6-5f55cd-688df7c466-t26tq"
Oct 30 19:32:21.613572 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:32:21.613545    4892 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4286dc2a-9c31-454f-891d-31d896d7a66b-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w6-5f55cd-85cb4d6b67-9qrfs\" (UID: \"4286dc2a-9c31-454f-891d-31d896d7a66b\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w6-5f55cd-85cb4d6b67-9qrfs"
Oct 30 19:32:21.613672 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:32:21.613575    4892 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tf5gc\" (UniqueName: \"kubernetes.io/projected/4286dc2a-9c31-454f-891d-31d896d7a66b-kube-api-access-tf5gc\") pod \"payload-pre-processing-e2e-worker-w6-5f55cd-85cb4d6b67-9qrfs\" (UID: \"4286dc2a-9c31-454f-891d-31d896d7a66b\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w6-5f55cd-85cb4d6b67-9qrfs"
Oct 30 19:32:21.614237 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:32:21.614202    4892 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4286dc2a-9c31-454f-891d-31d896d7a66b-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w6-5f55cd-85cb4d6b67-9qrfs\" (UID: \"4286dc2a-9c31-454f-891d-31d896d7a66b\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w6-5f55cd-85cb4d6b67-9qrfs"
Oct 30 19:32:21.625845 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:32:21.625816    4892 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w6-5f55cd-688df7c466-t26tq"]
Oct 30 19:32:21.671014 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:32:21.670901    4892 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tf5gc\" (UniqueName: \"kubernetes.io/projected/4286dc2a-9c31-454f-891d-31d896d7a66b-kube-api-access-tf5gc\") pod \"payload-pre-processing-e2e-worker-w6-5f55cd-85cb4d6b67-9qrfs\" (UID: \"4286dc2a-9c31-454f-891d-31d896d7a66b\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w6-5f55cd-85cb4d6b67-9qrfs"
Oct 30 19:32:21.714788 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:32:21.714761    4892 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b31aad4b-7f5c-46c1-9c46-e39ea2ea5803-plugins-config-volume\") pod \"payload-processing-e2e-worker-w6-5f55cd-688df7c466-t26tq\" (UID: \"b31aad4b-7f5c-46c1-9c46-e39ea2ea5803\") " pod="openshift-ingress/payload-processing-e2e-worker-w6-5f55cd-688df7c466-t26tq"
Oct 30 19:32:21.714919 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:32:21.714800    4892 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-r6ksk\" (UniqueName: \"kubernetes.io/projected/b31aad4b-7f5c-46c1-9c46-e39ea2ea5803-kube-api-access-r6ksk\") pod \"payload-processing-e2e-worker-w6-5f55cd-688df7c466-t26tq\" (UID: \"b31aad4b-7f5c-46c1-9c46-e39ea2ea5803\") " pod="openshift-ingress/payload-processing-e2e-worker-w6-5f55cd-688df7c466-t26tq"
Oct 30 19:32:21.737053 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:32:21.737023    4892 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w6-5f55cd-85cb4d6b67-9qrfs"
Oct 30 19:32:21.816404 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:32:21.816373    4892 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b31aad4b-7f5c-46c1-9c46-e39ea2ea5803-plugins-config-volume\") pod \"payload-processing-e2e-worker-w6-5f55cd-688df7c466-t26tq\" (UID: \"b31aad4b-7f5c-46c1-9c46-e39ea2ea5803\") " pod="openshift-ingress/payload-processing-e2e-worker-w6-5f55cd-688df7c466-t26tq"
Oct 30 19:32:21.816685 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:32:21.816412    4892 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-r6ksk\" (UniqueName: \"kubernetes.io/projected/b31aad4b-7f5c-46c1-9c46-e39ea2ea5803-kube-api-access-r6ksk\") pod \"payload-processing-e2e-worker-w6-5f55cd-688df7c466-t26tq\" (UID: \"b31aad4b-7f5c-46c1-9c46-e39ea2ea5803\") " pod="openshift-ingress/payload-processing-e2e-worker-w6-5f55cd-688df7c466-t26tq"
Oct 30 19:32:21.817446 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:32:21.817387    4892 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b31aad4b-7f5c-46c1-9c46-e39ea2ea5803-plugins-config-volume\") pod \"payload-processing-e2e-worker-w6-5f55cd-688df7c466-t26tq\" (UID: \"b31aad4b-7f5c-46c1-9c46-e39ea2ea5803\") " pod="openshift-ingress/payload-processing-e2e-worker-w6-5f55cd-688df7c466-t26tq"
Oct 30 19:32:21.873652 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:32:21.873622    4892 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-r6ksk\" (UniqueName: \"kubernetes.io/projected/b31aad4b-7f5c-46c1-9c46-e39ea2ea5803-kube-api-access-r6ksk\") pod \"payload-processing-e2e-worker-w6-5f55cd-688df7c466-t26tq\" (UID: \"b31aad4b-7f5c-46c1-9c46-e39ea2ea5803\") " pod="openshift-ingress/payload-processing-e2e-worker-w6-5f55cd-688df7c466-t26tq"
Oct 30 19:32:21.896386 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:32:21.896361    4892 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w6-5f55cd-688df7c466-t26tq"
Oct 30 19:32:22.110893 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:32:22.110868    4892 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w6-5f55cd-85cb4d6b67-9qrfs"]
Oct 30 19:32:22.113126 ip-10-0-132-82 kubenswrapper[4892]: W1030 19:32:22.113098    4892 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4286dc2a_9c31_454f_891d_31d896d7a66b.slice/crio-bbfe9c659dd9e36206fe23a6f7202c8cd93eedd4c9f431539a77989a88aa290a WatchSource:0}: Error finding container bbfe9c659dd9e36206fe23a6f7202c8cd93eedd4c9f431539a77989a88aa290a: Status 404 returned error can't find the container with id bbfe9c659dd9e36206fe23a6f7202c8cd93eedd4c9f431539a77989a88aa290a
Oct 30 19:32:22.228081 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:32:22.228001    4892 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w6-5f55cd-85cb4d6b67-9qrfs" event={"ID":"4286dc2a-9c31-454f-891d-31d896d7a66b","Type":"ContainerStarted","Data":"bbfe9c659dd9e36206fe23a6f7202c8cd93eedd4c9f431539a77989a88aa290a"}
Oct 30 19:32:22.267070 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:32:22.267043    4892 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w6-5f55cd-688df7c466-t26tq"]
Oct 30 19:32:22.269469 ip-10-0-132-82 kubenswrapper[4892]: W1030 19:32:22.269446    4892 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb31aad4b_7f5c_46c1_9c46_e39ea2ea5803.slice/crio-3d65a5ff773b5255657dd74a7ec70775db634a7c3770f41b284fa644b47206fe WatchSource:0}: Error finding container 3d65a5ff773b5255657dd74a7ec70775db634a7c3770f41b284fa644b47206fe: Status 404 returned error can't find the container with id 3d65a5ff773b5255657dd74a7ec70775db634a7c3770f41b284fa644b47206fe
Oct 30 19:32:23.235771 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:32:23.235732    4892 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w6-5f55cd-688df7c466-t26tq" event={"ID":"b31aad4b-7f5c-46c1-9c46-e39ea2ea5803","Type":"ContainerStarted","Data":"3d65a5ff773b5255657dd74a7ec70775db634a7c3770f41b284fa644b47206fe"}
Oct 30 19:32:24.888377 ip-10-0-132-82 kubenswrapper[4892]: E1030 19:32:24.884358    4892 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-jrm87" containerName="extractor"
Oct 30 19:32:25.360806 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:32:25.360778    4892 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-fsfvk"]
Oct 30 19:32:26.262734 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:32:26.262692    4892 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w6-5f55cd-85cb4d6b67-9qrfs" event={"ID":"4286dc2a-9c31-454f-891d-31d896d7a66b","Type":"ContainerStarted","Data":"6ac8f7e45ada17a45ce0acac47bd7dc40934edee4d643c7161eeee66cb5817be"}
Oct 30 19:32:26.263222 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:32:26.262803    4892 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-worker-w6-5f55cd-85cb4d6b67-9qrfs"
Oct 30 19:32:26.264434 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:32:26.264405    4892 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w6-5f55cd-688df7c466-t26tq" event={"ID":"b31aad4b-7f5c-46c1-9c46-e39ea2ea5803","Type":"ContainerStarted","Data":"07f691a8d9b079f12d2fd30161803246153cf95492383e4e9879b569a667dd9e"}
Oct 30 19:32:26.264561 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:32:26.264469    4892 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-worker-w6-5f55cd-688df7c466-t26tq"
Oct 30 19:32:26.378626 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:32:26.378574    4892 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-worker-w6-5f55cd-85cb4d6b67-9qrfs" podStartSLOduration=2.15713806 podStartE2EDuration="5.378557418s" totalImagesPullingTime="3.221419358s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:32:21 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:32:26.376611083 +0000 UTC m=+993.947076694" watchObservedRunningTime="2026-10-30 19:32:26.378557418 +0000 UTC m=+993.949023017"
Oct 30 19:32:26.534928 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:32:26.534811    4892 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-worker-w6-5f55cd-688df7c466-t26tq" podStartSLOduration=2.590008128 podStartE2EDuration="5.53479142s" totalImagesPullingTime="2.944783292s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:32:21 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:32:26.534179301 +0000 UTC m=+994.104644912" watchObservedRunningTime="2026-10-30 19:32:26.53479142 +0000 UTC m=+994.105257020"
Oct 30 19:32:33.088053 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:32:33.088025    4892 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-fsfvk"]
Oct 30 19:32:36.871666 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:32:36.871626    4892 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w0-2d9861-78d7ffc559-lp7cm"]
Oct 30 19:32:36.875302 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:32:36.875279    4892 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-2d9861-78d7ffc559-lp7cm"
Oct 30 19:32:36.907155 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:32:36.907134    4892 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gll9v\" (UniqueName: \"kubernetes.io/projected/f1df6069-128b-4c8c-86ef-dae9bce19498-kube-api-access-gll9v\") pod \"payload-pre-processing-e2e-worker-w0-2d9861-78d7ffc559-lp7cm\" (UID: \"f1df6069-128b-4c8c-86ef-dae9bce19498\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-2d9861-78d7ffc559-lp7cm"
Oct 30 19:32:36.907282 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:32:36.907178    4892 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f1df6069-128b-4c8c-86ef-dae9bce19498-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w0-2d9861-78d7ffc559-lp7cm\" (UID: \"f1df6069-128b-4c8c-86ef-dae9bce19498\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-2d9861-78d7ffc559-lp7cm"
Oct 30 19:32:36.931704 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:32:36.931680    4892 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-worker-w0-2d9861\""
Oct 30 19:32:36.981867 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:32:36.981129    4892 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-worker-w0-2d9861-dockercfg-77lfz\""
Oct 30 19:32:36.984304 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:32:36.984272    4892 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w0-2d9861-78d7ffc559-lp7cm"]
Oct 30 19:32:37.008471 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:32:37.008434    4892 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gll9v\" (UniqueName: \"kubernetes.io/projected/f1df6069-128b-4c8c-86ef-dae9bce19498-kube-api-access-gll9v\") pod \"payload-pre-processing-e2e-worker-w0-2d9861-78d7ffc559-lp7cm\" (UID: \"f1df6069-128b-4c8c-86ef-dae9bce19498\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-2d9861-78d7ffc559-lp7cm"
Oct 30 19:32:37.008664 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:32:37.008504    4892 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f1df6069-128b-4c8c-86ef-dae9bce19498-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w0-2d9861-78d7ffc559-lp7cm\" (UID: \"f1df6069-128b-4c8c-86ef-dae9bce19498\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-2d9861-78d7ffc559-lp7cm"
Oct 30 19:32:37.009219 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:32:37.009200    4892 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f1df6069-128b-4c8c-86ef-dae9bce19498-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w0-2d9861-78d7ffc559-lp7cm\" (UID: \"f1df6069-128b-4c8c-86ef-dae9bce19498\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-2d9861-78d7ffc559-lp7cm"
Oct 30 19:32:37.036280 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:32:37.036252    4892 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w0-2d9861-d6c699c74-2cwqk"]
Oct 30 19:32:37.040251 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:32:37.040228    4892 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w0-2d9861-d6c699c74-2cwqk"
Oct 30 19:32:37.065073 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:32:37.065048    4892 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gll9v\" (UniqueName: \"kubernetes.io/projected/f1df6069-128b-4c8c-86ef-dae9bce19498-kube-api-access-gll9v\") pod \"payload-pre-processing-e2e-worker-w0-2d9861-78d7ffc559-lp7cm\" (UID: \"f1df6069-128b-4c8c-86ef-dae9bce19498\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-2d9861-78d7ffc559-lp7cm"
Oct 30 19:32:37.095801 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:32:37.095663    4892 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w0-2d9861-d6c699c74-2cwqk"]
Oct 30 19:32:37.109376 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:32:37.109352    4892 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7ecedcf3-79c1-4637-8f4c-eefc4f056d7b-plugins-config-volume\") pod \"payload-processing-e2e-worker-w0-2d9861-d6c699c74-2cwqk\" (UID: \"7ecedcf3-79c1-4637-8f4c-eefc4f056d7b\") " pod="openshift-ingress/payload-processing-e2e-worker-w0-2d9861-d6c699c74-2cwqk"
Oct 30 19:32:37.109481 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:32:37.109467    4892 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-g6mnq\" (UniqueName: \"kubernetes.io/projected/7ecedcf3-79c1-4637-8f4c-eefc4f056d7b-kube-api-access-g6mnq\") pod \"payload-processing-e2e-worker-w0-2d9861-d6c699c74-2cwqk\" (UID: \"7ecedcf3-79c1-4637-8f4c-eefc4f056d7b\") " pod="openshift-ingress/payload-processing-e2e-worker-w0-2d9861-d6c699c74-2cwqk"
Oct 30 19:32:37.188063 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:32:37.188042    4892 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-2d9861-78d7ffc559-lp7cm"
Oct 30 19:32:37.210248 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:32:37.210220    4892 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-g6mnq\" (UniqueName: \"kubernetes.io/projected/7ecedcf3-79c1-4637-8f4c-eefc4f056d7b-kube-api-access-g6mnq\") pod \"payload-processing-e2e-worker-w0-2d9861-d6c699c74-2cwqk\" (UID: \"7ecedcf3-79c1-4637-8f4c-eefc4f056d7b\") " pod="openshift-ingress/payload-processing-e2e-worker-w0-2d9861-d6c699c74-2cwqk"
Oct 30 19:32:37.210365 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:32:37.210298    4892 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7ecedcf3-79c1-4637-8f4c-eefc4f056d7b-plugins-config-volume\") pod \"payload-processing-e2e-worker-w0-2d9861-d6c699c74-2cwqk\" (UID: \"7ecedcf3-79c1-4637-8f4c-eefc4f056d7b\") " pod="openshift-ingress/payload-processing-e2e-worker-w0-2d9861-d6c699c74-2cwqk"
Oct 30 19:32:37.211233 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:32:37.211206    4892 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7ecedcf3-79c1-4637-8f4c-eefc4f056d7b-plugins-config-volume\") pod \"payload-processing-e2e-worker-w0-2d9861-d6c699c74-2cwqk\" (UID: \"7ecedcf3-79c1-4637-8f4c-eefc4f056d7b\") " pod="openshift-ingress/payload-processing-e2e-worker-w0-2d9861-d6c699c74-2cwqk"
Oct 30 19:32:37.267023 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:32:37.266993    4892 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-g6mnq\" (UniqueName: \"kubernetes.io/projected/7ecedcf3-79c1-4637-8f4c-eefc4f056d7b-kube-api-access-g6mnq\") pod \"payload-processing-e2e-worker-w0-2d9861-d6c699c74-2cwqk\" (UID: \"7ecedcf3-79c1-4637-8f4c-eefc4f056d7b\") " pod="openshift-ingress/payload-processing-e2e-worker-w0-2d9861-d6c699c74-2cwqk"
Oct 30 19:32:37.272392 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:32:37.272298    4892 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-worker-w6-5f55cd-85cb4d6b67-9qrfs"
Oct 30 19:32:37.272537 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:32:37.272512    4892 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-worker-w6-5f55cd-688df7c466-t26tq"
Oct 30 19:32:37.352904 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:32:37.352878    4892 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w0-2d9861-d6c699c74-2cwqk"
Oct 30 19:32:37.558771 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:32:37.558750    4892 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w0-2d9861-78d7ffc559-lp7cm"]
Oct 30 19:32:37.560851 ip-10-0-132-82 kubenswrapper[4892]: W1030 19:32:37.560830    4892 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf1df6069_128b_4c8c_86ef_dae9bce19498.slice/crio-4415f3b45d736f0b4e0d512ae98b7015dd9987a3d053a573abe4da0659a985c1 WatchSource:0}: Error finding container 4415f3b45d736f0b4e0d512ae98b7015dd9987a3d053a573abe4da0659a985c1: Status 404 returned error can't find the container with id 4415f3b45d736f0b4e0d512ae98b7015dd9987a3d053a573abe4da0659a985c1
Oct 30 19:32:37.731862 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:32:37.731826    4892 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w0-2d9861-d6c699c74-2cwqk"]
Oct 30 19:32:37.736581 ip-10-0-132-82 kubenswrapper[4892]: W1030 19:32:37.736548    4892 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7ecedcf3_79c1_4637_8f4c_eefc4f056d7b.slice/crio-8e6d18773dcc7db5fb710c735525d9c763912378d88de12fa5303e3a7033c800 WatchSource:0}: Error finding container 8e6d18773dcc7db5fb710c735525d9c763912378d88de12fa5303e3a7033c800: Status 404 returned error can't find the container with id 8e6d18773dcc7db5fb710c735525d9c763912378d88de12fa5303e3a7033c800
Oct 30 19:32:38.329771 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:32:38.329738    4892 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-2d9861-78d7ffc559-lp7cm" event={"ID":"f1df6069-128b-4c8c-86ef-dae9bce19498","Type":"ContainerStarted","Data":"d827b779fa16786b1f673fc81b92d9e095af48da65264094e22be8a2c16a559c"}
Oct 30 19:32:38.330266 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:32:38.329794    4892 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-2d9861-78d7ffc559-lp7cm" event={"ID":"f1df6069-128b-4c8c-86ef-dae9bce19498","Type":"ContainerStarted","Data":"4415f3b45d736f0b4e0d512ae98b7015dd9987a3d053a573abe4da0659a985c1"}
Oct 30 19:32:38.330266 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:32:38.329837    4892 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-2d9861-78d7ffc559-lp7cm"
Oct 30 19:32:38.331339 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:32:38.331316    4892 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w0-2d9861-d6c699c74-2cwqk" event={"ID":"7ecedcf3-79c1-4637-8f4c-eefc4f056d7b","Type":"ContainerStarted","Data":"98043c270dceec91d43f761ede75370fc06e3c2b3d23b18769f5ac469d1cbd0e"}
Oct 30 19:32:38.331472 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:32:38.331359    4892 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w0-2d9861-d6c699c74-2cwqk" event={"ID":"7ecedcf3-79c1-4637-8f4c-eefc4f056d7b","Type":"ContainerStarted","Data":"8e6d18773dcc7db5fb710c735525d9c763912378d88de12fa5303e3a7033c800"}
Oct 30 19:32:38.331472 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:32:38.331388    4892 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-worker-w0-2d9861-d6c699c74-2cwqk"
Oct 30 19:32:38.444571 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:32:38.444514    4892 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-2d9861-78d7ffc559-lp7cm" podStartSLOduration=2.444497861 podStartE2EDuration="2.444497861s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:32:36 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:32:38.442743941 +0000 UTC m=+1006.013209540" watchObservedRunningTime="2026-10-30 19:32:38.444497861 +0000 UTC m=+1006.014963460"
Oct 30 19:32:49.339656 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:32:49.339631    4892 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-2d9861-78d7ffc559-lp7cm"
Oct 30 19:32:49.340110 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:32:49.339695    4892 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-worker-w0-2d9861-d6c699c74-2cwqk"
Oct 30 19:32:49.456721 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:32:49.456556    4892 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-worker-w0-2d9861-d6c699c74-2cwqk" podStartSLOduration=13.456540162 podStartE2EDuration="13.456540162s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:32:36 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:32:38.563178216 +0000 UTC m=+1006.133643819" watchObservedRunningTime="2026-10-30 19:32:49.456540162 +0000 UTC m=+1017.027005762"
Oct 30 19:32:52.431674 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:32:52.431642    4892 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-fsfvk"]
Oct 30 19:32:54.843412 ip-10-0-132-82 kubenswrapper[4892]: E1030 19:32:54.843372    4892 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-jrm87" containerName="extractor"
Oct 30 19:33:00.947739 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:33:00.947704    4892 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-fsfvk"]
Oct 30 19:33:00.962126 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:33:00.962100    4892 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w3-ada8d7/simulator-w3-ada8d7-kserve-7d575cfcb5-5txgn"]
Oct 30 19:33:00.969571 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:33:00.969552    4892 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w3-ada8d7/simulator-w3-ada8d7-kserve-7d575cfcb5-5txgn"
Oct 30 19:33:01.021485 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:33:01.021420    4892 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w3-ada8d7\"/\"openshift-service-ca.crt\""
Oct 30 19:33:01.021655 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:33:01.021519    4892 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w3-ada8d7\"/\"kube-root-ca.crt\""
Oct 30 19:33:01.058792 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:33:01.058765    4892 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/10a5c7c8-6146-47b0-9643-bb137cf10bf5-model-cache\") pod \"simulator-w3-ada8d7-kserve-7d575cfcb5-5txgn\" (UID: \"10a5c7c8-6146-47b0-9643-bb137cf10bf5\") " pod="e2e-models-e2e-worker-w3-ada8d7/simulator-w3-ada8d7-kserve-7d575cfcb5-5txgn"
Oct 30 19:33:01.058792 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:33:01.058795    4892 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/10a5c7c8-6146-47b0-9643-bb137cf10bf5-tls-certs\") pod \"simulator-w3-ada8d7-kserve-7d575cfcb5-5txgn\" (UID: \"10a5c7c8-6146-47b0-9643-bb137cf10bf5\") " pod="e2e-models-e2e-worker-w3-ada8d7/simulator-w3-ada8d7-kserve-7d575cfcb5-5txgn"
Oct 30 19:33:01.059013 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:33:01.058939    4892 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/10a5c7c8-6146-47b0-9643-bb137cf10bf5-home\") pod \"simulator-w3-ada8d7-kserve-7d575cfcb5-5txgn\" (UID: \"10a5c7c8-6146-47b0-9643-bb137cf10bf5\") " pod="e2e-models-e2e-worker-w3-ada8d7/simulator-w3-ada8d7-kserve-7d575cfcb5-5txgn"
Oct 30 19:33:01.059013 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:33:01.058980    4892 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/10a5c7c8-6146-47b0-9643-bb137cf10bf5-dshm\") pod \"simulator-w3-ada8d7-kserve-7d575cfcb5-5txgn\" (UID: \"10a5c7c8-6146-47b0-9643-bb137cf10bf5\") " pod="e2e-models-e2e-worker-w3-ada8d7/simulator-w3-ada8d7-kserve-7d575cfcb5-5txgn"
Oct 30 19:33:01.059013 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:33:01.059005    4892 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/10a5c7c8-6146-47b0-9643-bb137cf10bf5-tmp-dir\") pod \"simulator-w3-ada8d7-kserve-7d575cfcb5-5txgn\" (UID: \"10a5c7c8-6146-47b0-9643-bb137cf10bf5\") " pod="e2e-models-e2e-worker-w3-ada8d7/simulator-w3-ada8d7-kserve-7d575cfcb5-5txgn"
Oct 30 19:33:01.059192 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:33:01.059072    4892 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qxp7s\" (UniqueName: \"kubernetes.io/projected/10a5c7c8-6146-47b0-9643-bb137cf10bf5-kube-api-access-qxp7s\") pod \"simulator-w3-ada8d7-kserve-7d575cfcb5-5txgn\" (UID: \"10a5c7c8-6146-47b0-9643-bb137cf10bf5\") " pod="e2e-models-e2e-worker-w3-ada8d7/simulator-w3-ada8d7-kserve-7d575cfcb5-5txgn"
Oct 30 19:33:01.070694 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:33:01.070668    4892 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w3-ada8d7\"/\"default-dockercfg-7ntd4\""
Oct 30 19:33:01.070694 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:33:01.070681    4892 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w3-ada8d7\"/\"simulator-w3-ada8d7-kserve-self-signed-certs\""
Oct 30 19:33:01.073003 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:33:01.072983    4892 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w3-ada8d7/simulator-w3-ada8d7-kserve-7d575cfcb5-5txgn"]
Oct 30 19:33:01.162854 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:33:01.162757    4892 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/10a5c7c8-6146-47b0-9643-bb137cf10bf5-home\") pod \"simulator-w3-ada8d7-kserve-7d575cfcb5-5txgn\" (UID: \"10a5c7c8-6146-47b0-9643-bb137cf10bf5\") " pod="e2e-models-e2e-worker-w3-ada8d7/simulator-w3-ada8d7-kserve-7d575cfcb5-5txgn"
Oct 30 19:33:01.162854 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:33:01.162801    4892 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/10a5c7c8-6146-47b0-9643-bb137cf10bf5-dshm\") pod \"simulator-w3-ada8d7-kserve-7d575cfcb5-5txgn\" (UID: \"10a5c7c8-6146-47b0-9643-bb137cf10bf5\") " pod="e2e-models-e2e-worker-w3-ada8d7/simulator-w3-ada8d7-kserve-7d575cfcb5-5txgn"
Oct 30 19:33:01.162854 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:33:01.162822    4892 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/10a5c7c8-6146-47b0-9643-bb137cf10bf5-tmp-dir\") pod \"simulator-w3-ada8d7-kserve-7d575cfcb5-5txgn\" (UID: \"10a5c7c8-6146-47b0-9643-bb137cf10bf5\") " pod="e2e-models-e2e-worker-w3-ada8d7/simulator-w3-ada8d7-kserve-7d575cfcb5-5txgn"
Oct 30 19:33:01.162854 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:33:01.162851    4892 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qxp7s\" (UniqueName: \"kubernetes.io/projected/10a5c7c8-6146-47b0-9643-bb137cf10bf5-kube-api-access-qxp7s\") pod \"simulator-w3-ada8d7-kserve-7d575cfcb5-5txgn\" (UID: \"10a5c7c8-6146-47b0-9643-bb137cf10bf5\") " pod="e2e-models-e2e-worker-w3-ada8d7/simulator-w3-ada8d7-kserve-7d575cfcb5-5txgn"
Oct 30 19:33:01.163247 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:33:01.162921    4892 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/10a5c7c8-6146-47b0-9643-bb137cf10bf5-model-cache\") pod \"simulator-w3-ada8d7-kserve-7d575cfcb5-5txgn\" (UID: \"10a5c7c8-6146-47b0-9643-bb137cf10bf5\") " pod="e2e-models-e2e-worker-w3-ada8d7/simulator-w3-ada8d7-kserve-7d575cfcb5-5txgn"
Oct 30 19:33:01.163247 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:33:01.162942    4892 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/10a5c7c8-6146-47b0-9643-bb137cf10bf5-tls-certs\") pod \"simulator-w3-ada8d7-kserve-7d575cfcb5-5txgn\" (UID: \"10a5c7c8-6146-47b0-9643-bb137cf10bf5\") " pod="e2e-models-e2e-worker-w3-ada8d7/simulator-w3-ada8d7-kserve-7d575cfcb5-5txgn"
Oct 30 19:33:01.163414 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:33:01.163390    4892 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/10a5c7c8-6146-47b0-9643-bb137cf10bf5-home\") pod \"simulator-w3-ada8d7-kserve-7d575cfcb5-5txgn\" (UID: \"10a5c7c8-6146-47b0-9643-bb137cf10bf5\") " pod="e2e-models-e2e-worker-w3-ada8d7/simulator-w3-ada8d7-kserve-7d575cfcb5-5txgn"
Oct 30 19:33:01.164210 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:33:01.164182    4892 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/10a5c7c8-6146-47b0-9643-bb137cf10bf5-model-cache\") pod \"simulator-w3-ada8d7-kserve-7d575cfcb5-5txgn\" (UID: \"10a5c7c8-6146-47b0-9643-bb137cf10bf5\") " pod="e2e-models-e2e-worker-w3-ada8d7/simulator-w3-ada8d7-kserve-7d575cfcb5-5txgn"
Oct 30 19:33:01.164827 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:33:01.164801    4892 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/10a5c7c8-6146-47b0-9643-bb137cf10bf5-tmp-dir\") pod \"simulator-w3-ada8d7-kserve-7d575cfcb5-5txgn\" (UID: \"10a5c7c8-6146-47b0-9643-bb137cf10bf5\") " pod="e2e-models-e2e-worker-w3-ada8d7/simulator-w3-ada8d7-kserve-7d575cfcb5-5txgn"
Oct 30 19:33:01.166780 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:33:01.166756    4892 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/10a5c7c8-6146-47b0-9643-bb137cf10bf5-tls-certs\") pod \"simulator-w3-ada8d7-kserve-7d575cfcb5-5txgn\" (UID: \"10a5c7c8-6146-47b0-9643-bb137cf10bf5\") " pod="e2e-models-e2e-worker-w3-ada8d7/simulator-w3-ada8d7-kserve-7d575cfcb5-5txgn"
Oct 30 19:33:01.167441 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:33:01.167413    4892 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/10a5c7c8-6146-47b0-9643-bb137cf10bf5-dshm\") pod \"simulator-w3-ada8d7-kserve-7d575cfcb5-5txgn\" (UID: \"10a5c7c8-6146-47b0-9643-bb137cf10bf5\") " pod="e2e-models-e2e-worker-w3-ada8d7/simulator-w3-ada8d7-kserve-7d575cfcb5-5txgn"
Oct 30 19:33:01.220889 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:33:01.220824    4892 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qxp7s\" (UniqueName: \"kubernetes.io/projected/10a5c7c8-6146-47b0-9643-bb137cf10bf5-kube-api-access-qxp7s\") pod \"simulator-w3-ada8d7-kserve-7d575cfcb5-5txgn\" (UID: \"10a5c7c8-6146-47b0-9643-bb137cf10bf5\") " pod="e2e-models-e2e-worker-w3-ada8d7/simulator-w3-ada8d7-kserve-7d575cfcb5-5txgn"
Oct 30 19:33:01.281115 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:33:01.281082    4892 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w3-ada8d7/simulator-w3-ada8d7-kserve-7d575cfcb5-5txgn"
Oct 30 19:33:01.654306 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:33:01.654282    4892 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w3-ada8d7/simulator-w3-ada8d7-kserve-7d575cfcb5-5txgn"]
Oct 30 19:33:01.656042 ip-10-0-132-82 kubenswrapper[4892]: W1030 19:33:01.656014    4892 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod10a5c7c8_6146_47b0_9643_bb137cf10bf5.slice/crio-d9b691aaff95d751183bfca03c9a3e6374fee23cd47652d8fd1d36b85c1b22d2 WatchSource:0}: Error finding container d9b691aaff95d751183bfca03c9a3e6374fee23cd47652d8fd1d36b85c1b22d2: Status 404 returned error can't find the container with id d9b691aaff95d751183bfca03c9a3e6374fee23cd47652d8fd1d36b85c1b22d2
Oct 30 19:33:02.468779 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:33:02.468730    4892 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w3-ada8d7/simulator-w3-ada8d7-kserve-7d575cfcb5-5txgn" event={"ID":"10a5c7c8-6146-47b0-9643-bb137cf10bf5","Type":"ContainerStarted","Data":"d9b691aaff95d751183bfca03c9a3e6374fee23cd47652d8fd1d36b85c1b22d2"}
Oct 30 19:33:04.481641 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:33:04.481601    4892 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w3-ada8d7/simulator-w3-ada8d7-kserve-7d575cfcb5-5txgn" event={"ID":"10a5c7c8-6146-47b0-9643-bb137cf10bf5","Type":"ContainerStarted","Data":"b16fc553522b87ac281529364f1d4a580ead182ca2d88e0462cce03fa304a0b9"}
Oct 30 19:33:04.604148 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:33:04.604081    4892 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w3-ada8d7/simulator-w3-ada8d7-kserve-7d575cfcb5-5txgn" podStartSLOduration=2.611568889 podStartE2EDuration="4.604061482s" totalImagesPullingTime="1.992492593s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:33:00 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:33:04.60342415 +0000 UTC m=+1032.173889751" watchObservedRunningTime="2026-10-30 19:33:04.604061482 +0000 UTC m=+1032.174527089"
Oct 30 19:33:05.281257 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:33:05.281214    4892 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-ada8d7/simulator-w3-ada8d7-kserve-7d575cfcb5-5txgn"
Oct 30 19:33:11.281734 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:33:11.281697    4892 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w3-ada8d7/simulator-w3-ada8d7-kserve-7d575cfcb5-5txgn"
Oct 30 19:33:11.294827 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:33:11.294804    4892 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w3-ada8d7/simulator-w3-ada8d7-kserve-7d575cfcb5-5txgn"
Oct 30 19:33:11.533596 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:33:11.533509    4892 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-ada8d7/simulator-w3-ada8d7-kserve-7d575cfcb5-5txgn"
Oct 30 19:33:14.047982 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:33:14.047938    4892 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-fsfvk"]
Oct 30 19:33:17.489027 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:33:17.488992    4892 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w3-ada8d7-openshift-default-5f8b49bdfc-khngd"]
Oct 30 19:33:17.492655 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:33:17.492639    4892 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-ada8d7-openshift-default-5f8b49bdfc-khngd"
Oct 30 19:33:17.513639 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:33:17.513614    4892 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/eb58d944-e2ad-4505-a5df-d5c617c1dbf2-workload-certs\") pod \"e2e-worker-w3-ada8d7-openshift-default-5f8b49bdfc-khngd\" (UID: \"eb58d944-e2ad-4505-a5df-d5c617c1dbf2\") " pod="openshift-ingress/e2e-worker-w3-ada8d7-openshift-default-5f8b49bdfc-khngd"
Oct 30 19:33:17.513766 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:33:17.513644    4892 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/eb58d944-e2ad-4505-a5df-d5c617c1dbf2-istio-token\") pod \"e2e-worker-w3-ada8d7-openshift-default-5f8b49bdfc-khngd\" (UID: \"eb58d944-e2ad-4505-a5df-d5c617c1dbf2\") " pod="openshift-ingress/e2e-worker-w3-ada8d7-openshift-default-5f8b49bdfc-khngd"
Oct 30 19:33:17.513766 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:33:17.513681    4892 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/eb58d944-e2ad-4505-a5df-d5c617c1dbf2-workload-socket\") pod \"e2e-worker-w3-ada8d7-openshift-default-5f8b49bdfc-khngd\" (UID: \"eb58d944-e2ad-4505-a5df-d5c617c1dbf2\") " pod="openshift-ingress/e2e-worker-w3-ada8d7-openshift-default-5f8b49bdfc-khngd"
Oct 30 19:33:17.513880 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:33:17.513763    4892 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7qxtg\" (UniqueName: \"kubernetes.io/projected/eb58d944-e2ad-4505-a5df-d5c617c1dbf2-kube-api-access-7qxtg\") pod \"e2e-worker-w3-ada8d7-openshift-default-5f8b49bdfc-khngd\" (UID: \"eb58d944-e2ad-4505-a5df-d5c617c1dbf2\") " pod="openshift-ingress/e2e-worker-w3-ada8d7-openshift-default-5f8b49bdfc-khngd"
Oct 30 19:33:17.513936 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:33:17.513914    4892 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/eb58d944-e2ad-4505-a5df-d5c617c1dbf2-istiod-ca-cert\") pod \"e2e-worker-w3-ada8d7-openshift-default-5f8b49bdfc-khngd\" (UID: \"eb58d944-e2ad-4505-a5df-d5c617c1dbf2\") " pod="openshift-ingress/e2e-worker-w3-ada8d7-openshift-default-5f8b49bdfc-khngd"
Oct 30 19:33:17.514017 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:33:17.513977    4892 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/eb58d944-e2ad-4505-a5df-d5c617c1dbf2-credential-socket\") pod \"e2e-worker-w3-ada8d7-openshift-default-5f8b49bdfc-khngd\" (UID: \"eb58d944-e2ad-4505-a5df-d5c617c1dbf2\") " pod="openshift-ingress/e2e-worker-w3-ada8d7-openshift-default-5f8b49bdfc-khngd"
Oct 30 19:33:17.514085 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:33:17.514031    4892 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/eb58d944-e2ad-4505-a5df-d5c617c1dbf2-istio-data\") pod \"e2e-worker-w3-ada8d7-openshift-default-5f8b49bdfc-khngd\" (UID: \"eb58d944-e2ad-4505-a5df-d5c617c1dbf2\") " pod="openshift-ingress/e2e-worker-w3-ada8d7-openshift-default-5f8b49bdfc-khngd"
Oct 30 19:33:17.514142 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:33:17.514098    4892 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/eb58d944-e2ad-4505-a5df-d5c617c1dbf2-istio-podinfo\") pod \"e2e-worker-w3-ada8d7-openshift-default-5f8b49bdfc-khngd\" (UID: \"eb58d944-e2ad-4505-a5df-d5c617c1dbf2\") " pod="openshift-ingress/e2e-worker-w3-ada8d7-openshift-default-5f8b49bdfc-khngd"
Oct 30 19:33:17.514191 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:33:17.514155    4892 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/eb58d944-e2ad-4505-a5df-d5c617c1dbf2-istio-envoy\") pod \"e2e-worker-w3-ada8d7-openshift-default-5f8b49bdfc-khngd\" (UID: \"eb58d944-e2ad-4505-a5df-d5c617c1dbf2\") " pod="openshift-ingress/e2e-worker-w3-ada8d7-openshift-default-5f8b49bdfc-khngd"
Oct 30 19:33:17.544635 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:33:17.544615    4892 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w3-ada8d7-openshift-default-dockercfg-wtmwk\""
Oct 30 19:33:17.596272 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:33:17.596247    4892 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-ada8d7-openshift-default-5f8b49bdfc-khngd"]
Oct 30 19:33:17.615283 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:33:17.615252    4892 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7qxtg\" (UniqueName: \"kubernetes.io/projected/eb58d944-e2ad-4505-a5df-d5c617c1dbf2-kube-api-access-7qxtg\") pod \"e2e-worker-w3-ada8d7-openshift-default-5f8b49bdfc-khngd\" (UID: \"eb58d944-e2ad-4505-a5df-d5c617c1dbf2\") " pod="openshift-ingress/e2e-worker-w3-ada8d7-openshift-default-5f8b49bdfc-khngd"
Oct 30 19:33:17.615421 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:33:17.615301    4892 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/eb58d944-e2ad-4505-a5df-d5c617c1dbf2-istiod-ca-cert\") pod \"e2e-worker-w3-ada8d7-openshift-default-5f8b49bdfc-khngd\" (UID: \"eb58d944-e2ad-4505-a5df-d5c617c1dbf2\") " pod="openshift-ingress/e2e-worker-w3-ada8d7-openshift-default-5f8b49bdfc-khngd"
Oct 30 19:33:17.615421 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:33:17.615331    4892 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/eb58d944-e2ad-4505-a5df-d5c617c1dbf2-credential-socket\") pod \"e2e-worker-w3-ada8d7-openshift-default-5f8b49bdfc-khngd\" (UID: \"eb58d944-e2ad-4505-a5df-d5c617c1dbf2\") " pod="openshift-ingress/e2e-worker-w3-ada8d7-openshift-default-5f8b49bdfc-khngd"
Oct 30 19:33:17.615421 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:33:17.615377    4892 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/eb58d944-e2ad-4505-a5df-d5c617c1dbf2-istio-data\") pod \"e2e-worker-w3-ada8d7-openshift-default-5f8b49bdfc-khngd\" (UID: \"eb58d944-e2ad-4505-a5df-d5c617c1dbf2\") " pod="openshift-ingress/e2e-worker-w3-ada8d7-openshift-default-5f8b49bdfc-khngd"
Oct 30 19:33:17.615587 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:33:17.615446    4892 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/eb58d944-e2ad-4505-a5df-d5c617c1dbf2-istio-podinfo\") pod \"e2e-worker-w3-ada8d7-openshift-default-5f8b49bdfc-khngd\" (UID: \"eb58d944-e2ad-4505-a5df-d5c617c1dbf2\") " pod="openshift-ingress/e2e-worker-w3-ada8d7-openshift-default-5f8b49bdfc-khngd"
Oct 30 19:33:17.615587 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:33:17.615507    4892 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/eb58d944-e2ad-4505-a5df-d5c617c1dbf2-istio-envoy\") pod \"e2e-worker-w3-ada8d7-openshift-default-5f8b49bdfc-khngd\" (UID: \"eb58d944-e2ad-4505-a5df-d5c617c1dbf2\") " pod="openshift-ingress/e2e-worker-w3-ada8d7-openshift-default-5f8b49bdfc-khngd"
Oct 30 19:33:17.615587 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:33:17.615548    4892 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/eb58d944-e2ad-4505-a5df-d5c617c1dbf2-workload-certs\") pod \"e2e-worker-w3-ada8d7-openshift-default-5f8b49bdfc-khngd\" (UID: \"eb58d944-e2ad-4505-a5df-d5c617c1dbf2\") " pod="openshift-ingress/e2e-worker-w3-ada8d7-openshift-default-5f8b49bdfc-khngd"
Oct 30 19:33:17.615587 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:33:17.615575    4892 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/eb58d944-e2ad-4505-a5df-d5c617c1dbf2-istio-token\") pod \"e2e-worker-w3-ada8d7-openshift-default-5f8b49bdfc-khngd\" (UID: \"eb58d944-e2ad-4505-a5df-d5c617c1dbf2\") " pod="openshift-ingress/e2e-worker-w3-ada8d7-openshift-default-5f8b49bdfc-khngd"
Oct 30 19:33:17.615792 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:33:17.615636    4892 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/eb58d944-e2ad-4505-a5df-d5c617c1dbf2-workload-socket\") pod \"e2e-worker-w3-ada8d7-openshift-default-5f8b49bdfc-khngd\" (UID: \"eb58d944-e2ad-4505-a5df-d5c617c1dbf2\") " pod="openshift-ingress/e2e-worker-w3-ada8d7-openshift-default-5f8b49bdfc-khngd"
Oct 30 19:33:17.615855 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:33:17.615784    4892 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/eb58d944-e2ad-4505-a5df-d5c617c1dbf2-credential-socket\") pod \"e2e-worker-w3-ada8d7-openshift-default-5f8b49bdfc-khngd\" (UID: \"eb58d944-e2ad-4505-a5df-d5c617c1dbf2\") " pod="openshift-ingress/e2e-worker-w3-ada8d7-openshift-default-5f8b49bdfc-khngd"
Oct 30 19:33:17.615855 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:33:17.615814    4892 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/eb58d944-e2ad-4505-a5df-d5c617c1dbf2-istio-data\") pod \"e2e-worker-w3-ada8d7-openshift-default-5f8b49bdfc-khngd\" (UID: \"eb58d944-e2ad-4505-a5df-d5c617c1dbf2\") " pod="openshift-ingress/e2e-worker-w3-ada8d7-openshift-default-5f8b49bdfc-khngd"
Oct 30 19:33:17.616187 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:33:17.616140    4892 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/eb58d944-e2ad-4505-a5df-d5c617c1dbf2-workload-socket\") pod \"e2e-worker-w3-ada8d7-openshift-default-5f8b49bdfc-khngd\" (UID: \"eb58d944-e2ad-4505-a5df-d5c617c1dbf2\") " pod="openshift-ingress/e2e-worker-w3-ada8d7-openshift-default-5f8b49bdfc-khngd"
Oct 30 19:33:17.616276 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:33:17.616228    4892 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/eb58d944-e2ad-4505-a5df-d5c617c1dbf2-workload-certs\") pod \"e2e-worker-w3-ada8d7-openshift-default-5f8b49bdfc-khngd\" (UID: \"eb58d944-e2ad-4505-a5df-d5c617c1dbf2\") " pod="openshift-ingress/e2e-worker-w3-ada8d7-openshift-default-5f8b49bdfc-khngd"
Oct 30 19:33:17.616380 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:33:17.616362    4892 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/eb58d944-e2ad-4505-a5df-d5c617c1dbf2-istiod-ca-cert\") pod \"e2e-worker-w3-ada8d7-openshift-default-5f8b49bdfc-khngd\" (UID: \"eb58d944-e2ad-4505-a5df-d5c617c1dbf2\") " pod="openshift-ingress/e2e-worker-w3-ada8d7-openshift-default-5f8b49bdfc-khngd"
Oct 30 19:33:17.617875 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:33:17.617854    4892 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/eb58d944-e2ad-4505-a5df-d5c617c1dbf2-istio-envoy\") pod \"e2e-worker-w3-ada8d7-openshift-default-5f8b49bdfc-khngd\" (UID: \"eb58d944-e2ad-4505-a5df-d5c617c1dbf2\") " pod="openshift-ingress/e2e-worker-w3-ada8d7-openshift-default-5f8b49bdfc-khngd"
Oct 30 19:33:17.618272 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:33:17.618251    4892 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/eb58d944-e2ad-4505-a5df-d5c617c1dbf2-istio-podinfo\") pod \"e2e-worker-w3-ada8d7-openshift-default-5f8b49bdfc-khngd\" (UID: \"eb58d944-e2ad-4505-a5df-d5c617c1dbf2\") " pod="openshift-ingress/e2e-worker-w3-ada8d7-openshift-default-5f8b49bdfc-khngd"
Oct 30 19:33:17.672421 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:33:17.672380    4892 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/eb58d944-e2ad-4505-a5df-d5c617c1dbf2-istio-token\") pod \"e2e-worker-w3-ada8d7-openshift-default-5f8b49bdfc-khngd\" (UID: \"eb58d944-e2ad-4505-a5df-d5c617c1dbf2\") " pod="openshift-ingress/e2e-worker-w3-ada8d7-openshift-default-5f8b49bdfc-khngd"
Oct 30 19:33:17.672570 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:33:17.672504    4892 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7qxtg\" (UniqueName: \"kubernetes.io/projected/eb58d944-e2ad-4505-a5df-d5c617c1dbf2-kube-api-access-7qxtg\") pod \"e2e-worker-w3-ada8d7-openshift-default-5f8b49bdfc-khngd\" (UID: \"eb58d944-e2ad-4505-a5df-d5c617c1dbf2\") " pod="openshift-ingress/e2e-worker-w3-ada8d7-openshift-default-5f8b49bdfc-khngd"
Oct 30 19:33:17.806473 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:33:17.806403    4892 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-ada8d7-openshift-default-5f8b49bdfc-khngd"
Oct 30 19:33:18.229659 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:33:18.229638    4892 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-ada8d7-openshift-default-5f8b49bdfc-khngd"]
Oct 30 19:33:18.231545 ip-10-0-132-82 kubenswrapper[4892]: W1030 19:33:18.231513    4892 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podeb58d944_e2ad_4505_a5df_d5c617c1dbf2.slice/crio-e0903356a050cf11a595cf8ccfcc67e01f97f287dadb23554b4d22e1c2777400 WatchSource:0}: Error finding container e0903356a050cf11a595cf8ccfcc67e01f97f287dadb23554b4d22e1c2777400: Status 404 returned error can't find the container with id e0903356a050cf11a595cf8ccfcc67e01f97f287dadb23554b4d22e1c2777400
Oct 30 19:33:18.233847 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:33:18.233812    4892 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892000Ki","pods":"250"}
Oct 30 19:33:18.233924 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:33:18.233879    4892 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892000Ki","pods":"250"}
Oct 30 19:33:18.233924 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:33:18.233907    4892 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892000Ki","pods":"250"}
Oct 30 19:33:18.570273 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:33:18.570189    4892 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-ada8d7-openshift-default-5f8b49bdfc-khngd" event={"ID":"eb58d944-e2ad-4505-a5df-d5c617c1dbf2","Type":"ContainerStarted","Data":"c20bba13fe09724c6650e876534a4db34e2ba5a5a45fe865f2d69660b9d213d6"}
Oct 30 19:33:18.570749 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:33:18.570287    4892 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-ada8d7-openshift-default-5f8b49bdfc-khngd" event={"ID":"eb58d944-e2ad-4505-a5df-d5c617c1dbf2","Type":"ContainerStarted","Data":"e0903356a050cf11a595cf8ccfcc67e01f97f287dadb23554b4d22e1c2777400"}
Oct 30 19:33:18.687322 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:33:18.687270    4892 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w3-ada8d7-openshift-default-5f8b49bdfc-khngd" podStartSLOduration=1.687254372 podStartE2EDuration="1.687254372s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:33:17 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:33:18.686361548 +0000 UTC m=+1046.256827162" watchObservedRunningTime="2026-10-30 19:33:18.687254372 +0000 UTC m=+1046.257719972"
Oct 30 19:33:18.807292 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:33:18.807264    4892 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w3-ada8d7-openshift-default-5f8b49bdfc-khngd"
Oct 30 19:33:19.808311 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:33:19.808278    4892 patch_prober.go:29] interesting pod/e2e-worker-w3-ada8d7-openshift-default-5f8b49bdfc-khngd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.56:15021/healthz/ready\": dial tcp 10.132.0.56:15021: connect: connection refused" start-of-body=
Oct 30 19:33:19.808668 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:33:19.808340    4892 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-ada8d7-openshift-default-5f8b49bdfc-khngd" podUID="eb58d944-e2ad-4505-a5df-d5c617c1dbf2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.56:15021/healthz/ready\": dial tcp 10.132.0.56:15021: connect: connection refused"
Oct 30 19:33:20.807005 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:33:20.806949    4892 patch_prober.go:29] interesting pod/e2e-worker-w3-ada8d7-openshift-default-5f8b49bdfc-khngd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.56:15021/healthz/ready\": dial tcp 10.132.0.56:15021: connect: connection refused" start-of-body=
Oct 30 19:33:20.807204 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:33:20.807024    4892 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-ada8d7-openshift-default-5f8b49bdfc-khngd" podUID="eb58d944-e2ad-4505-a5df-d5c617c1dbf2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.56:15021/healthz/ready\": dial tcp 10.132.0.56:15021: connect: connection refused"
Oct 30 19:33:21.807009 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:33:21.806954    4892 patch_prober.go:29] interesting pod/e2e-worker-w3-ada8d7-openshift-default-5f8b49bdfc-khngd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.56:15021/healthz/ready\": dial tcp 10.132.0.56:15021: connect: connection refused" start-of-body=
Oct 30 19:33:21.807482 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:33:21.807048    4892 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-ada8d7-openshift-default-5f8b49bdfc-khngd" podUID="eb58d944-e2ad-4505-a5df-d5c617c1dbf2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.56:15021/healthz/ready\": dial tcp 10.132.0.56:15021: connect: connection refused"
Oct 30 19:33:22.807761 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:33:22.807726    4892 patch_prober.go:29] interesting pod/e2e-worker-w3-ada8d7-openshift-default-5f8b49bdfc-khngd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.56:15021/healthz/ready\": dial tcp 10.132.0.56:15021: connect: connection refused" start-of-body=
Oct 30 19:33:22.808216 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:33:22.807835    4892 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-ada8d7-openshift-default-5f8b49bdfc-khngd" podUID="eb58d944-e2ad-4505-a5df-d5c617c1dbf2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.56:15021/healthz/ready\": dial tcp 10.132.0.56:15021: connect: connection refused"
Oct 30 19:33:23.807360 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:33:23.807324    4892 patch_prober.go:29] interesting pod/e2e-worker-w3-ada8d7-openshift-default-5f8b49bdfc-khngd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.56:15021/healthz/ready\": dial tcp 10.132.0.56:15021: connect: connection refused" start-of-body=
Oct 30 19:33:23.807536 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:33:23.807378    4892 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-ada8d7-openshift-default-5f8b49bdfc-khngd" podUID="eb58d944-e2ad-4505-a5df-d5c617c1dbf2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.56:15021/healthz/ready\": dial tcp 10.132.0.56:15021: connect: connection refused"
Oct 30 19:33:24.846940 ip-10-0-132-82 kubenswrapper[4892]: E1030 19:33:24.846893    4892 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-jrm87" containerName="extractor"
Oct 30 19:33:24.882434 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:33:24.882396    4892 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w3-ada8d7-openshift-default-5f8b49bdfc-khngd"
Oct 30 19:33:24.883037 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:33:24.883007    4892 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w3-ada8d7-openshift-default-5f8b49bdfc-khngd"
Oct 30 19:33:24.884536 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:33:24.884513    4892 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w3-ada8d7-openshift-default-5f8b49bdfc-khngd"
Oct 30 19:33:29.200992 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:33:29.200935    4892 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w1-d52df0/simulator-w1-d52df0-kserve-67f74bc577-q5bjz"]
Oct 30 19:33:29.204499 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:33:29.204466    4892 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w1-d52df0/simulator-w1-d52df0-kserve-67f74bc577-q5bjz"
Oct 30 19:33:29.264200 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:33:29.264168    4892 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/81b7bb84-1003-4452-b7e0-b4aaba6fefa4-model-cache\") pod \"simulator-w1-d52df0-kserve-67f74bc577-q5bjz\" (UID: \"81b7bb84-1003-4452-b7e0-b4aaba6fefa4\") " pod="e2e-models-e2e-worker-w1-d52df0/simulator-w1-d52df0-kserve-67f74bc577-q5bjz"
Oct 30 19:33:29.264347 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:33:29.264297    4892 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-l2tmb\" (UniqueName: \"kubernetes.io/projected/81b7bb84-1003-4452-b7e0-b4aaba6fefa4-kube-api-access-l2tmb\") pod \"simulator-w1-d52df0-kserve-67f74bc577-q5bjz\" (UID: \"81b7bb84-1003-4452-b7e0-b4aaba6fefa4\") " pod="e2e-models-e2e-worker-w1-d52df0/simulator-w1-d52df0-kserve-67f74bc577-q5bjz"
Oct 30 19:33:29.264347 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:33:29.264338    4892 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/81b7bb84-1003-4452-b7e0-b4aaba6fefa4-home\") pod \"simulator-w1-d52df0-kserve-67f74bc577-q5bjz\" (UID: \"81b7bb84-1003-4452-b7e0-b4aaba6fefa4\") " pod="e2e-models-e2e-worker-w1-d52df0/simulator-w1-d52df0-kserve-67f74bc577-q5bjz"
Oct 30 19:33:29.264437 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:33:29.264357    4892 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/81b7bb84-1003-4452-b7e0-b4aaba6fefa4-tmp-dir\") pod \"simulator-w1-d52df0-kserve-67f74bc577-q5bjz\" (UID: \"81b7bb84-1003-4452-b7e0-b4aaba6fefa4\") " pod="e2e-models-e2e-worker-w1-d52df0/simulator-w1-d52df0-kserve-67f74bc577-q5bjz"
Oct 30 19:33:29.264437 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:33:29.264381    4892 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/81b7bb84-1003-4452-b7e0-b4aaba6fefa4-tls-certs\") pod \"simulator-w1-d52df0-kserve-67f74bc577-q5bjz\" (UID: \"81b7bb84-1003-4452-b7e0-b4aaba6fefa4\") " pod="e2e-models-e2e-worker-w1-d52df0/simulator-w1-d52df0-kserve-67f74bc577-q5bjz"
Oct 30 19:33:29.264543 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:33:29.264521    4892 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/81b7bb84-1003-4452-b7e0-b4aaba6fefa4-dshm\") pod \"simulator-w1-d52df0-kserve-67f74bc577-q5bjz\" (UID: \"81b7bb84-1003-4452-b7e0-b4aaba6fefa4\") " pod="e2e-models-e2e-worker-w1-d52df0/simulator-w1-d52df0-kserve-67f74bc577-q5bjz"
Oct 30 19:33:29.306097 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:33:29.306062    4892 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w1-d52df0\"/\"kube-root-ca.crt\""
Oct 30 19:33:29.306097 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:33:29.306062    4892 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w1-d52df0\"/\"simulator-w1-d52df0-kserve-self-signed-certs\""
Oct 30 19:33:29.306303 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:33:29.306102    4892 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w1-d52df0\"/\"openshift-service-ca.crt\""
Oct 30 19:33:29.306303 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:33:29.306194    4892 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w1-d52df0\"/\"default-dockercfg-9vbhz\""
Oct 30 19:33:29.308419 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:33:29.308400    4892 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w1-d52df0/simulator-w1-d52df0-kserve-67f74bc577-q5bjz"]
Oct 30 19:33:29.368144 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:33:29.367081    4892 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/81b7bb84-1003-4452-b7e0-b4aaba6fefa4-dshm\") pod \"simulator-w1-d52df0-kserve-67f74bc577-q5bjz\" (UID: \"81b7bb84-1003-4452-b7e0-b4aaba6fefa4\") " pod="e2e-models-e2e-worker-w1-d52df0/simulator-w1-d52df0-kserve-67f74bc577-q5bjz"
Oct 30 19:33:29.368144 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:33:29.367154    4892 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/81b7bb84-1003-4452-b7e0-b4aaba6fefa4-model-cache\") pod \"simulator-w1-d52df0-kserve-67f74bc577-q5bjz\" (UID: \"81b7bb84-1003-4452-b7e0-b4aaba6fefa4\") " pod="e2e-models-e2e-worker-w1-d52df0/simulator-w1-d52df0-kserve-67f74bc577-q5bjz"
Oct 30 19:33:29.368553 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:33:29.368519    4892 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-l2tmb\" (UniqueName: \"kubernetes.io/projected/81b7bb84-1003-4452-b7e0-b4aaba6fefa4-kube-api-access-l2tmb\") pod \"simulator-w1-d52df0-kserve-67f74bc577-q5bjz\" (UID: \"81b7bb84-1003-4452-b7e0-b4aaba6fefa4\") " pod="e2e-models-e2e-worker-w1-d52df0/simulator-w1-d52df0-kserve-67f74bc577-q5bjz"
Oct 30 19:33:29.368693 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:33:29.368589    4892 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/81b7bb84-1003-4452-b7e0-b4aaba6fefa4-home\") pod \"simulator-w1-d52df0-kserve-67f74bc577-q5bjz\" (UID: \"81b7bb84-1003-4452-b7e0-b4aaba6fefa4\") " pod="e2e-models-e2e-worker-w1-d52df0/simulator-w1-d52df0-kserve-67f74bc577-q5bjz"
Oct 30 19:33:29.368693 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:33:29.368664    4892 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/81b7bb84-1003-4452-b7e0-b4aaba6fefa4-tmp-dir\") pod \"simulator-w1-d52df0-kserve-67f74bc577-q5bjz\" (UID: \"81b7bb84-1003-4452-b7e0-b4aaba6fefa4\") " pod="e2e-models-e2e-worker-w1-d52df0/simulator-w1-d52df0-kserve-67f74bc577-q5bjz"
Oct 30 19:33:29.368871 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:33:29.368811    4892 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/81b7bb84-1003-4452-b7e0-b4aaba6fefa4-tls-certs\") pod \"simulator-w1-d52df0-kserve-67f74bc577-q5bjz\" (UID: \"81b7bb84-1003-4452-b7e0-b4aaba6fefa4\") " pod="e2e-models-e2e-worker-w1-d52df0/simulator-w1-d52df0-kserve-67f74bc577-q5bjz"
Oct 30 19:33:29.370897 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:33:29.370819    4892 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/81b7bb84-1003-4452-b7e0-b4aaba6fefa4-home\") pod \"simulator-w1-d52df0-kserve-67f74bc577-q5bjz\" (UID: \"81b7bb84-1003-4452-b7e0-b4aaba6fefa4\") " pod="e2e-models-e2e-worker-w1-d52df0/simulator-w1-d52df0-kserve-67f74bc577-q5bjz"
Oct 30 19:33:29.370897 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:33:29.370827    4892 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/81b7bb84-1003-4452-b7e0-b4aaba6fefa4-model-cache\") pod \"simulator-w1-d52df0-kserve-67f74bc577-q5bjz\" (UID: \"81b7bb84-1003-4452-b7e0-b4aaba6fefa4\") " pod="e2e-models-e2e-worker-w1-d52df0/simulator-w1-d52df0-kserve-67f74bc577-q5bjz"
Oct 30 19:33:29.370897 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:33:29.370886    4892 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/81b7bb84-1003-4452-b7e0-b4aaba6fefa4-tmp-dir\") pod \"simulator-w1-d52df0-kserve-67f74bc577-q5bjz\" (UID: \"81b7bb84-1003-4452-b7e0-b4aaba6fefa4\") " pod="e2e-models-e2e-worker-w1-d52df0/simulator-w1-d52df0-kserve-67f74bc577-q5bjz"
Oct 30 19:33:29.373172 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:33:29.373147    4892 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/81b7bb84-1003-4452-b7e0-b4aaba6fefa4-tls-certs\") pod \"simulator-w1-d52df0-kserve-67f74bc577-q5bjz\" (UID: \"81b7bb84-1003-4452-b7e0-b4aaba6fefa4\") " pod="e2e-models-e2e-worker-w1-d52df0/simulator-w1-d52df0-kserve-67f74bc577-q5bjz"
Oct 30 19:33:29.381489 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:33:29.381466    4892 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/81b7bb84-1003-4452-b7e0-b4aaba6fefa4-dshm\") pod \"simulator-w1-d52df0-kserve-67f74bc577-q5bjz\" (UID: \"81b7bb84-1003-4452-b7e0-b4aaba6fefa4\") " pod="e2e-models-e2e-worker-w1-d52df0/simulator-w1-d52df0-kserve-67f74bc577-q5bjz"
Oct 30 19:33:29.427412 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:33:29.427375    4892 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-l2tmb\" (UniqueName: \"kubernetes.io/projected/81b7bb84-1003-4452-b7e0-b4aaba6fefa4-kube-api-access-l2tmb\") pod \"simulator-w1-d52df0-kserve-67f74bc577-q5bjz\" (UID: \"81b7bb84-1003-4452-b7e0-b4aaba6fefa4\") " pod="e2e-models-e2e-worker-w1-d52df0/simulator-w1-d52df0-kserve-67f74bc577-q5bjz"
Oct 30 19:33:29.517531 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:33:29.517447    4892 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w1-d52df0/simulator-w1-d52df0-kserve-67f74bc577-q5bjz"
Oct 30 19:33:29.901402 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:33:29.901378    4892 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w1-d52df0/simulator-w1-d52df0-kserve-67f74bc577-q5bjz"]
Oct 30 19:33:29.902929 ip-10-0-132-82 kubenswrapper[4892]: W1030 19:33:29.902899    4892 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod81b7bb84_1003_4452_b7e0_b4aaba6fefa4.slice/crio-def8789d082e6d6ef7854065da675231870b8fb10903ad1f890cebcec9aead50 WatchSource:0}: Error finding container def8789d082e6d6ef7854065da675231870b8fb10903ad1f890cebcec9aead50: Status 404 returned error can't find the container with id def8789d082e6d6ef7854065da675231870b8fb10903ad1f890cebcec9aead50
Oct 30 19:33:30.642150 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:33:30.642106    4892 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w1-d52df0/simulator-w1-d52df0-kserve-67f74bc577-q5bjz" event={"ID":"81b7bb84-1003-4452-b7e0-b4aaba6fefa4","Type":"ContainerStarted","Data":"0b4b42a7b4b61a403cc70c8100f05eff14970c55a7a43951f2c8d8ecda92219c"}
Oct 30 19:33:30.642558 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:33:30.642163    4892 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w1-d52df0/simulator-w1-d52df0-kserve-67f74bc577-q5bjz" event={"ID":"81b7bb84-1003-4452-b7e0-b4aaba6fefa4","Type":"ContainerStarted","Data":"def8789d082e6d6ef7854065da675231870b8fb10903ad1f890cebcec9aead50"}
Oct 30 19:33:30.758846 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:33:30.758792    4892 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w1-d52df0/simulator-w1-d52df0-kserve-67f74bc577-q5bjz" podStartSLOduration=1.758777816 podStartE2EDuration="1.758777816s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:33:29 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:33:30.756912572 +0000 UTC m=+1058.327378175" watchObservedRunningTime="2026-10-30 19:33:30.758777816 +0000 UTC m=+1058.329243414"
Oct 30 19:33:31.518147 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:33:31.518115    4892 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-d52df0/simulator-w1-d52df0-kserve-67f74bc577-q5bjz"
Oct 30 19:33:34.780359 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:33:34.780326    4892 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-fsfvk"]
Oct 30 19:33:39.517859 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:33:39.517823    4892 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w1-d52df0/simulator-w1-d52df0-kserve-67f74bc577-q5bjz"
Oct 30 19:33:39.531417 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:33:39.531394    4892 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w1-d52df0/simulator-w1-d52df0-kserve-67f74bc577-q5bjz"
Oct 30 19:33:39.704558 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:33:39.704528    4892 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-d52df0/simulator-w1-d52df0-kserve-67f74bc577-q5bjz"
Oct 30 19:33:49.128009 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:33:49.127959    4892 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w6-5f55cd/simulator-w6-5f55cd-kserve-5ff86b7447-h9tm7"]
Oct 30 19:33:49.131850 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:33:49.131831    4892 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w6-5f55cd/simulator-w6-5f55cd-kserve-5ff86b7447-h9tm7"
Oct 30 19:33:49.211689 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:33:49.211665    4892 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w6-5f55cd\"/\"openshift-service-ca.crt\""
Oct 30 19:33:49.233059 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:33:49.233028    4892 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w6-5f55cd\"/\"kube-root-ca.crt\""
Oct 30 19:33:49.233059 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:33:49.233031    4892 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w6-5f55cd\"/\"simulator-w6-5f55cd-kserve-self-signed-certs\""
Oct 30 19:33:49.233222 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:33:49.233071    4892 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w6-5f55cd\"/\"default-dockercfg-z7q6k\""
Oct 30 19:33:49.235477 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:33:49.235459    4892 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w6-5f55cd/simulator-w6-5f55cd-kserve-5ff86b7447-h9tm7"]
Oct 30 19:33:49.257271 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:33:49.257248    4892 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/3cb47a99-8be1-4756-b31a-43b58cc6b8dc-dshm\") pod \"simulator-w6-5f55cd-kserve-5ff86b7447-h9tm7\" (UID: \"3cb47a99-8be1-4756-b31a-43b58cc6b8dc\") " pod="e2e-models-e2e-worker-w6-5f55cd/simulator-w6-5f55cd-kserve-5ff86b7447-h9tm7"
Oct 30 19:33:49.257410 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:33:49.257288    4892 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/3cb47a99-8be1-4756-b31a-43b58cc6b8dc-model-cache\") pod \"simulator-w6-5f55cd-kserve-5ff86b7447-h9tm7\" (UID: \"3cb47a99-8be1-4756-b31a-43b58cc6b8dc\") " pod="e2e-models-e2e-worker-w6-5f55cd/simulator-w6-5f55cd-kserve-5ff86b7447-h9tm7"
Oct 30 19:33:49.257486 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:33:49.257466    4892 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3cb47a99-8be1-4756-b31a-43b58cc6b8dc-tmp-dir\") pod \"simulator-w6-5f55cd-kserve-5ff86b7447-h9tm7\" (UID: \"3cb47a99-8be1-4756-b31a-43b58cc6b8dc\") " pod="e2e-models-e2e-worker-w6-5f55cd/simulator-w6-5f55cd-kserve-5ff86b7447-h9tm7"
Oct 30 19:33:49.257543 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:33:49.257514    4892 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/3cb47a99-8be1-4756-b31a-43b58cc6b8dc-home\") pod \"simulator-w6-5f55cd-kserve-5ff86b7447-h9tm7\" (UID: \"3cb47a99-8be1-4756-b31a-43b58cc6b8dc\") " pod="e2e-models-e2e-worker-w6-5f55cd/simulator-w6-5f55cd-kserve-5ff86b7447-h9tm7"
Oct 30 19:33:49.257601 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:33:49.257581    4892 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/3cb47a99-8be1-4756-b31a-43b58cc6b8dc-tls-certs\") pod \"simulator-w6-5f55cd-kserve-5ff86b7447-h9tm7\" (UID: \"3cb47a99-8be1-4756-b31a-43b58cc6b8dc\") " pod="e2e-models-e2e-worker-w6-5f55cd/simulator-w6-5f55cd-kserve-5ff86b7447-h9tm7"
Oct 30 19:33:49.257640 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:33:49.257617    4892 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5bw6x\" (UniqueName: \"kubernetes.io/projected/3cb47a99-8be1-4756-b31a-43b58cc6b8dc-kube-api-access-5bw6x\") pod \"simulator-w6-5f55cd-kserve-5ff86b7447-h9tm7\" (UID: \"3cb47a99-8be1-4756-b31a-43b58cc6b8dc\") " pod="e2e-models-e2e-worker-w6-5f55cd/simulator-w6-5f55cd-kserve-5ff86b7447-h9tm7"
Oct 30 19:33:49.359033 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:33:49.358996    4892 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3cb47a99-8be1-4756-b31a-43b58cc6b8dc-tmp-dir\") pod \"simulator-w6-5f55cd-kserve-5ff86b7447-h9tm7\" (UID: \"3cb47a99-8be1-4756-b31a-43b58cc6b8dc\") " pod="e2e-models-e2e-worker-w6-5f55cd/simulator-w6-5f55cd-kserve-5ff86b7447-h9tm7"
Oct 30 19:33:49.359195 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:33:49.359054    4892 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/3cb47a99-8be1-4756-b31a-43b58cc6b8dc-home\") pod \"simulator-w6-5f55cd-kserve-5ff86b7447-h9tm7\" (UID: \"3cb47a99-8be1-4756-b31a-43b58cc6b8dc\") " pod="e2e-models-e2e-worker-w6-5f55cd/simulator-w6-5f55cd-kserve-5ff86b7447-h9tm7"
Oct 30 19:33:49.359195 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:33:49.359092    4892 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/3cb47a99-8be1-4756-b31a-43b58cc6b8dc-tls-certs\") pod \"simulator-w6-5f55cd-kserve-5ff86b7447-h9tm7\" (UID: \"3cb47a99-8be1-4756-b31a-43b58cc6b8dc\") " pod="e2e-models-e2e-worker-w6-5f55cd/simulator-w6-5f55cd-kserve-5ff86b7447-h9tm7"
Oct 30 19:33:49.359195 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:33:49.359149    4892 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5bw6x\" (UniqueName: \"kubernetes.io/projected/3cb47a99-8be1-4756-b31a-43b58cc6b8dc-kube-api-access-5bw6x\") pod \"simulator-w6-5f55cd-kserve-5ff86b7447-h9tm7\" (UID: \"3cb47a99-8be1-4756-b31a-43b58cc6b8dc\") " pod="e2e-models-e2e-worker-w6-5f55cd/simulator-w6-5f55cd-kserve-5ff86b7447-h9tm7"
Oct 30 19:33:49.359357 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:33:49.359195    4892 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/3cb47a99-8be1-4756-b31a-43b58cc6b8dc-dshm\") pod \"simulator-w6-5f55cd-kserve-5ff86b7447-h9tm7\" (UID: \"3cb47a99-8be1-4756-b31a-43b58cc6b8dc\") " pod="e2e-models-e2e-worker-w6-5f55cd/simulator-w6-5f55cd-kserve-5ff86b7447-h9tm7"
Oct 30 19:33:49.359438 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:33:49.359413    4892 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3cb47a99-8be1-4756-b31a-43b58cc6b8dc-tmp-dir\") pod \"simulator-w6-5f55cd-kserve-5ff86b7447-h9tm7\" (UID: \"3cb47a99-8be1-4756-b31a-43b58cc6b8dc\") " pod="e2e-models-e2e-worker-w6-5f55cd/simulator-w6-5f55cd-kserve-5ff86b7447-h9tm7"
Oct 30 19:33:49.359501 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:33:49.359431    4892 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/3cb47a99-8be1-4756-b31a-43b58cc6b8dc-home\") pod \"simulator-w6-5f55cd-kserve-5ff86b7447-h9tm7\" (UID: \"3cb47a99-8be1-4756-b31a-43b58cc6b8dc\") " pod="e2e-models-e2e-worker-w6-5f55cd/simulator-w6-5f55cd-kserve-5ff86b7447-h9tm7"
Oct 30 19:33:49.359501 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:33:49.359468    4892 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/3cb47a99-8be1-4756-b31a-43b58cc6b8dc-model-cache\") pod \"simulator-w6-5f55cd-kserve-5ff86b7447-h9tm7\" (UID: \"3cb47a99-8be1-4756-b31a-43b58cc6b8dc\") " pod="e2e-models-e2e-worker-w6-5f55cd/simulator-w6-5f55cd-kserve-5ff86b7447-h9tm7"
Oct 30 19:33:49.359743 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:33:49.359718    4892 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/3cb47a99-8be1-4756-b31a-43b58cc6b8dc-model-cache\") pod \"simulator-w6-5f55cd-kserve-5ff86b7447-h9tm7\" (UID: \"3cb47a99-8be1-4756-b31a-43b58cc6b8dc\") " pod="e2e-models-e2e-worker-w6-5f55cd/simulator-w6-5f55cd-kserve-5ff86b7447-h9tm7"
Oct 30 19:33:49.361687 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:33:49.361666    4892 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/3cb47a99-8be1-4756-b31a-43b58cc6b8dc-dshm\") pod \"simulator-w6-5f55cd-kserve-5ff86b7447-h9tm7\" (UID: \"3cb47a99-8be1-4756-b31a-43b58cc6b8dc\") " pod="e2e-models-e2e-worker-w6-5f55cd/simulator-w6-5f55cd-kserve-5ff86b7447-h9tm7"
Oct 30 19:33:49.363183 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:33:49.363163    4892 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/3cb47a99-8be1-4756-b31a-43b58cc6b8dc-tls-certs\") pod \"simulator-w6-5f55cd-kserve-5ff86b7447-h9tm7\" (UID: \"3cb47a99-8be1-4756-b31a-43b58cc6b8dc\") " pod="e2e-models-e2e-worker-w6-5f55cd/simulator-w6-5f55cd-kserve-5ff86b7447-h9tm7"
Oct 30 19:33:49.425178 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:33:49.425149    4892 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5bw6x\" (UniqueName: \"kubernetes.io/projected/3cb47a99-8be1-4756-b31a-43b58cc6b8dc-kube-api-access-5bw6x\") pod \"simulator-w6-5f55cd-kserve-5ff86b7447-h9tm7\" (UID: \"3cb47a99-8be1-4756-b31a-43b58cc6b8dc\") " pod="e2e-models-e2e-worker-w6-5f55cd/simulator-w6-5f55cd-kserve-5ff86b7447-h9tm7"
Oct 30 19:33:49.445084 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:33:49.445061    4892 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w6-5f55cd/simulator-w6-5f55cd-kserve-5ff86b7447-h9tm7"
Oct 30 19:33:49.820847 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:33:49.820818    4892 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w6-5f55cd/simulator-w6-5f55cd-kserve-5ff86b7447-h9tm7"]
Oct 30 19:33:49.823042 ip-10-0-132-82 kubenswrapper[4892]: W1030 19:33:49.823013    4892 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod3cb47a99_8be1_4756_b31a_43b58cc6b8dc.slice/crio-2e98566bd9e10fbf0e8b578d31f14ac5c2f80e0d7ca7f2f1ea90d2804cd2503c WatchSource:0}: Error finding container 2e98566bd9e10fbf0e8b578d31f14ac5c2f80e0d7ca7f2f1ea90d2804cd2503c: Status 404 returned error can't find the container with id 2e98566bd9e10fbf0e8b578d31f14ac5c2f80e0d7ca7f2f1ea90d2804cd2503c
Oct 30 19:33:50.754628 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:33:50.754600    4892 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w6-5f55cd/simulator-w6-5f55cd-kserve-5ff86b7447-h9tm7" event={"ID":"3cb47a99-8be1-4756-b31a-43b58cc6b8dc","Type":"ContainerStarted","Data":"7fb9393a625a7900385890a2e7393fa67154902bd3f995959de80d728f5fe573"}
Oct 30 19:33:50.755040 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:33:50.754643    4892 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w6-5f55cd/simulator-w6-5f55cd-kserve-5ff86b7447-h9tm7" event={"ID":"3cb47a99-8be1-4756-b31a-43b58cc6b8dc","Type":"ContainerStarted","Data":"2e98566bd9e10fbf0e8b578d31f14ac5c2f80e0d7ca7f2f1ea90d2804cd2503c"}
Oct 30 19:33:50.870989 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:33:50.870918    4892 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w6-5f55cd/simulator-w6-5f55cd-kserve-5ff86b7447-h9tm7" podStartSLOduration=1.870904587 podStartE2EDuration="1.870904587s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:33:49 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:33:50.869591908 +0000 UTC m=+1078.440057507" watchObservedRunningTime="2026-10-30 19:33:50.870904587 +0000 UTC m=+1078.441370185"
Oct 30 19:33:51.445399 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:33:51.445367    4892 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w6-5f55cd/simulator-w6-5f55cd-kserve-5ff86b7447-h9tm7"
Oct 30 19:33:54.843841 ip-10-0-132-82 kubenswrapper[4892]: E1030 19:33:54.843784    4892 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-jrm87" containerName="extractor"
Oct 30 19:33:56.773920 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:33:56.773880    4892 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-fsfvk"]
Oct 30 19:33:59.445565 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:33:59.445525    4892 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w6-5f55cd/simulator-w6-5f55cd-kserve-5ff86b7447-h9tm7"
Oct 30 19:33:59.463884 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:33:59.463856    4892 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w6-5f55cd/simulator-w6-5f55cd-kserve-5ff86b7447-h9tm7"
Oct 30 19:33:59.825804 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:33:59.825723    4892 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w6-5f55cd/simulator-w6-5f55cd-kserve-5ff86b7447-h9tm7"
Oct 30 19:34:07.102192 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:34:07.102160    4892 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w2-a3f9b7/simulator-w2-a3f9b7-kserve-79bbc4cb5f-8m488"]
Oct 30 19:34:07.105834 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:34:07.105814    4892 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w2-a3f9b7/simulator-w2-a3f9b7-kserve-79bbc4cb5f-8m488"
Oct 30 19:34:07.149425 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:34:07.149393    4892 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/50fb2283-bf3f-41da-835b-f407ee1a04f3-home\") pod \"simulator-w2-a3f9b7-kserve-79bbc4cb5f-8m488\" (UID: \"50fb2283-bf3f-41da-835b-f407ee1a04f3\") " pod="e2e-models-e2e-worker-w2-a3f9b7/simulator-w2-a3f9b7-kserve-79bbc4cb5f-8m488"
Oct 30 19:34:07.149425 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:34:07.149432    4892 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/50fb2283-bf3f-41da-835b-f407ee1a04f3-model-cache\") pod \"simulator-w2-a3f9b7-kserve-79bbc4cb5f-8m488\" (UID: \"50fb2283-bf3f-41da-835b-f407ee1a04f3\") " pod="e2e-models-e2e-worker-w2-a3f9b7/simulator-w2-a3f9b7-kserve-79bbc4cb5f-8m488"
Oct 30 19:34:07.149650 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:34:07.149478    4892 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/50fb2283-bf3f-41da-835b-f407ee1a04f3-tls-certs\") pod \"simulator-w2-a3f9b7-kserve-79bbc4cb5f-8m488\" (UID: \"50fb2283-bf3f-41da-835b-f407ee1a04f3\") " pod="e2e-models-e2e-worker-w2-a3f9b7/simulator-w2-a3f9b7-kserve-79bbc4cb5f-8m488"
Oct 30 19:34:07.149650 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:34:07.149572    4892 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/50fb2283-bf3f-41da-835b-f407ee1a04f3-dshm\") pod \"simulator-w2-a3f9b7-kserve-79bbc4cb5f-8m488\" (UID: \"50fb2283-bf3f-41da-835b-f407ee1a04f3\") " pod="e2e-models-e2e-worker-w2-a3f9b7/simulator-w2-a3f9b7-kserve-79bbc4cb5f-8m488"
Oct 30 19:34:07.149824 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:34:07.149662    4892 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qx6vd\" (UniqueName: \"kubernetes.io/projected/50fb2283-bf3f-41da-835b-f407ee1a04f3-kube-api-access-qx6vd\") pod \"simulator-w2-a3f9b7-kserve-79bbc4cb5f-8m488\" (UID: \"50fb2283-bf3f-41da-835b-f407ee1a04f3\") " pod="e2e-models-e2e-worker-w2-a3f9b7/simulator-w2-a3f9b7-kserve-79bbc4cb5f-8m488"
Oct 30 19:34:07.149824 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:34:07.149711    4892 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/50fb2283-bf3f-41da-835b-f407ee1a04f3-tmp-dir\") pod \"simulator-w2-a3f9b7-kserve-79bbc4cb5f-8m488\" (UID: \"50fb2283-bf3f-41da-835b-f407ee1a04f3\") " pod="e2e-models-e2e-worker-w2-a3f9b7/simulator-w2-a3f9b7-kserve-79bbc4cb5f-8m488"
Oct 30 19:34:07.157731 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:34:07.157708    4892 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w2-a3f9b7\"/\"kube-root-ca.crt\""
Oct 30 19:34:07.207045 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:34:07.207020    4892 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w2-a3f9b7\"/\"simulator-w2-a3f9b7-kserve-self-signed-certs\""
Oct 30 19:34:07.207045 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:34:07.207036    4892 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w2-a3f9b7\"/\"default-dockercfg-x47rt\""
Oct 30 19:34:07.207212 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:34:07.207107    4892 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w2-a3f9b7\"/\"openshift-service-ca.crt\""
Oct 30 19:34:07.209369 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:34:07.209350    4892 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w2-a3f9b7/simulator-w2-a3f9b7-kserve-79bbc4cb5f-8m488"]
Oct 30 19:34:07.271130 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:34:07.271093    4892 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qx6vd\" (UniqueName: \"kubernetes.io/projected/50fb2283-bf3f-41da-835b-f407ee1a04f3-kube-api-access-qx6vd\") pod \"simulator-w2-a3f9b7-kserve-79bbc4cb5f-8m488\" (UID: \"50fb2283-bf3f-41da-835b-f407ee1a04f3\") " pod="e2e-models-e2e-worker-w2-a3f9b7/simulator-w2-a3f9b7-kserve-79bbc4cb5f-8m488"
Oct 30 19:34:07.271331 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:34:07.271165    4892 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/50fb2283-bf3f-41da-835b-f407ee1a04f3-tmp-dir\") pod \"simulator-w2-a3f9b7-kserve-79bbc4cb5f-8m488\" (UID: \"50fb2283-bf3f-41da-835b-f407ee1a04f3\") " pod="e2e-models-e2e-worker-w2-a3f9b7/simulator-w2-a3f9b7-kserve-79bbc4cb5f-8m488"
Oct 30 19:34:07.271331 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:34:07.271275    4892 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/50fb2283-bf3f-41da-835b-f407ee1a04f3-home\") pod \"simulator-w2-a3f9b7-kserve-79bbc4cb5f-8m488\" (UID: \"50fb2283-bf3f-41da-835b-f407ee1a04f3\") " pod="e2e-models-e2e-worker-w2-a3f9b7/simulator-w2-a3f9b7-kserve-79bbc4cb5f-8m488"
Oct 30 19:34:07.271331 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:34:07.271308    4892 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/50fb2283-bf3f-41da-835b-f407ee1a04f3-model-cache\") pod \"simulator-w2-a3f9b7-kserve-79bbc4cb5f-8m488\" (UID: \"50fb2283-bf3f-41da-835b-f407ee1a04f3\") " pod="e2e-models-e2e-worker-w2-a3f9b7/simulator-w2-a3f9b7-kserve-79bbc4cb5f-8m488"
Oct 30 19:34:07.271509 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:34:07.271334    4892 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/50fb2283-bf3f-41da-835b-f407ee1a04f3-tls-certs\") pod \"simulator-w2-a3f9b7-kserve-79bbc4cb5f-8m488\" (UID: \"50fb2283-bf3f-41da-835b-f407ee1a04f3\") " pod="e2e-models-e2e-worker-w2-a3f9b7/simulator-w2-a3f9b7-kserve-79bbc4cb5f-8m488"
Oct 30 19:34:07.271509 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:34:07.271364    4892 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/50fb2283-bf3f-41da-835b-f407ee1a04f3-dshm\") pod \"simulator-w2-a3f9b7-kserve-79bbc4cb5f-8m488\" (UID: \"50fb2283-bf3f-41da-835b-f407ee1a04f3\") " pod="e2e-models-e2e-worker-w2-a3f9b7/simulator-w2-a3f9b7-kserve-79bbc4cb5f-8m488"
Oct 30 19:34:07.272308 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:34:07.272237    4892 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/50fb2283-bf3f-41da-835b-f407ee1a04f3-home\") pod \"simulator-w2-a3f9b7-kserve-79bbc4cb5f-8m488\" (UID: \"50fb2283-bf3f-41da-835b-f407ee1a04f3\") " pod="e2e-models-e2e-worker-w2-a3f9b7/simulator-w2-a3f9b7-kserve-79bbc4cb5f-8m488"
Oct 30 19:34:07.272846 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:34:07.272821    4892 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/50fb2283-bf3f-41da-835b-f407ee1a04f3-tmp-dir\") pod \"simulator-w2-a3f9b7-kserve-79bbc4cb5f-8m488\" (UID: \"50fb2283-bf3f-41da-835b-f407ee1a04f3\") " pod="e2e-models-e2e-worker-w2-a3f9b7/simulator-w2-a3f9b7-kserve-79bbc4cb5f-8m488"
Oct 30 19:34:07.273483 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:34:07.273426    4892 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/50fb2283-bf3f-41da-835b-f407ee1a04f3-model-cache\") pod \"simulator-w2-a3f9b7-kserve-79bbc4cb5f-8m488\" (UID: \"50fb2283-bf3f-41da-835b-f407ee1a04f3\") " pod="e2e-models-e2e-worker-w2-a3f9b7/simulator-w2-a3f9b7-kserve-79bbc4cb5f-8m488"
Oct 30 19:34:07.280326 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:34:07.280293    4892 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/50fb2283-bf3f-41da-835b-f407ee1a04f3-dshm\") pod \"simulator-w2-a3f9b7-kserve-79bbc4cb5f-8m488\" (UID: \"50fb2283-bf3f-41da-835b-f407ee1a04f3\") " pod="e2e-models-e2e-worker-w2-a3f9b7/simulator-w2-a3f9b7-kserve-79bbc4cb5f-8m488"
Oct 30 19:34:07.280826 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:34:07.280797    4892 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/50fb2283-bf3f-41da-835b-f407ee1a04f3-tls-certs\") pod \"simulator-w2-a3f9b7-kserve-79bbc4cb5f-8m488\" (UID: \"50fb2283-bf3f-41da-835b-f407ee1a04f3\") " pod="e2e-models-e2e-worker-w2-a3f9b7/simulator-w2-a3f9b7-kserve-79bbc4cb5f-8m488"
Oct 30 19:34:07.329188 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:34:07.329164    4892 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qx6vd\" (UniqueName: \"kubernetes.io/projected/50fb2283-bf3f-41da-835b-f407ee1a04f3-kube-api-access-qx6vd\") pod \"simulator-w2-a3f9b7-kserve-79bbc4cb5f-8m488\" (UID: \"50fb2283-bf3f-41da-835b-f407ee1a04f3\") " pod="e2e-models-e2e-worker-w2-a3f9b7/simulator-w2-a3f9b7-kserve-79bbc4cb5f-8m488"
Oct 30 19:34:07.417415 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:34:07.417368    4892 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w2-a3f9b7/simulator-w2-a3f9b7-kserve-79bbc4cb5f-8m488"
Oct 30 19:34:07.800771 ip-10-0-132-82 kubenswrapper[4892]: W1030 19:34:07.800739    4892 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod50fb2283_bf3f_41da_835b_f407ee1a04f3.slice/crio-feb4c41f668c3ab5520bd5aad6c2b701f236b226b31ee5b48d3c5f77b93c3eb5 WatchSource:0}: Error finding container feb4c41f668c3ab5520bd5aad6c2b701f236b226b31ee5b48d3c5f77b93c3eb5: Status 404 returned error can't find the container with id feb4c41f668c3ab5520bd5aad6c2b701f236b226b31ee5b48d3c5f77b93c3eb5
Oct 30 19:34:07.810068 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:34:07.810041    4892 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w2-a3f9b7/simulator-w2-a3f9b7-kserve-79bbc4cb5f-8m488"]
Oct 30 19:34:07.859214 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:34:07.859184    4892 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w2-a3f9b7/simulator-w2-a3f9b7-kserve-79bbc4cb5f-8m488" event={"ID":"50fb2283-bf3f-41da-835b-f407ee1a04f3","Type":"ContainerStarted","Data":"feb4c41f668c3ab5520bd5aad6c2b701f236b226b31ee5b48d3c5f77b93c3eb5"}
Oct 30 19:34:08.866475 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:34:08.866434    4892 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w2-a3f9b7/simulator-w2-a3f9b7-kserve-79bbc4cb5f-8m488" event={"ID":"50fb2283-bf3f-41da-835b-f407ee1a04f3","Type":"ContainerStarted","Data":"222b1703c0f545b6c40b1052718766033672f7cc4aacaa0061ab5fafce867dc9"}
Oct 30 19:34:08.981935 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:34:08.981879    4892 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w2-a3f9b7/simulator-w2-a3f9b7-kserve-79bbc4cb5f-8m488" podStartSLOduration=1.981865327 podStartE2EDuration="1.981865327s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:34:07 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:34:08.980249246 +0000 UTC m=+1096.550714845" watchObservedRunningTime="2026-10-30 19:34:08.981865327 +0000 UTC m=+1096.552330926"
Oct 30 19:34:09.417729 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:34:09.417702    4892 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-a3f9b7/simulator-w2-a3f9b7-kserve-79bbc4cb5f-8m488"
Oct 30 19:34:16.290185 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:34:16.290150    4892 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-fsfvk"]
Oct 30 19:34:17.418158 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:34:17.418122    4892 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w2-a3f9b7/simulator-w2-a3f9b7-kserve-79bbc4cb5f-8m488"
Oct 30 19:34:17.433680 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:34:17.433657    4892 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w2-a3f9b7/simulator-w2-a3f9b7-kserve-79bbc4cb5f-8m488"
Oct 30 19:34:17.932568 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:34:17.932540    4892 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-a3f9b7/simulator-w2-a3f9b7-kserve-79bbc4cb5f-8m488"
Oct 30 19:34:24.889681 ip-10-0-132-82 kubenswrapper[4892]: E1030 19:34:24.889639    4892 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-jrm87" containerName="extractor"
Oct 30 19:34:32.285549 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:34:32.285475    4892 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-fsfvk"]
Oct 30 19:34:48.830247 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:34:48.830210    4892 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-fsfvk"]
Oct 30 19:34:54.844046 ip-10-0-132-82 kubenswrapper[4892]: E1030 19:34:54.844006    4892 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-jrm87" containerName="extractor"
Oct 30 19:34:57.569804 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:34:57.569767    4892 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w6-5f55cd/premium-w6-5f55cd-kserve-7f785c544b-pcvhb"]
Oct 30 19:34:57.573126 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:34:57.573108    4892 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w6-5f55cd/premium-w6-5f55cd-kserve-7f785c544b-pcvhb"
Oct 30 19:34:57.674556 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:34:57.674527    4892 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w6-5f55cd\"/\"premium-w6-5f55cd-kserve-self-signed-certs\""
Oct 30 19:34:57.676986 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:34:57.676952    4892 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w6-5f55cd/premium-w6-5f55cd-kserve-7f785c544b-pcvhb"]
Oct 30 19:34:57.682995 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:34:57.682941    4892 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/191593a2-e3a9-4ce4-a1c6-c089723b0237-dshm\") pod \"premium-w6-5f55cd-kserve-7f785c544b-pcvhb\" (UID: \"191593a2-e3a9-4ce4-a1c6-c089723b0237\") " pod="e2e-models-e2e-worker-w6-5f55cd/premium-w6-5f55cd-kserve-7f785c544b-pcvhb"
Oct 30 19:34:57.683119 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:34:57.683001    4892 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/191593a2-e3a9-4ce4-a1c6-c089723b0237-home\") pod \"premium-w6-5f55cd-kserve-7f785c544b-pcvhb\" (UID: \"191593a2-e3a9-4ce4-a1c6-c089723b0237\") " pod="e2e-models-e2e-worker-w6-5f55cd/premium-w6-5f55cd-kserve-7f785c544b-pcvhb"
Oct 30 19:34:57.683169 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:34:57.683117    4892 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/191593a2-e3a9-4ce4-a1c6-c089723b0237-tmp-dir\") pod \"premium-w6-5f55cd-kserve-7f785c544b-pcvhb\" (UID: \"191593a2-e3a9-4ce4-a1c6-c089723b0237\") " pod="e2e-models-e2e-worker-w6-5f55cd/premium-w6-5f55cd-kserve-7f785c544b-pcvhb"
Oct 30 19:34:57.683169 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:34:57.683149    4892 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-g5bj7\" (UniqueName: \"kubernetes.io/projected/191593a2-e3a9-4ce4-a1c6-c089723b0237-kube-api-access-g5bj7\") pod \"premium-w6-5f55cd-kserve-7f785c544b-pcvhb\" (UID: \"191593a2-e3a9-4ce4-a1c6-c089723b0237\") " pod="e2e-models-e2e-worker-w6-5f55cd/premium-w6-5f55cd-kserve-7f785c544b-pcvhb"
Oct 30 19:34:57.683241 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:34:57.683177    4892 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/191593a2-e3a9-4ce4-a1c6-c089723b0237-tls-certs\") pod \"premium-w6-5f55cd-kserve-7f785c544b-pcvhb\" (UID: \"191593a2-e3a9-4ce4-a1c6-c089723b0237\") " pod="e2e-models-e2e-worker-w6-5f55cd/premium-w6-5f55cd-kserve-7f785c544b-pcvhb"
Oct 30 19:34:57.683370 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:34:57.683355    4892 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/191593a2-e3a9-4ce4-a1c6-c089723b0237-model-cache\") pod \"premium-w6-5f55cd-kserve-7f785c544b-pcvhb\" (UID: \"191593a2-e3a9-4ce4-a1c6-c089723b0237\") " pod="e2e-models-e2e-worker-w6-5f55cd/premium-w6-5f55cd-kserve-7f785c544b-pcvhb"
Oct 30 19:34:57.784845 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:34:57.784807    4892 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/191593a2-e3a9-4ce4-a1c6-c089723b0237-dshm\") pod \"premium-w6-5f55cd-kserve-7f785c544b-pcvhb\" (UID: \"191593a2-e3a9-4ce4-a1c6-c089723b0237\") " pod="e2e-models-e2e-worker-w6-5f55cd/premium-w6-5f55cd-kserve-7f785c544b-pcvhb"
Oct 30 19:34:57.785056 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:34:57.784864    4892 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/191593a2-e3a9-4ce4-a1c6-c089723b0237-home\") pod \"premium-w6-5f55cd-kserve-7f785c544b-pcvhb\" (UID: \"191593a2-e3a9-4ce4-a1c6-c089723b0237\") " pod="e2e-models-e2e-worker-w6-5f55cd/premium-w6-5f55cd-kserve-7f785c544b-pcvhb"
Oct 30 19:34:57.785056 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:34:57.784942    4892 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/191593a2-e3a9-4ce4-a1c6-c089723b0237-tmp-dir\") pod \"premium-w6-5f55cd-kserve-7f785c544b-pcvhb\" (UID: \"191593a2-e3a9-4ce4-a1c6-c089723b0237\") " pod="e2e-models-e2e-worker-w6-5f55cd/premium-w6-5f55cd-kserve-7f785c544b-pcvhb"
Oct 30 19:34:57.785056 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:34:57.784993    4892 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-g5bj7\" (UniqueName: \"kubernetes.io/projected/191593a2-e3a9-4ce4-a1c6-c089723b0237-kube-api-access-g5bj7\") pod \"premium-w6-5f55cd-kserve-7f785c544b-pcvhb\" (UID: \"191593a2-e3a9-4ce4-a1c6-c089723b0237\") " pod="e2e-models-e2e-worker-w6-5f55cd/premium-w6-5f55cd-kserve-7f785c544b-pcvhb"
Oct 30 19:34:57.785056 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:34:57.785022    4892 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/191593a2-e3a9-4ce4-a1c6-c089723b0237-tls-certs\") pod \"premium-w6-5f55cd-kserve-7f785c544b-pcvhb\" (UID: \"191593a2-e3a9-4ce4-a1c6-c089723b0237\") " pod="e2e-models-e2e-worker-w6-5f55cd/premium-w6-5f55cd-kserve-7f785c544b-pcvhb"
Oct 30 19:34:57.785056 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:34:57.785046    4892 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/191593a2-e3a9-4ce4-a1c6-c089723b0237-model-cache\") pod \"premium-w6-5f55cd-kserve-7f785c544b-pcvhb\" (UID: \"191593a2-e3a9-4ce4-a1c6-c089723b0237\") " pod="e2e-models-e2e-worker-w6-5f55cd/premium-w6-5f55cd-kserve-7f785c544b-pcvhb"
Oct 30 19:34:57.785395 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:34:57.785364    4892 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/191593a2-e3a9-4ce4-a1c6-c089723b0237-tmp-dir\") pod \"premium-w6-5f55cd-kserve-7f785c544b-pcvhb\" (UID: \"191593a2-e3a9-4ce4-a1c6-c089723b0237\") " pod="e2e-models-e2e-worker-w6-5f55cd/premium-w6-5f55cd-kserve-7f785c544b-pcvhb"
Oct 30 19:34:57.785550 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:34:57.785522    4892 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/191593a2-e3a9-4ce4-a1c6-c089723b0237-model-cache\") pod \"premium-w6-5f55cd-kserve-7f785c544b-pcvhb\" (UID: \"191593a2-e3a9-4ce4-a1c6-c089723b0237\") " pod="e2e-models-e2e-worker-w6-5f55cd/premium-w6-5f55cd-kserve-7f785c544b-pcvhb"
Oct 30 19:34:57.785676 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:34:57.785560    4892 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/191593a2-e3a9-4ce4-a1c6-c089723b0237-home\") pod \"premium-w6-5f55cd-kserve-7f785c544b-pcvhb\" (UID: \"191593a2-e3a9-4ce4-a1c6-c089723b0237\") " pod="e2e-models-e2e-worker-w6-5f55cd/premium-w6-5f55cd-kserve-7f785c544b-pcvhb"
Oct 30 19:34:57.787581 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:34:57.787561    4892 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/191593a2-e3a9-4ce4-a1c6-c089723b0237-dshm\") pod \"premium-w6-5f55cd-kserve-7f785c544b-pcvhb\" (UID: \"191593a2-e3a9-4ce4-a1c6-c089723b0237\") " pod="e2e-models-e2e-worker-w6-5f55cd/premium-w6-5f55cd-kserve-7f785c544b-pcvhb"
Oct 30 19:34:57.787884 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:34:57.787861    4892 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/191593a2-e3a9-4ce4-a1c6-c089723b0237-tls-certs\") pod \"premium-w6-5f55cd-kserve-7f785c544b-pcvhb\" (UID: \"191593a2-e3a9-4ce4-a1c6-c089723b0237\") " pod="e2e-models-e2e-worker-w6-5f55cd/premium-w6-5f55cd-kserve-7f785c544b-pcvhb"
Oct 30 19:34:57.842200 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:34:57.842118    4892 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-g5bj7\" (UniqueName: \"kubernetes.io/projected/191593a2-e3a9-4ce4-a1c6-c089723b0237-kube-api-access-g5bj7\") pod \"premium-w6-5f55cd-kserve-7f785c544b-pcvhb\" (UID: \"191593a2-e3a9-4ce4-a1c6-c089723b0237\") " pod="e2e-models-e2e-worker-w6-5f55cd/premium-w6-5f55cd-kserve-7f785c544b-pcvhb"
Oct 30 19:34:57.885765 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:34:57.885506    4892 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w6-5f55cd/premium-w6-5f55cd-kserve-7f785c544b-pcvhb"
Oct 30 19:34:58.324993 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:34:58.324937    4892 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w6-5f55cd/premium-w6-5f55cd-kserve-7f785c544b-pcvhb"]
Oct 30 19:34:58.326888 ip-10-0-132-82 kubenswrapper[4892]: W1030 19:34:58.326849    4892 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod191593a2_e3a9_4ce4_a1c6_c089723b0237.slice/crio-5a6384cc7c027b634daa052bc2016391b33e529904c98c29118b9d39683ea050 WatchSource:0}: Error finding container 5a6384cc7c027b634daa052bc2016391b33e529904c98c29118b9d39683ea050: Status 404 returned error can't find the container with id 5a6384cc7c027b634daa052bc2016391b33e529904c98c29118b9d39683ea050
Oct 30 19:34:59.160555 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:34:59.160519    4892 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w6-5f55cd/premium-w6-5f55cd-kserve-7f785c544b-pcvhb" event={"ID":"191593a2-e3a9-4ce4-a1c6-c089723b0237","Type":"ContainerStarted","Data":"1704c766f16e7a726d7dcfdc4866656f49d41dcd941c72047c420a8145a5e9f2"}
Oct 30 19:34:59.160555 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:34:59.160563    4892 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w6-5f55cd/premium-w6-5f55cd-kserve-7f785c544b-pcvhb" event={"ID":"191593a2-e3a9-4ce4-a1c6-c089723b0237","Type":"ContainerStarted","Data":"5a6384cc7c027b634daa052bc2016391b33e529904c98c29118b9d39683ea050"}
Oct 30 19:34:59.276422 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:34:59.276369    4892 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w6-5f55cd/premium-w6-5f55cd-kserve-7f785c544b-pcvhb" podStartSLOduration=2.2763554040000002 podStartE2EDuration="2.276355404s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:34:57 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:34:59.273541974 +0000 UTC m=+1146.844007557" watchObservedRunningTime="2026-10-30 19:34:59.276355404 +0000 UTC m=+1146.846821036"
Oct 30 19:34:59.885668 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:34:59.885638    4892 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w6-5f55cd/premium-w6-5f55cd-kserve-7f785c544b-pcvhb"
Oct 30 19:35:05.842063 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:35:05.842027    4892 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-fsfvk"]
Oct 30 19:35:07.885818 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:35:07.885787    4892 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w6-5f55cd/premium-w6-5f55cd-kserve-7f785c544b-pcvhb"
Oct 30 19:35:07.899214 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:35:07.899183    4892 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w6-5f55cd/premium-w6-5f55cd-kserve-7f785c544b-pcvhb"
Oct 30 19:35:08.221689 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:35:08.221611    4892 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w6-5f55cd/premium-w6-5f55cd-kserve-7f785c544b-pcvhb"
Oct 30 19:35:15.267592 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:35:15.267559    4892 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-6e1ad926-4tgkm"]
Oct 30 19:35:15.271349 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:35:15.271332    4892 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6e1ad926-4tgkm"
Oct 30 19:35:15.323134 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:35:15.323109    4892 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 19:35:15.323256 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:35:15.323203    4892 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 30 19:35:15.370391 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:35:15.370361    4892 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/87c18cb1-32c8-4596-84cc-05375b9e760c-maas-api-xks-ca\") pod \"maas-api-revoke-sub-6e1ad926-4tgkm\" (UID: \"87c18cb1-32c8-4596-84cc-05375b9e760c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6e1ad926-4tgkm"
Oct 30 19:35:15.370545 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:35:15.370529    4892 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jlrv8\" (UniqueName: \"kubernetes.io/projected/87c18cb1-32c8-4596-84cc-05375b9e760c-kube-api-access-jlrv8\") pod \"maas-api-revoke-sub-6e1ad926-4tgkm\" (UID: \"87c18cb1-32c8-4596-84cc-05375b9e760c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6e1ad926-4tgkm"
Oct 30 19:35:15.370623 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:35:15.370604    4892 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/87c18cb1-32c8-4596-84cc-05375b9e760c-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-6e1ad926-4tgkm\" (UID: \"87c18cb1-32c8-4596-84cc-05375b9e760c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6e1ad926-4tgkm"
Oct 30 19:35:15.372547 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:35:15.372520    4892 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 30 19:35:15.372638 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:35:15.372577    4892 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-nlxp4\""
Oct 30 19:35:15.375061 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:35:15.375036    4892 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-6e1ad926-4tgkm"]
Oct 30 19:35:15.471577 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:35:15.471540    4892 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jlrv8\" (UniqueName: \"kubernetes.io/projected/87c18cb1-32c8-4596-84cc-05375b9e760c-kube-api-access-jlrv8\") pod \"maas-api-revoke-sub-6e1ad926-4tgkm\" (UID: \"87c18cb1-32c8-4596-84cc-05375b9e760c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6e1ad926-4tgkm"
Oct 30 19:35:15.471753 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:35:15.471585    4892 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/87c18cb1-32c8-4596-84cc-05375b9e760c-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-6e1ad926-4tgkm\" (UID: \"87c18cb1-32c8-4596-84cc-05375b9e760c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6e1ad926-4tgkm"
Oct 30 19:35:15.471753 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:35:15.471645    4892 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/87c18cb1-32c8-4596-84cc-05375b9e760c-maas-api-xks-ca\") pod \"maas-api-revoke-sub-6e1ad926-4tgkm\" (UID: \"87c18cb1-32c8-4596-84cc-05375b9e760c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6e1ad926-4tgkm"
Oct 30 19:35:15.472553 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:35:15.472523    4892 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/87c18cb1-32c8-4596-84cc-05375b9e760c-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-6e1ad926-4tgkm\" (UID: \"87c18cb1-32c8-4596-84cc-05375b9e760c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6e1ad926-4tgkm"
Oct 30 19:35:15.474167 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:35:15.474145    4892 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/87c18cb1-32c8-4596-84cc-05375b9e760c-maas-api-xks-ca\") pod \"maas-api-revoke-sub-6e1ad926-4tgkm\" (UID: \"87c18cb1-32c8-4596-84cc-05375b9e760c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6e1ad926-4tgkm"
Oct 30 19:35:15.528719 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:35:15.528646    4892 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jlrv8\" (UniqueName: \"kubernetes.io/projected/87c18cb1-32c8-4596-84cc-05375b9e760c-kube-api-access-jlrv8\") pod \"maas-api-revoke-sub-6e1ad926-4tgkm\" (UID: \"87c18cb1-32c8-4596-84cc-05375b9e760c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6e1ad926-4tgkm"
Oct 30 19:35:15.585223 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:35:15.585200    4892 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6e1ad926-4tgkm"
Oct 30 19:35:15.986608 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:35:15.986580    4892 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-6e1ad926-4tgkm"]
Oct 30 19:35:15.991576 ip-10-0-132-82 kubenswrapper[4892]: W1030 19:35:15.991547    4892 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod87c18cb1_32c8_4596_84cc_05375b9e760c.slice/crio-964c5cc08f701e9928505ffb3d2362d7133a3d672d2ce1cd06ef49b687a9ab09 WatchSource:0}: Error finding container 964c5cc08f701e9928505ffb3d2362d7133a3d672d2ce1cd06ef49b687a9ab09: Status 404 returned error can't find the container with id 964c5cc08f701e9928505ffb3d2362d7133a3d672d2ce1cd06ef49b687a9ab09
Oct 30 19:35:15.994019 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:35:15.994001    4892 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Oct 30 19:35:16.260461 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:35:16.260379    4892 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6e1ad926-4tgkm" event={"ID":"87c18cb1-32c8-4596-84cc-05375b9e760c","Type":"ContainerStarted","Data":"964c5cc08f701e9928505ffb3d2362d7133a3d672d2ce1cd06ef49b687a9ab09"}
Oct 30 19:35:23.301760 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:35:23.301729    4892 generic.go:356] "Generic (PLEG): container finished" podID="87c18cb1-32c8-4596-84cc-05375b9e760c" containerID="17ad1117604568a4d162450050b83f2cc2b23174d347e12d57d9963279e1bbb6" exitCode=0
Oct 30 19:35:23.302156 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:35:23.301812    4892 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6e1ad926-4tgkm" event={"ID":"87c18cb1-32c8-4596-84cc-05375b9e760c","Type":"ContainerDied","Data":"17ad1117604568a4d162450050b83f2cc2b23174d347e12d57d9963279e1bbb6"}
Oct 30 19:35:23.791717 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:35:23.791673    4892 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-fsfvk"]
Oct 30 19:35:24.544590 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:35:24.544338    4892 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6e1ad926-4tgkm"
Oct 30 19:35:24.666510 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:35:24.666479    4892 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/87c18cb1-32c8-4596-84cc-05375b9e760c-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/87c18cb1-32c8-4596-84cc-05375b9e760c-maas-api-xks-ca\") pod \"87c18cb1-32c8-4596-84cc-05375b9e760c\" (UID: \"87c18cb1-32c8-4596-84cc-05375b9e760c\") "
Oct 30 19:35:24.666710 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:35:24.666550    4892 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/87c18cb1-32c8-4596-84cc-05375b9e760c-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/87c18cb1-32c8-4596-84cc-05375b9e760c-maas-api-openshift-ca\") pod \"87c18cb1-32c8-4596-84cc-05375b9e760c\" (UID: \"87c18cb1-32c8-4596-84cc-05375b9e760c\") "
Oct 30 19:35:24.666710 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:35:24.666685    4892 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/87c18cb1-32c8-4596-84cc-05375b9e760c-kube-api-access-jlrv8\" (UniqueName: \"kubernetes.io/projected/87c18cb1-32c8-4596-84cc-05375b9e760c-kube-api-access-jlrv8\") pod \"87c18cb1-32c8-4596-84cc-05375b9e760c\" (UID: \"87c18cb1-32c8-4596-84cc-05375b9e760c\") "
Oct 30 19:35:24.667492 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:35:24.667455    4892 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/87c18cb1-32c8-4596-84cc-05375b9e760c-maas-api-openshift-ca" pod "87c18cb1-32c8-4596-84cc-05375b9e760c" (UID: "87c18cb1-32c8-4596-84cc-05375b9e760c"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 19:35:24.669430 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:35:24.669402    4892 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/87c18cb1-32c8-4596-84cc-05375b9e760c-maas-api-xks-ca" pod "87c18cb1-32c8-4596-84cc-05375b9e760c" (UID: "87c18cb1-32c8-4596-84cc-05375b9e760c"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 19:35:24.669737 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:35:24.669712    4892 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/87c18cb1-32c8-4596-84cc-05375b9e760c-kube-api-access-jlrv8" pod "87c18cb1-32c8-4596-84cc-05375b9e760c" (UID: "87c18cb1-32c8-4596-84cc-05375b9e760c"). InnerVolumeSpecName "kube-api-access-jlrv8". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:35:24.771654 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:35:24.771610    4892 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jlrv8\" (UniqueName: \"kubernetes.io/projected/87c18cb1-32c8-4596-84cc-05375b9e760c-kube-api-access-jlrv8\") on node \"ip-10-0-132-82.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 19:35:24.771915 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:35:24.771894    4892 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/87c18cb1-32c8-4596-84cc-05375b9e760c-maas-api-xks-ca\") on node \"ip-10-0-132-82.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 19:35:24.772053 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:35:24.772038    4892 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/87c18cb1-32c8-4596-84cc-05375b9e760c-maas-api-openshift-ca\") on node \"ip-10-0-132-82.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 19:35:24.843054 ip-10-0-132-82 kubenswrapper[4892]: E1030 19:35:24.843020    4892 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-jrm87" containerName="extractor"
Oct 30 19:35:25.315568 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:35:25.315540    4892 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6e1ad926-4tgkm"
Oct 30 19:35:25.315762 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:35:25.315540    4892 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6e1ad926-4tgkm" event={"ID":"87c18cb1-32c8-4596-84cc-05375b9e760c","Type":"ContainerDied","Data":"964c5cc08f701e9928505ffb3d2362d7133a3d672d2ce1cd06ef49b687a9ab09"}
Oct 30 19:35:25.315762 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:35:25.315654    4892 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="964c5cc08f701e9928505ffb3d2362d7133a3d672d2ce1cd06ef49b687a9ab09"
Oct 30 19:35:30.354522 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:35:30.354488    4892 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w0-2d9861/premium-w0-2d9861-kserve-67945c6848-4n6kz"]
Oct 30 19:35:30.354933 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:35:30.354895    4892 cpu_manager.go:414] "containerMap: removing container" podUID="87c18cb1-32c8-4596-84cc-05375b9e760c" containerName="revoke-keys"
Oct 30 19:35:30.354933 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:35:30.354905    4892 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="87c18cb1-32c8-4596-84cc-05375b9e760c" containerName="revoke-keys"
Oct 30 19:35:30.355031 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:35:30.355014    4892 memory_manager.go:365] "RemoveStaleState removing state" podUID="87c18cb1-32c8-4596-84cc-05375b9e760c" containerName="revoke-keys"
Oct 30 19:35:30.358427 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:35:30.358411    4892 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-2d9861/premium-w0-2d9861-kserve-67945c6848-4n6kz"
Oct 30 19:35:30.413121 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:35:30.413059    4892 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w0-2d9861\"/\"openshift-service-ca.crt\""
Oct 30 19:35:30.413335 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:35:30.413205    4892 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w0-2d9861\"/\"kube-root-ca.crt\""
Oct 30 19:35:30.423674 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:35:30.423647    4892 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/798988b2-c769-4f73-a7c8-ca15de0beef1-home\") pod \"premium-w0-2d9861-kserve-67945c6848-4n6kz\" (UID: \"798988b2-c769-4f73-a7c8-ca15de0beef1\") " pod="e2e-models-e2e-worker-w0-2d9861/premium-w0-2d9861-kserve-67945c6848-4n6kz"
Oct 30 19:35:30.423810 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:35:30.423689    4892 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bpvwz\" (UniqueName: \"kubernetes.io/projected/798988b2-c769-4f73-a7c8-ca15de0beef1-kube-api-access-bpvwz\") pod \"premium-w0-2d9861-kserve-67945c6848-4n6kz\" (UID: \"798988b2-c769-4f73-a7c8-ca15de0beef1\") " pod="e2e-models-e2e-worker-w0-2d9861/premium-w0-2d9861-kserve-67945c6848-4n6kz"
Oct 30 19:35:30.423890 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:35:30.423875    4892 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/798988b2-c769-4f73-a7c8-ca15de0beef1-tls-certs\") pod \"premium-w0-2d9861-kserve-67945c6848-4n6kz\" (UID: \"798988b2-c769-4f73-a7c8-ca15de0beef1\") " pod="e2e-models-e2e-worker-w0-2d9861/premium-w0-2d9861-kserve-67945c6848-4n6kz"
Oct 30 19:35:30.423953 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:35:30.423915    4892 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/798988b2-c769-4f73-a7c8-ca15de0beef1-model-cache\") pod \"premium-w0-2d9861-kserve-67945c6848-4n6kz\" (UID: \"798988b2-c769-4f73-a7c8-ca15de0beef1\") " pod="e2e-models-e2e-worker-w0-2d9861/premium-w0-2d9861-kserve-67945c6848-4n6kz"
Oct 30 19:35:30.424021 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:35:30.424002    4892 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/798988b2-c769-4f73-a7c8-ca15de0beef1-tmp-dir\") pod \"premium-w0-2d9861-kserve-67945c6848-4n6kz\" (UID: \"798988b2-c769-4f73-a7c8-ca15de0beef1\") " pod="e2e-models-e2e-worker-w0-2d9861/premium-w0-2d9861-kserve-67945c6848-4n6kz"
Oct 30 19:35:30.424068 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:35:30.424031    4892 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/798988b2-c769-4f73-a7c8-ca15de0beef1-dshm\") pod \"premium-w0-2d9861-kserve-67945c6848-4n6kz\" (UID: \"798988b2-c769-4f73-a7c8-ca15de0beef1\") " pod="e2e-models-e2e-worker-w0-2d9861/premium-w0-2d9861-kserve-67945c6848-4n6kz"
Oct 30 19:35:30.457767 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:35:30.457738    4892 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w0-2d9861\"/\"default-dockercfg-4lbfj\""
Oct 30 19:35:30.460240 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:35:30.460221    4892 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w0-2d9861/premium-w0-2d9861-kserve-67945c6848-4n6kz"]
Oct 30 19:35:30.462030 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:35:30.462015    4892 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w0-2d9861\"/\"premium-w0-2d9861-kserve-self-signed-certs\""
Oct 30 19:35:30.524619 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:35:30.524584    4892 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/798988b2-c769-4f73-a7c8-ca15de0beef1-home\") pod \"premium-w0-2d9861-kserve-67945c6848-4n6kz\" (UID: \"798988b2-c769-4f73-a7c8-ca15de0beef1\") " pod="e2e-models-e2e-worker-w0-2d9861/premium-w0-2d9861-kserve-67945c6848-4n6kz"
Oct 30 19:35:30.524821 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:35:30.524629    4892 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bpvwz\" (UniqueName: \"kubernetes.io/projected/798988b2-c769-4f73-a7c8-ca15de0beef1-kube-api-access-bpvwz\") pod \"premium-w0-2d9861-kserve-67945c6848-4n6kz\" (UID: \"798988b2-c769-4f73-a7c8-ca15de0beef1\") " pod="e2e-models-e2e-worker-w0-2d9861/premium-w0-2d9861-kserve-67945c6848-4n6kz"
Oct 30 19:35:30.524821 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:35:30.524727    4892 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/798988b2-c769-4f73-a7c8-ca15de0beef1-tls-certs\") pod \"premium-w0-2d9861-kserve-67945c6848-4n6kz\" (UID: \"798988b2-c769-4f73-a7c8-ca15de0beef1\") " pod="e2e-models-e2e-worker-w0-2d9861/premium-w0-2d9861-kserve-67945c6848-4n6kz"
Oct 30 19:35:30.524821 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:35:30.524760    4892 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/798988b2-c769-4f73-a7c8-ca15de0beef1-model-cache\") pod \"premium-w0-2d9861-kserve-67945c6848-4n6kz\" (UID: \"798988b2-c769-4f73-a7c8-ca15de0beef1\") " pod="e2e-models-e2e-worker-w0-2d9861/premium-w0-2d9861-kserve-67945c6848-4n6kz"
Oct 30 19:35:30.524821 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:35:30.524786    4892 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/798988b2-c769-4f73-a7c8-ca15de0beef1-tmp-dir\") pod \"premium-w0-2d9861-kserve-67945c6848-4n6kz\" (UID: \"798988b2-c769-4f73-a7c8-ca15de0beef1\") " pod="e2e-models-e2e-worker-w0-2d9861/premium-w0-2d9861-kserve-67945c6848-4n6kz"
Oct 30 19:35:30.524821 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:35:30.524809    4892 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/798988b2-c769-4f73-a7c8-ca15de0beef1-dshm\") pod \"premium-w0-2d9861-kserve-67945c6848-4n6kz\" (UID: \"798988b2-c769-4f73-a7c8-ca15de0beef1\") " pod="e2e-models-e2e-worker-w0-2d9861/premium-w0-2d9861-kserve-67945c6848-4n6kz"
Oct 30 19:35:30.525145 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:35:30.525114    4892 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/798988b2-c769-4f73-a7c8-ca15de0beef1-home\") pod \"premium-w0-2d9861-kserve-67945c6848-4n6kz\" (UID: \"798988b2-c769-4f73-a7c8-ca15de0beef1\") " pod="e2e-models-e2e-worker-w0-2d9861/premium-w0-2d9861-kserve-67945c6848-4n6kz"
Oct 30 19:35:30.525474 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:35:30.525451    4892 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/798988b2-c769-4f73-a7c8-ca15de0beef1-model-cache\") pod \"premium-w0-2d9861-kserve-67945c6848-4n6kz\" (UID: \"798988b2-c769-4f73-a7c8-ca15de0beef1\") " pod="e2e-models-e2e-worker-w0-2d9861/premium-w0-2d9861-kserve-67945c6848-4n6kz"
Oct 30 19:35:30.525909 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:35:30.525890    4892 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/798988b2-c769-4f73-a7c8-ca15de0beef1-tmp-dir\") pod \"premium-w0-2d9861-kserve-67945c6848-4n6kz\" (UID: \"798988b2-c769-4f73-a7c8-ca15de0beef1\") " pod="e2e-models-e2e-worker-w0-2d9861/premium-w0-2d9861-kserve-67945c6848-4n6kz"
Oct 30 19:35:30.527550 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:35:30.527530    4892 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/798988b2-c769-4f73-a7c8-ca15de0beef1-dshm\") pod \"premium-w0-2d9861-kserve-67945c6848-4n6kz\" (UID: \"798988b2-c769-4f73-a7c8-ca15de0beef1\") " pod="e2e-models-e2e-worker-w0-2d9861/premium-w0-2d9861-kserve-67945c6848-4n6kz"
Oct 30 19:35:30.528239 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:35:30.528217    4892 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/798988b2-c769-4f73-a7c8-ca15de0beef1-tls-certs\") pod \"premium-w0-2d9861-kserve-67945c6848-4n6kz\" (UID: \"798988b2-c769-4f73-a7c8-ca15de0beef1\") " pod="e2e-models-e2e-worker-w0-2d9861/premium-w0-2d9861-kserve-67945c6848-4n6kz"
Oct 30 19:35:30.581446 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:35:30.581413    4892 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bpvwz\" (UniqueName: \"kubernetes.io/projected/798988b2-c769-4f73-a7c8-ca15de0beef1-kube-api-access-bpvwz\") pod \"premium-w0-2d9861-kserve-67945c6848-4n6kz\" (UID: \"798988b2-c769-4f73-a7c8-ca15de0beef1\") " pod="e2e-models-e2e-worker-w0-2d9861/premium-w0-2d9861-kserve-67945c6848-4n6kz"
Oct 30 19:35:30.669983 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:35:30.669933    4892 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-2d9861/premium-w0-2d9861-kserve-67945c6848-4n6kz"
Oct 30 19:35:31.081096 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:35:31.081070    4892 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w0-2d9861/premium-w0-2d9861-kserve-67945c6848-4n6kz"]
Oct 30 19:35:31.082674 ip-10-0-132-82 kubenswrapper[4892]: W1030 19:35:31.082640    4892 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod798988b2_c769_4f73_a7c8_ca15de0beef1.slice/crio-81935d55ef963ebc8372addb46ae497b06bd3006acc6bf54203797c976d5c709 WatchSource:0}: Error finding container 81935d55ef963ebc8372addb46ae497b06bd3006acc6bf54203797c976d5c709: Status 404 returned error can't find the container with id 81935d55ef963ebc8372addb46ae497b06bd3006acc6bf54203797c976d5c709
Oct 30 19:35:31.350402 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:35:31.350321    4892 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-2d9861/premium-w0-2d9861-kserve-67945c6848-4n6kz" event={"ID":"798988b2-c769-4f73-a7c8-ca15de0beef1","Type":"ContainerStarted","Data":"8d4825891af93b96a32d886ec52a856fd76916d0c7acaf73ef5311b66ae1bc6f"}
Oct 30 19:35:31.350402 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:35:31.350362    4892 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-2d9861/premium-w0-2d9861-kserve-67945c6848-4n6kz" event={"ID":"798988b2-c769-4f73-a7c8-ca15de0beef1","Type":"ContainerStarted","Data":"81935d55ef963ebc8372addb46ae497b06bd3006acc6bf54203797c976d5c709"}
Oct 30 19:35:31.464009 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:35:31.463923    4892 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w0-2d9861/premium-w0-2d9861-kserve-67945c6848-4n6kz" podStartSLOduration=1.463907064 podStartE2EDuration="1.463907064s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:35:30 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:35:31.462849776 +0000 UTC m=+1179.033315377" watchObservedRunningTime="2026-10-30 19:35:31.463907064 +0000 UTC m=+1179.034372664"
Oct 30 19:35:31.670874 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:35:31.670842    4892 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-2d9861/premium-w0-2d9861-kserve-67945c6848-4n6kz"
Oct 30 19:35:40.436865 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:35:40.436826    4892 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-be1662e4-7l2wh"]
Oct 30 19:35:40.441216 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:35:40.441198    4892 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-be1662e4-7l2wh"
Oct 30 19:35:40.527673 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:35:40.527643    4892 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-25894\" (UniqueName: \"kubernetes.io/projected/f4922dc5-e50a-4758-8098-aad71d12fa83-kube-api-access-25894\") pod \"maas-api-revoke-sub-be1662e4-7l2wh\" (UID: \"f4922dc5-e50a-4758-8098-aad71d12fa83\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-be1662e4-7l2wh"
Oct 30 19:35:40.527863 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:35:40.527836    4892 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f4922dc5-e50a-4758-8098-aad71d12fa83-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-be1662e4-7l2wh\" (UID: \"f4922dc5-e50a-4758-8098-aad71d12fa83\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-be1662e4-7l2wh"
Oct 30 19:35:40.527926 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:35:40.527884    4892 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f4922dc5-e50a-4758-8098-aad71d12fa83-maas-api-xks-ca\") pod \"maas-api-revoke-sub-be1662e4-7l2wh\" (UID: \"f4922dc5-e50a-4758-8098-aad71d12fa83\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-be1662e4-7l2wh"
Oct 30 19:35:40.542884 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:35:40.542860    4892 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 30 19:35:40.543111 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:35:40.543089    4892 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 19:35:40.543111 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:35:40.543099    4892 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 30 19:35:40.543299 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:35:40.543154    4892 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-nlxp4\""
Oct 30 19:35:40.545889 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:35:40.545869    4892 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-be1662e4-7l2wh"]
Oct 30 19:35:40.629326 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:35:40.629288    4892 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-25894\" (UniqueName: \"kubernetes.io/projected/f4922dc5-e50a-4758-8098-aad71d12fa83-kube-api-access-25894\") pod \"maas-api-revoke-sub-be1662e4-7l2wh\" (UID: \"f4922dc5-e50a-4758-8098-aad71d12fa83\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-be1662e4-7l2wh"
Oct 30 19:35:40.629613 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:35:40.629594    4892 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f4922dc5-e50a-4758-8098-aad71d12fa83-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-be1662e4-7l2wh\" (UID: \"f4922dc5-e50a-4758-8098-aad71d12fa83\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-be1662e4-7l2wh"
Oct 30 19:35:40.629692 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:35:40.629634    4892 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f4922dc5-e50a-4758-8098-aad71d12fa83-maas-api-xks-ca\") pod \"maas-api-revoke-sub-be1662e4-7l2wh\" (UID: \"f4922dc5-e50a-4758-8098-aad71d12fa83\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-be1662e4-7l2wh"
Oct 30 19:35:40.630323 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:35:40.630297    4892 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f4922dc5-e50a-4758-8098-aad71d12fa83-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-be1662e4-7l2wh\" (UID: \"f4922dc5-e50a-4758-8098-aad71d12fa83\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-be1662e4-7l2wh"
Oct 30 19:35:40.632348 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:35:40.632330    4892 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f4922dc5-e50a-4758-8098-aad71d12fa83-maas-api-xks-ca\") pod \"maas-api-revoke-sub-be1662e4-7l2wh\" (UID: \"f4922dc5-e50a-4758-8098-aad71d12fa83\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-be1662e4-7l2wh"
Oct 30 19:35:40.670412 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:35:40.670387    4892 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w0-2d9861/premium-w0-2d9861-kserve-67945c6848-4n6kz"
Oct 30 19:35:40.683107 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:35:40.683089    4892 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w0-2d9861/premium-w0-2d9861-kserve-67945c6848-4n6kz"
Oct 30 19:35:40.686430 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:35:40.686408    4892 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-25894\" (UniqueName: \"kubernetes.io/projected/f4922dc5-e50a-4758-8098-aad71d12fa83-kube-api-access-25894\") pod \"maas-api-revoke-sub-be1662e4-7l2wh\" (UID: \"f4922dc5-e50a-4758-8098-aad71d12fa83\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-be1662e4-7l2wh"
Oct 30 19:35:40.758723 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:35:40.758657    4892 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-be1662e4-7l2wh"
Oct 30 19:35:41.141391 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:35:41.141364    4892 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-be1662e4-7l2wh"]
Oct 30 19:35:41.143217 ip-10-0-132-82 kubenswrapper[4892]: W1030 19:35:41.143191    4892 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf4922dc5_e50a_4758_8098_aad71d12fa83.slice/crio-f1cca8a52063f34e2848381a064641bceadbdbdb02c3e35a91e174bc2e981d8e WatchSource:0}: Error finding container f1cca8a52063f34e2848381a064641bceadbdbdb02c3e35a91e174bc2e981d8e: Status 404 returned error can't find the container with id f1cca8a52063f34e2848381a064641bceadbdbdb02c3e35a91e174bc2e981d8e
Oct 30 19:35:41.463706 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:35:41.463620    4892 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-be1662e4-7l2wh" event={"ID":"f4922dc5-e50a-4758-8098-aad71d12fa83","Type":"ContainerStarted","Data":"f1cca8a52063f34e2848381a064641bceadbdbdb02c3e35a91e174bc2e981d8e"}
Oct 30 19:35:41.481253 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:35:41.481222    4892 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-2d9861/premium-w0-2d9861-kserve-67945c6848-4n6kz"
Oct 30 19:35:42.472390 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:35:42.472357    4892 generic.go:356] "Generic (PLEG): container finished" podID="f4922dc5-e50a-4758-8098-aad71d12fa83" containerID="2ddbe404d830c1724af73dd8448c6d113b26b80efe430f8d14c4787890731bd8" exitCode=0
Oct 30 19:35:42.472854 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:35:42.472449    4892 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-be1662e4-7l2wh" event={"ID":"f4922dc5-e50a-4758-8098-aad71d12fa83","Type":"ContainerDied","Data":"2ddbe404d830c1724af73dd8448c6d113b26b80efe430f8d14c4787890731bd8"}
Oct 30 19:35:43.620204 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:35:43.620180    4892 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-be1662e4-7l2wh"
Oct 30 19:35:43.765815 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:35:43.765746    4892 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/f4922dc5-e50a-4758-8098-aad71d12fa83-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f4922dc5-e50a-4758-8098-aad71d12fa83-maas-api-xks-ca\") pod \"f4922dc5-e50a-4758-8098-aad71d12fa83\" (UID: \"f4922dc5-e50a-4758-8098-aad71d12fa83\") "
Oct 30 19:35:43.766018 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:35:43.765827    4892 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f4922dc5-e50a-4758-8098-aad71d12fa83-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f4922dc5-e50a-4758-8098-aad71d12fa83-maas-api-openshift-ca\") pod \"f4922dc5-e50a-4758-8098-aad71d12fa83\" (UID: \"f4922dc5-e50a-4758-8098-aad71d12fa83\") "
Oct 30 19:35:43.766018 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:35:43.765887    4892 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f4922dc5-e50a-4758-8098-aad71d12fa83-kube-api-access-25894\" (UniqueName: \"kubernetes.io/projected/f4922dc5-e50a-4758-8098-aad71d12fa83-kube-api-access-25894\") pod \"f4922dc5-e50a-4758-8098-aad71d12fa83\" (UID: \"f4922dc5-e50a-4758-8098-aad71d12fa83\") "
Oct 30 19:35:43.766233 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:35:43.766208    4892 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f4922dc5-e50a-4758-8098-aad71d12fa83-maas-api-openshift-ca" pod "f4922dc5-e50a-4758-8098-aad71d12fa83" (UID: "f4922dc5-e50a-4758-8098-aad71d12fa83"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 19:35:43.768141 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:35:43.768120    4892 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f4922dc5-e50a-4758-8098-aad71d12fa83-maas-api-xks-ca" pod "f4922dc5-e50a-4758-8098-aad71d12fa83" (UID: "f4922dc5-e50a-4758-8098-aad71d12fa83"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 19:35:43.768223 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:35:43.768158    4892 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f4922dc5-e50a-4758-8098-aad71d12fa83-kube-api-access-25894" pod "f4922dc5-e50a-4758-8098-aad71d12fa83" (UID: "f4922dc5-e50a-4758-8098-aad71d12fa83"). InnerVolumeSpecName "kube-api-access-25894". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:35:43.867583 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:35:43.867544    4892 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f4922dc5-e50a-4758-8098-aad71d12fa83-maas-api-xks-ca\") on node \"ip-10-0-132-82.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 19:35:43.867583 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:35:43.867571    4892 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f4922dc5-e50a-4758-8098-aad71d12fa83-maas-api-openshift-ca\") on node \"ip-10-0-132-82.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 19:35:43.867583 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:35:43.867581    4892 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-25894\" (UniqueName: \"kubernetes.io/projected/f4922dc5-e50a-4758-8098-aad71d12fa83-kube-api-access-25894\") on node \"ip-10-0-132-82.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 19:35:44.484616 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:35:44.484591    4892 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-be1662e4-7l2wh"
Oct 30 19:35:44.484616 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:35:44.484600    4892 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-be1662e4-7l2wh" event={"ID":"f4922dc5-e50a-4758-8098-aad71d12fa83","Type":"ContainerDied","Data":"f1cca8a52063f34e2848381a064641bceadbdbdb02c3e35a91e174bc2e981d8e"}
Oct 30 19:35:44.484810 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:35:44.484644    4892 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="f1cca8a52063f34e2848381a064641bceadbdbdb02c3e35a91e174bc2e981d8e"
Oct 30 19:35:49.062249 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:35:49.062216    4892 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w1-d52df0-openshift-default-6d5f887b45-985gh"]
Oct 30 19:35:49.062717 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:35:49.062633    4892 cpu_manager.go:414] "containerMap: removing container" podUID="f4922dc5-e50a-4758-8098-aad71d12fa83" containerName="revoke-keys"
Oct 30 19:35:49.062717 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:35:49.062644    4892 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f4922dc5-e50a-4758-8098-aad71d12fa83" containerName="revoke-keys"
Oct 30 19:35:49.062787 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:35:49.062739    4892 memory_manager.go:365] "RemoveStaleState removing state" podUID="f4922dc5-e50a-4758-8098-aad71d12fa83" containerName="revoke-keys"
Oct 30 19:35:49.066103 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:35:49.066081    4892 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-d52df0-openshift-default-6d5f887b45-985gh"
Oct 30 19:35:49.118168 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:35:49.118146    4892 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w1-d52df0-openshift-default-dockercfg-dwj6l\""
Oct 30 19:35:49.171646 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:35:49.171613    4892 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-d52df0-openshift-default-6d5f887b45-985gh"]
Oct 30 19:35:49.237443 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:35:49.237407    4892 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e453311b-d7b1-4819-a57a-d1de4886de26-workload-socket\") pod \"e2e-worker-w1-d52df0-openshift-default-6d5f887b45-985gh\" (UID: \"e453311b-d7b1-4819-a57a-d1de4886de26\") " pod="openshift-ingress/e2e-worker-w1-d52df0-openshift-default-6d5f887b45-985gh"
Oct 30 19:35:49.237628 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:35:49.237451    4892 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e453311b-d7b1-4819-a57a-d1de4886de26-workload-certs\") pod \"e2e-worker-w1-d52df0-openshift-default-6d5f887b45-985gh\" (UID: \"e453311b-d7b1-4819-a57a-d1de4886de26\") " pod="openshift-ingress/e2e-worker-w1-d52df0-openshift-default-6d5f887b45-985gh"
Oct 30 19:35:49.237628 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:35:49.237477    4892 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e453311b-d7b1-4819-a57a-d1de4886de26-istio-token\") pod \"e2e-worker-w1-d52df0-openshift-default-6d5f887b45-985gh\" (UID: \"e453311b-d7b1-4819-a57a-d1de4886de26\") " pod="openshift-ingress/e2e-worker-w1-d52df0-openshift-default-6d5f887b45-985gh"
Oct 30 19:35:49.237628 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:35:49.237543    4892 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bkvrn\" (UniqueName: \"kubernetes.io/projected/e453311b-d7b1-4819-a57a-d1de4886de26-kube-api-access-bkvrn\") pod \"e2e-worker-w1-d52df0-openshift-default-6d5f887b45-985gh\" (UID: \"e453311b-d7b1-4819-a57a-d1de4886de26\") " pod="openshift-ingress/e2e-worker-w1-d52df0-openshift-default-6d5f887b45-985gh"
Oct 30 19:35:49.237801 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:35:49.237665    4892 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e453311b-d7b1-4819-a57a-d1de4886de26-istiod-ca-cert\") pod \"e2e-worker-w1-d52df0-openshift-default-6d5f887b45-985gh\" (UID: \"e453311b-d7b1-4819-a57a-d1de4886de26\") " pod="openshift-ingress/e2e-worker-w1-d52df0-openshift-default-6d5f887b45-985gh"
Oct 30 19:35:49.237801 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:35:49.237705    4892 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e453311b-d7b1-4819-a57a-d1de4886de26-istio-envoy\") pod \"e2e-worker-w1-d52df0-openshift-default-6d5f887b45-985gh\" (UID: \"e453311b-d7b1-4819-a57a-d1de4886de26\") " pod="openshift-ingress/e2e-worker-w1-d52df0-openshift-default-6d5f887b45-985gh"
Oct 30 19:35:49.237801 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:35:49.237759    4892 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e453311b-d7b1-4819-a57a-d1de4886de26-istio-podinfo\") pod \"e2e-worker-w1-d52df0-openshift-default-6d5f887b45-985gh\" (UID: \"e453311b-d7b1-4819-a57a-d1de4886de26\") " pod="openshift-ingress/e2e-worker-w1-d52df0-openshift-default-6d5f887b45-985gh"
Oct 30 19:35:49.238007 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:35:49.237849    4892 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e453311b-d7b1-4819-a57a-d1de4886de26-credential-socket\") pod \"e2e-worker-w1-d52df0-openshift-default-6d5f887b45-985gh\" (UID: \"e453311b-d7b1-4819-a57a-d1de4886de26\") " pod="openshift-ingress/e2e-worker-w1-d52df0-openshift-default-6d5f887b45-985gh"
Oct 30 19:35:49.238007 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:35:49.237886    4892 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e453311b-d7b1-4819-a57a-d1de4886de26-istio-data\") pod \"e2e-worker-w1-d52df0-openshift-default-6d5f887b45-985gh\" (UID: \"e453311b-d7b1-4819-a57a-d1de4886de26\") " pod="openshift-ingress/e2e-worker-w1-d52df0-openshift-default-6d5f887b45-985gh"
Oct 30 19:35:49.338558 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:35:49.338484    4892 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e453311b-d7b1-4819-a57a-d1de4886de26-credential-socket\") pod \"e2e-worker-w1-d52df0-openshift-default-6d5f887b45-985gh\" (UID: \"e453311b-d7b1-4819-a57a-d1de4886de26\") " pod="openshift-ingress/e2e-worker-w1-d52df0-openshift-default-6d5f887b45-985gh"
Oct 30 19:35:49.338558 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:35:49.338523    4892 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e453311b-d7b1-4819-a57a-d1de4886de26-istio-data\") pod \"e2e-worker-w1-d52df0-openshift-default-6d5f887b45-985gh\" (UID: \"e453311b-d7b1-4819-a57a-d1de4886de26\") " pod="openshift-ingress/e2e-worker-w1-d52df0-openshift-default-6d5f887b45-985gh"
Oct 30 19:35:49.338768 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:35:49.338570    4892 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e453311b-d7b1-4819-a57a-d1de4886de26-workload-socket\") pod \"e2e-worker-w1-d52df0-openshift-default-6d5f887b45-985gh\" (UID: \"e453311b-d7b1-4819-a57a-d1de4886de26\") " pod="openshift-ingress/e2e-worker-w1-d52df0-openshift-default-6d5f887b45-985gh"
Oct 30 19:35:49.338768 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:35:49.338598    4892 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e453311b-d7b1-4819-a57a-d1de4886de26-workload-certs\") pod \"e2e-worker-w1-d52df0-openshift-default-6d5f887b45-985gh\" (UID: \"e453311b-d7b1-4819-a57a-d1de4886de26\") " pod="openshift-ingress/e2e-worker-w1-d52df0-openshift-default-6d5f887b45-985gh"
Oct 30 19:35:49.338768 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:35:49.338632    4892 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e453311b-d7b1-4819-a57a-d1de4886de26-istio-token\") pod \"e2e-worker-w1-d52df0-openshift-default-6d5f887b45-985gh\" (UID: \"e453311b-d7b1-4819-a57a-d1de4886de26\") " pod="openshift-ingress/e2e-worker-w1-d52df0-openshift-default-6d5f887b45-985gh"
Oct 30 19:35:49.338768 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:35:49.338659    4892 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bkvrn\" (UniqueName: \"kubernetes.io/projected/e453311b-d7b1-4819-a57a-d1de4886de26-kube-api-access-bkvrn\") pod \"e2e-worker-w1-d52df0-openshift-default-6d5f887b45-985gh\" (UID: \"e453311b-d7b1-4819-a57a-d1de4886de26\") " pod="openshift-ingress/e2e-worker-w1-d52df0-openshift-default-6d5f887b45-985gh"
Oct 30 19:35:49.338768 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:35:49.338685    4892 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e453311b-d7b1-4819-a57a-d1de4886de26-istiod-ca-cert\") pod \"e2e-worker-w1-d52df0-openshift-default-6d5f887b45-985gh\" (UID: \"e453311b-d7b1-4819-a57a-d1de4886de26\") " pod="openshift-ingress/e2e-worker-w1-d52df0-openshift-default-6d5f887b45-985gh"
Oct 30 19:35:49.338768 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:35:49.338713    4892 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e453311b-d7b1-4819-a57a-d1de4886de26-istio-envoy\") pod \"e2e-worker-w1-d52df0-openshift-default-6d5f887b45-985gh\" (UID: \"e453311b-d7b1-4819-a57a-d1de4886de26\") " pod="openshift-ingress/e2e-worker-w1-d52df0-openshift-default-6d5f887b45-985gh"
Oct 30 19:35:49.339106 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:35:49.338895    4892 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e453311b-d7b1-4819-a57a-d1de4886de26-credential-socket\") pod \"e2e-worker-w1-d52df0-openshift-default-6d5f887b45-985gh\" (UID: \"e453311b-d7b1-4819-a57a-d1de4886de26\") " pod="openshift-ingress/e2e-worker-w1-d52df0-openshift-default-6d5f887b45-985gh"
Oct 30 19:35:49.339106 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:35:49.338954    4892 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e453311b-d7b1-4819-a57a-d1de4886de26-istio-podinfo\") pod \"e2e-worker-w1-d52df0-openshift-default-6d5f887b45-985gh\" (UID: \"e453311b-d7b1-4819-a57a-d1de4886de26\") " pod="openshift-ingress/e2e-worker-w1-d52df0-openshift-default-6d5f887b45-985gh"
Oct 30 19:35:49.339306 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:35:49.339282    4892 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e453311b-d7b1-4819-a57a-d1de4886de26-istio-data\") pod \"e2e-worker-w1-d52df0-openshift-default-6d5f887b45-985gh\" (UID: \"e453311b-d7b1-4819-a57a-d1de4886de26\") " pod="openshift-ingress/e2e-worker-w1-d52df0-openshift-default-6d5f887b45-985gh"
Oct 30 19:35:49.339364 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:35:49.339331    4892 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e453311b-d7b1-4819-a57a-d1de4886de26-workload-socket\") pod \"e2e-worker-w1-d52df0-openshift-default-6d5f887b45-985gh\" (UID: \"e453311b-d7b1-4819-a57a-d1de4886de26\") " pod="openshift-ingress/e2e-worker-w1-d52df0-openshift-default-6d5f887b45-985gh"
Oct 30 19:35:49.339420 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:35:49.339357    4892 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e453311b-d7b1-4819-a57a-d1de4886de26-workload-certs\") pod \"e2e-worker-w1-d52df0-openshift-default-6d5f887b45-985gh\" (UID: \"e453311b-d7b1-4819-a57a-d1de4886de26\") " pod="openshift-ingress/e2e-worker-w1-d52df0-openshift-default-6d5f887b45-985gh"
Oct 30 19:35:49.339527 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:35:49.339506    4892 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e453311b-d7b1-4819-a57a-d1de4886de26-istiod-ca-cert\") pod \"e2e-worker-w1-d52df0-openshift-default-6d5f887b45-985gh\" (UID: \"e453311b-d7b1-4819-a57a-d1de4886de26\") " pod="openshift-ingress/e2e-worker-w1-d52df0-openshift-default-6d5f887b45-985gh"
Oct 30 19:35:49.341528 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:35:49.341500    4892 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e453311b-d7b1-4819-a57a-d1de4886de26-istio-envoy\") pod \"e2e-worker-w1-d52df0-openshift-default-6d5f887b45-985gh\" (UID: \"e453311b-d7b1-4819-a57a-d1de4886de26\") " pod="openshift-ingress/e2e-worker-w1-d52df0-openshift-default-6d5f887b45-985gh"
Oct 30 19:35:49.341653 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:35:49.341535    4892 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e453311b-d7b1-4819-a57a-d1de4886de26-istio-podinfo\") pod \"e2e-worker-w1-d52df0-openshift-default-6d5f887b45-985gh\" (UID: \"e453311b-d7b1-4819-a57a-d1de4886de26\") " pod="openshift-ingress/e2e-worker-w1-d52df0-openshift-default-6d5f887b45-985gh"
Oct 30 19:35:49.396059 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:35:49.396024    4892 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e453311b-d7b1-4819-a57a-d1de4886de26-istio-token\") pod \"e2e-worker-w1-d52df0-openshift-default-6d5f887b45-985gh\" (UID: \"e453311b-d7b1-4819-a57a-d1de4886de26\") " pod="openshift-ingress/e2e-worker-w1-d52df0-openshift-default-6d5f887b45-985gh"
Oct 30 19:35:49.396188 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:35:49.396140    4892 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bkvrn\" (UniqueName: \"kubernetes.io/projected/e453311b-d7b1-4819-a57a-d1de4886de26-kube-api-access-bkvrn\") pod \"e2e-worker-w1-d52df0-openshift-default-6d5f887b45-985gh\" (UID: \"e453311b-d7b1-4819-a57a-d1de4886de26\") " pod="openshift-ingress/e2e-worker-w1-d52df0-openshift-default-6d5f887b45-985gh"
Oct 30 19:35:49.634389 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:35:49.634352    4892 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-99f6dcfa-whqwx"]
Oct 30 19:35:49.639271 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:35:49.639242    4892 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-99f6dcfa-whqwx"
Oct 30 19:35:49.682759 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:35:49.682731    4892 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-d52df0-openshift-default-6d5f887b45-985gh"
Oct 30 19:35:49.692224 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:35:49.692199    4892 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 30 19:35:49.692329 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:35:49.692199    4892 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 19:35:49.695697 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:35:49.695676    4892 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 30 19:35:49.702633 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:35:49.702608    4892 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-99f6dcfa-whqwx"]
Oct 30 19:35:49.743447 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:35:49.743409    4892 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ml9xl\" (UniqueName: \"kubernetes.io/projected/a2107422-c567-4aad-94ca-3b8b38b4dc9d-kube-api-access-ml9xl\") pod \"maas-api-revoke-sub-99f6dcfa-whqwx\" (UID: \"a2107422-c567-4aad-94ca-3b8b38b4dc9d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-99f6dcfa-whqwx"
Oct 30 19:35:49.743627 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:35:49.743474    4892 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a2107422-c567-4aad-94ca-3b8b38b4dc9d-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-99f6dcfa-whqwx\" (UID: \"a2107422-c567-4aad-94ca-3b8b38b4dc9d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-99f6dcfa-whqwx"
Oct 30 19:35:49.743627 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:35:49.743499    4892 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a2107422-c567-4aad-94ca-3b8b38b4dc9d-maas-api-xks-ca\") pod \"maas-api-revoke-sub-99f6dcfa-whqwx\" (UID: \"a2107422-c567-4aad-94ca-3b8b38b4dc9d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-99f6dcfa-whqwx"
Oct 30 19:35:49.792907 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:35:49.792881    4892 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-nlxp4\""
Oct 30 19:35:49.845151 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:35:49.845117    4892 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a2107422-c567-4aad-94ca-3b8b38b4dc9d-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-99f6dcfa-whqwx\" (UID: \"a2107422-c567-4aad-94ca-3b8b38b4dc9d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-99f6dcfa-whqwx"
Oct 30 19:35:49.845314 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:35:49.845168    4892 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a2107422-c567-4aad-94ca-3b8b38b4dc9d-maas-api-xks-ca\") pod \"maas-api-revoke-sub-99f6dcfa-whqwx\" (UID: \"a2107422-c567-4aad-94ca-3b8b38b4dc9d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-99f6dcfa-whqwx"
Oct 30 19:35:49.845314 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:35:49.845305    4892 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ml9xl\" (UniqueName: \"kubernetes.io/projected/a2107422-c567-4aad-94ca-3b8b38b4dc9d-kube-api-access-ml9xl\") pod \"maas-api-revoke-sub-99f6dcfa-whqwx\" (UID: \"a2107422-c567-4aad-94ca-3b8b38b4dc9d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-99f6dcfa-whqwx"
Oct 30 19:35:49.845805 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:35:49.845783    4892 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a2107422-c567-4aad-94ca-3b8b38b4dc9d-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-99f6dcfa-whqwx\" (UID: \"a2107422-c567-4aad-94ca-3b8b38b4dc9d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-99f6dcfa-whqwx"
Oct 30 19:35:49.847945 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:35:49.847919    4892 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a2107422-c567-4aad-94ca-3b8b38b4dc9d-maas-api-xks-ca\") pod \"maas-api-revoke-sub-99f6dcfa-whqwx\" (UID: \"a2107422-c567-4aad-94ca-3b8b38b4dc9d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-99f6dcfa-whqwx"
Oct 30 19:35:49.909680 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:35:49.909608    4892 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ml9xl\" (UniqueName: \"kubernetes.io/projected/a2107422-c567-4aad-94ca-3b8b38b4dc9d-kube-api-access-ml9xl\") pod \"maas-api-revoke-sub-99f6dcfa-whqwx\" (UID: \"a2107422-c567-4aad-94ca-3b8b38b4dc9d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-99f6dcfa-whqwx"
Oct 30 19:35:49.957161 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:35:49.957121    4892 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-99f6dcfa-whqwx"
Oct 30 19:35:50.148119 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:35:50.148091    4892 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-d52df0-openshift-default-6d5f887b45-985gh"]
Oct 30 19:35:50.149552 ip-10-0-132-82 kubenswrapper[4892]: W1030 19:35:50.149525    4892 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode453311b_d7b1_4819_a57a_d1de4886de26.slice/crio-0c8b344fc0851738484e86eefa65a91e967c0b555e45d5247d8c8346ad6f4af9 WatchSource:0}: Error finding container 0c8b344fc0851738484e86eefa65a91e967c0b555e45d5247d8c8346ad6f4af9: Status 404 returned error can't find the container with id 0c8b344fc0851738484e86eefa65a91e967c0b555e45d5247d8c8346ad6f4af9
Oct 30 19:35:50.152112 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:35:50.152074    4892 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892000Ki","pods":"250"}
Oct 30 19:35:50.152221 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:35:50.152147    4892 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892000Ki","pods":"250"}
Oct 30 19:35:50.152221 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:35:50.152176    4892 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892000Ki","pods":"250"}
Oct 30 19:35:50.334781 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:35:50.334752    4892 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-99f6dcfa-whqwx"]
Oct 30 19:35:50.336437 ip-10-0-132-82 kubenswrapper[4892]: W1030 19:35:50.336412    4892 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda2107422_c567_4aad_94ca_3b8b38b4dc9d.slice/crio-d212b69f1b4c01287b8ddef477bd90d579b98cee37720a9d2035565cc0b461da WatchSource:0}: Error finding container d212b69f1b4c01287b8ddef477bd90d579b98cee37720a9d2035565cc0b461da: Status 404 returned error can't find the container with id d212b69f1b4c01287b8ddef477bd90d579b98cee37720a9d2035565cc0b461da
Oct 30 19:35:50.522581 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:35:50.522483    4892 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-d52df0-openshift-default-6d5f887b45-985gh" event={"ID":"e453311b-d7b1-4819-a57a-d1de4886de26","Type":"ContainerStarted","Data":"cb0272a78a5802f04e51979e7d05a3ba3bf4351cfe7d1f4180da73935a42deb6"}
Oct 30 19:35:50.522581 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:35:50.522553    4892 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-d52df0-openshift-default-6d5f887b45-985gh" event={"ID":"e453311b-d7b1-4819-a57a-d1de4886de26","Type":"ContainerStarted","Data":"0c8b344fc0851738484e86eefa65a91e967c0b555e45d5247d8c8346ad6f4af9"}
Oct 30 19:35:50.523884 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:35:50.523858    4892 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-99f6dcfa-whqwx" event={"ID":"a2107422-c567-4aad-94ca-3b8b38b4dc9d","Type":"ContainerStarted","Data":"d212b69f1b4c01287b8ddef477bd90d579b98cee37720a9d2035565cc0b461da"}
Oct 30 19:35:50.639056 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:35:50.638876    4892 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w1-d52df0-openshift-default-6d5f887b45-985gh" podStartSLOduration=1.638856143 podStartE2EDuration="1.638856143s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:35:49 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:35:50.637655641 +0000 UTC m=+1198.208121267" watchObservedRunningTime="2026-10-30 19:35:50.638856143 +0000 UTC m=+1198.209321744"
Oct 30 19:35:50.683624 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:35:50.683597    4892 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w1-d52df0-openshift-default-6d5f887b45-985gh"
Oct 30 19:35:51.568057 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:35:51.566501    4892 generic.go:356] "Generic (PLEG): container finished" podID="a2107422-c567-4aad-94ca-3b8b38b4dc9d" containerID="95b632ddf7475722c4c29c718c4e858de1f38ddb6f4db069809f0cc09b3de4f7" exitCode=0
Oct 30 19:35:51.568057 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:35:51.566580    4892 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-99f6dcfa-whqwx" event={"ID":"a2107422-c567-4aad-94ca-3b8b38b4dc9d","Type":"ContainerDied","Data":"95b632ddf7475722c4c29c718c4e858de1f38ddb6f4db069809f0cc09b3de4f7"}
Oct 30 19:35:51.684888 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:35:51.684853    4892 patch_prober.go:29] interesting pod/e2e-worker-w1-d52df0-openshift-default-6d5f887b45-985gh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.64:15021/healthz/ready\": dial tcp 10.132.0.64:15021: connect: connection refused" start-of-body=
Oct 30 19:35:51.685076 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:35:51.684919    4892 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-d52df0-openshift-default-6d5f887b45-985gh" podUID="e453311b-d7b1-4819-a57a-d1de4886de26" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.64:15021/healthz/ready\": dial tcp 10.132.0.64:15021: connect: connection refused"
Oct 30 19:35:51.692940 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:35:51.692915    4892 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-fsfvk"]
Oct 30 19:35:52.683573 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:35:52.683478    4892 patch_prober.go:29] interesting pod/e2e-worker-w1-d52df0-openshift-default-6d5f887b45-985gh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.64:15021/healthz/ready\": dial tcp 10.132.0.64:15021: connect: connection refused" start-of-body=
Oct 30 19:35:52.683573 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:35:52.683531    4892 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-d52df0-openshift-default-6d5f887b45-985gh" podUID="e453311b-d7b1-4819-a57a-d1de4886de26" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.64:15021/healthz/ready\": dial tcp 10.132.0.64:15021: connect: connection refused"
Oct 30 19:35:52.982989 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:35:52.975897    4892 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-99f6dcfa-whqwx"
Oct 30 19:35:52.992494 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:35:52.992465    4892 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-tg5xj_11db9cec-653f-4aee-8937-0cf5b367efac/console-operator/3.log"
Oct 30 19:35:52.993072 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:35:52.993042    4892 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/a2107422-c567-4aad-94ca-3b8b38b4dc9d-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a2107422-c567-4aad-94ca-3b8b38b4dc9d-maas-api-xks-ca\") pod \"a2107422-c567-4aad-94ca-3b8b38b4dc9d\" (UID: \"a2107422-c567-4aad-94ca-3b8b38b4dc9d\") "
Oct 30 19:35:52.993329 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:35:52.993308    4892 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a2107422-c567-4aad-94ca-3b8b38b4dc9d-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a2107422-c567-4aad-94ca-3b8b38b4dc9d-maas-api-openshift-ca\") pod \"a2107422-c567-4aad-94ca-3b8b38b4dc9d\" (UID: \"a2107422-c567-4aad-94ca-3b8b38b4dc9d\") "
Oct 30 19:35:52.993566 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:35:52.993546    4892 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a2107422-c567-4aad-94ca-3b8b38b4dc9d-kube-api-access-ml9xl\" (UniqueName: \"kubernetes.io/projected/a2107422-c567-4aad-94ca-3b8b38b4dc9d-kube-api-access-ml9xl\") pod \"a2107422-c567-4aad-94ca-3b8b38b4dc9d\" (UID: \"a2107422-c567-4aad-94ca-3b8b38b4dc9d\") "
Oct 30 19:35:53.001680 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:35:53.001646    4892 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a2107422-c567-4aad-94ca-3b8b38b4dc9d-maas-api-openshift-ca" pod "a2107422-c567-4aad-94ca-3b8b38b4dc9d" (UID: "a2107422-c567-4aad-94ca-3b8b38b4dc9d"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 19:35:53.027823 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:35:53.027788    4892 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-tg5xj_11db9cec-653f-4aee-8937-0cf5b367efac/console-operator/3.log"
Oct 30 19:35:53.051184 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:35:53.044079    4892 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a2107422-c567-4aad-94ca-3b8b38b4dc9d-maas-api-xks-ca" pod "a2107422-c567-4aad-94ca-3b8b38b4dc9d" (UID: "a2107422-c567-4aad-94ca-3b8b38b4dc9d"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 19:35:53.051184 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:35:53.046503    4892 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a2107422-c567-4aad-94ca-3b8b38b4dc9d-kube-api-access-ml9xl" pod "a2107422-c567-4aad-94ca-3b8b38b4dc9d" (UID: "a2107422-c567-4aad-94ca-3b8b38b4dc9d"). InnerVolumeSpecName "kube-api-access-ml9xl". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:35:53.095988 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:35:53.095352    4892 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ml9xl\" (UniqueName: \"kubernetes.io/projected/a2107422-c567-4aad-94ca-3b8b38b4dc9d-kube-api-access-ml9xl\") on node \"ip-10-0-132-82.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 19:35:53.095988 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:35:53.095378    4892 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a2107422-c567-4aad-94ca-3b8b38b4dc9d-maas-api-xks-ca\") on node \"ip-10-0-132-82.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 19:35:53.095988 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:35:53.095388    4892 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a2107422-c567-4aad-94ca-3b8b38b4dc9d-maas-api-openshift-ca\") on node \"ip-10-0-132-82.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 19:35:53.590145 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:35:53.590106    4892 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-99f6dcfa-whqwx" event={"ID":"a2107422-c567-4aad-94ca-3b8b38b4dc9d","Type":"ContainerDied","Data":"d212b69f1b4c01287b8ddef477bd90d579b98cee37720a9d2035565cc0b461da"}
Oct 30 19:35:53.590145 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:35:53.590135    4892 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-99f6dcfa-whqwx"
Oct 30 19:35:53.590371 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:35:53.590168    4892 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="d212b69f1b4c01287b8ddef477bd90d579b98cee37720a9d2035565cc0b461da"
Oct 30 19:35:53.683419 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:35:53.683384    4892 patch_prober.go:29] interesting pod/e2e-worker-w1-d52df0-openshift-default-6d5f887b45-985gh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.64:15021/healthz/ready\": dial tcp 10.132.0.64:15021: connect: connection refused" start-of-body=
Oct 30 19:35:53.683581 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:35:53.683442    4892 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-d52df0-openshift-default-6d5f887b45-985gh" podUID="e453311b-d7b1-4819-a57a-d1de4886de26" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.64:15021/healthz/ready\": dial tcp 10.132.0.64:15021: connect: connection refused"
Oct 30 19:35:54.683250 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:35:54.683214    4892 patch_prober.go:29] interesting pod/e2e-worker-w1-d52df0-openshift-default-6d5f887b45-985gh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.64:15021/healthz/ready\": dial tcp 10.132.0.64:15021: connect: connection refused" start-of-body=
Oct 30 19:35:54.683642 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:35:54.683266    4892 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-d52df0-openshift-default-6d5f887b45-985gh" podUID="e453311b-d7b1-4819-a57a-d1de4886de26" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.64:15021/healthz/ready\": dial tcp 10.132.0.64:15021: connect: connection refused"
Oct 30 19:35:54.867093 ip-10-0-132-82 kubenswrapper[4892]: E1030 19:35:54.867056    4892 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-jrm87" containerName="extractor"
Oct 30 19:35:55.683461 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:35:55.683427    4892 patch_prober.go:29] interesting pod/e2e-worker-w1-d52df0-openshift-default-6d5f887b45-985gh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.64:15021/healthz/ready\": dial tcp 10.132.0.64:15021: connect: connection refused" start-of-body=
Oct 30 19:35:55.683859 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:35:55.683482    4892 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-d52df0-openshift-default-6d5f887b45-985gh" podUID="e453311b-d7b1-4819-a57a-d1de4886de26" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.64:15021/healthz/ready\": dial tcp 10.132.0.64:15021: connect: connection refused"
Oct 30 19:35:56.683760 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:35:56.683723    4892 patch_prober.go:29] interesting pod/e2e-worker-w1-d52df0-openshift-default-6d5f887b45-985gh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.64:15021/healthz/ready\": dial tcp 10.132.0.64:15021: connect: connection refused" start-of-body=
Oct 30 19:35:56.684164 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:35:56.683801    4892 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-d52df0-openshift-default-6d5f887b45-985gh" podUID="e453311b-d7b1-4819-a57a-d1de4886de26" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.64:15021/healthz/ready\": dial tcp 10.132.0.64:15021: connect: connection refused"
Oct 30 19:35:57.874009 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:35:57.873948    4892 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w1-d52df0-openshift-default-6d5f887b45-985gh"
Oct 30 19:35:57.874414 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:35:57.874390    4892 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w1-d52df0-openshift-default-6d5f887b45-985gh"
Oct 30 19:35:57.875232 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:35:57.875212    4892 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w1-d52df0-openshift-default-6d5f887b45-985gh"
Oct 30 19:36:05.737362 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:36:05.737270    4892 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-eea179bf-l894m"]
Oct 30 19:36:05.737909 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:36:05.737888    4892 cpu_manager.go:414] "containerMap: removing container" podUID="a2107422-c567-4aad-94ca-3b8b38b4dc9d" containerName="revoke-keys"
Oct 30 19:36:05.737996 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:36:05.737911    4892 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a2107422-c567-4aad-94ca-3b8b38b4dc9d" containerName="revoke-keys"
Oct 30 19:36:05.738135 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:36:05.738121    4892 memory_manager.go:365] "RemoveStaleState removing state" podUID="a2107422-c567-4aad-94ca-3b8b38b4dc9d" containerName="revoke-keys"
Oct 30 19:36:05.745522 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:36:05.745495    4892 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-eea179bf-l894m"
Oct 30 19:36:05.796613 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:36:05.796582    4892 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-eea179bf-l894m"]
Oct 30 19:36:05.826884 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:36:05.826855    4892 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9c5741cb-f9d5-4a5e-a121-9b689a263a74-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-eea179bf-l894m\" (UID: \"9c5741cb-f9d5-4a5e-a121-9b689a263a74\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-eea179bf-l894m"
Oct 30 19:36:05.827064 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:36:05.826909    4892 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9c5741cb-f9d5-4a5e-a121-9b689a263a74-maas-api-xks-ca\") pod \"maas-api-revoke-sub-eea179bf-l894m\" (UID: \"9c5741cb-f9d5-4a5e-a121-9b689a263a74\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-eea179bf-l894m"
Oct 30 19:36:05.827145 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:36:05.827086    4892 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7nxf8\" (UniqueName: \"kubernetes.io/projected/9c5741cb-f9d5-4a5e-a121-9b689a263a74-kube-api-access-7nxf8\") pod \"maas-api-revoke-sub-eea179bf-l894m\" (UID: \"9c5741cb-f9d5-4a5e-a121-9b689a263a74\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-eea179bf-l894m"
Oct 30 19:36:05.842876 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:36:05.842845    4892 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 19:36:05.843025 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:36:05.842941    4892 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 30 19:36:05.843099 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:36:05.843087    4892 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-nlxp4\""
Oct 30 19:36:05.846102 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:36:05.846078    4892 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 30 19:36:05.928183 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:36:05.928158    4892 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9c5741cb-f9d5-4a5e-a121-9b689a263a74-maas-api-xks-ca\") pod \"maas-api-revoke-sub-eea179bf-l894m\" (UID: \"9c5741cb-f9d5-4a5e-a121-9b689a263a74\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-eea179bf-l894m"
Oct 30 19:36:05.928335 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:36:05.928212    4892 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7nxf8\" (UniqueName: \"kubernetes.io/projected/9c5741cb-f9d5-4a5e-a121-9b689a263a74-kube-api-access-7nxf8\") pod \"maas-api-revoke-sub-eea179bf-l894m\" (UID: \"9c5741cb-f9d5-4a5e-a121-9b689a263a74\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-eea179bf-l894m"
Oct 30 19:36:05.928335 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:36:05.928286    4892 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9c5741cb-f9d5-4a5e-a121-9b689a263a74-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-eea179bf-l894m\" (UID: \"9c5741cb-f9d5-4a5e-a121-9b689a263a74\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-eea179bf-l894m"
Oct 30 19:36:05.928914 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:36:05.928868    4892 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9c5741cb-f9d5-4a5e-a121-9b689a263a74-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-eea179bf-l894m\" (UID: \"9c5741cb-f9d5-4a5e-a121-9b689a263a74\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-eea179bf-l894m"
Oct 30 19:36:05.930697 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:36:05.930675    4892 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9c5741cb-f9d5-4a5e-a121-9b689a263a74-maas-api-xks-ca\") pod \"maas-api-revoke-sub-eea179bf-l894m\" (UID: \"9c5741cb-f9d5-4a5e-a121-9b689a263a74\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-eea179bf-l894m"
Oct 30 19:36:06.002572 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:36:05.999377    4892 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7nxf8\" (UniqueName: \"kubernetes.io/projected/9c5741cb-f9d5-4a5e-a121-9b689a263a74-kube-api-access-7nxf8\") pod \"maas-api-revoke-sub-eea179bf-l894m\" (UID: \"9c5741cb-f9d5-4a5e-a121-9b689a263a74\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-eea179bf-l894m"
Oct 30 19:36:06.058781 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:36:06.058754    4892 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-eea179bf-l894m"
Oct 30 19:36:06.434646 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:36:06.434619    4892 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-eea179bf-l894m"]
Oct 30 19:36:06.435370 ip-10-0-132-82 kubenswrapper[4892]: W1030 19:36:06.435344    4892 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9c5741cb_f9d5_4a5e_a121_9b689a263a74.slice/crio-32168122d40775af11b7bd9467cf4737e25db5aae11db9654e2f6791391369ba WatchSource:0}: Error finding container 32168122d40775af11b7bd9467cf4737e25db5aae11db9654e2f6791391369ba: Status 404 returned error can't find the container with id 32168122d40775af11b7bd9467cf4737e25db5aae11db9654e2f6791391369ba
Oct 30 19:36:06.666558 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:36:06.666529    4892 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-eea179bf-l894m" event={"ID":"9c5741cb-f9d5-4a5e-a121-9b689a263a74","Type":"ContainerStarted","Data":"32168122d40775af11b7bd9467cf4737e25db5aae11db9654e2f6791391369ba"}
Oct 30 19:36:07.673825 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:36:07.673789    4892 generic.go:356] "Generic (PLEG): container finished" podID="9c5741cb-f9d5-4a5e-a121-9b689a263a74" containerID="4e0bf3926c794ba22485d31a5ecfbda3b9e8741ea12e1d4fb4a821873eb480e3" exitCode=0
Oct 30 19:36:07.674232 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:36:07.673870    4892 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-eea179bf-l894m" event={"ID":"9c5741cb-f9d5-4a5e-a121-9b689a263a74","Type":"ContainerDied","Data":"4e0bf3926c794ba22485d31a5ecfbda3b9e8741ea12e1d4fb4a821873eb480e3"}
Oct 30 19:36:08.825038 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:36:08.825015    4892 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-eea179bf-l894m"
Oct 30 19:36:08.857799 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:36:08.857767    4892 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9c5741cb-f9d5-4a5e-a121-9b689a263a74-kube-api-access-7nxf8\" (UniqueName: \"kubernetes.io/projected/9c5741cb-f9d5-4a5e-a121-9b689a263a74-kube-api-access-7nxf8\") pod \"9c5741cb-f9d5-4a5e-a121-9b689a263a74\" (UID: \"9c5741cb-f9d5-4a5e-a121-9b689a263a74\") "
Oct 30 19:36:08.857940 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:36:08.857825    4892 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/9c5741cb-f9d5-4a5e-a121-9b689a263a74-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9c5741cb-f9d5-4a5e-a121-9b689a263a74-maas-api-xks-ca\") pod \"9c5741cb-f9d5-4a5e-a121-9b689a263a74\" (UID: \"9c5741cb-f9d5-4a5e-a121-9b689a263a74\") "
Oct 30 19:36:08.857940 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:36:08.857909    4892 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/9c5741cb-f9d5-4a5e-a121-9b689a263a74-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9c5741cb-f9d5-4a5e-a121-9b689a263a74-maas-api-openshift-ca\") pod \"9c5741cb-f9d5-4a5e-a121-9b689a263a74\" (UID: \"9c5741cb-f9d5-4a5e-a121-9b689a263a74\") "
Oct 30 19:36:08.858281 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:36:08.858256    4892 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9c5741cb-f9d5-4a5e-a121-9b689a263a74-maas-api-openshift-ca" pod "9c5741cb-f9d5-4a5e-a121-9b689a263a74" (UID: "9c5741cb-f9d5-4a5e-a121-9b689a263a74"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 19:36:08.860004 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:36:08.859976    4892 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9c5741cb-f9d5-4a5e-a121-9b689a263a74-maas-api-xks-ca" pod "9c5741cb-f9d5-4a5e-a121-9b689a263a74" (UID: "9c5741cb-f9d5-4a5e-a121-9b689a263a74"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 19:36:08.860138 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:36:08.860115    4892 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9c5741cb-f9d5-4a5e-a121-9b689a263a74-kube-api-access-7nxf8" pod "9c5741cb-f9d5-4a5e-a121-9b689a263a74" (UID: "9c5741cb-f9d5-4a5e-a121-9b689a263a74"). InnerVolumeSpecName "kube-api-access-7nxf8". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:36:08.959281 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:36:08.959195    4892 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9c5741cb-f9d5-4a5e-a121-9b689a263a74-maas-api-openshift-ca\") on node \"ip-10-0-132-82.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 19:36:08.959281 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:36:08.959224    4892 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7nxf8\" (UniqueName: \"kubernetes.io/projected/9c5741cb-f9d5-4a5e-a121-9b689a263a74-kube-api-access-7nxf8\") on node \"ip-10-0-132-82.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 19:36:08.959281 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:36:08.959234    4892 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9c5741cb-f9d5-4a5e-a121-9b689a263a74-maas-api-xks-ca\") on node \"ip-10-0-132-82.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 19:36:09.281949 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:36:09.281870    4892 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2cbf0d2e-x2csg"]
Oct 30 19:36:09.282310 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:36:09.282297    4892 cpu_manager.go:414] "containerMap: removing container" podUID="9c5741cb-f9d5-4a5e-a121-9b689a263a74" containerName="revoke-keys"
Oct 30 19:36:09.282364 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:36:09.282312    4892 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9c5741cb-f9d5-4a5e-a121-9b689a263a74" containerName="revoke-keys"
Oct 30 19:36:09.282425 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:36:09.282415    4892 memory_manager.go:365] "RemoveStaleState removing state" podUID="9c5741cb-f9d5-4a5e-a121-9b689a263a74" containerName="revoke-keys"
Oct 30 19:36:09.285676 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:36:09.285654    4892 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2cbf0d2e-x2csg"
Oct 30 19:36:09.342901 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:36:09.342871    4892 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2cbf0d2e-x2csg"]
Oct 30 19:36:09.362895 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:36:09.362870    4892 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c9cadeb0-5278-4c8d-b5fc-9c345360ca3f-maas-api-xks-ca\") pod \"maas-api-revoke-sub-2cbf0d2e-x2csg\" (UID: \"c9cadeb0-5278-4c8d-b5fc-9c345360ca3f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2cbf0d2e-x2csg"
Oct 30 19:36:09.363049 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:36:09.362923    4892 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c9cadeb0-5278-4c8d-b5fc-9c345360ca3f-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-2cbf0d2e-x2csg\" (UID: \"c9cadeb0-5278-4c8d-b5fc-9c345360ca3f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2cbf0d2e-x2csg"
Oct 30 19:36:09.363049 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:36:09.363034    4892 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9kfpm\" (UniqueName: \"kubernetes.io/projected/c9cadeb0-5278-4c8d-b5fc-9c345360ca3f-kube-api-access-9kfpm\") pod \"maas-api-revoke-sub-2cbf0d2e-x2csg\" (UID: \"c9cadeb0-5278-4c8d-b5fc-9c345360ca3f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2cbf0d2e-x2csg"
Oct 30 19:36:09.464589 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:36:09.464561    4892 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c9cadeb0-5278-4c8d-b5fc-9c345360ca3f-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-2cbf0d2e-x2csg\" (UID: \"c9cadeb0-5278-4c8d-b5fc-9c345360ca3f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2cbf0d2e-x2csg"
Oct 30 19:36:09.464730 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:36:09.464603    4892 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9kfpm\" (UniqueName: \"kubernetes.io/projected/c9cadeb0-5278-4c8d-b5fc-9c345360ca3f-kube-api-access-9kfpm\") pod \"maas-api-revoke-sub-2cbf0d2e-x2csg\" (UID: \"c9cadeb0-5278-4c8d-b5fc-9c345360ca3f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2cbf0d2e-x2csg"
Oct 30 19:36:09.464730 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:36:09.464685    4892 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c9cadeb0-5278-4c8d-b5fc-9c345360ca3f-maas-api-xks-ca\") pod \"maas-api-revoke-sub-2cbf0d2e-x2csg\" (UID: \"c9cadeb0-5278-4c8d-b5fc-9c345360ca3f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2cbf0d2e-x2csg"
Oct 30 19:36:09.465255 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:36:09.465230    4892 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c9cadeb0-5278-4c8d-b5fc-9c345360ca3f-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-2cbf0d2e-x2csg\" (UID: \"c9cadeb0-5278-4c8d-b5fc-9c345360ca3f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2cbf0d2e-x2csg"
Oct 30 19:36:09.467241 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:36:09.467219    4892 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c9cadeb0-5278-4c8d-b5fc-9c345360ca3f-maas-api-xks-ca\") pod \"maas-api-revoke-sub-2cbf0d2e-x2csg\" (UID: \"c9cadeb0-5278-4c8d-b5fc-9c345360ca3f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2cbf0d2e-x2csg"
Oct 30 19:36:09.520857 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:36:09.520834    4892 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9kfpm\" (UniqueName: \"kubernetes.io/projected/c9cadeb0-5278-4c8d-b5fc-9c345360ca3f-kube-api-access-9kfpm\") pod \"maas-api-revoke-sub-2cbf0d2e-x2csg\" (UID: \"c9cadeb0-5278-4c8d-b5fc-9c345360ca3f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2cbf0d2e-x2csg"
Oct 30 19:36:09.602180 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:36:09.602123    4892 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2cbf0d2e-x2csg"
Oct 30 19:36:09.687496 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:36:09.687471    4892 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-eea179bf-l894m"
Oct 30 19:36:09.687627 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:36:09.687470    4892 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-eea179bf-l894m" event={"ID":"9c5741cb-f9d5-4a5e-a121-9b689a263a74","Type":"ContainerDied","Data":"32168122d40775af11b7bd9467cf4737e25db5aae11db9654e2f6791391369ba"}
Oct 30 19:36:09.687627 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:36:09.687607    4892 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="32168122d40775af11b7bd9467cf4737e25db5aae11db9654e2f6791391369ba"
Oct 30 19:36:09.972705 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:36:09.972672    4892 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2cbf0d2e-x2csg"]
Oct 30 19:36:10.015243 ip-10-0-132-82 kubenswrapper[4892]: W1030 19:36:10.015217    4892 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc9cadeb0_5278_4c8d_b5fc_9c345360ca3f.slice/crio-14f84e9d23fdb306d84b5cb20b50644a756d0c6e81c927958faa98a61e30b73f WatchSource:0}: Error finding container 14f84e9d23fdb306d84b5cb20b50644a756d0c6e81c927958faa98a61e30b73f: Status 404 returned error can't find the container with id 14f84e9d23fdb306d84b5cb20b50644a756d0c6e81c927958faa98a61e30b73f
Oct 30 19:36:10.694887 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:36:10.694851    4892 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2cbf0d2e-x2csg" event={"ID":"c9cadeb0-5278-4c8d-b5fc-9c345360ca3f","Type":"ContainerStarted","Data":"4e7988d1b2868d1314e3ff433a0991b33726e9bd5a37b988bcb217de1a3d3c5d"}
Oct 30 19:36:10.695075 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:36:10.694910    4892 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2cbf0d2e-x2csg" event={"ID":"c9cadeb0-5278-4c8d-b5fc-9c345360ca3f","Type":"ContainerStarted","Data":"14f84e9d23fdb306d84b5cb20b50644a756d0c6e81c927958faa98a61e30b73f"}
Oct 30 19:36:10.809652 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:36:10.809592    4892 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2cbf0d2e-x2csg" podStartSLOduration=1.809577658 podStartE2EDuration="1.809577658s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:36:09 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:36:10.808832855 +0000 UTC m=+1218.379298453" watchObservedRunningTime="2026-10-30 19:36:10.809577658 +0000 UTC m=+1218.380043256"
Oct 30 19:36:11.703178 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:36:11.703141    4892 generic.go:356] "Generic (PLEG): container finished" podID="c9cadeb0-5278-4c8d-b5fc-9c345360ca3f" containerID="4e7988d1b2868d1314e3ff433a0991b33726e9bd5a37b988bcb217de1a3d3c5d" exitCode=0
Oct 30 19:36:11.703562 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:36:11.703225    4892 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2cbf0d2e-x2csg" event={"ID":"c9cadeb0-5278-4c8d-b5fc-9c345360ca3f","Type":"ContainerDied","Data":"4e7988d1b2868d1314e3ff433a0991b33726e9bd5a37b988bcb217de1a3d3c5d"}
Oct 30 19:36:12.868735 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:36:12.868704    4892 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2cbf0d2e-x2csg"
Oct 30 19:36:12.944919 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:36:12.944889    4892 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-fsfvk"]
Oct 30 19:36:12.999750 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:36:12.999674    4892 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/c9cadeb0-5278-4c8d-b5fc-9c345360ca3f-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c9cadeb0-5278-4c8d-b5fc-9c345360ca3f-maas-api-xks-ca\") pod \"c9cadeb0-5278-4c8d-b5fc-9c345360ca3f\" (UID: \"c9cadeb0-5278-4c8d-b5fc-9c345360ca3f\") "
Oct 30 19:36:12.999750 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:36:12.999724    4892 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c9cadeb0-5278-4c8d-b5fc-9c345360ca3f-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c9cadeb0-5278-4c8d-b5fc-9c345360ca3f-maas-api-openshift-ca\") pod \"c9cadeb0-5278-4c8d-b5fc-9c345360ca3f\" (UID: \"c9cadeb0-5278-4c8d-b5fc-9c345360ca3f\") "
Oct 30 19:36:12.999953 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:36:12.999769    4892 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c9cadeb0-5278-4c8d-b5fc-9c345360ca3f-kube-api-access-9kfpm\" (UniqueName: \"kubernetes.io/projected/c9cadeb0-5278-4c8d-b5fc-9c345360ca3f-kube-api-access-9kfpm\") pod \"c9cadeb0-5278-4c8d-b5fc-9c345360ca3f\" (UID: \"c9cadeb0-5278-4c8d-b5fc-9c345360ca3f\") "
Oct 30 19:36:13.000435 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:36:13.000407    4892 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c9cadeb0-5278-4c8d-b5fc-9c345360ca3f-maas-api-openshift-ca" pod "c9cadeb0-5278-4c8d-b5fc-9c345360ca3f" (UID: "c9cadeb0-5278-4c8d-b5fc-9c345360ca3f"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 19:36:13.007007 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:36:13.002387    4892 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c9cadeb0-5278-4c8d-b5fc-9c345360ca3f-maas-api-xks-ca" pod "c9cadeb0-5278-4c8d-b5fc-9c345360ca3f" (UID: "c9cadeb0-5278-4c8d-b5fc-9c345360ca3f"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 19:36:13.007007 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:36:13.002511    4892 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c9cadeb0-5278-4c8d-b5fc-9c345360ca3f-kube-api-access-9kfpm" pod "c9cadeb0-5278-4c8d-b5fc-9c345360ca3f" (UID: "c9cadeb0-5278-4c8d-b5fc-9c345360ca3f"). InnerVolumeSpecName "kube-api-access-9kfpm". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:36:13.101429 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:36:13.101400    4892 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c9cadeb0-5278-4c8d-b5fc-9c345360ca3f-maas-api-xks-ca\") on node \"ip-10-0-132-82.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 19:36:13.101429 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:36:13.101424    4892 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c9cadeb0-5278-4c8d-b5fc-9c345360ca3f-maas-api-openshift-ca\") on node \"ip-10-0-132-82.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 19:36:13.101429 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:36:13.101433    4892 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9kfpm\" (UniqueName: \"kubernetes.io/projected/c9cadeb0-5278-4c8d-b5fc-9c345360ca3f-kube-api-access-9kfpm\") on node \"ip-10-0-132-82.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 19:36:13.582683 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:36:13.581999    4892 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-vvbng"]
Oct 30 19:36:13.583316 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:36:13.583269    4892 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-vvbng" podUID="52ecec31-2002-4855-a815-74843a6c459a" containerName="istio-proxy" containerID="cri-o://68b7c92a120af3bacf8a9de93fd54371cd68d3bb70405545e367fd5bcfed4d80" gracePeriod=30
Oct 30 19:36:13.722070 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:36:13.722036    4892 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2cbf0d2e-x2csg" event={"ID":"c9cadeb0-5278-4c8d-b5fc-9c345360ca3f","Type":"ContainerDied","Data":"14f84e9d23fdb306d84b5cb20b50644a756d0c6e81c927958faa98a61e30b73f"}
Oct 30 19:36:13.722259 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:36:13.722082    4892 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2cbf0d2e-x2csg"
Oct 30 19:36:13.722259 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:36:13.722108    4892 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="14f84e9d23fdb306d84b5cb20b50644a756d0c6e81c927958faa98a61e30b73f"
Oct 30 19:36:17.195923 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:36:17.195890    4892 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6db586dd6c-vvbng container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Oct 30 19:36:17.196324 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:36:17.195941    4892 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-vvbng" podUID="52ecec31-2002-4855-a815-74843a6c459a" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Oct 30 19:36:18.760249 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:36:18.760033    4892 generic.go:356] "Generic (PLEG): container finished" podID="52ecec31-2002-4855-a815-74843a6c459a" containerID="68b7c92a120af3bacf8a9de93fd54371cd68d3bb70405545e367fd5bcfed4d80" exitCode=0
Oct 30 19:36:18.760249 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:36:18.760095    4892 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-vvbng" event={"ID":"52ecec31-2002-4855-a815-74843a6c459a","Type":"ContainerDied","Data":"68b7c92a120af3bacf8a9de93fd54371cd68d3bb70405545e367fd5bcfed4d80"}
Oct 30 19:36:18.760249 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:36:18.760147    4892 scope.go:122] "RemoveContainer" containerID="d551544ddfb7049fef6df2b158d82d907308a9e655c11de82a56c8d500a691b7"
Oct 30 19:36:18.876698 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:36:18.876673    4892 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-vvbng"
Oct 30 19:36:18.983950 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:36:18.983875    4892 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/52ecec31-2002-4855-a815-74843a6c459a-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/52ecec31-2002-4855-a815-74843a6c459a-credential-socket\") pod \"52ecec31-2002-4855-a815-74843a6c459a\" (UID: \"52ecec31-2002-4855-a815-74843a6c459a\") "
Oct 30 19:36:18.983950 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:36:18.983913    4892 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/52ecec31-2002-4855-a815-74843a6c459a-istio-token\" (UniqueName: \"kubernetes.io/projected/52ecec31-2002-4855-a815-74843a6c459a-istio-token\") pod \"52ecec31-2002-4855-a815-74843a6c459a\" (UID: \"52ecec31-2002-4855-a815-74843a6c459a\") "
Oct 30 19:36:18.984243 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:36:18.983953    4892 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/52ecec31-2002-4855-a815-74843a6c459a-kube-api-access-t95n7\" (UniqueName: \"kubernetes.io/projected/52ecec31-2002-4855-a815-74843a6c459a-kube-api-access-t95n7\") pod \"52ecec31-2002-4855-a815-74843a6c459a\" (UID: \"52ecec31-2002-4855-a815-74843a6c459a\") "
Oct 30 19:36:18.984243 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:36:18.984004    4892 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/52ecec31-2002-4855-a815-74843a6c459a-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/52ecec31-2002-4855-a815-74843a6c459a-istio-envoy\") pod \"52ecec31-2002-4855-a815-74843a6c459a\" (UID: \"52ecec31-2002-4855-a815-74843a6c459a\") "
Oct 30 19:36:18.984243 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:36:18.984044    4892 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/52ecec31-2002-4855-a815-74843a6c459a-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/52ecec31-2002-4855-a815-74843a6c459a-istio-data\") pod \"52ecec31-2002-4855-a815-74843a6c459a\" (UID: \"52ecec31-2002-4855-a815-74843a6c459a\") "
Oct 30 19:36:18.984243 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:36:18.984124    4892 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/52ecec31-2002-4855-a815-74843a6c459a-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/52ecec31-2002-4855-a815-74843a6c459a-istiod-ca-cert\") pod \"52ecec31-2002-4855-a815-74843a6c459a\" (UID: \"52ecec31-2002-4855-a815-74843a6c459a\") "
Oct 30 19:36:18.984243 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:36:18.984192    4892 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/52ecec31-2002-4855-a815-74843a6c459a-credential-socket" pod "52ecec31-2002-4855-a815-74843a6c459a" (UID: "52ecec31-2002-4855-a815-74843a6c459a"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:36:18.984243 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:36:18.984203    4892 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/52ecec31-2002-4855-a815-74843a6c459a-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/52ecec31-2002-4855-a815-74843a6c459a-workload-certs\") pod \"52ecec31-2002-4855-a815-74843a6c459a\" (UID: \"52ecec31-2002-4855-a815-74843a6c459a\") "
Oct 30 19:36:18.984243 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:36:18.984229    4892 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/52ecec31-2002-4855-a815-74843a6c459a-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/52ecec31-2002-4855-a815-74843a6c459a-workload-socket\") pod \"52ecec31-2002-4855-a815-74843a6c459a\" (UID: \"52ecec31-2002-4855-a815-74843a6c459a\") "
Oct 30 19:36:18.984588 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:36:18.984302    4892 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/52ecec31-2002-4855-a815-74843a6c459a-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/52ecec31-2002-4855-a815-74843a6c459a-istio-podinfo\") pod \"52ecec31-2002-4855-a815-74843a6c459a\" (UID: \"52ecec31-2002-4855-a815-74843a6c459a\") "
Oct 30 19:36:18.984588 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:36:18.984379    4892 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/52ecec31-2002-4855-a815-74843a6c459a-istio-data" pod "52ecec31-2002-4855-a815-74843a6c459a" (UID: "52ecec31-2002-4855-a815-74843a6c459a"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:36:18.984588 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:36:18.984496    4892 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/52ecec31-2002-4855-a815-74843a6c459a-istiod-ca-cert" pod "52ecec31-2002-4855-a815-74843a6c459a" (UID: "52ecec31-2002-4855-a815-74843a6c459a"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 19:36:18.984588 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:36:18.984576    4892 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/52ecec31-2002-4855-a815-74843a6c459a-workload-socket" pod "52ecec31-2002-4855-a815-74843a6c459a" (UID: "52ecec31-2002-4855-a815-74843a6c459a"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:36:18.984980 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:36:18.984879    4892 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/52ecec31-2002-4855-a815-74843a6c459a-workload-socket\") on node \"ip-10-0-132-82.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 19:36:18.984980 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:36:18.984901    4892 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/52ecec31-2002-4855-a815-74843a6c459a-credential-socket\") on node \"ip-10-0-132-82.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 19:36:18.984980 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:36:18.984915    4892 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/52ecec31-2002-4855-a815-74843a6c459a-istio-data\") on node \"ip-10-0-132-82.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 19:36:18.984980 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:36:18.984930    4892 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/52ecec31-2002-4855-a815-74843a6c459a-istiod-ca-cert\") on node \"ip-10-0-132-82.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 19:36:18.985201 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:36:18.985011    4892 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/52ecec31-2002-4855-a815-74843a6c459a-workload-certs" pod "52ecec31-2002-4855-a815-74843a6c459a" (UID: "52ecec31-2002-4855-a815-74843a6c459a"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:36:18.986358 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:36:18.986330    4892 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/52ecec31-2002-4855-a815-74843a6c459a-istio-envoy" pod "52ecec31-2002-4855-a815-74843a6c459a" (UID: "52ecec31-2002-4855-a815-74843a6c459a"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:36:18.986678 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:36:18.986657    4892 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/52ecec31-2002-4855-a815-74843a6c459a-istio-token" pod "52ecec31-2002-4855-a815-74843a6c459a" (UID: "52ecec31-2002-4855-a815-74843a6c459a"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:36:18.986678 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:36:18.986666    4892 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/52ecec31-2002-4855-a815-74843a6c459a-kube-api-access-t95n7" pod "52ecec31-2002-4855-a815-74843a6c459a" (UID: "52ecec31-2002-4855-a815-74843a6c459a"). InnerVolumeSpecName "kube-api-access-t95n7". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:36:18.986787 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:36:18.986684    4892 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/52ecec31-2002-4855-a815-74843a6c459a-istio-podinfo" pod "52ecec31-2002-4855-a815-74843a6c459a" (UID: "52ecec31-2002-4855-a815-74843a6c459a"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 19:36:19.086242 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:36:19.086213    4892 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/52ecec31-2002-4855-a815-74843a6c459a-istio-podinfo\") on node \"ip-10-0-132-82.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 19:36:19.086242 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:36:19.086243    4892 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/52ecec31-2002-4855-a815-74843a6c459a-istio-token\") on node \"ip-10-0-132-82.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 19:36:19.086453 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:36:19.086256    4892 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-t95n7\" (UniqueName: \"kubernetes.io/projected/52ecec31-2002-4855-a815-74843a6c459a-kube-api-access-t95n7\") on node \"ip-10-0-132-82.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 19:36:19.086453 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:36:19.086271    4892 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/52ecec31-2002-4855-a815-74843a6c459a-istio-envoy\") on node \"ip-10-0-132-82.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 19:36:19.086453 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:36:19.086282    4892 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/52ecec31-2002-4855-a815-74843a6c459a-workload-certs\") on node \"ip-10-0-132-82.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 19:36:19.768148 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:36:19.768119    4892 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-vvbng"
Oct 30 19:36:19.768148 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:36:19.768141    4892 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-vvbng" event={"ID":"52ecec31-2002-4855-a815-74843a6c459a","Type":"ContainerDied","Data":"67cd7f02907b95733b592d528871c7be106356f54c1408fb286e0b91a2dbb581"}
Oct 30 19:36:19.768608 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:36:19.768193    4892 scope.go:122] "RemoveContainer" containerID="68b7c92a120af3bacf8a9de93fd54371cd68d3bb70405545e367fd5bcfed4d80"
Oct 30 19:36:19.977508 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:36:19.977479    4892 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f92547ee-ldpfj"]
Oct 30 19:36:19.978064 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:36:19.977958    4892 cpu_manager.go:414] "containerMap: removing container" podUID="c9cadeb0-5278-4c8d-b5fc-9c345360ca3f" containerName="revoke-keys"
Oct 30 19:36:19.978134 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:36:19.978068    4892 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c9cadeb0-5278-4c8d-b5fc-9c345360ca3f" containerName="revoke-keys"
Oct 30 19:36:19.978134 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:36:19.978094    4892 cpu_manager.go:414] "containerMap: removing container" podUID="52ecec31-2002-4855-a815-74843a6c459a" containerName="istio-proxy"
Oct 30 19:36:19.978134 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:36:19.978099    4892 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="52ecec31-2002-4855-a815-74843a6c459a" containerName="istio-proxy"
Oct 30 19:36:19.978134 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:36:19.978110    4892 cpu_manager.go:414] "containerMap: removing container" podUID="52ecec31-2002-4855-a815-74843a6c459a" containerName="istio-proxy"
Oct 30 19:36:19.978134 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:36:19.978115    4892 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="52ecec31-2002-4855-a815-74843a6c459a" containerName="istio-proxy"
Oct 30 19:36:19.978279 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:36:19.978243    4892 memory_manager.go:365] "RemoveStaleState removing state" podUID="52ecec31-2002-4855-a815-74843a6c459a" containerName="istio-proxy"
Oct 30 19:36:19.978279 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:36:19.978251    4892 memory_manager.go:365] "RemoveStaleState removing state" podUID="52ecec31-2002-4855-a815-74843a6c459a" containerName="istio-proxy"
Oct 30 19:36:19.978279 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:36:19.978263    4892 memory_manager.go:365] "RemoveStaleState removing state" podUID="c9cadeb0-5278-4c8d-b5fc-9c345360ca3f" containerName="revoke-keys"
Oct 30 19:36:19.983112 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:36:19.983089    4892 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-vvbng"]
Oct 30 19:36:19.983261 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:36:19.983124    4892 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f92547ee-ldpfj"
Oct 30 19:36:20.032094 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:36:20.032019    4892 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-vvbng"]
Oct 30 19:36:20.035308 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:36:20.035279    4892 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 19:36:20.035482 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:36:20.035340    4892 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 30 19:36:20.035482 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:36:20.035399    4892 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-nlxp4\""
Oct 30 19:36:20.035482 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:36:20.035443    4892 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 30 19:36:20.038762 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:36:20.038737    4892 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f92547ee-ldpfj"]
Oct 30 19:36:20.096818 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:36:20.096788    4892 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/01dbf08c-a038-4419-bacb-ec4e2df61a57-maas-api-xks-ca\") pod \"maas-api-revoke-sub-f92547ee-ldpfj\" (UID: \"01dbf08c-a038-4419-bacb-ec4e2df61a57\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f92547ee-ldpfj"
Oct 30 19:36:20.097090 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:36:20.096827    4892 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/01dbf08c-a038-4419-bacb-ec4e2df61a57-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-f92547ee-ldpfj\" (UID: \"01dbf08c-a038-4419-bacb-ec4e2df61a57\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f92547ee-ldpfj"
Oct 30 19:36:20.097090 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:36:20.096848    4892 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-88ksq\" (UniqueName: \"kubernetes.io/projected/01dbf08c-a038-4419-bacb-ec4e2df61a57-kube-api-access-88ksq\") pod \"maas-api-revoke-sub-f92547ee-ldpfj\" (UID: \"01dbf08c-a038-4419-bacb-ec4e2df61a57\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f92547ee-ldpfj"
Oct 30 19:36:20.198441 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:36:20.198412    4892 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/01dbf08c-a038-4419-bacb-ec4e2df61a57-maas-api-xks-ca\") pod \"maas-api-revoke-sub-f92547ee-ldpfj\" (UID: \"01dbf08c-a038-4419-bacb-ec4e2df61a57\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f92547ee-ldpfj"
Oct 30 19:36:20.198617 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:36:20.198452    4892 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/01dbf08c-a038-4419-bacb-ec4e2df61a57-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-f92547ee-ldpfj\" (UID: \"01dbf08c-a038-4419-bacb-ec4e2df61a57\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f92547ee-ldpfj"
Oct 30 19:36:20.198617 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:36:20.198472    4892 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-88ksq\" (UniqueName: \"kubernetes.io/projected/01dbf08c-a038-4419-bacb-ec4e2df61a57-kube-api-access-88ksq\") pod \"maas-api-revoke-sub-f92547ee-ldpfj\" (UID: \"01dbf08c-a038-4419-bacb-ec4e2df61a57\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f92547ee-ldpfj"
Oct 30 19:36:20.199198 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:36:20.199172    4892 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/01dbf08c-a038-4419-bacb-ec4e2df61a57-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-f92547ee-ldpfj\" (UID: \"01dbf08c-a038-4419-bacb-ec4e2df61a57\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f92547ee-ldpfj"
Oct 30 19:36:20.200829 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:36:20.200804    4892 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/01dbf08c-a038-4419-bacb-ec4e2df61a57-maas-api-xks-ca\") pod \"maas-api-revoke-sub-f92547ee-ldpfj\" (UID: \"01dbf08c-a038-4419-bacb-ec4e2df61a57\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f92547ee-ldpfj"
Oct 30 19:36:20.256702 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:36:20.256670    4892 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-88ksq\" (UniqueName: \"kubernetes.io/projected/01dbf08c-a038-4419-bacb-ec4e2df61a57-kube-api-access-88ksq\") pod \"maas-api-revoke-sub-f92547ee-ldpfj\" (UID: \"01dbf08c-a038-4419-bacb-ec4e2df61a57\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f92547ee-ldpfj"
Oct 30 19:36:20.296677 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:36:20.296620    4892 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f92547ee-ldpfj"
Oct 30 19:36:20.570546 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:36:20.570475    4892 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="52ecec31-2002-4855-a815-74843a6c459a" path="/var/lib/kubelet/pods/52ecec31-2002-4855-a815-74843a6c459a/volumes"
Oct 30 19:36:20.675149 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:36:20.675123    4892 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f92547ee-ldpfj"]
Oct 30 19:36:20.677325 ip-10-0-132-82 kubenswrapper[4892]: W1030 19:36:20.677300    4892 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod01dbf08c_a038_4419_bacb_ec4e2df61a57.slice/crio-dac106a8320294f02150a5fa1779b1252c3411e2eae3f40333e4507f67d75325 WatchSource:0}: Error finding container dac106a8320294f02150a5fa1779b1252c3411e2eae3f40333e4507f67d75325: Status 404 returned error can't find the container with id dac106a8320294f02150a5fa1779b1252c3411e2eae3f40333e4507f67d75325
Oct 30 19:36:20.774847 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:36:20.774820    4892 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f92547ee-ldpfj" event={"ID":"01dbf08c-a038-4419-bacb-ec4e2df61a57","Type":"ContainerStarted","Data":"dac106a8320294f02150a5fa1779b1252c3411e2eae3f40333e4507f67d75325"}
Oct 30 19:36:21.783873 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:36:21.783783    4892 generic.go:356] "Generic (PLEG): container finished" podID="01dbf08c-a038-4419-bacb-ec4e2df61a57" containerID="3190864233b07b72014fee8365fbcee348c60add7739c13f42a6127c662dda91" exitCode=0
Oct 30 19:36:21.784263 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:36:21.783876    4892 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f92547ee-ldpfj" event={"ID":"01dbf08c-a038-4419-bacb-ec4e2df61a57","Type":"ContainerDied","Data":"3190864233b07b72014fee8365fbcee348c60add7739c13f42a6127c662dda91"}
Oct 30 19:36:22.943787 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:36:22.943764    4892 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f92547ee-ldpfj"
Oct 30 19:36:23.124344 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:36:23.124267    4892 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/01dbf08c-a038-4419-bacb-ec4e2df61a57-kube-api-access-88ksq\" (UniqueName: \"kubernetes.io/projected/01dbf08c-a038-4419-bacb-ec4e2df61a57-kube-api-access-88ksq\") pod \"01dbf08c-a038-4419-bacb-ec4e2df61a57\" (UID: \"01dbf08c-a038-4419-bacb-ec4e2df61a57\") "
Oct 30 19:36:23.124505 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:36:23.124364    4892 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/01dbf08c-a038-4419-bacb-ec4e2df61a57-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/01dbf08c-a038-4419-bacb-ec4e2df61a57-maas-api-xks-ca\") pod \"01dbf08c-a038-4419-bacb-ec4e2df61a57\" (UID: \"01dbf08c-a038-4419-bacb-ec4e2df61a57\") "
Oct 30 19:36:23.124505 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:36:23.124430    4892 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/01dbf08c-a038-4419-bacb-ec4e2df61a57-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/01dbf08c-a038-4419-bacb-ec4e2df61a57-maas-api-openshift-ca\") pod \"01dbf08c-a038-4419-bacb-ec4e2df61a57\" (UID: \"01dbf08c-a038-4419-bacb-ec4e2df61a57\") "
Oct 30 19:36:23.124818 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:36:23.124792    4892 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/01dbf08c-a038-4419-bacb-ec4e2df61a57-maas-api-openshift-ca" pod "01dbf08c-a038-4419-bacb-ec4e2df61a57" (UID: "01dbf08c-a038-4419-bacb-ec4e2df61a57"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 19:36:23.126467 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:36:23.126445    4892 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/01dbf08c-a038-4419-bacb-ec4e2df61a57-maas-api-xks-ca" pod "01dbf08c-a038-4419-bacb-ec4e2df61a57" (UID: "01dbf08c-a038-4419-bacb-ec4e2df61a57"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 19:36:23.126543 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:36:23.126485    4892 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/01dbf08c-a038-4419-bacb-ec4e2df61a57-kube-api-access-88ksq" pod "01dbf08c-a038-4419-bacb-ec4e2df61a57" (UID: "01dbf08c-a038-4419-bacb-ec4e2df61a57"). InnerVolumeSpecName "kube-api-access-88ksq". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:36:23.226262 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:36:23.226233    4892 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-88ksq\" (UniqueName: \"kubernetes.io/projected/01dbf08c-a038-4419-bacb-ec4e2df61a57-kube-api-access-88ksq\") on node \"ip-10-0-132-82.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 19:36:23.226262 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:36:23.226259    4892 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/01dbf08c-a038-4419-bacb-ec4e2df61a57-maas-api-xks-ca\") on node \"ip-10-0-132-82.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 19:36:23.226450 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:36:23.226268    4892 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/01dbf08c-a038-4419-bacb-ec4e2df61a57-maas-api-openshift-ca\") on node \"ip-10-0-132-82.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 19:36:23.798853 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:36:23.798821    4892 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f92547ee-ldpfj" event={"ID":"01dbf08c-a038-4419-bacb-ec4e2df61a57","Type":"ContainerDied","Data":"dac106a8320294f02150a5fa1779b1252c3411e2eae3f40333e4507f67d75325"}
Oct 30 19:36:23.798853 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:36:23.798851    4892 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f92547ee-ldpfj"
Oct 30 19:36:23.799077 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:36:23.798876    4892 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="dac106a8320294f02150a5fa1779b1252c3411e2eae3f40333e4507f67d75325"
Oct 30 19:36:24.859585 ip-10-0-132-82 kubenswrapper[4892]: E1030 19:36:24.859549    4892 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-jrm87" containerName="extractor"
Oct 30 19:36:26.313726 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:36:26.313689    4892 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-cf3dab52-ghxkm"]
Oct 30 19:36:26.314190 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:36:26.314175    4892 cpu_manager.go:414] "containerMap: removing container" podUID="01dbf08c-a038-4419-bacb-ec4e2df61a57" containerName="revoke-keys"
Oct 30 19:36:26.314246 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:36:26.314192    4892 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="01dbf08c-a038-4419-bacb-ec4e2df61a57" containerName="revoke-keys"
Oct 30 19:36:26.314300 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:36:26.314291    4892 memory_manager.go:365] "RemoveStaleState removing state" podUID="01dbf08c-a038-4419-bacb-ec4e2df61a57" containerName="revoke-keys"
Oct 30 19:36:26.317635 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:36:26.317621    4892 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-cf3dab52-ghxkm"
Oct 30 19:36:26.369866 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:36:26.369840    4892 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cmhpx\" (UniqueName: \"kubernetes.io/projected/dc25af73-f73d-49c8-aa34-a03de0cd2297-kube-api-access-cmhpx\") pod \"maas-api-revoke-sub-cf3dab52-ghxkm\" (UID: \"dc25af73-f73d-49c8-aa34-a03de0cd2297\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-cf3dab52-ghxkm"
Oct 30 19:36:26.370009 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:36:26.369952    4892 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/dc25af73-f73d-49c8-aa34-a03de0cd2297-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-cf3dab52-ghxkm\" (UID: \"dc25af73-f73d-49c8-aa34-a03de0cd2297\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-cf3dab52-ghxkm"
Oct 30 19:36:26.370055 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:36:26.370015    4892 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/dc25af73-f73d-49c8-aa34-a03de0cd2297-maas-api-xks-ca\") pod \"maas-api-revoke-sub-cf3dab52-ghxkm\" (UID: \"dc25af73-f73d-49c8-aa34-a03de0cd2297\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-cf3dab52-ghxkm"
Oct 30 19:36:26.420375 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:36:26.420335    4892 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 19:36:26.420375 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:36:26.420353    4892 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-nlxp4\""
Oct 30 19:36:26.420375 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:36:26.420353    4892 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 30 19:36:26.420578 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:36:26.420399    4892 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 30 19:36:26.423188 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:36:26.423168    4892 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-cf3dab52-ghxkm"]
Oct 30 19:36:26.471118 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:36:26.471094    4892 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/dc25af73-f73d-49c8-aa34-a03de0cd2297-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-cf3dab52-ghxkm\" (UID: \"dc25af73-f73d-49c8-aa34-a03de0cd2297\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-cf3dab52-ghxkm"
Oct 30 19:36:26.471229 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:36:26.471129    4892 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/dc25af73-f73d-49c8-aa34-a03de0cd2297-maas-api-xks-ca\") pod \"maas-api-revoke-sub-cf3dab52-ghxkm\" (UID: \"dc25af73-f73d-49c8-aa34-a03de0cd2297\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-cf3dab52-ghxkm"
Oct 30 19:36:26.471229 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:36:26.471207    4892 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cmhpx\" (UniqueName: \"kubernetes.io/projected/dc25af73-f73d-49c8-aa34-a03de0cd2297-kube-api-access-cmhpx\") pod \"maas-api-revoke-sub-cf3dab52-ghxkm\" (UID: \"dc25af73-f73d-49c8-aa34-a03de0cd2297\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-cf3dab52-ghxkm"
Oct 30 19:36:26.471934 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:36:26.471876    4892 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/dc25af73-f73d-49c8-aa34-a03de0cd2297-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-cf3dab52-ghxkm\" (UID: \"dc25af73-f73d-49c8-aa34-a03de0cd2297\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-cf3dab52-ghxkm"
Oct 30 19:36:26.473746 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:36:26.473726    4892 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/dc25af73-f73d-49c8-aa34-a03de0cd2297-maas-api-xks-ca\") pod \"maas-api-revoke-sub-cf3dab52-ghxkm\" (UID: \"dc25af73-f73d-49c8-aa34-a03de0cd2297\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-cf3dab52-ghxkm"
Oct 30 19:36:26.527545 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:36:26.527519    4892 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-cmhpx\" (UniqueName: \"kubernetes.io/projected/dc25af73-f73d-49c8-aa34-a03de0cd2297-kube-api-access-cmhpx\") pod \"maas-api-revoke-sub-cf3dab52-ghxkm\" (UID: \"dc25af73-f73d-49c8-aa34-a03de0cd2297\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-cf3dab52-ghxkm"
Oct 30 19:36:26.630641 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:36:26.630575    4892 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-cf3dab52-ghxkm"
Oct 30 19:36:27.016552 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:36:27.016524    4892 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-cf3dab52-ghxkm"]
Oct 30 19:36:27.017823 ip-10-0-132-82 kubenswrapper[4892]: W1030 19:36:27.017793    4892 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddc25af73_f73d_49c8_aa34_a03de0cd2297.slice/crio-c619bc89c0bd55a4c4407e36cc8d36db326dcea566f3d460193e6f7204e9a5a3 WatchSource:0}: Error finding container c619bc89c0bd55a4c4407e36cc8d36db326dcea566f3d460193e6f7204e9a5a3: Status 404 returned error can't find the container with id c619bc89c0bd55a4c4407e36cc8d36db326dcea566f3d460193e6f7204e9a5a3
Oct 30 19:36:27.835218 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:36:27.835128    4892 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-cf3dab52-ghxkm" event={"ID":"dc25af73-f73d-49c8-aa34-a03de0cd2297","Type":"ContainerStarted","Data":"f0364ff0cfe6520dce00be5b77b09e5b0f6ac68d4cac81f0bf1eb7d3ecd0b820"}
Oct 30 19:36:27.835614 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:36:27.835271    4892 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-cf3dab52-ghxkm" event={"ID":"dc25af73-f73d-49c8-aa34-a03de0cd2297","Type":"ContainerStarted","Data":"c619bc89c0bd55a4c4407e36cc8d36db326dcea566f3d460193e6f7204e9a5a3"}
Oct 30 19:36:27.950949 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:36:27.950900    4892 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-cf3dab52-ghxkm" podStartSLOduration=1.950887477 podStartE2EDuration="1.950887477s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:36:26 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:36:27.948258613 +0000 UTC m=+1235.518724213" watchObservedRunningTime="2026-10-30 19:36:27.950887477 +0000 UTC m=+1235.521353080"
Oct 30 19:36:28.000192 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:36:28.000167    4892 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-15133760-rwwjv"]
Oct 30 19:36:28.003880 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:36:28.003864    4892 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-15133760-rwwjv"
Oct 30 19:36:28.015464 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:36:28.015440    4892 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0f728833-f24c-48d8-acd2-37e2a738ffc8-maas-api-xks-ca\") pod \"maas-api-revoke-sub-15133760-rwwjv\" (UID: \"0f728833-f24c-48d8-acd2-37e2a738ffc8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-15133760-rwwjv"
Oct 30 19:36:28.015548 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:36:28.015485    4892 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0f728833-f24c-48d8-acd2-37e2a738ffc8-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-15133760-rwwjv\" (UID: \"0f728833-f24c-48d8-acd2-37e2a738ffc8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-15133760-rwwjv"
Oct 30 19:36:28.015628 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:36:28.015608    4892 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jhxjx\" (UniqueName: \"kubernetes.io/projected/0f728833-f24c-48d8-acd2-37e2a738ffc8-kube-api-access-jhxjx\") pod \"maas-api-revoke-sub-15133760-rwwjv\" (UID: \"0f728833-f24c-48d8-acd2-37e2a738ffc8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-15133760-rwwjv"
Oct 30 19:36:28.064661 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:36:28.064635    4892 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-15133760-rwwjv"]
Oct 30 19:36:28.116374 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:36:28.116305    4892 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0f728833-f24c-48d8-acd2-37e2a738ffc8-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-15133760-rwwjv\" (UID: \"0f728833-f24c-48d8-acd2-37e2a738ffc8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-15133760-rwwjv"
Oct 30 19:36:28.116537 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:36:28.116444    4892 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jhxjx\" (UniqueName: \"kubernetes.io/projected/0f728833-f24c-48d8-acd2-37e2a738ffc8-kube-api-access-jhxjx\") pod \"maas-api-revoke-sub-15133760-rwwjv\" (UID: \"0f728833-f24c-48d8-acd2-37e2a738ffc8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-15133760-rwwjv"
Oct 30 19:36:28.116537 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:36:28.116502    4892 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0f728833-f24c-48d8-acd2-37e2a738ffc8-maas-api-xks-ca\") pod \"maas-api-revoke-sub-15133760-rwwjv\" (UID: \"0f728833-f24c-48d8-acd2-37e2a738ffc8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-15133760-rwwjv"
Oct 30 19:36:28.117089 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:36:28.117062    4892 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0f728833-f24c-48d8-acd2-37e2a738ffc8-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-15133760-rwwjv\" (UID: \"0f728833-f24c-48d8-acd2-37e2a738ffc8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-15133760-rwwjv"
Oct 30 19:36:28.119026 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:36:28.118999    4892 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0f728833-f24c-48d8-acd2-37e2a738ffc8-maas-api-xks-ca\") pod \"maas-api-revoke-sub-15133760-rwwjv\" (UID: \"0f728833-f24c-48d8-acd2-37e2a738ffc8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-15133760-rwwjv"
Oct 30 19:36:28.172217 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:36:28.172182    4892 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jhxjx\" (UniqueName: \"kubernetes.io/projected/0f728833-f24c-48d8-acd2-37e2a738ffc8-kube-api-access-jhxjx\") pod \"maas-api-revoke-sub-15133760-rwwjv\" (UID: \"0f728833-f24c-48d8-acd2-37e2a738ffc8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-15133760-rwwjv"
Oct 30 19:36:28.316843 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:36:28.316818    4892 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-15133760-rwwjv"
Oct 30 19:36:28.695897 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:36:28.695867    4892 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-15133760-rwwjv"]
Oct 30 19:36:28.697446 ip-10-0-132-82 kubenswrapper[4892]: W1030 19:36:28.697421    4892 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0f728833_f24c_48d8_acd2_37e2a738ffc8.slice/crio-73500731111b5e66a76ccc7f1d2bdb5995fa8873672cee78b4cea600f5f53da3 WatchSource:0}: Error finding container 73500731111b5e66a76ccc7f1d2bdb5995fa8873672cee78b4cea600f5f53da3: Status 404 returned error can't find the container with id 73500731111b5e66a76ccc7f1d2bdb5995fa8873672cee78b4cea600f5f53da3
Oct 30 19:36:28.843723 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:36:28.843684    4892 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-15133760-rwwjv" event={"ID":"0f728833-f24c-48d8-acd2-37e2a738ffc8","Type":"ContainerStarted","Data":"73500731111b5e66a76ccc7f1d2bdb5995fa8873672cee78b4cea600f5f53da3"}
Oct 30 19:36:28.845521 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:36:28.845485    4892 generic.go:356] "Generic (PLEG): container finished" podID="dc25af73-f73d-49c8-aa34-a03de0cd2297" containerID="f0364ff0cfe6520dce00be5b77b09e5b0f6ac68d4cac81f0bf1eb7d3ecd0b820" exitCode=0
Oct 30 19:36:28.845643 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:36:28.845576    4892 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-cf3dab52-ghxkm" event={"ID":"dc25af73-f73d-49c8-aa34-a03de0cd2297","Type":"ContainerDied","Data":"f0364ff0cfe6520dce00be5b77b09e5b0f6ac68d4cac81f0bf1eb7d3ecd0b820"}
Oct 30 19:36:29.854913 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:36:29.854875    4892 generic.go:356] "Generic (PLEG): container finished" podID="0f728833-f24c-48d8-acd2-37e2a738ffc8" containerID="93be56df23857f3096cc7aadba9a1d6221683899fc0d3cf07cade71a8f99d832" exitCode=0
Oct 30 19:36:29.855341 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:36:29.854958    4892 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-15133760-rwwjv" event={"ID":"0f728833-f24c-48d8-acd2-37e2a738ffc8","Type":"ContainerDied","Data":"93be56df23857f3096cc7aadba9a1d6221683899fc0d3cf07cade71a8f99d832"}
Oct 30 19:36:29.999956 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:36:29.999930    4892 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-cf3dab52-ghxkm"
Oct 30 19:36:30.055369 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:36:30.055336    4892 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/dc25af73-f73d-49c8-aa34-a03de0cd2297-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/dc25af73-f73d-49c8-aa34-a03de0cd2297-maas-api-xks-ca\") pod \"dc25af73-f73d-49c8-aa34-a03de0cd2297\" (UID: \"dc25af73-f73d-49c8-aa34-a03de0cd2297\") "
Oct 30 19:36:30.055523 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:36:30.055451    4892 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/dc25af73-f73d-49c8-aa34-a03de0cd2297-kube-api-access-cmhpx\" (UniqueName: \"kubernetes.io/projected/dc25af73-f73d-49c8-aa34-a03de0cd2297-kube-api-access-cmhpx\") pod \"dc25af73-f73d-49c8-aa34-a03de0cd2297\" (UID: \"dc25af73-f73d-49c8-aa34-a03de0cd2297\") "
Oct 30 19:36:30.055606 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:36:30.055520    4892 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/dc25af73-f73d-49c8-aa34-a03de0cd2297-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/dc25af73-f73d-49c8-aa34-a03de0cd2297-maas-api-openshift-ca\") pod \"dc25af73-f73d-49c8-aa34-a03de0cd2297\" (UID: \"dc25af73-f73d-49c8-aa34-a03de0cd2297\") "
Oct 30 19:36:30.055916 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:36:30.055887    4892 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/dc25af73-f73d-49c8-aa34-a03de0cd2297-maas-api-openshift-ca" pod "dc25af73-f73d-49c8-aa34-a03de0cd2297" (UID: "dc25af73-f73d-49c8-aa34-a03de0cd2297"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 19:36:30.057787 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:36:30.057757    4892 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/dc25af73-f73d-49c8-aa34-a03de0cd2297-maas-api-xks-ca" pod "dc25af73-f73d-49c8-aa34-a03de0cd2297" (UID: "dc25af73-f73d-49c8-aa34-a03de0cd2297"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 19:36:30.057892 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:36:30.057786    4892 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/dc25af73-f73d-49c8-aa34-a03de0cd2297-kube-api-access-cmhpx" pod "dc25af73-f73d-49c8-aa34-a03de0cd2297" (UID: "dc25af73-f73d-49c8-aa34-a03de0cd2297"). InnerVolumeSpecName "kube-api-access-cmhpx". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:36:30.157272 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:36:30.157218    4892 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/dc25af73-f73d-49c8-aa34-a03de0cd2297-maas-api-openshift-ca\") on node \"ip-10-0-132-82.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 19:36:30.157272 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:36:30.157262    4892 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/dc25af73-f73d-49c8-aa34-a03de0cd2297-maas-api-xks-ca\") on node \"ip-10-0-132-82.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 19:36:30.157272 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:36:30.157272    4892 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-cmhpx\" (UniqueName: \"kubernetes.io/projected/dc25af73-f73d-49c8-aa34-a03de0cd2297-kube-api-access-cmhpx\") on node \"ip-10-0-132-82.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 19:36:30.865189 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:36:30.865107    4892 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-cf3dab52-ghxkm"
Oct 30 19:36:30.865189 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:36:30.865095    4892 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-cf3dab52-ghxkm" event={"ID":"dc25af73-f73d-49c8-aa34-a03de0cd2297","Type":"ContainerDied","Data":"c619bc89c0bd55a4c4407e36cc8d36db326dcea566f3d460193e6f7204e9a5a3"}
Oct 30 19:36:30.865718 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:36:30.865217    4892 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="c619bc89c0bd55a4c4407e36cc8d36db326dcea566f3d460193e6f7204e9a5a3"
Oct 30 19:36:31.013435 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:36:31.013412    4892 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-15133760-rwwjv"
Oct 30 19:36:31.064867 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:36:31.064841    4892 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/0f728833-f24c-48d8-acd2-37e2a738ffc8-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0f728833-f24c-48d8-acd2-37e2a738ffc8-maas-api-xks-ca\") pod \"0f728833-f24c-48d8-acd2-37e2a738ffc8\" (UID: \"0f728833-f24c-48d8-acd2-37e2a738ffc8\") "
Oct 30 19:36:31.065056 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:36:31.064907    4892 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/0f728833-f24c-48d8-acd2-37e2a738ffc8-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0f728833-f24c-48d8-acd2-37e2a738ffc8-maas-api-openshift-ca\") pod \"0f728833-f24c-48d8-acd2-37e2a738ffc8\" (UID: \"0f728833-f24c-48d8-acd2-37e2a738ffc8\") "
Oct 30 19:36:31.065056 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:36:31.065029    4892 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0f728833-f24c-48d8-acd2-37e2a738ffc8-kube-api-access-jhxjx\" (UniqueName: \"kubernetes.io/projected/0f728833-f24c-48d8-acd2-37e2a738ffc8-kube-api-access-jhxjx\") pod \"0f728833-f24c-48d8-acd2-37e2a738ffc8\" (UID: \"0f728833-f24c-48d8-acd2-37e2a738ffc8\") "
Oct 30 19:36:31.065290 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:36:31.065267    4892 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0f728833-f24c-48d8-acd2-37e2a738ffc8-maas-api-openshift-ca" pod "0f728833-f24c-48d8-acd2-37e2a738ffc8" (UID: "0f728833-f24c-48d8-acd2-37e2a738ffc8"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 19:36:31.065499 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:36:31.065479    4892 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0f728833-f24c-48d8-acd2-37e2a738ffc8-maas-api-openshift-ca\") on node \"ip-10-0-132-82.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 19:36:31.067294 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:36:31.067260    4892 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0f728833-f24c-48d8-acd2-37e2a738ffc8-maas-api-xks-ca" pod "0f728833-f24c-48d8-acd2-37e2a738ffc8" (UID: "0f728833-f24c-48d8-acd2-37e2a738ffc8"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 19:36:31.067514 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:36:31.067487    4892 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0f728833-f24c-48d8-acd2-37e2a738ffc8-kube-api-access-jhxjx" pod "0f728833-f24c-48d8-acd2-37e2a738ffc8" (UID: "0f728833-f24c-48d8-acd2-37e2a738ffc8"). InnerVolumeSpecName "kube-api-access-jhxjx". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:36:31.166437 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:36:31.166410    4892 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0f728833-f24c-48d8-acd2-37e2a738ffc8-maas-api-xks-ca\") on node \"ip-10-0-132-82.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 19:36:31.166437 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:36:31.166438    4892 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jhxjx\" (UniqueName: \"kubernetes.io/projected/0f728833-f24c-48d8-acd2-37e2a738ffc8-kube-api-access-jhxjx\") on node \"ip-10-0-132-82.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 19:36:31.872284 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:36:31.872254    4892 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-15133760-rwwjv" event={"ID":"0f728833-f24c-48d8-acd2-37e2a738ffc8","Type":"ContainerDied","Data":"73500731111b5e66a76ccc7f1d2bdb5995fa8873672cee78b4cea600f5f53da3"}
Oct 30 19:36:31.872284 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:36:31.872271    4892 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-15133760-rwwjv"
Oct 30 19:36:31.872736 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:36:31.872305    4892 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="73500731111b5e66a76ccc7f1d2bdb5995fa8873672cee78b4cea600f5f53da3"
Oct 30 19:36:43.785477 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:36:43.785436    4892 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w2-a3f9b7-openshift-default-7d6d8b8fb5-sfm8n"]
Oct 30 19:36:43.786086 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:36:43.786067    4892 cpu_manager.go:414] "containerMap: removing container" podUID="0f728833-f24c-48d8-acd2-37e2a738ffc8" containerName="revoke-keys"
Oct 30 19:36:43.786167 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:36:43.786088    4892 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0f728833-f24c-48d8-acd2-37e2a738ffc8" containerName="revoke-keys"
Oct 30 19:36:43.786167 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:36:43.786147    4892 cpu_manager.go:414] "containerMap: removing container" podUID="dc25af73-f73d-49c8-aa34-a03de0cd2297" containerName="revoke-keys"
Oct 30 19:36:43.786167 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:36:43.786156    4892 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="dc25af73-f73d-49c8-aa34-a03de0cd2297" containerName="revoke-keys"
Oct 30 19:36:43.786326 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:36:43.786293    4892 memory_manager.go:365] "RemoveStaleState removing state" podUID="0f728833-f24c-48d8-acd2-37e2a738ffc8" containerName="revoke-keys"
Oct 30 19:36:43.786326 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:36:43.786310    4892 memory_manager.go:365] "RemoveStaleState removing state" podUID="dc25af73-f73d-49c8-aa34-a03de0cd2297" containerName="revoke-keys"
Oct 30 19:36:43.789675 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:36:43.789655    4892 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-a3f9b7-openshift-default-7d6d8b8fb5-sfm8n"
Oct 30 19:36:43.842555 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:36:43.842533    4892 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w2-a3f9b7-openshift-default-dockercfg-9v6f2\""
Oct 30 19:36:43.889093 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:36:43.889066    4892 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1c3346aa-a989-495e-9d54-66c8ec632d91-istio-podinfo\") pod \"e2e-worker-w2-a3f9b7-openshift-default-7d6d8b8fb5-sfm8n\" (UID: \"1c3346aa-a989-495e-9d54-66c8ec632d91\") " pod="openshift-ingress/e2e-worker-w2-a3f9b7-openshift-default-7d6d8b8fb5-sfm8n"
Oct 30 19:36:43.889232 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:36:43.889108    4892 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1c3346aa-a989-495e-9d54-66c8ec632d91-istio-data\") pod \"e2e-worker-w2-a3f9b7-openshift-default-7d6d8b8fb5-sfm8n\" (UID: \"1c3346aa-a989-495e-9d54-66c8ec632d91\") " pod="openshift-ingress/e2e-worker-w2-a3f9b7-openshift-default-7d6d8b8fb5-sfm8n"
Oct 30 19:36:43.889232 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:36:43.889124    4892 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1c3346aa-a989-495e-9d54-66c8ec632d91-workload-socket\") pod \"e2e-worker-w2-a3f9b7-openshift-default-7d6d8b8fb5-sfm8n\" (UID: \"1c3346aa-a989-495e-9d54-66c8ec632d91\") " pod="openshift-ingress/e2e-worker-w2-a3f9b7-openshift-default-7d6d8b8fb5-sfm8n"
Oct 30 19:36:43.889232 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:36:43.889146    4892 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1c3346aa-a989-495e-9d54-66c8ec632d91-istio-envoy\") pod \"e2e-worker-w2-a3f9b7-openshift-default-7d6d8b8fb5-sfm8n\" (UID: \"1c3346aa-a989-495e-9d54-66c8ec632d91\") " pod="openshift-ingress/e2e-worker-w2-a3f9b7-openshift-default-7d6d8b8fb5-sfm8n"
Oct 30 19:36:43.889232 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:36:43.889162    4892 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1c3346aa-a989-495e-9d54-66c8ec632d91-istio-token\") pod \"e2e-worker-w2-a3f9b7-openshift-default-7d6d8b8fb5-sfm8n\" (UID: \"1c3346aa-a989-495e-9d54-66c8ec632d91\") " pod="openshift-ingress/e2e-worker-w2-a3f9b7-openshift-default-7d6d8b8fb5-sfm8n"
Oct 30 19:36:43.889437 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:36:43.889223    4892 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1c3346aa-a989-495e-9d54-66c8ec632d91-credential-socket\") pod \"e2e-worker-w2-a3f9b7-openshift-default-7d6d8b8fb5-sfm8n\" (UID: \"1c3346aa-a989-495e-9d54-66c8ec632d91\") " pod="openshift-ingress/e2e-worker-w2-a3f9b7-openshift-default-7d6d8b8fb5-sfm8n"
Oct 30 19:36:43.889437 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:36:43.889412    4892 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1c3346aa-a989-495e-9d54-66c8ec632d91-workload-certs\") pod \"e2e-worker-w2-a3f9b7-openshift-default-7d6d8b8fb5-sfm8n\" (UID: \"1c3346aa-a989-495e-9d54-66c8ec632d91\") " pod="openshift-ingress/e2e-worker-w2-a3f9b7-openshift-default-7d6d8b8fb5-sfm8n"
Oct 30 19:36:43.889591 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:36:43.889473    4892 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1c3346aa-a989-495e-9d54-66c8ec632d91-istiod-ca-cert\") pod \"e2e-worker-w2-a3f9b7-openshift-default-7d6d8b8fb5-sfm8n\" (UID: \"1c3346aa-a989-495e-9d54-66c8ec632d91\") " pod="openshift-ingress/e2e-worker-w2-a3f9b7-openshift-default-7d6d8b8fb5-sfm8n"
Oct 30 19:36:43.889591 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:36:43.889505    4892 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9mxx5\" (UniqueName: \"kubernetes.io/projected/1c3346aa-a989-495e-9d54-66c8ec632d91-kube-api-access-9mxx5\") pod \"e2e-worker-w2-a3f9b7-openshift-default-7d6d8b8fb5-sfm8n\" (UID: \"1c3346aa-a989-495e-9d54-66c8ec632d91\") " pod="openshift-ingress/e2e-worker-w2-a3f9b7-openshift-default-7d6d8b8fb5-sfm8n"
Oct 30 19:36:43.895775 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:36:43.895751    4892 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-a3f9b7-openshift-default-7d6d8b8fb5-sfm8n"]
Oct 30 19:36:43.991089 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:36:43.991050    4892 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1c3346aa-a989-495e-9d54-66c8ec632d91-workload-certs\") pod \"e2e-worker-w2-a3f9b7-openshift-default-7d6d8b8fb5-sfm8n\" (UID: \"1c3346aa-a989-495e-9d54-66c8ec632d91\") " pod="openshift-ingress/e2e-worker-w2-a3f9b7-openshift-default-7d6d8b8fb5-sfm8n"
Oct 30 19:36:43.991089 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:36:43.991096    4892 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1c3346aa-a989-495e-9d54-66c8ec632d91-istiod-ca-cert\") pod \"e2e-worker-w2-a3f9b7-openshift-default-7d6d8b8fb5-sfm8n\" (UID: \"1c3346aa-a989-495e-9d54-66c8ec632d91\") " pod="openshift-ingress/e2e-worker-w2-a3f9b7-openshift-default-7d6d8b8fb5-sfm8n"
Oct 30 19:36:43.991346 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:36:43.991116    4892 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9mxx5\" (UniqueName: \"kubernetes.io/projected/1c3346aa-a989-495e-9d54-66c8ec632d91-kube-api-access-9mxx5\") pod \"e2e-worker-w2-a3f9b7-openshift-default-7d6d8b8fb5-sfm8n\" (UID: \"1c3346aa-a989-495e-9d54-66c8ec632d91\") " pod="openshift-ingress/e2e-worker-w2-a3f9b7-openshift-default-7d6d8b8fb5-sfm8n"
Oct 30 19:36:43.991346 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:36:43.991192    4892 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1c3346aa-a989-495e-9d54-66c8ec632d91-istio-podinfo\") pod \"e2e-worker-w2-a3f9b7-openshift-default-7d6d8b8fb5-sfm8n\" (UID: \"1c3346aa-a989-495e-9d54-66c8ec632d91\") " pod="openshift-ingress/e2e-worker-w2-a3f9b7-openshift-default-7d6d8b8fb5-sfm8n"
Oct 30 19:36:43.991346 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:36:43.991217    4892 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1c3346aa-a989-495e-9d54-66c8ec632d91-istio-data\") pod \"e2e-worker-w2-a3f9b7-openshift-default-7d6d8b8fb5-sfm8n\" (UID: \"1c3346aa-a989-495e-9d54-66c8ec632d91\") " pod="openshift-ingress/e2e-worker-w2-a3f9b7-openshift-default-7d6d8b8fb5-sfm8n"
Oct 30 19:36:43.991346 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:36:43.991234    4892 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1c3346aa-a989-495e-9d54-66c8ec632d91-workload-socket\") pod \"e2e-worker-w2-a3f9b7-openshift-default-7d6d8b8fb5-sfm8n\" (UID: \"1c3346aa-a989-495e-9d54-66c8ec632d91\") " pod="openshift-ingress/e2e-worker-w2-a3f9b7-openshift-default-7d6d8b8fb5-sfm8n"
Oct 30 19:36:43.991346 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:36:43.991249    4892 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1c3346aa-a989-495e-9d54-66c8ec632d91-istio-envoy\") pod \"e2e-worker-w2-a3f9b7-openshift-default-7d6d8b8fb5-sfm8n\" (UID: \"1c3346aa-a989-495e-9d54-66c8ec632d91\") " pod="openshift-ingress/e2e-worker-w2-a3f9b7-openshift-default-7d6d8b8fb5-sfm8n"
Oct 30 19:36:43.991346 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:36:43.991264    4892 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1c3346aa-a989-495e-9d54-66c8ec632d91-istio-token\") pod \"e2e-worker-w2-a3f9b7-openshift-default-7d6d8b8fb5-sfm8n\" (UID: \"1c3346aa-a989-495e-9d54-66c8ec632d91\") " pod="openshift-ingress/e2e-worker-w2-a3f9b7-openshift-default-7d6d8b8fb5-sfm8n"
Oct 30 19:36:43.991346 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:36:43.991285    4892 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1c3346aa-a989-495e-9d54-66c8ec632d91-credential-socket\") pod \"e2e-worker-w2-a3f9b7-openshift-default-7d6d8b8fb5-sfm8n\" (UID: \"1c3346aa-a989-495e-9d54-66c8ec632d91\") " pod="openshift-ingress/e2e-worker-w2-a3f9b7-openshift-default-7d6d8b8fb5-sfm8n"
Oct 30 19:36:43.991690 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:36:43.991561    4892 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1c3346aa-a989-495e-9d54-66c8ec632d91-workload-certs\") pod \"e2e-worker-w2-a3f9b7-openshift-default-7d6d8b8fb5-sfm8n\" (UID: \"1c3346aa-a989-495e-9d54-66c8ec632d91\") " pod="openshift-ingress/e2e-worker-w2-a3f9b7-openshift-default-7d6d8b8fb5-sfm8n"
Oct 30 19:36:43.991690 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:36:43.991662    4892 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1c3346aa-a989-495e-9d54-66c8ec632d91-istio-data\") pod \"e2e-worker-w2-a3f9b7-openshift-default-7d6d8b8fb5-sfm8n\" (UID: \"1c3346aa-a989-495e-9d54-66c8ec632d91\") " pod="openshift-ingress/e2e-worker-w2-a3f9b7-openshift-default-7d6d8b8fb5-sfm8n"
Oct 30 19:36:43.991901 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:36:43.991880    4892 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1c3346aa-a989-495e-9d54-66c8ec632d91-workload-socket\") pod \"e2e-worker-w2-a3f9b7-openshift-default-7d6d8b8fb5-sfm8n\" (UID: \"1c3346aa-a989-495e-9d54-66c8ec632d91\") " pod="openshift-ingress/e2e-worker-w2-a3f9b7-openshift-default-7d6d8b8fb5-sfm8n"
Oct 30 19:36:43.991996 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:36:43.991915    4892 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1c3346aa-a989-495e-9d54-66c8ec632d91-credential-socket\") pod \"e2e-worker-w2-a3f9b7-openshift-default-7d6d8b8fb5-sfm8n\" (UID: \"1c3346aa-a989-495e-9d54-66c8ec632d91\") " pod="openshift-ingress/e2e-worker-w2-a3f9b7-openshift-default-7d6d8b8fb5-sfm8n"
Oct 30 19:36:43.992045 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:36:43.992000    4892 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1c3346aa-a989-495e-9d54-66c8ec632d91-istiod-ca-cert\") pod \"e2e-worker-w2-a3f9b7-openshift-default-7d6d8b8fb5-sfm8n\" (UID: \"1c3346aa-a989-495e-9d54-66c8ec632d91\") " pod="openshift-ingress/e2e-worker-w2-a3f9b7-openshift-default-7d6d8b8fb5-sfm8n"
Oct 30 19:36:43.993735 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:36:43.993714    4892 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1c3346aa-a989-495e-9d54-66c8ec632d91-istio-envoy\") pod \"e2e-worker-w2-a3f9b7-openshift-default-7d6d8b8fb5-sfm8n\" (UID: \"1c3346aa-a989-495e-9d54-66c8ec632d91\") " pod="openshift-ingress/e2e-worker-w2-a3f9b7-openshift-default-7d6d8b8fb5-sfm8n"
Oct 30 19:36:43.994085 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:36:43.994064    4892 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1c3346aa-a989-495e-9d54-66c8ec632d91-istio-podinfo\") pod \"e2e-worker-w2-a3f9b7-openshift-default-7d6d8b8fb5-sfm8n\" (UID: \"1c3346aa-a989-495e-9d54-66c8ec632d91\") " pod="openshift-ingress/e2e-worker-w2-a3f9b7-openshift-default-7d6d8b8fb5-sfm8n"
Oct 30 19:36:44.047995 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:36:44.047914    4892 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1c3346aa-a989-495e-9d54-66c8ec632d91-istio-token\") pod \"e2e-worker-w2-a3f9b7-openshift-default-7d6d8b8fb5-sfm8n\" (UID: \"1c3346aa-a989-495e-9d54-66c8ec632d91\") " pod="openshift-ingress/e2e-worker-w2-a3f9b7-openshift-default-7d6d8b8fb5-sfm8n"
Oct 30 19:36:44.048188 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:36:44.048163    4892 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9mxx5\" (UniqueName: \"kubernetes.io/projected/1c3346aa-a989-495e-9d54-66c8ec632d91-kube-api-access-9mxx5\") pod \"e2e-worker-w2-a3f9b7-openshift-default-7d6d8b8fb5-sfm8n\" (UID: \"1c3346aa-a989-495e-9d54-66c8ec632d91\") " pod="openshift-ingress/e2e-worker-w2-a3f9b7-openshift-default-7d6d8b8fb5-sfm8n"
Oct 30 19:36:44.103018 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:36:44.102996    4892 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-a3f9b7-openshift-default-7d6d8b8fb5-sfm8n"
Oct 30 19:36:44.356515 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:36:44.356429    4892 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-insights/periodic-gathering-58sbq-z4rz6"]
Oct 30 19:36:44.360254 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:36:44.360229    4892 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/periodic-gathering-58sbq-z4rz6"
Oct 30 19:36:44.414124 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:36:44.414098    4892 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"operator-dockercfg-4qn9x\""
Oct 30 19:36:44.417500 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:36:44.417473    4892 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-insights/periodic-gathering-58sbq-z4rz6"]
Oct 30 19:36:44.497088 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:36:44.497054    4892 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/1ba4f185-0574-403d-8db6-77f802d0dc1f-service-ca-bundle\") pod \"periodic-gathering-58sbq-z4rz6\" (UID: \"1ba4f185-0574-403d-8db6-77f802d0dc1f\") " pod="openshift-insights/periodic-gathering-58sbq-z4rz6"
Oct 30 19:36:44.497234 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:36:44.497131    4892 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lf796\" (UniqueName: \"kubernetes.io/projected/1ba4f185-0574-403d-8db6-77f802d0dc1f-kube-api-access-lf796\") pod \"periodic-gathering-58sbq-z4rz6\" (UID: \"1ba4f185-0574-403d-8db6-77f802d0dc1f\") " pod="openshift-insights/periodic-gathering-58sbq-z4rz6"
Oct 30 19:36:44.497285 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:36:44.497233    4892 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/1ba4f185-0574-403d-8db6-77f802d0dc1f-trusted-ca-bundle\") pod \"periodic-gathering-58sbq-z4rz6\" (UID: \"1ba4f185-0574-403d-8db6-77f802d0dc1f\") " pod="openshift-insights/periodic-gathering-58sbq-z4rz6"
Oct 30 19:36:44.497285 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:36:44.497275    4892 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"archives-path\" (UniqueName: \"kubernetes.io/empty-dir/1ba4f185-0574-403d-8db6-77f802d0dc1f-archives-path\") pod \"periodic-gathering-58sbq-z4rz6\" (UID: \"1ba4f185-0574-403d-8db6-77f802d0dc1f\") " pod="openshift-insights/periodic-gathering-58sbq-z4rz6"
Oct 30 19:36:44.538739 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:36:44.538624    4892 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-a3f9b7-openshift-default-7d6d8b8fb5-sfm8n"]
Oct 30 19:36:44.541534 ip-10-0-132-82 kubenswrapper[4892]: W1030 19:36:44.541508    4892 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1c3346aa_a989_495e_9d54_66c8ec632d91.slice/crio-2113779abae42b583b9336b113c943eeb62b8e81401d928b1da7c9abf7cdc5ed WatchSource:0}: Error finding container 2113779abae42b583b9336b113c943eeb62b8e81401d928b1da7c9abf7cdc5ed: Status 404 returned error can't find the container with id 2113779abae42b583b9336b113c943eeb62b8e81401d928b1da7c9abf7cdc5ed
Oct 30 19:36:44.544085 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:36:44.544050    4892 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892000Ki","pods":"250"}
Oct 30 19:36:44.544172 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:36:44.544115    4892 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892000Ki","pods":"250"}
Oct 30 19:36:44.544172 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:36:44.544145    4892 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892000Ki","pods":"250"}
Oct 30 19:36:44.597866 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:36:44.597843    4892 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/1ba4f185-0574-403d-8db6-77f802d0dc1f-service-ca-bundle\") pod \"periodic-gathering-58sbq-z4rz6\" (UID: \"1ba4f185-0574-403d-8db6-77f802d0dc1f\") " pod="openshift-insights/periodic-gathering-58sbq-z4rz6"
Oct 30 19:36:44.598001 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:36:44.597910    4892 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lf796\" (UniqueName: \"kubernetes.io/projected/1ba4f185-0574-403d-8db6-77f802d0dc1f-kube-api-access-lf796\") pod \"periodic-gathering-58sbq-z4rz6\" (UID: \"1ba4f185-0574-403d-8db6-77f802d0dc1f\") " pod="openshift-insights/periodic-gathering-58sbq-z4rz6"
Oct 30 19:36:44.598001 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:36:44.597986    4892 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/1ba4f185-0574-403d-8db6-77f802d0dc1f-trusted-ca-bundle\") pod \"periodic-gathering-58sbq-z4rz6\" (UID: \"1ba4f185-0574-403d-8db6-77f802d0dc1f\") " pod="openshift-insights/periodic-gathering-58sbq-z4rz6"
Oct 30 19:36:44.598115 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:36:44.598031    4892 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"archives-path\" (UniqueName: \"kubernetes.io/empty-dir/1ba4f185-0574-403d-8db6-77f802d0dc1f-archives-path\") pod \"periodic-gathering-58sbq-z4rz6\" (UID: \"1ba4f185-0574-403d-8db6-77f802d0dc1f\") " pod="openshift-insights/periodic-gathering-58sbq-z4rz6"
Oct 30 19:36:44.598431 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:36:44.598411    4892 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"archives-path\" (UniqueName: \"kubernetes.io/empty-dir/1ba4f185-0574-403d-8db6-77f802d0dc1f-archives-path\") pod \"periodic-gathering-58sbq-z4rz6\" (UID: \"1ba4f185-0574-403d-8db6-77f802d0dc1f\") " pod="openshift-insights/periodic-gathering-58sbq-z4rz6"
Oct 30 19:36:44.598652 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:36:44.598636    4892 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/1ba4f185-0574-403d-8db6-77f802d0dc1f-service-ca-bundle\") pod \"periodic-gathering-58sbq-z4rz6\" (UID: \"1ba4f185-0574-403d-8db6-77f802d0dc1f\") " pod="openshift-insights/periodic-gathering-58sbq-z4rz6"
Oct 30 19:36:44.598693 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:36:44.598683    4892 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/1ba4f185-0574-403d-8db6-77f802d0dc1f-trusted-ca-bundle\") pod \"periodic-gathering-58sbq-z4rz6\" (UID: \"1ba4f185-0574-403d-8db6-77f802d0dc1f\") " pod="openshift-insights/periodic-gathering-58sbq-z4rz6"
Oct 30 19:36:44.656743 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:36:44.656662    4892 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lf796\" (UniqueName: \"kubernetes.io/projected/1ba4f185-0574-403d-8db6-77f802d0dc1f-kube-api-access-lf796\") pod \"periodic-gathering-58sbq-z4rz6\" (UID: \"1ba4f185-0574-403d-8db6-77f802d0dc1f\") " pod="openshift-insights/periodic-gathering-58sbq-z4rz6"
Oct 30 19:36:44.674589 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:36:44.674553    4892 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/periodic-gathering-58sbq-z4rz6"
Oct 30 19:36:44.974736 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:36:44.974647    4892 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-a3f9b7-openshift-default-7d6d8b8fb5-sfm8n" event={"ID":"1c3346aa-a989-495e-9d54-66c8ec632d91","Type":"ContainerStarted","Data":"0c4417cc4bb8af483b3db4a9fd5ae18b438a366697c09861876a6f417453a841"}
Oct 30 19:36:44.975137 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:36:44.974745    4892 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-a3f9b7-openshift-default-7d6d8b8fb5-sfm8n" event={"ID":"1c3346aa-a989-495e-9d54-66c8ec632d91","Type":"ContainerStarted","Data":"2113779abae42b583b9336b113c943eeb62b8e81401d928b1da7c9abf7cdc5ed"}
Oct 30 19:36:45.066016 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:36:45.065956    4892 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-insights/periodic-gathering-58sbq-z4rz6"]
Oct 30 19:36:45.080470 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:36:45.080444    4892 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-26f039ec-7kmdh"]
Oct 30 19:36:45.085404 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:36:45.085386    4892 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-26f039ec-7kmdh"
Oct 30 19:36:45.103382 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:36:45.103358    4892 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w2-a3f9b7-openshift-default-7d6d8b8fb5-sfm8n"
Oct 30 19:36:45.104471 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:36:45.104448    4892 patch_prober.go:29] interesting pod/e2e-worker-w2-a3f9b7-openshift-default-7d6d8b8fb5-sfm8n container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.71:15021/healthz/ready\": dial tcp 10.132.0.71:15021: connect: connection refused" start-of-body=
Oct 30 19:36:45.104606 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:36:45.104492    4892 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-a3f9b7-openshift-default-7d6d8b8fb5-sfm8n" podUID="1c3346aa-a989-495e-9d54-66c8ec632d91" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.71:15021/healthz/ready\": dial tcp 10.132.0.71:15021: connect: connection refused"
Oct 30 19:36:45.113272 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:36:45.113209    4892 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w2-a3f9b7-openshift-default-7d6d8b8fb5-sfm8n" podStartSLOduration=2.113196191 podStartE2EDuration="2.113196191s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:36:43 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:36:45.09054063 +0000 UTC m=+1252.661006270" watchObservedRunningTime="2026-10-30 19:36:45.113196191 +0000 UTC m=+1252.683661790"
Oct 30 19:36:45.142993 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:36:45.139241    4892 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 30 19:36:45.142993 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:36:45.139322    4892 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 19:36:45.142993 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:36:45.139423    4892 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 30 19:36:45.142993 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:36:45.139519    4892 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-nlxp4\""
Oct 30 19:36:45.170238 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:36:45.170210    4892 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-26f039ec-7kmdh"]
Oct 30 19:36:45.207006 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:36:45.205939    4892 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/79ebfc06-f72b-48af-849d-d81baf3f9765-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-26f039ec-7kmdh\" (UID: \"79ebfc06-f72b-48af-849d-d81baf3f9765\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-26f039ec-7kmdh"
Oct 30 19:36:45.207006 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:36:45.206006    4892 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xrpgh\" (UniqueName: \"kubernetes.io/projected/79ebfc06-f72b-48af-849d-d81baf3f9765-kube-api-access-xrpgh\") pod \"maas-api-revoke-sub-26f039ec-7kmdh\" (UID: \"79ebfc06-f72b-48af-849d-d81baf3f9765\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-26f039ec-7kmdh"
Oct 30 19:36:45.207006 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:36:45.206027    4892 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/79ebfc06-f72b-48af-849d-d81baf3f9765-maas-api-xks-ca\") pod \"maas-api-revoke-sub-26f039ec-7kmdh\" (UID: \"79ebfc06-f72b-48af-849d-d81baf3f9765\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-26f039ec-7kmdh"
Oct 30 19:36:45.282547 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:36:45.282460    4892 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-fsfvk"]
Oct 30 19:36:45.306544 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:36:45.306517    4892 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/79ebfc06-f72b-48af-849d-d81baf3f9765-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-26f039ec-7kmdh\" (UID: \"79ebfc06-f72b-48af-849d-d81baf3f9765\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-26f039ec-7kmdh"
Oct 30 19:36:45.306698 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:36:45.306560    4892 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xrpgh\" (UniqueName: \"kubernetes.io/projected/79ebfc06-f72b-48af-849d-d81baf3f9765-kube-api-access-xrpgh\") pod \"maas-api-revoke-sub-26f039ec-7kmdh\" (UID: \"79ebfc06-f72b-48af-849d-d81baf3f9765\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-26f039ec-7kmdh"
Oct 30 19:36:45.306698 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:36:45.306579    4892 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/79ebfc06-f72b-48af-849d-d81baf3f9765-maas-api-xks-ca\") pod \"maas-api-revoke-sub-26f039ec-7kmdh\" (UID: \"79ebfc06-f72b-48af-849d-d81baf3f9765\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-26f039ec-7kmdh"
Oct 30 19:36:45.307194 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:36:45.307173    4892 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/79ebfc06-f72b-48af-849d-d81baf3f9765-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-26f039ec-7kmdh\" (UID: \"79ebfc06-f72b-48af-849d-d81baf3f9765\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-26f039ec-7kmdh"
Oct 30 19:36:45.309166 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:36:45.309141    4892 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/79ebfc06-f72b-48af-849d-d81baf3f9765-maas-api-xks-ca\") pod \"maas-api-revoke-sub-26f039ec-7kmdh\" (UID: \"79ebfc06-f72b-48af-849d-d81baf3f9765\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-26f039ec-7kmdh"
Oct 30 19:36:45.362694 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:36:45.362660    4892 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xrpgh\" (UniqueName: \"kubernetes.io/projected/79ebfc06-f72b-48af-849d-d81baf3f9765-kube-api-access-xrpgh\") pod \"maas-api-revoke-sub-26f039ec-7kmdh\" (UID: \"79ebfc06-f72b-48af-849d-d81baf3f9765\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-26f039ec-7kmdh"
Oct 30 19:36:45.401596 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:36:45.401569    4892 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-26f039ec-7kmdh"
Oct 30 19:36:45.803899 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:36:45.803875    4892 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-26f039ec-7kmdh"]
Oct 30 19:36:45.804385 ip-10-0-132-82 kubenswrapper[4892]: W1030 19:36:45.804346    4892 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod79ebfc06_f72b_48af_849d_d81baf3f9765.slice/crio-cf49b3f9e722402b8bbca53d5577e32826e3de05294fe29167aad4732e14cc32 WatchSource:0}: Error finding container cf49b3f9e722402b8bbca53d5577e32826e3de05294fe29167aad4732e14cc32: Status 404 returned error can't find the container with id cf49b3f9e722402b8bbca53d5577e32826e3de05294fe29167aad4732e14cc32
Oct 30 19:36:45.982161 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:36:45.982122    4892 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-26f039ec-7kmdh" event={"ID":"79ebfc06-f72b-48af-849d-d81baf3f9765","Type":"ContainerStarted","Data":"cf49b3f9e722402b8bbca53d5577e32826e3de05294fe29167aad4732e14cc32"}
Oct 30 19:36:45.983937 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:36:45.983913    4892 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/periodic-gathering-58sbq-z4rz6" event={"ID":"1ba4f185-0574-403d-8db6-77f802d0dc1f","Type":"ContainerStarted","Data":"e589a555c9e7ceee632261f13909a33ab7f9559346388dd0b5d9dcab3ae2f6b8"}
Oct 30 19:36:45.984102 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:36:45.983955    4892 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/periodic-gathering-58sbq-z4rz6" event={"ID":"1ba4f185-0574-403d-8db6-77f802d0dc1f","Type":"ContainerStarted","Data":"834c91c83f3d78d68c1f049c34bc383fc7f7e38b911e2c4369ad860755794a81"}
Oct 30 19:36:46.098162 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:36:46.098073    4892 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-insights/periodic-gathering-58sbq-z4rz6" podStartSLOduration=2.098058768 podStartE2EDuration="2.098058768s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:36:44 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:36:46.097796686 +0000 UTC m=+1253.668262285" watchObservedRunningTime="2026-10-30 19:36:46.098058768 +0000 UTC m=+1253.668524366"
Oct 30 19:36:46.104181 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:36:46.104154    4892 patch_prober.go:29] interesting pod/e2e-worker-w2-a3f9b7-openshift-default-7d6d8b8fb5-sfm8n container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.71:15021/healthz/ready\": dial tcp 10.132.0.71:15021: connect: connection refused" start-of-body=
Oct 30 19:36:46.104311 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:36:46.104199    4892 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-a3f9b7-openshift-default-7d6d8b8fb5-sfm8n" podUID="1c3346aa-a989-495e-9d54-66c8ec632d91" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.71:15021/healthz/ready\": dial tcp 10.132.0.71:15021: connect: connection refused"
Oct 30 19:36:46.995527 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:36:46.995488    4892 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-26f039ec-7kmdh" event={"ID":"79ebfc06-f72b-48af-849d-d81baf3f9765","Type":"ContainerStarted","Data":"3dc82df4c54ac64bceccf03c8b67e3113c865dcfdd564eaf679f747e6d89fde7"}
Oct 30 19:36:47.104151 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:36:47.104114    4892 patch_prober.go:29] interesting pod/e2e-worker-w2-a3f9b7-openshift-default-7d6d8b8fb5-sfm8n container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.71:15021/healthz/ready\": dial tcp 10.132.0.71:15021: connect: connection refused" start-of-body=
Oct 30 19:36:47.104293 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:36:47.104168    4892 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-a3f9b7-openshift-default-7d6d8b8fb5-sfm8n" podUID="1c3346aa-a989-495e-9d54-66c8ec632d91" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.71:15021/healthz/ready\": dial tcp 10.132.0.71:15021: connect: connection refused"
Oct 30 19:36:47.122435 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:36:47.122382    4892 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-26f039ec-7kmdh" podStartSLOduration=2.122369395 podStartE2EDuration="2.122369395s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:36:45 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:36:47.120030677 +0000 UTC m=+1254.690496274" watchObservedRunningTime="2026-10-30 19:36:47.122369395 +0000 UTC m=+1254.692834993"
Oct 30 19:36:48.002494 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:36:48.002452    4892 generic.go:356] "Generic (PLEG): container finished" podID="79ebfc06-f72b-48af-849d-d81baf3f9765" containerID="3dc82df4c54ac64bceccf03c8b67e3113c865dcfdd564eaf679f747e6d89fde7" exitCode=0
Oct 30 19:36:48.002882 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:36:48.002574    4892 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-26f039ec-7kmdh" event={"ID":"79ebfc06-f72b-48af-849d-d81baf3f9765","Type":"ContainerDied","Data":"3dc82df4c54ac64bceccf03c8b67e3113c865dcfdd564eaf679f747e6d89fde7"}
Oct 30 19:36:48.104226 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:36:48.104195    4892 patch_prober.go:29] interesting pod/e2e-worker-w2-a3f9b7-openshift-default-7d6d8b8fb5-sfm8n container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.71:15021/healthz/ready\": dial tcp 10.132.0.71:15021: connect: connection refused" start-of-body=
Oct 30 19:36:48.104387 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:36:48.104245    4892 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-a3f9b7-openshift-default-7d6d8b8fb5-sfm8n" podUID="1c3346aa-a989-495e-9d54-66c8ec632d91" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.71:15021/healthz/ready\": dial tcp 10.132.0.71:15021: connect: connection refused"
Oct 30 19:36:49.103467 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:36:49.103435    4892 patch_prober.go:29] interesting pod/e2e-worker-w2-a3f9b7-openshift-default-7d6d8b8fb5-sfm8n container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.71:15021/healthz/ready\": dial tcp 10.132.0.71:15021: connect: connection refused" start-of-body=
Oct 30 19:36:49.103889 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:36:49.103855    4892 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-a3f9b7-openshift-default-7d6d8b8fb5-sfm8n" podUID="1c3346aa-a989-495e-9d54-66c8ec632d91" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.71:15021/healthz/ready\": dial tcp 10.132.0.71:15021: connect: connection refused"
Oct 30 19:36:49.222637 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:36:49.222605    4892 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-26f039ec-7kmdh"
Oct 30 19:36:49.375809 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:36:49.375710    4892 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/79ebfc06-f72b-48af-849d-d81baf3f9765-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/79ebfc06-f72b-48af-849d-d81baf3f9765-maas-api-openshift-ca\") pod \"79ebfc06-f72b-48af-849d-d81baf3f9765\" (UID: \"79ebfc06-f72b-48af-849d-d81baf3f9765\") "
Oct 30 19:36:49.375809 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:36:49.375766    4892 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/79ebfc06-f72b-48af-849d-d81baf3f9765-kube-api-access-xrpgh\" (UniqueName: \"kubernetes.io/projected/79ebfc06-f72b-48af-849d-d81baf3f9765-kube-api-access-xrpgh\") pod \"79ebfc06-f72b-48af-849d-d81baf3f9765\" (UID: \"79ebfc06-f72b-48af-849d-d81baf3f9765\") "
Oct 30 19:36:49.375809 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:36:49.375806    4892 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/79ebfc06-f72b-48af-849d-d81baf3f9765-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/79ebfc06-f72b-48af-849d-d81baf3f9765-maas-api-xks-ca\") pod \"79ebfc06-f72b-48af-849d-d81baf3f9765\" (UID: \"79ebfc06-f72b-48af-849d-d81baf3f9765\") "
Oct 30 19:36:49.376155 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:36:49.376126    4892 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/79ebfc06-f72b-48af-849d-d81baf3f9765-maas-api-openshift-ca" pod "79ebfc06-f72b-48af-849d-d81baf3f9765" (UID: "79ebfc06-f72b-48af-849d-d81baf3f9765"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 19:36:49.378439 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:36:49.378401    4892 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/79ebfc06-f72b-48af-849d-d81baf3f9765-maas-api-xks-ca" pod "79ebfc06-f72b-48af-849d-d81baf3f9765" (UID: "79ebfc06-f72b-48af-849d-d81baf3f9765"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 19:36:49.378616 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:36:49.378427    4892 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/79ebfc06-f72b-48af-849d-d81baf3f9765-kube-api-access-xrpgh" pod "79ebfc06-f72b-48af-849d-d81baf3f9765" (UID: "79ebfc06-f72b-48af-849d-d81baf3f9765"). InnerVolumeSpecName "kube-api-access-xrpgh". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:36:49.477506 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:36:49.477464    4892 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/79ebfc06-f72b-48af-849d-d81baf3f9765-maas-api-openshift-ca\") on node \"ip-10-0-132-82.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 19:36:49.477506 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:36:49.477499    4892 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xrpgh\" (UniqueName: \"kubernetes.io/projected/79ebfc06-f72b-48af-849d-d81baf3f9765-kube-api-access-xrpgh\") on node \"ip-10-0-132-82.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 19:36:49.477506 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:36:49.477514    4892 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/79ebfc06-f72b-48af-849d-d81baf3f9765-maas-api-xks-ca\") on node \"ip-10-0-132-82.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 19:36:50.015117 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:36:50.015086    4892 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-26f039ec-7kmdh"
Oct 30 19:36:50.015271 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:36:50.015118    4892 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-26f039ec-7kmdh" event={"ID":"79ebfc06-f72b-48af-849d-d81baf3f9765","Type":"ContainerDied","Data":"cf49b3f9e722402b8bbca53d5577e32826e3de05294fe29167aad4732e14cc32"}
Oct 30 19:36:50.015271 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:36:50.015160    4892 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="cf49b3f9e722402b8bbca53d5577e32826e3de05294fe29167aad4732e14cc32"
Oct 30 19:36:50.103748 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:36:50.103713    4892 patch_prober.go:29] interesting pod/e2e-worker-w2-a3f9b7-openshift-default-7d6d8b8fb5-sfm8n container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.71:15021/healthz/ready\": dial tcp 10.132.0.71:15021: connect: connection refused" start-of-body=
Oct 30 19:36:50.104144 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:36:50.103778    4892 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-a3f9b7-openshift-default-7d6d8b8fb5-sfm8n" podUID="1c3346aa-a989-495e-9d54-66c8ec632d91" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.71:15021/healthz/ready\": dial tcp 10.132.0.71:15021: connect: connection refused"
Oct 30 19:36:50.383010 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:36:50.382984    4892 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-1ad201c0-gzpm5"]
Oct 30 19:36:50.383485 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:36:50.383471    4892 cpu_manager.go:414] "containerMap: removing container" podUID="79ebfc06-f72b-48af-849d-d81baf3f9765" containerName="revoke-keys"
Oct 30 19:36:50.383539 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:36:50.383487    4892 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="79ebfc06-f72b-48af-849d-d81baf3f9765" containerName="revoke-keys"
Oct 30 19:36:50.383583 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:36:50.383574    4892 memory_manager.go:365] "RemoveStaleState removing state" podUID="79ebfc06-f72b-48af-849d-d81baf3f9765" containerName="revoke-keys"
Oct 30 19:36:50.387306 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:36:50.387286    4892 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1ad201c0-gzpm5"
Oct 30 19:36:50.445558 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:36:50.440019    4892 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-nlxp4\""
Oct 30 19:36:50.445558 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:36:50.440107    4892 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 30 19:36:50.445558 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:36:50.440164    4892 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 30 19:36:50.445558 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:36:50.440249    4892 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 19:36:50.480089 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:36:50.480047    4892 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-1ad201c0-gzpm5"]
Oct 30 19:36:50.489460 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:36:50.489426    4892 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f9aff12a-9664-4d53-ac43-4b0667eccbc8-maas-api-xks-ca\") pod \"maas-api-revoke-sub-1ad201c0-gzpm5\" (UID: \"f9aff12a-9664-4d53-ac43-4b0667eccbc8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1ad201c0-gzpm5"
Oct 30 19:36:50.489630 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:36:50.489610    4892 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f9aff12a-9664-4d53-ac43-4b0667eccbc8-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-1ad201c0-gzpm5\" (UID: \"f9aff12a-9664-4d53-ac43-4b0667eccbc8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1ad201c0-gzpm5"
Oct 30 19:36:50.489739 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:36:50.489720    4892 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-v4xjt\" (UniqueName: \"kubernetes.io/projected/f9aff12a-9664-4d53-ac43-4b0667eccbc8-kube-api-access-v4xjt\") pod \"maas-api-revoke-sub-1ad201c0-gzpm5\" (UID: \"f9aff12a-9664-4d53-ac43-4b0667eccbc8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1ad201c0-gzpm5"
Oct 30 19:36:50.590476 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:36:50.590441    4892 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-v4xjt\" (UniqueName: \"kubernetes.io/projected/f9aff12a-9664-4d53-ac43-4b0667eccbc8-kube-api-access-v4xjt\") pod \"maas-api-revoke-sub-1ad201c0-gzpm5\" (UID: \"f9aff12a-9664-4d53-ac43-4b0667eccbc8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1ad201c0-gzpm5"
Oct 30 19:36:50.590638 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:36:50.590574    4892 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f9aff12a-9664-4d53-ac43-4b0667eccbc8-maas-api-xks-ca\") pod \"maas-api-revoke-sub-1ad201c0-gzpm5\" (UID: \"f9aff12a-9664-4d53-ac43-4b0667eccbc8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1ad201c0-gzpm5"
Oct 30 19:36:50.590691 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:36:50.590666    4892 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f9aff12a-9664-4d53-ac43-4b0667eccbc8-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-1ad201c0-gzpm5\" (UID: \"f9aff12a-9664-4d53-ac43-4b0667eccbc8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1ad201c0-gzpm5"
Oct 30 19:36:50.591294 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:36:50.591268    4892 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f9aff12a-9664-4d53-ac43-4b0667eccbc8-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-1ad201c0-gzpm5\" (UID: \"f9aff12a-9664-4d53-ac43-4b0667eccbc8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1ad201c0-gzpm5"
Oct 30 19:36:50.592992 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:36:50.592944    4892 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f9aff12a-9664-4d53-ac43-4b0667eccbc8-maas-api-xks-ca\") pod \"maas-api-revoke-sub-1ad201c0-gzpm5\" (UID: \"f9aff12a-9664-4d53-ac43-4b0667eccbc8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1ad201c0-gzpm5"
Oct 30 19:36:50.647989 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:36:50.647901    4892 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-v4xjt\" (UniqueName: \"kubernetes.io/projected/f9aff12a-9664-4d53-ac43-4b0667eccbc8-kube-api-access-v4xjt\") pod \"maas-api-revoke-sub-1ad201c0-gzpm5\" (UID: \"f9aff12a-9664-4d53-ac43-4b0667eccbc8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1ad201c0-gzpm5"
Oct 30 19:36:50.700355 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:36:50.700330    4892 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1ad201c0-gzpm5"
Oct 30 19:36:51.081336 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:36:51.081296    4892 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-1ad201c0-gzpm5"]
Oct 30 19:36:51.082778 ip-10-0-132-82 kubenswrapper[4892]: W1030 19:36:51.082749    4892 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf9aff12a_9664_4d53_ac43_4b0667eccbc8.slice/crio-5e931498b3d64ec4719452d629cbc707d94cc0f9edb2eba8afe08712d1e81d8a WatchSource:0}: Error finding container 5e931498b3d64ec4719452d629cbc707d94cc0f9edb2eba8afe08712d1e81d8a: Status 404 returned error can't find the container with id 5e931498b3d64ec4719452d629cbc707d94cc0f9edb2eba8afe08712d1e81d8a
Oct 30 19:36:52.033751 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:36:52.033707    4892 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1ad201c0-gzpm5" event={"ID":"f9aff12a-9664-4d53-ac43-4b0667eccbc8","Type":"ContainerStarted","Data":"fbb40fdacbc99b04faeb2b4df8b21e3a8e820389a34e98a87d07f6bc13f75398"}
Oct 30 19:36:52.034487 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:36:52.034450    4892 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1ad201c0-gzpm5" event={"ID":"f9aff12a-9664-4d53-ac43-4b0667eccbc8","Type":"ContainerStarted","Data":"5e931498b3d64ec4719452d629cbc707d94cc0f9edb2eba8afe08712d1e81d8a"}
Oct 30 19:36:52.099260 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:36:52.099231    4892 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w2-a3f9b7-openshift-default-7d6d8b8fb5-sfm8n"
Oct 30 19:36:52.099581 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:36:52.099561    4892 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w2-a3f9b7-openshift-default-7d6d8b8fb5-sfm8n"
Oct 30 19:36:52.100316 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:36:52.100295    4892 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w2-a3f9b7-openshift-default-7d6d8b8fb5-sfm8n"
Oct 30 19:36:52.147689 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:36:52.147645    4892 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1ad201c0-gzpm5" podStartSLOduration=2.147630643 podStartE2EDuration="2.147630643s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:36:50 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:36:52.146502143 +0000 UTC m=+1259.716967742" watchObservedRunningTime="2026-10-30 19:36:52.147630643 +0000 UTC m=+1259.718096242"
Oct 30 19:36:53.041880 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:36:53.041837    4892 generic.go:356] "Generic (PLEG): container finished" podID="f9aff12a-9664-4d53-ac43-4b0667eccbc8" containerID="fbb40fdacbc99b04faeb2b4df8b21e3a8e820389a34e98a87d07f6bc13f75398" exitCode=0
Oct 30 19:36:53.042364 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:36:53.041891    4892 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1ad201c0-gzpm5" event={"ID":"f9aff12a-9664-4d53-ac43-4b0667eccbc8","Type":"ContainerDied","Data":"fbb40fdacbc99b04faeb2b4df8b21e3a8e820389a34e98a87d07f6bc13f75398"}
Oct 30 19:36:54.207189 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:36:54.207158    4892 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1ad201c0-gzpm5"
Oct 30 19:36:54.228895 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:36:54.228867    4892 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f9aff12a-9664-4d53-ac43-4b0667eccbc8-kube-api-access-v4xjt\" (UniqueName: \"kubernetes.io/projected/f9aff12a-9664-4d53-ac43-4b0667eccbc8-kube-api-access-v4xjt\") pod \"f9aff12a-9664-4d53-ac43-4b0667eccbc8\" (UID: \"f9aff12a-9664-4d53-ac43-4b0667eccbc8\") "
Oct 30 19:36:54.229036 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:36:54.229011    4892 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/f9aff12a-9664-4d53-ac43-4b0667eccbc8-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f9aff12a-9664-4d53-ac43-4b0667eccbc8-maas-api-xks-ca\") pod \"f9aff12a-9664-4d53-ac43-4b0667eccbc8\" (UID: \"f9aff12a-9664-4d53-ac43-4b0667eccbc8\") "
Oct 30 19:36:54.229084 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:36:54.229035    4892 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f9aff12a-9664-4d53-ac43-4b0667eccbc8-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f9aff12a-9664-4d53-ac43-4b0667eccbc8-maas-api-openshift-ca\") pod \"f9aff12a-9664-4d53-ac43-4b0667eccbc8\" (UID: \"f9aff12a-9664-4d53-ac43-4b0667eccbc8\") "
Oct 30 19:36:54.229417 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:36:54.229394    4892 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f9aff12a-9664-4d53-ac43-4b0667eccbc8-maas-api-openshift-ca" pod "f9aff12a-9664-4d53-ac43-4b0667eccbc8" (UID: "f9aff12a-9664-4d53-ac43-4b0667eccbc8"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 19:36:54.231055 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:36:54.231032    4892 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f9aff12a-9664-4d53-ac43-4b0667eccbc8-maas-api-xks-ca" pod "f9aff12a-9664-4d53-ac43-4b0667eccbc8" (UID: "f9aff12a-9664-4d53-ac43-4b0667eccbc8"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 19:36:54.231205 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:36:54.231180    4892 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f9aff12a-9664-4d53-ac43-4b0667eccbc8-kube-api-access-v4xjt" pod "f9aff12a-9664-4d53-ac43-4b0667eccbc8" (UID: "f9aff12a-9664-4d53-ac43-4b0667eccbc8"). InnerVolumeSpecName "kube-api-access-v4xjt". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:36:54.329837 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:36:54.329775    4892 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-v4xjt\" (UniqueName: \"kubernetes.io/projected/f9aff12a-9664-4d53-ac43-4b0667eccbc8-kube-api-access-v4xjt\") on node \"ip-10-0-132-82.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 19:36:54.329837 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:36:54.329811    4892 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f9aff12a-9664-4d53-ac43-4b0667eccbc8-maas-api-xks-ca\") on node \"ip-10-0-132-82.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 19:36:54.329837 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:36:54.329826    4892 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f9aff12a-9664-4d53-ac43-4b0667eccbc8-maas-api-openshift-ca\") on node \"ip-10-0-132-82.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 19:36:54.843167 ip-10-0-132-82 kubenswrapper[4892]: E1030 19:36:54.843130    4892 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-jrm87" containerName="extractor"
Oct 30 19:36:55.055449 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:36:55.055426    4892 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1ad201c0-gzpm5"
Oct 30 19:36:55.055611 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:36:55.055460    4892 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1ad201c0-gzpm5" event={"ID":"f9aff12a-9664-4d53-ac43-4b0667eccbc8","Type":"ContainerDied","Data":"5e931498b3d64ec4719452d629cbc707d94cc0f9edb2eba8afe08712d1e81d8a"}
Oct 30 19:36:55.055611 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:36:55.055529    4892 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="5e931498b3d64ec4719452d629cbc707d94cc0f9edb2eba8afe08712d1e81d8a"
Oct 30 19:37:09.129422 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:37:09.129379    4892 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w4-b5178f-openshift-default-c8fbdff8c-bjzz7"]
Oct 30 19:37:09.129901 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:37:09.129880    4892 cpu_manager.go:414] "containerMap: removing container" podUID="f9aff12a-9664-4d53-ac43-4b0667eccbc8" containerName="revoke-keys"
Oct 30 19:37:09.129901 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:37:09.129896    4892 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f9aff12a-9664-4d53-ac43-4b0667eccbc8" containerName="revoke-keys"
Oct 30 19:37:09.130041 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:37:09.130022    4892 memory_manager.go:365] "RemoveStaleState removing state" podUID="f9aff12a-9664-4d53-ac43-4b0667eccbc8" containerName="revoke-keys"
Oct 30 19:37:09.133419 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:37:09.133400    4892 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-b5178f-openshift-default-c8fbdff8c-bjzz7"
Oct 30 19:37:09.185566 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:37:09.185543    4892 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w4-b5178f-openshift-default-dockercfg-k2spr\""
Oct 30 19:37:09.195000 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:37:09.194956    4892 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8l2c6\" (UniqueName: \"kubernetes.io/projected/98aae325-07a2-49fc-a4ba-3e6709dc7c31-kube-api-access-8l2c6\") pod \"e2e-worker-w4-b5178f-openshift-default-c8fbdff8c-bjzz7\" (UID: \"98aae325-07a2-49fc-a4ba-3e6709dc7c31\") " pod="openshift-ingress/e2e-worker-w4-b5178f-openshift-default-c8fbdff8c-bjzz7"
Oct 30 19:37:09.195134 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:37:09.195010    4892 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/98aae325-07a2-49fc-a4ba-3e6709dc7c31-istio-data\") pod \"e2e-worker-w4-b5178f-openshift-default-c8fbdff8c-bjzz7\" (UID: \"98aae325-07a2-49fc-a4ba-3e6709dc7c31\") " pod="openshift-ingress/e2e-worker-w4-b5178f-openshift-default-c8fbdff8c-bjzz7"
Oct 30 19:37:09.195134 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:37:09.195092    4892 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/98aae325-07a2-49fc-a4ba-3e6709dc7c31-istio-token\") pod \"e2e-worker-w4-b5178f-openshift-default-c8fbdff8c-bjzz7\" (UID: \"98aae325-07a2-49fc-a4ba-3e6709dc7c31\") " pod="openshift-ingress/e2e-worker-w4-b5178f-openshift-default-c8fbdff8c-bjzz7"
Oct 30 19:37:09.195249 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:37:09.195151    4892 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/98aae325-07a2-49fc-a4ba-3e6709dc7c31-istio-envoy\") pod \"e2e-worker-w4-b5178f-openshift-default-c8fbdff8c-bjzz7\" (UID: \"98aae325-07a2-49fc-a4ba-3e6709dc7c31\") " pod="openshift-ingress/e2e-worker-w4-b5178f-openshift-default-c8fbdff8c-bjzz7"
Oct 30 19:37:09.195314 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:37:09.195261    4892 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/98aae325-07a2-49fc-a4ba-3e6709dc7c31-workload-certs\") pod \"e2e-worker-w4-b5178f-openshift-default-c8fbdff8c-bjzz7\" (UID: \"98aae325-07a2-49fc-a4ba-3e6709dc7c31\") " pod="openshift-ingress/e2e-worker-w4-b5178f-openshift-default-c8fbdff8c-bjzz7"
Oct 30 19:37:09.195314 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:37:09.195301    4892 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/98aae325-07a2-49fc-a4ba-3e6709dc7c31-workload-socket\") pod \"e2e-worker-w4-b5178f-openshift-default-c8fbdff8c-bjzz7\" (UID: \"98aae325-07a2-49fc-a4ba-3e6709dc7c31\") " pod="openshift-ingress/e2e-worker-w4-b5178f-openshift-default-c8fbdff8c-bjzz7"
Oct 30 19:37:09.195417 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:37:09.195324    4892 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/98aae325-07a2-49fc-a4ba-3e6709dc7c31-istiod-ca-cert\") pod \"e2e-worker-w4-b5178f-openshift-default-c8fbdff8c-bjzz7\" (UID: \"98aae325-07a2-49fc-a4ba-3e6709dc7c31\") " pod="openshift-ingress/e2e-worker-w4-b5178f-openshift-default-c8fbdff8c-bjzz7"
Oct 30 19:37:09.195417 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:37:09.195386    4892 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/98aae325-07a2-49fc-a4ba-3e6709dc7c31-credential-socket\") pod \"e2e-worker-w4-b5178f-openshift-default-c8fbdff8c-bjzz7\" (UID: \"98aae325-07a2-49fc-a4ba-3e6709dc7c31\") " pod="openshift-ingress/e2e-worker-w4-b5178f-openshift-default-c8fbdff8c-bjzz7"
Oct 30 19:37:09.195523 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:37:09.195415    4892 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/98aae325-07a2-49fc-a4ba-3e6709dc7c31-istio-podinfo\") pod \"e2e-worker-w4-b5178f-openshift-default-c8fbdff8c-bjzz7\" (UID: \"98aae325-07a2-49fc-a4ba-3e6709dc7c31\") " pod="openshift-ingress/e2e-worker-w4-b5178f-openshift-default-c8fbdff8c-bjzz7"
Oct 30 19:37:09.240338 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:37:09.240312    4892 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-b5178f-openshift-default-c8fbdff8c-bjzz7"]
Oct 30 19:37:09.296525 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:37:09.296496    4892 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/98aae325-07a2-49fc-a4ba-3e6709dc7c31-workload-certs\") pod \"e2e-worker-w4-b5178f-openshift-default-c8fbdff8c-bjzz7\" (UID: \"98aae325-07a2-49fc-a4ba-3e6709dc7c31\") " pod="openshift-ingress/e2e-worker-w4-b5178f-openshift-default-c8fbdff8c-bjzz7"
Oct 30 19:37:09.296682 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:37:09.296540    4892 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/98aae325-07a2-49fc-a4ba-3e6709dc7c31-workload-socket\") pod \"e2e-worker-w4-b5178f-openshift-default-c8fbdff8c-bjzz7\" (UID: \"98aae325-07a2-49fc-a4ba-3e6709dc7c31\") " pod="openshift-ingress/e2e-worker-w4-b5178f-openshift-default-c8fbdff8c-bjzz7"
Oct 30 19:37:09.296682 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:37:09.296556    4892 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/98aae325-07a2-49fc-a4ba-3e6709dc7c31-istiod-ca-cert\") pod \"e2e-worker-w4-b5178f-openshift-default-c8fbdff8c-bjzz7\" (UID: \"98aae325-07a2-49fc-a4ba-3e6709dc7c31\") " pod="openshift-ingress/e2e-worker-w4-b5178f-openshift-default-c8fbdff8c-bjzz7"
Oct 30 19:37:09.296682 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:37:09.296584    4892 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/98aae325-07a2-49fc-a4ba-3e6709dc7c31-credential-socket\") pod \"e2e-worker-w4-b5178f-openshift-default-c8fbdff8c-bjzz7\" (UID: \"98aae325-07a2-49fc-a4ba-3e6709dc7c31\") " pod="openshift-ingress/e2e-worker-w4-b5178f-openshift-default-c8fbdff8c-bjzz7"
Oct 30 19:37:09.296682 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:37:09.296603    4892 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/98aae325-07a2-49fc-a4ba-3e6709dc7c31-istio-podinfo\") pod \"e2e-worker-w4-b5178f-openshift-default-c8fbdff8c-bjzz7\" (UID: \"98aae325-07a2-49fc-a4ba-3e6709dc7c31\") " pod="openshift-ingress/e2e-worker-w4-b5178f-openshift-default-c8fbdff8c-bjzz7"
Oct 30 19:37:09.296682 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:37:09.296627    4892 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8l2c6\" (UniqueName: \"kubernetes.io/projected/98aae325-07a2-49fc-a4ba-3e6709dc7c31-kube-api-access-8l2c6\") pod \"e2e-worker-w4-b5178f-openshift-default-c8fbdff8c-bjzz7\" (UID: \"98aae325-07a2-49fc-a4ba-3e6709dc7c31\") " pod="openshift-ingress/e2e-worker-w4-b5178f-openshift-default-c8fbdff8c-bjzz7"
Oct 30 19:37:09.296682 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:37:09.296646    4892 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/98aae325-07a2-49fc-a4ba-3e6709dc7c31-istio-data\") pod \"e2e-worker-w4-b5178f-openshift-default-c8fbdff8c-bjzz7\" (UID: \"98aae325-07a2-49fc-a4ba-3e6709dc7c31\") " pod="openshift-ingress/e2e-worker-w4-b5178f-openshift-default-c8fbdff8c-bjzz7"
Oct 30 19:37:09.296682 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:37:09.296664    4892 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/98aae325-07a2-49fc-a4ba-3e6709dc7c31-istio-token\") pod \"e2e-worker-w4-b5178f-openshift-default-c8fbdff8c-bjzz7\" (UID: \"98aae325-07a2-49fc-a4ba-3e6709dc7c31\") " pod="openshift-ingress/e2e-worker-w4-b5178f-openshift-default-c8fbdff8c-bjzz7"
Oct 30 19:37:09.297142 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:37:09.296698    4892 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/98aae325-07a2-49fc-a4ba-3e6709dc7c31-istio-envoy\") pod \"e2e-worker-w4-b5178f-openshift-default-c8fbdff8c-bjzz7\" (UID: \"98aae325-07a2-49fc-a4ba-3e6709dc7c31\") " pod="openshift-ingress/e2e-worker-w4-b5178f-openshift-default-c8fbdff8c-bjzz7"
Oct 30 19:37:09.297142 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:37:09.296951    4892 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/98aae325-07a2-49fc-a4ba-3e6709dc7c31-workload-certs\") pod \"e2e-worker-w4-b5178f-openshift-default-c8fbdff8c-bjzz7\" (UID: \"98aae325-07a2-49fc-a4ba-3e6709dc7c31\") " pod="openshift-ingress/e2e-worker-w4-b5178f-openshift-default-c8fbdff8c-bjzz7"
Oct 30 19:37:09.297142 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:37:09.297025    4892 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/98aae325-07a2-49fc-a4ba-3e6709dc7c31-workload-socket\") pod \"e2e-worker-w4-b5178f-openshift-default-c8fbdff8c-bjzz7\" (UID: \"98aae325-07a2-49fc-a4ba-3e6709dc7c31\") " pod="openshift-ingress/e2e-worker-w4-b5178f-openshift-default-c8fbdff8c-bjzz7"
Oct 30 19:37:09.297142 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:37:09.297087    4892 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/98aae325-07a2-49fc-a4ba-3e6709dc7c31-credential-socket\") pod \"e2e-worker-w4-b5178f-openshift-default-c8fbdff8c-bjzz7\" (UID: \"98aae325-07a2-49fc-a4ba-3e6709dc7c31\") " pod="openshift-ingress/e2e-worker-w4-b5178f-openshift-default-c8fbdff8c-bjzz7"
Oct 30 19:37:09.297336 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:37:09.297160    4892 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/98aae325-07a2-49fc-a4ba-3e6709dc7c31-istio-data\") pod \"e2e-worker-w4-b5178f-openshift-default-c8fbdff8c-bjzz7\" (UID: \"98aae325-07a2-49fc-a4ba-3e6709dc7c31\") " pod="openshift-ingress/e2e-worker-w4-b5178f-openshift-default-c8fbdff8c-bjzz7"
Oct 30 19:37:09.297336 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:37:09.297323    4892 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/98aae325-07a2-49fc-a4ba-3e6709dc7c31-istiod-ca-cert\") pod \"e2e-worker-w4-b5178f-openshift-default-c8fbdff8c-bjzz7\" (UID: \"98aae325-07a2-49fc-a4ba-3e6709dc7c31\") " pod="openshift-ingress/e2e-worker-w4-b5178f-openshift-default-c8fbdff8c-bjzz7"
Oct 30 19:37:09.299399 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:37:09.299377    4892 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/98aae325-07a2-49fc-a4ba-3e6709dc7c31-istio-envoy\") pod \"e2e-worker-w4-b5178f-openshift-default-c8fbdff8c-bjzz7\" (UID: \"98aae325-07a2-49fc-a4ba-3e6709dc7c31\") " pod="openshift-ingress/e2e-worker-w4-b5178f-openshift-default-c8fbdff8c-bjzz7"
Oct 30 19:37:09.299494 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:37:09.299483    4892 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/98aae325-07a2-49fc-a4ba-3e6709dc7c31-istio-podinfo\") pod \"e2e-worker-w4-b5178f-openshift-default-c8fbdff8c-bjzz7\" (UID: \"98aae325-07a2-49fc-a4ba-3e6709dc7c31\") " pod="openshift-ingress/e2e-worker-w4-b5178f-openshift-default-c8fbdff8c-bjzz7"
Oct 30 19:37:09.354402 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:37:09.354371    4892 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/98aae325-07a2-49fc-a4ba-3e6709dc7c31-istio-token\") pod \"e2e-worker-w4-b5178f-openshift-default-c8fbdff8c-bjzz7\" (UID: \"98aae325-07a2-49fc-a4ba-3e6709dc7c31\") " pod="openshift-ingress/e2e-worker-w4-b5178f-openshift-default-c8fbdff8c-bjzz7"
Oct 30 19:37:09.354620 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:37:09.354603    4892 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8l2c6\" (UniqueName: \"kubernetes.io/projected/98aae325-07a2-49fc-a4ba-3e6709dc7c31-kube-api-access-8l2c6\") pod \"e2e-worker-w4-b5178f-openshift-default-c8fbdff8c-bjzz7\" (UID: \"98aae325-07a2-49fc-a4ba-3e6709dc7c31\") " pod="openshift-ingress/e2e-worker-w4-b5178f-openshift-default-c8fbdff8c-bjzz7"
Oct 30 19:37:09.446270 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:37:09.446242    4892 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-b5178f-openshift-default-c8fbdff8c-bjzz7"
Oct 30 19:37:09.875654 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:37:09.875626    4892 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-b5178f-openshift-default-c8fbdff8c-bjzz7"]
Oct 30 19:37:09.877038 ip-10-0-132-82 kubenswrapper[4892]: W1030 19:37:09.876952    4892 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod98aae325_07a2_49fc_a4ba_3e6709dc7c31.slice/crio-970b23dd993461e6cf6b45bf7d9f34c1e22de65696203557aa76aaf20474b6f6 WatchSource:0}: Error finding container 970b23dd993461e6cf6b45bf7d9f34c1e22de65696203557aa76aaf20474b6f6: Status 404 returned error can't find the container with id 970b23dd993461e6cf6b45bf7d9f34c1e22de65696203557aa76aaf20474b6f6
Oct 30 19:37:09.879377 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:37:09.879343    4892 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892000Ki","pods":"250"}
Oct 30 19:37:09.879450 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:37:09.879412    4892 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892000Ki","pods":"250"}
Oct 30 19:37:09.879450 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:37:09.879441    4892 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892000Ki","pods":"250"}
Oct 30 19:37:10.160834 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:37:10.160797    4892 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-b5178f-openshift-default-c8fbdff8c-bjzz7" event={"ID":"98aae325-07a2-49fc-a4ba-3e6709dc7c31","Type":"ContainerStarted","Data":"7ef76347a62902de8860eacaff47801435076f847731251161c12a26d4f90b60"}
Oct 30 19:37:10.161361 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:37:10.160853    4892 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-b5178f-openshift-default-c8fbdff8c-bjzz7" event={"ID":"98aae325-07a2-49fc-a4ba-3e6709dc7c31","Type":"ContainerStarted","Data":"970b23dd993461e6cf6b45bf7d9f34c1e22de65696203557aa76aaf20474b6f6"}
Oct 30 19:37:10.279595 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:37:10.279543    4892 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w4-b5178f-openshift-default-c8fbdff8c-bjzz7" podStartSLOduration=1.279529485 podStartE2EDuration="1.279529485s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:37:09 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:37:10.276557307 +0000 UTC m=+1277.847022916" watchObservedRunningTime="2026-10-30 19:37:10.279529485 +0000 UTC m=+1277.849995150"
Oct 30 19:37:10.449047 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:37:10.448883    4892 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w4-b5178f-openshift-default-c8fbdff8c-bjzz7"
Oct 30 19:37:10.449215 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:37:10.449046    4892 patch_prober.go:29] interesting pod/e2e-worker-w4-b5178f-openshift-default-c8fbdff8c-bjzz7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.75:15021/healthz/ready\": dial tcp 10.132.0.75:15021: connect: connection refused" start-of-body=
Oct 30 19:37:10.449215 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:37:10.449090    4892 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-b5178f-openshift-default-c8fbdff8c-bjzz7" podUID="98aae325-07a2-49fc-a4ba-3e6709dc7c31" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.75:15021/healthz/ready\": dial tcp 10.132.0.75:15021: connect: connection refused"
Oct 30 19:37:11.450985 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:37:11.450297    4892 patch_prober.go:29] interesting pod/e2e-worker-w4-b5178f-openshift-default-c8fbdff8c-bjzz7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.75:15021/healthz/ready\": dial tcp 10.132.0.75:15021: connect: connection refused" start-of-body=
Oct 30 19:37:11.450985 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:37:11.450356    4892 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-b5178f-openshift-default-c8fbdff8c-bjzz7" podUID="98aae325-07a2-49fc-a4ba-3e6709dc7c31" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.75:15021/healthz/ready\": dial tcp 10.132.0.75:15021: connect: connection refused"
Oct 30 19:37:12.449991 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:37:12.449312    4892 patch_prober.go:29] interesting pod/e2e-worker-w4-b5178f-openshift-default-c8fbdff8c-bjzz7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.75:15021/healthz/ready\": dial tcp 10.132.0.75:15021: connect: connection refused" start-of-body=
Oct 30 19:37:12.449991 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:37:12.449374    4892 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-b5178f-openshift-default-c8fbdff8c-bjzz7" podUID="98aae325-07a2-49fc-a4ba-3e6709dc7c31" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.75:15021/healthz/ready\": dial tcp 10.132.0.75:15021: connect: connection refused"
Oct 30 19:37:13.449991 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:37:13.449334    4892 patch_prober.go:29] interesting pod/e2e-worker-w4-b5178f-openshift-default-c8fbdff8c-bjzz7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.75:15021/healthz/ready\": dial tcp 10.132.0.75:15021: connect: connection refused" start-of-body=
Oct 30 19:37:13.449991 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:37:13.449398    4892 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-b5178f-openshift-default-c8fbdff8c-bjzz7" podUID="98aae325-07a2-49fc-a4ba-3e6709dc7c31" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.75:15021/healthz/ready\": dial tcp 10.132.0.75:15021: connect: connection refused"
Oct 30 19:37:14.448954 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:37:14.448916    4892 patch_prober.go:29] interesting pod/e2e-worker-w4-b5178f-openshift-default-c8fbdff8c-bjzz7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.75:15021/healthz/ready\": dial tcp 10.132.0.75:15021: connect: connection refused" start-of-body=
Oct 30 19:37:14.449307 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:37:14.449276    4892 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-b5178f-openshift-default-c8fbdff8c-bjzz7" podUID="98aae325-07a2-49fc-a4ba-3e6709dc7c31" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.75:15021/healthz/ready\": dial tcp 10.132.0.75:15021: connect: connection refused"
Oct 30 19:37:15.447987 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:37:15.447303    4892 patch_prober.go:29] interesting pod/e2e-worker-w4-b5178f-openshift-default-c8fbdff8c-bjzz7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.75:15021/healthz/ready\": dial tcp 10.132.0.75:15021: connect: connection refused" start-of-body=
Oct 30 19:37:15.447987 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:37:15.447373    4892 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-b5178f-openshift-default-c8fbdff8c-bjzz7" podUID="98aae325-07a2-49fc-a4ba-3e6709dc7c31" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.75:15021/healthz/ready\": dial tcp 10.132.0.75:15021: connect: connection refused"
Oct 30 19:37:15.540551 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:37:15.540513    4892 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-fsfvk"]
Oct 30 19:37:16.447331 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:37:16.447297    4892 patch_prober.go:29] interesting pod/e2e-worker-w4-b5178f-openshift-default-c8fbdff8c-bjzz7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.75:15021/healthz/ready\": dial tcp 10.132.0.75:15021: connect: connection refused" start-of-body=
Oct 30 19:37:16.447538 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:37:16.447364    4892 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-b5178f-openshift-default-c8fbdff8c-bjzz7" podUID="98aae325-07a2-49fc-a4ba-3e6709dc7c31" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.75:15021/healthz/ready\": dial tcp 10.132.0.75:15021: connect: connection refused"
Oct 30 19:37:17.629861 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:37:17.629830    4892 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w4-b5178f-openshift-default-c8fbdff8c-bjzz7"
Oct 30 19:37:17.630718 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:37:17.630691    4892 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w4-b5178f-openshift-default-c8fbdff8c-bjzz7"
Oct 30 19:37:17.631936 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:37:17.631918    4892 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w4-b5178f-openshift-default-c8fbdff8c-bjzz7"
Oct 30 19:37:24.845366 ip-10-0-132-82 kubenswrapper[4892]: E1030 19:37:24.845324    4892 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-jrm87" containerName="extractor"
Oct 30 19:37:36.964555 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:37:36.964454    4892 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c24a2487-rpvhb"]
Oct 30 19:37:36.969696 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:37:36.969678    4892 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c24a2487-rpvhb"
Oct 30 19:37:36.997449 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:37:36.997423    4892 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c55219bc-f5e4-4a58-8c3a-51485d360d49-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-c24a2487-rpvhb\" (UID: \"c55219bc-f5e4-4a58-8c3a-51485d360d49\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c24a2487-rpvhb"
Oct 30 19:37:36.997568 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:37:36.997498    4892 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c55219bc-f5e4-4a58-8c3a-51485d360d49-maas-api-xks-ca\") pod \"maas-api-revoke-sub-c24a2487-rpvhb\" (UID: \"c55219bc-f5e4-4a58-8c3a-51485d360d49\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c24a2487-rpvhb"
Oct 30 19:37:36.997633 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:37:36.997582    4892 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6hrtv\" (UniqueName: \"kubernetes.io/projected/c55219bc-f5e4-4a58-8c3a-51485d360d49-kube-api-access-6hrtv\") pod \"maas-api-revoke-sub-c24a2487-rpvhb\" (UID: \"c55219bc-f5e4-4a58-8c3a-51485d360d49\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c24a2487-rpvhb"
Oct 30 19:37:37.022294 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:37:37.022263    4892 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 30 19:37:37.071351 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:37:37.071329    4892 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 19:37:37.071615 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:37:37.071586    4892 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-nlxp4\""
Oct 30 19:37:37.071725 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:37:37.071671    4892 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 30 19:37:37.074891 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:37:37.074859    4892 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c24a2487-rpvhb"]
Oct 30 19:37:37.098885 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:37:37.098860    4892 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c55219bc-f5e4-4a58-8c3a-51485d360d49-maas-api-xks-ca\") pod \"maas-api-revoke-sub-c24a2487-rpvhb\" (UID: \"c55219bc-f5e4-4a58-8c3a-51485d360d49\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c24a2487-rpvhb"
Oct 30 19:37:37.099006 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:37:37.098892    4892 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6hrtv\" (UniqueName: \"kubernetes.io/projected/c55219bc-f5e4-4a58-8c3a-51485d360d49-kube-api-access-6hrtv\") pod \"maas-api-revoke-sub-c24a2487-rpvhb\" (UID: \"c55219bc-f5e4-4a58-8c3a-51485d360d49\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c24a2487-rpvhb"
Oct 30 19:37:37.099006 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:37:37.098994    4892 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c55219bc-f5e4-4a58-8c3a-51485d360d49-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-c24a2487-rpvhb\" (UID: \"c55219bc-f5e4-4a58-8c3a-51485d360d49\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c24a2487-rpvhb"
Oct 30 19:37:37.099569 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:37:37.099537    4892 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c55219bc-f5e4-4a58-8c3a-51485d360d49-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-c24a2487-rpvhb\" (UID: \"c55219bc-f5e4-4a58-8c3a-51485d360d49\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c24a2487-rpvhb"
Oct 30 19:37:37.101656 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:37:37.101633    4892 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c55219bc-f5e4-4a58-8c3a-51485d360d49-maas-api-xks-ca\") pod \"maas-api-revoke-sub-c24a2487-rpvhb\" (UID: \"c55219bc-f5e4-4a58-8c3a-51485d360d49\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c24a2487-rpvhb"
Oct 30 19:37:37.155071 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:37:37.155047    4892 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6hrtv\" (UniqueName: \"kubernetes.io/projected/c55219bc-f5e4-4a58-8c3a-51485d360d49-kube-api-access-6hrtv\") pod \"maas-api-revoke-sub-c24a2487-rpvhb\" (UID: \"c55219bc-f5e4-4a58-8c3a-51485d360d49\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c24a2487-rpvhb"
Oct 30 19:37:37.283318 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:37:37.283252    4892 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c24a2487-rpvhb"
Oct 30 19:37:37.661556 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:37:37.660779    4892 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c24a2487-rpvhb"]
Oct 30 19:37:37.665213 ip-10-0-132-82 kubenswrapper[4892]: W1030 19:37:37.665182    4892 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc55219bc_f5e4_4a58_8c3a_51485d360d49.slice/crio-b624e953a741335d7edf37955a266447708320e6395eab05b8e3b1848e0e9650 WatchSource:0}: Error finding container b624e953a741335d7edf37955a266447708320e6395eab05b8e3b1848e0e9650: Status 404 returned error can't find the container with id b624e953a741335d7edf37955a266447708320e6395eab05b8e3b1848e0e9650
Oct 30 19:37:38.357449 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:37:38.357341    4892 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c24a2487-rpvhb" event={"ID":"c55219bc-f5e4-4a58-8c3a-51485d360d49","Type":"ContainerStarted","Data":"9fee3f79a37da6e1108cb6ff7056fee62b1a22fd6cbb811d782289d007f5ce9c"}
Oct 30 19:37:38.357449 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:37:38.357419    4892 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c24a2487-rpvhb" event={"ID":"c55219bc-f5e4-4a58-8c3a-51485d360d49","Type":"ContainerStarted","Data":"b624e953a741335d7edf37955a266447708320e6395eab05b8e3b1848e0e9650"}
Oct 30 19:37:38.472126 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:37:38.472061    4892 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c24a2487-rpvhb" podStartSLOduration=2.472043546 podStartE2EDuration="2.472043546s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:37:36 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:37:38.470959579 +0000 UTC m=+1306.041425179" watchObservedRunningTime="2026-10-30 19:37:38.472043546 +0000 UTC m=+1306.042509147"
Oct 30 19:37:39.366734 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:37:39.366701    4892 generic.go:356] "Generic (PLEG): container finished" podID="c55219bc-f5e4-4a58-8c3a-51485d360d49" containerID="9fee3f79a37da6e1108cb6ff7056fee62b1a22fd6cbb811d782289d007f5ce9c" exitCode=0
Oct 30 19:37:39.367255 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:37:39.366799    4892 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c24a2487-rpvhb" event={"ID":"c55219bc-f5e4-4a58-8c3a-51485d360d49","Type":"ContainerDied","Data":"9fee3f79a37da6e1108cb6ff7056fee62b1a22fd6cbb811d782289d007f5ce9c"}
Oct 30 19:37:40.545946 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:37:40.545915    4892 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c24a2487-rpvhb"
Oct 30 19:37:40.680043 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:37:40.680012    4892 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c55219bc-f5e4-4a58-8c3a-51485d360d49-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c55219bc-f5e4-4a58-8c3a-51485d360d49-maas-api-openshift-ca\") pod \"c55219bc-f5e4-4a58-8c3a-51485d360d49\" (UID: \"c55219bc-f5e4-4a58-8c3a-51485d360d49\") "
Oct 30 19:37:40.680225 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:37:40.680196    4892 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/c55219bc-f5e4-4a58-8c3a-51485d360d49-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c55219bc-f5e4-4a58-8c3a-51485d360d49-maas-api-xks-ca\") pod \"c55219bc-f5e4-4a58-8c3a-51485d360d49\" (UID: \"c55219bc-f5e4-4a58-8c3a-51485d360d49\") "
Oct 30 19:37:40.680293 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:37:40.680247    4892 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c55219bc-f5e4-4a58-8c3a-51485d360d49-kube-api-access-6hrtv\" (UniqueName: \"kubernetes.io/projected/c55219bc-f5e4-4a58-8c3a-51485d360d49-kube-api-access-6hrtv\") pod \"c55219bc-f5e4-4a58-8c3a-51485d360d49\" (UID: \"c55219bc-f5e4-4a58-8c3a-51485d360d49\") "
Oct 30 19:37:40.680647 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:37:40.680624    4892 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c55219bc-f5e4-4a58-8c3a-51485d360d49-maas-api-openshift-ca" pod "c55219bc-f5e4-4a58-8c3a-51485d360d49" (UID: "c55219bc-f5e4-4a58-8c3a-51485d360d49"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 19:37:40.685033 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:37:40.684451    4892 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c55219bc-f5e4-4a58-8c3a-51485d360d49-kube-api-access-6hrtv" pod "c55219bc-f5e4-4a58-8c3a-51485d360d49" (UID: "c55219bc-f5e4-4a58-8c3a-51485d360d49"). InnerVolumeSpecName "kube-api-access-6hrtv". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:37:40.690375 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:37:40.690347    4892 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c55219bc-f5e4-4a58-8c3a-51485d360d49-maas-api-xks-ca" pod "c55219bc-f5e4-4a58-8c3a-51485d360d49" (UID: "c55219bc-f5e4-4a58-8c3a-51485d360d49"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 19:37:40.781596 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:37:40.781559    4892 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c55219bc-f5e4-4a58-8c3a-51485d360d49-maas-api-openshift-ca\") on node \"ip-10-0-132-82.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 19:37:40.781596 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:37:40.781588    4892 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c55219bc-f5e4-4a58-8c3a-51485d360d49-maas-api-xks-ca\") on node \"ip-10-0-132-82.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 19:37:40.781596 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:37:40.781598    4892 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6hrtv\" (UniqueName: \"kubernetes.io/projected/c55219bc-f5e4-4a58-8c3a-51485d360d49-kube-api-access-6hrtv\") on node \"ip-10-0-132-82.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 19:37:41.000665 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:37:41.000568    4892 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-fsfvk"]
Oct 30 19:37:41.384360 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:37:41.384320    4892 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c24a2487-rpvhb" event={"ID":"c55219bc-f5e4-4a58-8c3a-51485d360d49","Type":"ContainerDied","Data":"b624e953a741335d7edf37955a266447708320e6395eab05b8e3b1848e0e9650"}
Oct 30 19:37:41.384527 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:37:41.384370    4892 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="b624e953a741335d7edf37955a266447708320e6395eab05b8e3b1848e0e9650"
Oct 30 19:37:41.384527 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:37:41.384367    4892 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c24a2487-rpvhb"
Oct 30 19:37:47.668404 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:37:47.668370    4892 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a5b01630-p8ch7"]
Oct 30 19:37:47.668847 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:37:47.668833    4892 cpu_manager.go:414] "containerMap: removing container" podUID="c55219bc-f5e4-4a58-8c3a-51485d360d49" containerName="revoke-keys"
Oct 30 19:37:47.668892 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:37:47.668847    4892 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c55219bc-f5e4-4a58-8c3a-51485d360d49" containerName="revoke-keys"
Oct 30 19:37:47.668937 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:37:47.668928    4892 memory_manager.go:365] "RemoveStaleState removing state" podUID="c55219bc-f5e4-4a58-8c3a-51485d360d49" containerName="revoke-keys"
Oct 30 19:37:47.673284 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:37:47.673264    4892 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a5b01630-p8ch7"
Oct 30 19:37:47.726308 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:37:47.726284    4892 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 30 19:37:47.775542 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:37:47.775514    4892 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 19:37:47.775647 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:37:47.775575    4892 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 30 19:37:47.775708 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:37:47.775694    4892 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-nlxp4\""
Oct 30 19:37:47.778776 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:37:47.778759    4892 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a5b01630-p8ch7"]
Oct 30 19:37:47.862395 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:37:47.862362    4892 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ee5f11d0-f573-4583-93af-1a7ec44d49ea-maas-api-xks-ca\") pod \"maas-api-revoke-sub-a5b01630-p8ch7\" (UID: \"ee5f11d0-f573-4583-93af-1a7ec44d49ea\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a5b01630-p8ch7"
Oct 30 19:37:47.862586 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:37:47.862409    4892 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ee5f11d0-f573-4583-93af-1a7ec44d49ea-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-a5b01630-p8ch7\" (UID: \"ee5f11d0-f573-4583-93af-1a7ec44d49ea\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a5b01630-p8ch7"
Oct 30 19:37:47.862586 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:37:47.862532    4892 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-w7kt9\" (UniqueName: \"kubernetes.io/projected/ee5f11d0-f573-4583-93af-1a7ec44d49ea-kube-api-access-w7kt9\") pod \"maas-api-revoke-sub-a5b01630-p8ch7\" (UID: \"ee5f11d0-f573-4583-93af-1a7ec44d49ea\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a5b01630-p8ch7"
Oct 30 19:37:47.963917 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:37:47.963850    4892 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ee5f11d0-f573-4583-93af-1a7ec44d49ea-maas-api-xks-ca\") pod \"maas-api-revoke-sub-a5b01630-p8ch7\" (UID: \"ee5f11d0-f573-4583-93af-1a7ec44d49ea\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a5b01630-p8ch7"
Oct 30 19:37:47.963917 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:37:47.963899    4892 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ee5f11d0-f573-4583-93af-1a7ec44d49ea-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-a5b01630-p8ch7\" (UID: \"ee5f11d0-f573-4583-93af-1a7ec44d49ea\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a5b01630-p8ch7"
Oct 30 19:37:47.964093 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:37:47.964014    4892 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-w7kt9\" (UniqueName: \"kubernetes.io/projected/ee5f11d0-f573-4583-93af-1a7ec44d49ea-kube-api-access-w7kt9\") pod \"maas-api-revoke-sub-a5b01630-p8ch7\" (UID: \"ee5f11d0-f573-4583-93af-1a7ec44d49ea\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a5b01630-p8ch7"
Oct 30 19:37:47.964510 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:37:47.964473    4892 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ee5f11d0-f573-4583-93af-1a7ec44d49ea-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-a5b01630-p8ch7\" (UID: \"ee5f11d0-f573-4583-93af-1a7ec44d49ea\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a5b01630-p8ch7"
Oct 30 19:37:47.966345 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:37:47.966328    4892 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ee5f11d0-f573-4583-93af-1a7ec44d49ea-maas-api-xks-ca\") pod \"maas-api-revoke-sub-a5b01630-p8ch7\" (UID: \"ee5f11d0-f573-4583-93af-1a7ec44d49ea\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a5b01630-p8ch7"
Oct 30 19:37:48.020784 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:37:48.020747    4892 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-w7kt9\" (UniqueName: \"kubernetes.io/projected/ee5f11d0-f573-4583-93af-1a7ec44d49ea-kube-api-access-w7kt9\") pod \"maas-api-revoke-sub-a5b01630-p8ch7\" (UID: \"ee5f11d0-f573-4583-93af-1a7ec44d49ea\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a5b01630-p8ch7"
Oct 30 19:37:48.286674 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:37:48.286595    4892 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a5b01630-p8ch7"
Oct 30 19:37:48.662143 ip-10-0-132-82 kubenswrapper[4892]: W1030 19:37:48.662117    4892 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podee5f11d0_f573_4583_93af_1a7ec44d49ea.slice/crio-6dc1236a609aada20366b30db59c4eb064f971d65009c13da80ea61df1e060c2 WatchSource:0}: Error finding container 6dc1236a609aada20366b30db59c4eb064f971d65009c13da80ea61df1e060c2: Status 404 returned error can't find the container with id 6dc1236a609aada20366b30db59c4eb064f971d65009c13da80ea61df1e060c2
Oct 30 19:37:48.662143 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:37:48.662121    4892 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a5b01630-p8ch7"]
Oct 30 19:37:49.435708 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:37:49.435677    4892 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a5b01630-p8ch7" event={"ID":"ee5f11d0-f573-4583-93af-1a7ec44d49ea","Type":"ContainerStarted","Data":"3933dfedcf561e55b4d9ec4f30a9ccaf8ed137755b4d03b662f3067b9a2d692b"}
Oct 30 19:37:49.436130 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:37:49.435730    4892 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a5b01630-p8ch7" event={"ID":"ee5f11d0-f573-4583-93af-1a7ec44d49ea","Type":"ContainerStarted","Data":"6dc1236a609aada20366b30db59c4eb064f971d65009c13da80ea61df1e060c2"}
Oct 30 19:37:49.550776 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:37:49.550723    4892 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a5b01630-p8ch7" podStartSLOduration=2.55070886 podStartE2EDuration="2.55070886s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:37:47 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:37:49.547616384 +0000 UTC m=+1317.118081981" watchObservedRunningTime="2026-10-30 19:37:49.55070886 +0000 UTC m=+1317.121174458"
Oct 30 19:37:50.443075 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:37:50.443041    4892 generic.go:356] "Generic (PLEG): container finished" podID="ee5f11d0-f573-4583-93af-1a7ec44d49ea" containerID="3933dfedcf561e55b4d9ec4f30a9ccaf8ed137755b4d03b662f3067b9a2d692b" exitCode=0
Oct 30 19:37:50.443465 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:37:50.443137    4892 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a5b01630-p8ch7" event={"ID":"ee5f11d0-f573-4583-93af-1a7ec44d49ea","Type":"ContainerDied","Data":"3933dfedcf561e55b4d9ec4f30a9ccaf8ed137755b4d03b662f3067b9a2d692b"}
Oct 30 19:37:51.609768 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:37:51.609742    4892 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a5b01630-p8ch7"
Oct 30 19:37:51.799460 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:37:51.799386    4892 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ee5f11d0-f573-4583-93af-1a7ec44d49ea-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ee5f11d0-f573-4583-93af-1a7ec44d49ea-maas-api-openshift-ca\") pod \"ee5f11d0-f573-4583-93af-1a7ec44d49ea\" (UID: \"ee5f11d0-f573-4583-93af-1a7ec44d49ea\") "
Oct 30 19:37:51.799603 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:37:51.799490    4892 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ee5f11d0-f573-4583-93af-1a7ec44d49ea-kube-api-access-w7kt9\" (UniqueName: \"kubernetes.io/projected/ee5f11d0-f573-4583-93af-1a7ec44d49ea-kube-api-access-w7kt9\") pod \"ee5f11d0-f573-4583-93af-1a7ec44d49ea\" (UID: \"ee5f11d0-f573-4583-93af-1a7ec44d49ea\") "
Oct 30 19:37:51.799603 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:37:51.799566    4892 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/ee5f11d0-f573-4583-93af-1a7ec44d49ea-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ee5f11d0-f573-4583-93af-1a7ec44d49ea-maas-api-xks-ca\") pod \"ee5f11d0-f573-4583-93af-1a7ec44d49ea\" (UID: \"ee5f11d0-f573-4583-93af-1a7ec44d49ea\") "
Oct 30 19:37:51.799816 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:37:51.799783    4892 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ee5f11d0-f573-4583-93af-1a7ec44d49ea-maas-api-openshift-ca" pod "ee5f11d0-f573-4583-93af-1a7ec44d49ea" (UID: "ee5f11d0-f573-4583-93af-1a7ec44d49ea"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 19:37:51.799920 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:37:51.799906    4892 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ee5f11d0-f573-4583-93af-1a7ec44d49ea-maas-api-openshift-ca\") on node \"ip-10-0-132-82.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 19:37:51.801699 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:37:51.801677    4892 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ee5f11d0-f573-4583-93af-1a7ec44d49ea-kube-api-access-w7kt9" pod "ee5f11d0-f573-4583-93af-1a7ec44d49ea" (UID: "ee5f11d0-f573-4583-93af-1a7ec44d49ea"). InnerVolumeSpecName "kube-api-access-w7kt9". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:37:51.801787 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:37:51.801700    4892 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ee5f11d0-f573-4583-93af-1a7ec44d49ea-maas-api-xks-ca" pod "ee5f11d0-f573-4583-93af-1a7ec44d49ea" (UID: "ee5f11d0-f573-4583-93af-1a7ec44d49ea"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 19:37:51.901290 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:37:51.901263    4892 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-w7kt9\" (UniqueName: \"kubernetes.io/projected/ee5f11d0-f573-4583-93af-1a7ec44d49ea-kube-api-access-w7kt9\") on node \"ip-10-0-132-82.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 19:37:51.901290 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:37:51.901289    4892 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ee5f11d0-f573-4583-93af-1a7ec44d49ea-maas-api-xks-ca\") on node \"ip-10-0-132-82.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 19:37:52.463927 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:37:52.463897    4892 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a5b01630-p8ch7"
Oct 30 19:37:52.463927 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:37:52.463912    4892 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a5b01630-p8ch7" event={"ID":"ee5f11d0-f573-4583-93af-1a7ec44d49ea","Type":"ContainerDied","Data":"6dc1236a609aada20366b30db59c4eb064f971d65009c13da80ea61df1e060c2"}
Oct 30 19:37:52.464143 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:37:52.464025    4892 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="6dc1236a609aada20366b30db59c4eb064f971d65009c13da80ea61df1e060c2"
Oct 30 19:37:54.845221 ip-10-0-132-82 kubenswrapper[4892]: E1030 19:37:54.845183    4892 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-jrm87" containerName="extractor"
Oct 30 19:37:59.418777 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:37:59.418738    4892 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-xqnps"]
Oct 30 19:37:59.419638 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:37:59.419014    4892 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-xqnps" podUID="4ac820b6-1c82-4e99-9a96-59ed74259ae7" containerName="istio-proxy" containerID="cri-o://0faea4937790e22d09d64392601c213319461aa958c6b746d3c018482a657cb8" gracePeriod=30
Oct 30 19:38:04.585530 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:38:04.585308    4892 generic.go:356] "Generic (PLEG): container finished" podID="4ac820b6-1c82-4e99-9a96-59ed74259ae7" containerID="0faea4937790e22d09d64392601c213319461aa958c6b746d3c018482a657cb8" exitCode=0
Oct 30 19:38:04.585530 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:38:04.585391    4892 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-xqnps" event={"ID":"4ac820b6-1c82-4e99-9a96-59ed74259ae7","Type":"ContainerDied","Data":"0faea4937790e22d09d64392601c213319461aa958c6b746d3c018482a657cb8"}
Oct 30 19:38:04.703231 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:38:04.703204    4892 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-xqnps"
Oct 30 19:38:04.767264 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:38:04.767187    4892 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4ac820b6-1c82-4e99-9a96-59ed74259ae7-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4ac820b6-1c82-4e99-9a96-59ed74259ae7-workload-socket\") pod \"4ac820b6-1c82-4e99-9a96-59ed74259ae7\" (UID: \"4ac820b6-1c82-4e99-9a96-59ed74259ae7\") "
Oct 30 19:38:04.767264 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:38:04.767228    4892 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/4ac820b6-1c82-4e99-9a96-59ed74259ae7-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4ac820b6-1c82-4e99-9a96-59ed74259ae7-istio-podinfo\") pod \"4ac820b6-1c82-4e99-9a96-59ed74259ae7\" (UID: \"4ac820b6-1c82-4e99-9a96-59ed74259ae7\") "
Oct 30 19:38:04.767473 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:38:04.767301    4892 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4ac820b6-1c82-4e99-9a96-59ed74259ae7-istio-token\" (UniqueName: \"kubernetes.io/projected/4ac820b6-1c82-4e99-9a96-59ed74259ae7-istio-token\") pod \"4ac820b6-1c82-4e99-9a96-59ed74259ae7\" (UID: \"4ac820b6-1c82-4e99-9a96-59ed74259ae7\") "
Oct 30 19:38:04.767473 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:38:04.767367    4892 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4ac820b6-1c82-4e99-9a96-59ed74259ae7-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4ac820b6-1c82-4e99-9a96-59ed74259ae7-credential-socket\") pod \"4ac820b6-1c82-4e99-9a96-59ed74259ae7\" (UID: \"4ac820b6-1c82-4e99-9a96-59ed74259ae7\") "
Oct 30 19:38:04.767473 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:38:04.767404    4892 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4ac820b6-1c82-4e99-9a96-59ed74259ae7-kube-api-access-jfrhb\" (UniqueName: \"kubernetes.io/projected/4ac820b6-1c82-4e99-9a96-59ed74259ae7-kube-api-access-jfrhb\") pod \"4ac820b6-1c82-4e99-9a96-59ed74259ae7\" (UID: \"4ac820b6-1c82-4e99-9a96-59ed74259ae7\") "
Oct 30 19:38:04.767473 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:38:04.767434    4892 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4ac820b6-1c82-4e99-9a96-59ed74259ae7-workload-socket" pod "4ac820b6-1c82-4e99-9a96-59ed74259ae7" (UID: "4ac820b6-1c82-4e99-9a96-59ed74259ae7"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:38:04.767684 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:38:04.767485    4892 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4ac820b6-1c82-4e99-9a96-59ed74259ae7-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4ac820b6-1c82-4e99-9a96-59ed74259ae7-istio-envoy\") pod \"4ac820b6-1c82-4e99-9a96-59ed74259ae7\" (UID: \"4ac820b6-1c82-4e99-9a96-59ed74259ae7\") "
Oct 30 19:38:04.767684 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:38:04.767524    4892 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4ac820b6-1c82-4e99-9a96-59ed74259ae7-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4ac820b6-1c82-4e99-9a96-59ed74259ae7-istio-data\") pod \"4ac820b6-1c82-4e99-9a96-59ed74259ae7\" (UID: \"4ac820b6-1c82-4e99-9a96-59ed74259ae7\") "
Oct 30 19:38:04.767684 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:38:04.767558    4892 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/4ac820b6-1c82-4e99-9a96-59ed74259ae7-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4ac820b6-1c82-4e99-9a96-59ed74259ae7-istiod-ca-cert\") pod \"4ac820b6-1c82-4e99-9a96-59ed74259ae7\" (UID: \"4ac820b6-1c82-4e99-9a96-59ed74259ae7\") "
Oct 30 19:38:04.767684 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:38:04.767663    4892 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4ac820b6-1c82-4e99-9a96-59ed74259ae7-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4ac820b6-1c82-4e99-9a96-59ed74259ae7-workload-certs\") pod \"4ac820b6-1c82-4e99-9a96-59ed74259ae7\" (UID: \"4ac820b6-1c82-4e99-9a96-59ed74259ae7\") "
Oct 30 19:38:04.767902 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:38:04.767682    4892 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4ac820b6-1c82-4e99-9a96-59ed74259ae7-credential-socket" pod "4ac820b6-1c82-4e99-9a96-59ed74259ae7" (UID: "4ac820b6-1c82-4e99-9a96-59ed74259ae7"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:38:04.767958 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:38:04.767893    4892 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4ac820b6-1c82-4e99-9a96-59ed74259ae7-istio-data" pod "4ac820b6-1c82-4e99-9a96-59ed74259ae7" (UID: "4ac820b6-1c82-4e99-9a96-59ed74259ae7"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:38:04.768048 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:38:04.768006    4892 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4ac820b6-1c82-4e99-9a96-59ed74259ae7-istiod-ca-cert" pod "4ac820b6-1c82-4e99-9a96-59ed74259ae7" (UID: "4ac820b6-1c82-4e99-9a96-59ed74259ae7"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 19:38:04.768150 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:38:04.768114    4892 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4ac820b6-1c82-4e99-9a96-59ed74259ae7-workload-certs" pod "4ac820b6-1c82-4e99-9a96-59ed74259ae7" (UID: "4ac820b6-1c82-4e99-9a96-59ed74259ae7"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:38:04.768233 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:38:04.768147    4892 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4ac820b6-1c82-4e99-9a96-59ed74259ae7-workload-socket\") on node \"ip-10-0-132-82.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 19:38:04.768233 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:38:04.768182    4892 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4ac820b6-1c82-4e99-9a96-59ed74259ae7-credential-socket\") on node \"ip-10-0-132-82.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 19:38:04.768233 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:38:04.768192    4892 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4ac820b6-1c82-4e99-9a96-59ed74259ae7-istio-data\") on node \"ip-10-0-132-82.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 19:38:04.768233 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:38:04.768200    4892 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4ac820b6-1c82-4e99-9a96-59ed74259ae7-istiod-ca-cert\") on node \"ip-10-0-132-82.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 19:38:04.770218 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:38:04.770179    4892 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4ac820b6-1c82-4e99-9a96-59ed74259ae7-kube-api-access-jfrhb" pod "4ac820b6-1c82-4e99-9a96-59ed74259ae7" (UID: "4ac820b6-1c82-4e99-9a96-59ed74259ae7"). InnerVolumeSpecName "kube-api-access-jfrhb". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:38:04.770218 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:38:04.770200    4892 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4ac820b6-1c82-4e99-9a96-59ed74259ae7-istio-token" pod "4ac820b6-1c82-4e99-9a96-59ed74259ae7" (UID: "4ac820b6-1c82-4e99-9a96-59ed74259ae7"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:38:04.770419 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:38:04.770400    4892 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4ac820b6-1c82-4e99-9a96-59ed74259ae7-istio-envoy" pod "4ac820b6-1c82-4e99-9a96-59ed74259ae7" (UID: "4ac820b6-1c82-4e99-9a96-59ed74259ae7"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:38:04.770518 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:38:04.770502    4892 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/4ac820b6-1c82-4e99-9a96-59ed74259ae7-istio-podinfo" pod "4ac820b6-1c82-4e99-9a96-59ed74259ae7" (UID: "4ac820b6-1c82-4e99-9a96-59ed74259ae7"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 19:38:04.869491 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:38:04.869456    4892 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jfrhb\" (UniqueName: \"kubernetes.io/projected/4ac820b6-1c82-4e99-9a96-59ed74259ae7-kube-api-access-jfrhb\") on node \"ip-10-0-132-82.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 19:38:04.869491 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:38:04.869487    4892 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4ac820b6-1c82-4e99-9a96-59ed74259ae7-istio-envoy\") on node \"ip-10-0-132-82.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 19:38:04.869491 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:38:04.869496    4892 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4ac820b6-1c82-4e99-9a96-59ed74259ae7-workload-certs\") on node \"ip-10-0-132-82.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 19:38:04.869727 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:38:04.869506    4892 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4ac820b6-1c82-4e99-9a96-59ed74259ae7-istio-podinfo\") on node \"ip-10-0-132-82.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 19:38:04.869727 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:38:04.869515    4892 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4ac820b6-1c82-4e99-9a96-59ed74259ae7-istio-token\") on node \"ip-10-0-132-82.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 19:38:05.593455 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:38:05.593425    4892 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-xqnps"
Oct 30 19:38:05.593455 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:38:05.593435    4892 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-xqnps" event={"ID":"4ac820b6-1c82-4e99-9a96-59ed74259ae7","Type":"ContainerDied","Data":"9ab21d3855dae5b7859fb614f4f3a92d864d4214a0eca491e894c343eb11bd24"}
Oct 30 19:38:05.593988 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:38:05.593497    4892 scope.go:122] "RemoveContainer" containerID="0faea4937790e22d09d64392601c213319461aa958c6b746d3c018482a657cb8"
Oct 30 19:38:05.806605 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:38:05.806578    4892 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-xqnps"]
Oct 30 19:38:05.855845 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:38:05.855767    4892 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-xqnps"]
Oct 30 19:38:06.569981 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:38:06.569936    4892 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="4ac820b6-1c82-4e99-9a96-59ed74259ae7" path="/var/lib/kubelet/pods/4ac820b6-1c82-4e99-9a96-59ed74259ae7/volumes"
Oct 30 19:38:10.696176 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:38:10.696144    4892 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-fsfvk"]
Oct 30 19:38:24.844739 ip-10-0-132-82 kubenswrapper[4892]: E1030 19:38:24.844697    4892 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-jrm87" containerName="extractor"
Oct 30 19:38:25.091600 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:38:25.091563    4892 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-6af60237-mnfct"]
Oct 30 19:38:25.092437 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:38:25.092415    4892 cpu_manager.go:414] "containerMap: removing container" podUID="ee5f11d0-f573-4583-93af-1a7ec44d49ea" containerName="revoke-keys"
Oct 30 19:38:25.092503 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:38:25.092441    4892 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ee5f11d0-f573-4583-93af-1a7ec44d49ea" containerName="revoke-keys"
Oct 30 19:38:25.092503 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:38:25.092461    4892 cpu_manager.go:414] "containerMap: removing container" podUID="4ac820b6-1c82-4e99-9a96-59ed74259ae7" containerName="istio-proxy"
Oct 30 19:38:25.092503 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:38:25.092469    4892 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4ac820b6-1c82-4e99-9a96-59ed74259ae7" containerName="istio-proxy"
Oct 30 19:38:25.092635 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:38:25.092622    4892 memory_manager.go:365] "RemoveStaleState removing state" podUID="ee5f11d0-f573-4583-93af-1a7ec44d49ea" containerName="revoke-keys"
Oct 30 19:38:25.092679 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:38:25.092640    4892 memory_manager.go:365] "RemoveStaleState removing state" podUID="4ac820b6-1c82-4e99-9a96-59ed74259ae7" containerName="istio-proxy"
Oct 30 19:38:25.095959 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:38:25.095907    4892 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6af60237-mnfct"
Oct 30 19:38:25.150845 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:38:25.150813    4892 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 30 19:38:25.177881 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:38:25.177846    4892 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f0f6128d-e2d6-4156-b261-c362b64ca062-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-6af60237-mnfct\" (UID: \"f0f6128d-e2d6-4156-b261-c362b64ca062\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6af60237-mnfct"
Oct 30 19:38:25.178054 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:38:25.177892    4892 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pjkfz\" (UniqueName: \"kubernetes.io/projected/f0f6128d-e2d6-4156-b261-c362b64ca062-kube-api-access-pjkfz\") pod \"maas-api-revoke-sub-6af60237-mnfct\" (UID: \"f0f6128d-e2d6-4156-b261-c362b64ca062\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6af60237-mnfct"
Oct 30 19:38:25.178054 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:38:25.178004    4892 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f0f6128d-e2d6-4156-b261-c362b64ca062-maas-api-xks-ca\") pod \"maas-api-revoke-sub-6af60237-mnfct\" (UID: \"f0f6128d-e2d6-4156-b261-c362b64ca062\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6af60237-mnfct"
Oct 30 19:38:25.197174 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:38:25.197143    4892 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 30 19:38:25.197174 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:38:25.197150    4892 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 19:38:25.197331 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:38:25.197211    4892 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-nlxp4\""
Oct 30 19:38:25.200780 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:38:25.200755    4892 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-6af60237-mnfct"]
Oct 30 19:38:25.279037 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:38:25.279010    4892 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f0f6128d-e2d6-4156-b261-c362b64ca062-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-6af60237-mnfct\" (UID: \"f0f6128d-e2d6-4156-b261-c362b64ca062\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6af60237-mnfct"
Oct 30 19:38:25.279037 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:38:25.279041    4892 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pjkfz\" (UniqueName: \"kubernetes.io/projected/f0f6128d-e2d6-4156-b261-c362b64ca062-kube-api-access-pjkfz\") pod \"maas-api-revoke-sub-6af60237-mnfct\" (UID: \"f0f6128d-e2d6-4156-b261-c362b64ca062\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6af60237-mnfct"
Oct 30 19:38:25.279213 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:38:25.279082    4892 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f0f6128d-e2d6-4156-b261-c362b64ca062-maas-api-xks-ca\") pod \"maas-api-revoke-sub-6af60237-mnfct\" (UID: \"f0f6128d-e2d6-4156-b261-c362b64ca062\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6af60237-mnfct"
Oct 30 19:38:25.279641 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:38:25.279622    4892 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f0f6128d-e2d6-4156-b261-c362b64ca062-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-6af60237-mnfct\" (UID: \"f0f6128d-e2d6-4156-b261-c362b64ca062\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6af60237-mnfct"
Oct 30 19:38:25.281518 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:38:25.281500    4892 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f0f6128d-e2d6-4156-b261-c362b64ca062-maas-api-xks-ca\") pod \"maas-api-revoke-sub-6af60237-mnfct\" (UID: \"f0f6128d-e2d6-4156-b261-c362b64ca062\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6af60237-mnfct"
Oct 30 19:38:25.344179 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:38:25.344153    4892 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-pjkfz\" (UniqueName: \"kubernetes.io/projected/f0f6128d-e2d6-4156-b261-c362b64ca062-kube-api-access-pjkfz\") pod \"maas-api-revoke-sub-6af60237-mnfct\" (UID: \"f0f6128d-e2d6-4156-b261-c362b64ca062\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6af60237-mnfct"
Oct 30 19:38:25.479609 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:38:25.479576    4892 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6af60237-mnfct"
Oct 30 19:38:25.722652 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:38:25.722613    4892 generic.go:356] "Generic (PLEG): container finished" podID="1ba4f185-0574-403d-8db6-77f802d0dc1f" containerID="e589a555c9e7ceee632261f13909a33ab7f9559346388dd0b5d9dcab3ae2f6b8" exitCode=0
Oct 30 19:38:25.722820 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:38:25.722685    4892 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/periodic-gathering-58sbq-z4rz6" event={"ID":"1ba4f185-0574-403d-8db6-77f802d0dc1f","Type":"ContainerDied","Data":"e589a555c9e7ceee632261f13909a33ab7f9559346388dd0b5d9dcab3ae2f6b8"}
Oct 30 19:38:25.860439 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:38:25.860415    4892 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-6af60237-mnfct"]
Oct 30 19:38:25.862316 ip-10-0-132-82 kubenswrapper[4892]: W1030 19:38:25.862288    4892 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf0f6128d_e2d6_4156_b261_c362b64ca062.slice/crio-0c27af56da934e9bbd09fba6549179c26d456ef8c04078ce0088b8ee3d508073 WatchSource:0}: Error finding container 0c27af56da934e9bbd09fba6549179c26d456ef8c04078ce0088b8ee3d508073: Status 404 returned error can't find the container with id 0c27af56da934e9bbd09fba6549179c26d456ef8c04078ce0088b8ee3d508073
Oct 30 19:38:26.730613 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:38:26.730586    4892 generic.go:356] "Generic (PLEG): container finished" podID="f0f6128d-e2d6-4156-b261-c362b64ca062" containerID="8e04dd468eafc106b973c87398222e366ead23eab487a8de1beed4e51918f279" exitCode=0
Oct 30 19:38:26.730716 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:38:26.730665    4892 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6af60237-mnfct" event={"ID":"f0f6128d-e2d6-4156-b261-c362b64ca062","Type":"ContainerDied","Data":"8e04dd468eafc106b973c87398222e366ead23eab487a8de1beed4e51918f279"}
Oct 30 19:38:26.730765 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:38:26.730714    4892 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6af60237-mnfct" event={"ID":"f0f6128d-e2d6-4156-b261-c362b64ca062","Type":"ContainerStarted","Data":"0c27af56da934e9bbd09fba6549179c26d456ef8c04078ce0088b8ee3d508073"}
Oct 30 19:38:26.752760 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:38:26.752732    4892 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b19201f6-kv7xb"]
Oct 30 19:38:26.756349 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:38:26.756334    4892 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b19201f6-kv7xb"
Oct 30 19:38:26.815934 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:38:26.815897    4892 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b19201f6-kv7xb"]
Oct 30 19:38:26.876307 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:38:26.876284    4892 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-insights/periodic-gathering-58sbq-z4rz6"
Oct 30 19:38:26.894465 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:38:26.894409    4892 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/aa084871-b2e9-4cb2-bae0-f4add975912b-maas-api-xks-ca\") pod \"maas-api-revoke-sub-b19201f6-kv7xb\" (UID: \"aa084871-b2e9-4cb2-bae0-f4add975912b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b19201f6-kv7xb"
Oct 30 19:38:26.894556 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:38:26.894488    4892 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-275nv\" (UniqueName: \"kubernetes.io/projected/aa084871-b2e9-4cb2-bae0-f4add975912b-kube-api-access-275nv\") pod \"maas-api-revoke-sub-b19201f6-kv7xb\" (UID: \"aa084871-b2e9-4cb2-bae0-f4add975912b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b19201f6-kv7xb"
Oct 30 19:38:26.894614 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:38:26.894554    4892 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/aa084871-b2e9-4cb2-bae0-f4add975912b-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-b19201f6-kv7xb\" (UID: \"aa084871-b2e9-4cb2-bae0-f4add975912b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b19201f6-kv7xb"
Oct 30 19:38:26.995552 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:38:26.995482    4892 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1ba4f185-0574-403d-8db6-77f802d0dc1f-kube-api-access-lf796\" (UniqueName: \"kubernetes.io/projected/1ba4f185-0574-403d-8db6-77f802d0dc1f-kube-api-access-lf796\") pod \"1ba4f185-0574-403d-8db6-77f802d0dc1f\" (UID: \"1ba4f185-0574-403d-8db6-77f802d0dc1f\") "
Oct 30 19:38:26.995552 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:38:26.995544    4892 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/1ba4f185-0574-403d-8db6-77f802d0dc1f-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/1ba4f185-0574-403d-8db6-77f802d0dc1f-trusted-ca-bundle\") pod \"1ba4f185-0574-403d-8db6-77f802d0dc1f\" (UID: \"1ba4f185-0574-403d-8db6-77f802d0dc1f\") "
Oct 30 19:38:26.995762 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:38:26.995622    4892 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/1ba4f185-0574-403d-8db6-77f802d0dc1f-service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/1ba4f185-0574-403d-8db6-77f802d0dc1f-service-ca-bundle\") pod \"1ba4f185-0574-403d-8db6-77f802d0dc1f\" (UID: \"1ba4f185-0574-403d-8db6-77f802d0dc1f\") "
Oct 30 19:38:26.995762 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:38:26.995666    4892 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1ba4f185-0574-403d-8db6-77f802d0dc1f-archives-path\" (UniqueName: \"kubernetes.io/empty-dir/1ba4f185-0574-403d-8db6-77f802d0dc1f-archives-path\") pod \"1ba4f185-0574-403d-8db6-77f802d0dc1f\" (UID: \"1ba4f185-0574-403d-8db6-77f802d0dc1f\") "
Oct 30 19:38:26.995923 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:38:26.995904    4892 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/aa084871-b2e9-4cb2-bae0-f4add975912b-maas-api-xks-ca\") pod \"maas-api-revoke-sub-b19201f6-kv7xb\" (UID: \"aa084871-b2e9-4cb2-bae0-f4add975912b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b19201f6-kv7xb"
Oct 30 19:38:26.996061 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:38:26.996014    4892 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1ba4f185-0574-403d-8db6-77f802d0dc1f-trusted-ca-bundle" pod "1ba4f185-0574-403d-8db6-77f802d0dc1f" (UID: "1ba4f185-0574-403d-8db6-77f802d0dc1f"). InnerVolumeSpecName "trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 19:38:26.996061 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:38:26.996033    4892 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1ba4f185-0574-403d-8db6-77f802d0dc1f-archives-path" pod "1ba4f185-0574-403d-8db6-77f802d0dc1f" (UID: "1ba4f185-0574-403d-8db6-77f802d0dc1f"). InnerVolumeSpecName "archives-path". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:38:26.996061 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:38:26.996023    4892 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1ba4f185-0574-403d-8db6-77f802d0dc1f-service-ca-bundle" pod "1ba4f185-0574-403d-8db6-77f802d0dc1f" (UID: "1ba4f185-0574-403d-8db6-77f802d0dc1f"). InnerVolumeSpecName "service-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 19:38:26.996236 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:38:26.996023    4892 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-275nv\" (UniqueName: \"kubernetes.io/projected/aa084871-b2e9-4cb2-bae0-f4add975912b-kube-api-access-275nv\") pod \"maas-api-revoke-sub-b19201f6-kv7xb\" (UID: \"aa084871-b2e9-4cb2-bae0-f4add975912b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b19201f6-kv7xb"
Oct 30 19:38:26.996236 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:38:26.996107    4892 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/aa084871-b2e9-4cb2-bae0-f4add975912b-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-b19201f6-kv7xb\" (UID: \"aa084871-b2e9-4cb2-bae0-f4add975912b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b19201f6-kv7xb"
Oct 30 19:38:26.996236 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:38:26.996214    4892 reconciler_common.go:299] "Volume detached for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/1ba4f185-0574-403d-8db6-77f802d0dc1f-trusted-ca-bundle\") on node \"ip-10-0-132-82.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 19:38:26.996236 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:38:26.996226    4892 reconciler_common.go:299] "Volume detached for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/1ba4f185-0574-403d-8db6-77f802d0dc1f-service-ca-bundle\") on node \"ip-10-0-132-82.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 19:38:26.996236 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:38:26.996235    4892 reconciler_common.go:299] "Volume detached for volume \"archives-path\" (UniqueName: \"kubernetes.io/empty-dir/1ba4f185-0574-403d-8db6-77f802d0dc1f-archives-path\") on node \"ip-10-0-132-82.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 19:38:26.996870 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:38:26.996850    4892 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/aa084871-b2e9-4cb2-bae0-f4add975912b-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-b19201f6-kv7xb\" (UID: \"aa084871-b2e9-4cb2-bae0-f4add975912b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b19201f6-kv7xb"
Oct 30 19:38:26.997995 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:38:26.997942    4892 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1ba4f185-0574-403d-8db6-77f802d0dc1f-kube-api-access-lf796" pod "1ba4f185-0574-403d-8db6-77f802d0dc1f" (UID: "1ba4f185-0574-403d-8db6-77f802d0dc1f"). InnerVolumeSpecName "kube-api-access-lf796". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:38:26.998568 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:38:26.998546    4892 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/aa084871-b2e9-4cb2-bae0-f4add975912b-maas-api-xks-ca\") pod \"maas-api-revoke-sub-b19201f6-kv7xb\" (UID: \"aa084871-b2e9-4cb2-bae0-f4add975912b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b19201f6-kv7xb"
Oct 30 19:38:27.053435 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:38:27.053411    4892 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-275nv\" (UniqueName: \"kubernetes.io/projected/aa084871-b2e9-4cb2-bae0-f4add975912b-kube-api-access-275nv\") pod \"maas-api-revoke-sub-b19201f6-kv7xb\" (UID: \"aa084871-b2e9-4cb2-bae0-f4add975912b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b19201f6-kv7xb"
Oct 30 19:38:27.097518 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:38:27.097493    4892 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lf796\" (UniqueName: \"kubernetes.io/projected/1ba4f185-0574-403d-8db6-77f802d0dc1f-kube-api-access-lf796\") on node \"ip-10-0-132-82.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 19:38:27.101283 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:38:27.101265    4892 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b19201f6-kv7xb"
Oct 30 19:38:27.505049 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:38:27.505020    4892 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b19201f6-kv7xb"]
Oct 30 19:38:27.505866 ip-10-0-132-82 kubenswrapper[4892]: W1030 19:38:27.505834    4892 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podaa084871_b2e9_4cb2_bae0_f4add975912b.slice/crio-64ab6e1481965cc433fe2cccedff09c00e69126c6a1abb8c140701fd0a408a68 WatchSource:0}: Error finding container 64ab6e1481965cc433fe2cccedff09c00e69126c6a1abb8c140701fd0a408a68: Status 404 returned error can't find the container with id 64ab6e1481965cc433fe2cccedff09c00e69126c6a1abb8c140701fd0a408a68
Oct 30 19:38:27.744298 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:38:27.744260    4892 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-insights/periodic-gathering-58sbq-z4rz6"
Oct 30 19:38:27.744298 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:38:27.744268    4892 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/periodic-gathering-58sbq-z4rz6" event={"ID":"1ba4f185-0574-403d-8db6-77f802d0dc1f","Type":"ContainerDied","Data":"834c91c83f3d78d68c1f049c34bc383fc7f7e38b911e2c4369ad860755794a81"}
Oct 30 19:38:27.744480 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:38:27.744328    4892 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="834c91c83f3d78d68c1f049c34bc383fc7f7e38b911e2c4369ad860755794a81"
Oct 30 19:38:27.748187 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:38:27.748164    4892 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b19201f6-kv7xb" event={"ID":"aa084871-b2e9-4cb2-bae0-f4add975912b","Type":"ContainerStarted","Data":"64ab6e1481965cc433fe2cccedff09c00e69126c6a1abb8c140701fd0a408a68"}
Oct 30 19:38:27.774354 ip-10-0-132-82 kubenswrapper[4892]: E1030 19:38:27.774326    4892 cadvisor_stats_provider.go:569] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1ba4f185_0574_403d_8db6_77f802d0dc1f.slice/crio-834c91c83f3d78d68c1f049c34bc383fc7f7e38b911e2c4369ad860755794a81\": RecentStats: unable to find data in memory cache]"
Oct 30 19:38:27.892235 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:38:27.892211    4892 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6af60237-mnfct"
Oct 30 19:38:28.005930 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:38:28.005845    4892 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f0f6128d-e2d6-4156-b261-c362b64ca062-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f0f6128d-e2d6-4156-b261-c362b64ca062-maas-api-openshift-ca\") pod \"f0f6128d-e2d6-4156-b261-c362b64ca062\" (UID: \"f0f6128d-e2d6-4156-b261-c362b64ca062\") "
Oct 30 19:38:28.005930 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:38:28.005900    4892 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/f0f6128d-e2d6-4156-b261-c362b64ca062-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f0f6128d-e2d6-4156-b261-c362b64ca062-maas-api-xks-ca\") pod \"f0f6128d-e2d6-4156-b261-c362b64ca062\" (UID: \"f0f6128d-e2d6-4156-b261-c362b64ca062\") "
Oct 30 19:38:28.005930 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:38:28.005927    4892 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f0f6128d-e2d6-4156-b261-c362b64ca062-kube-api-access-pjkfz\" (UniqueName: \"kubernetes.io/projected/f0f6128d-e2d6-4156-b261-c362b64ca062-kube-api-access-pjkfz\") pod \"f0f6128d-e2d6-4156-b261-c362b64ca062\" (UID: \"f0f6128d-e2d6-4156-b261-c362b64ca062\") "
Oct 30 19:38:28.006280 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:38:28.006251    4892 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f0f6128d-e2d6-4156-b261-c362b64ca062-maas-api-openshift-ca" pod "f0f6128d-e2d6-4156-b261-c362b64ca062" (UID: "f0f6128d-e2d6-4156-b261-c362b64ca062"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 19:38:28.006514 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:38:28.006487    4892 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f0f6128d-e2d6-4156-b261-c362b64ca062-maas-api-openshift-ca\") on node \"ip-10-0-132-82.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 19:38:28.008567 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:38:28.008530    4892 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f0f6128d-e2d6-4156-b261-c362b64ca062-maas-api-xks-ca" pod "f0f6128d-e2d6-4156-b261-c362b64ca062" (UID: "f0f6128d-e2d6-4156-b261-c362b64ca062"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 19:38:28.008844 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:38:28.008821    4892 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f0f6128d-e2d6-4156-b261-c362b64ca062-kube-api-access-pjkfz" pod "f0f6128d-e2d6-4156-b261-c362b64ca062" (UID: "f0f6128d-e2d6-4156-b261-c362b64ca062"). InnerVolumeSpecName "kube-api-access-pjkfz". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:38:28.107284 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:38:28.107258    4892 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f0f6128d-e2d6-4156-b261-c362b64ca062-maas-api-xks-ca\") on node \"ip-10-0-132-82.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 19:38:28.107392 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:38:28.107284    4892 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pjkfz\" (UniqueName: \"kubernetes.io/projected/f0f6128d-e2d6-4156-b261-c362b64ca062-kube-api-access-pjkfz\") on node \"ip-10-0-132-82.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 19:38:28.756330 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:38:28.756245    4892 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6af60237-mnfct"
Oct 30 19:38:28.756330 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:38:28.756275    4892 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6af60237-mnfct" event={"ID":"f0f6128d-e2d6-4156-b261-c362b64ca062","Type":"ContainerDied","Data":"0c27af56da934e9bbd09fba6549179c26d456ef8c04078ce0088b8ee3d508073"}
Oct 30 19:38:28.756330 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:38:28.756328    4892 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="0c27af56da934e9bbd09fba6549179c26d456ef8c04078ce0088b8ee3d508073"
Oct 30 19:38:28.758137 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:38:28.758110    4892 generic.go:356] "Generic (PLEG): container finished" podID="aa084871-b2e9-4cb2-bae0-f4add975912b" containerID="d4e63a468abe6bde7fe79679cfdcae37580628eb0787861866792f48124d2d43" exitCode=0
Oct 30 19:38:28.758288 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:38:28.758171    4892 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b19201f6-kv7xb" event={"ID":"aa084871-b2e9-4cb2-bae0-f4add975912b","Type":"ContainerDied","Data":"d4e63a468abe6bde7fe79679cfdcae37580628eb0787861866792f48124d2d43"}
Oct 30 19:38:29.912146 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:38:29.912120    4892 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b19201f6-kv7xb"
Oct 30 19:38:30.028818 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:38:30.028740    4892 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/aa084871-b2e9-4cb2-bae0-f4add975912b-kube-api-access-275nv\" (UniqueName: \"kubernetes.io/projected/aa084871-b2e9-4cb2-bae0-f4add975912b-kube-api-access-275nv\") pod \"aa084871-b2e9-4cb2-bae0-f4add975912b\" (UID: \"aa084871-b2e9-4cb2-bae0-f4add975912b\") "
Oct 30 19:38:30.028818 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:38:30.028804    4892 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/aa084871-b2e9-4cb2-bae0-f4add975912b-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/aa084871-b2e9-4cb2-bae0-f4add975912b-maas-api-xks-ca\") pod \"aa084871-b2e9-4cb2-bae0-f4add975912b\" (UID: \"aa084871-b2e9-4cb2-bae0-f4add975912b\") "
Oct 30 19:38:30.029131 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:38:30.028917    4892 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/aa084871-b2e9-4cb2-bae0-f4add975912b-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/aa084871-b2e9-4cb2-bae0-f4add975912b-maas-api-openshift-ca\") pod \"aa084871-b2e9-4cb2-bae0-f4add975912b\" (UID: \"aa084871-b2e9-4cb2-bae0-f4add975912b\") "
Oct 30 19:38:30.029577 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:38:30.029549    4892 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/aa084871-b2e9-4cb2-bae0-f4add975912b-maas-api-openshift-ca" pod "aa084871-b2e9-4cb2-bae0-f4add975912b" (UID: "aa084871-b2e9-4cb2-bae0-f4add975912b"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 19:38:30.031336 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:38:30.031307    4892 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/aa084871-b2e9-4cb2-bae0-f4add975912b-maas-api-xks-ca" pod "aa084871-b2e9-4cb2-bae0-f4add975912b" (UID: "aa084871-b2e9-4cb2-bae0-f4add975912b"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 19:38:30.031336 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:38:30.031314    4892 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/aa084871-b2e9-4cb2-bae0-f4add975912b-kube-api-access-275nv" pod "aa084871-b2e9-4cb2-bae0-f4add975912b" (UID: "aa084871-b2e9-4cb2-bae0-f4add975912b"). InnerVolumeSpecName "kube-api-access-275nv". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:38:30.130228 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:38:30.130196    4892 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-275nv\" (UniqueName: \"kubernetes.io/projected/aa084871-b2e9-4cb2-bae0-f4add975912b-kube-api-access-275nv\") on node \"ip-10-0-132-82.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 19:38:30.130228 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:38:30.130227    4892 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/aa084871-b2e9-4cb2-bae0-f4add975912b-maas-api-xks-ca\") on node \"ip-10-0-132-82.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 19:38:30.130482 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:38:30.130240    4892 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/aa084871-b2e9-4cb2-bae0-f4add975912b-maas-api-openshift-ca\") on node \"ip-10-0-132-82.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 19:38:30.773016 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:38:30.772979    4892 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b19201f6-kv7xb" event={"ID":"aa084871-b2e9-4cb2-bae0-f4add975912b","Type":"ContainerDied","Data":"64ab6e1481965cc433fe2cccedff09c00e69126c6a1abb8c140701fd0a408a68"}
Oct 30 19:38:30.773016 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:38:30.773004    4892 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b19201f6-kv7xb"
Oct 30 19:38:30.773270 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:38:30.773048    4892 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="64ab6e1481965cc433fe2cccedff09c00e69126c6a1abb8c140701fd0a408a68"
Oct 30 19:38:34.225670 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:38:34.225628    4892 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-fsfvk"]
Oct 30 19:38:34.627215 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:38:34.627129    4892 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w5-686689-openshift-default-5fbf5787bf-jrt7h"]
Oct 30 19:38:34.627650 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:38:34.627630    4892 cpu_manager.go:414] "containerMap: removing container" podUID="1ba4f185-0574-403d-8db6-77f802d0dc1f" containerName="insights-gathering"
Oct 30 19:38:34.627749 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:38:34.627652    4892 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1ba4f185-0574-403d-8db6-77f802d0dc1f" containerName="insights-gathering"
Oct 30 19:38:34.627749 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:38:34.627670    4892 cpu_manager.go:414] "containerMap: removing container" podUID="f0f6128d-e2d6-4156-b261-c362b64ca062" containerName="revoke-keys"
Oct 30 19:38:34.627749 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:38:34.627678    4892 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f0f6128d-e2d6-4156-b261-c362b64ca062" containerName="revoke-keys"
Oct 30 19:38:34.627749 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:38:34.627716    4892 cpu_manager.go:414] "containerMap: removing container" podUID="aa084871-b2e9-4cb2-bae0-f4add975912b" containerName="revoke-keys"
Oct 30 19:38:34.627749 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:38:34.627724    4892 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="aa084871-b2e9-4cb2-bae0-f4add975912b" containerName="revoke-keys"
Oct 30 19:38:34.628060 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:38:34.627863    4892 memory_manager.go:365] "RemoveStaleState removing state" podUID="aa084871-b2e9-4cb2-bae0-f4add975912b" containerName="revoke-keys"
Oct 30 19:38:34.628060 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:38:34.627883    4892 memory_manager.go:365] "RemoveStaleState removing state" podUID="1ba4f185-0574-403d-8db6-77f802d0dc1f" containerName="insights-gathering"
Oct 30 19:38:34.628060 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:38:34.627892    4892 memory_manager.go:365] "RemoveStaleState removing state" podUID="f0f6128d-e2d6-4156-b261-c362b64ca062" containerName="revoke-keys"
Oct 30 19:38:34.631671 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:38:34.631644    4892 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-686689-openshift-default-5fbf5787bf-jrt7h"
Oct 30 19:38:34.732918 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:38:34.732894    4892 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w5-686689-openshift-default-dockercfg-75mcj\""
Oct 30 19:38:34.736955 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:38:34.736930    4892 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w5-686689-openshift-default-5fbf5787bf-jrt7h"]
Oct 30 19:38:34.778515 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:38:34.778489    4892 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/fa668020-72be-46a7-aa53-e8249130c4f1-istio-envoy\") pod \"e2e-worker-w5-686689-openshift-default-5fbf5787bf-jrt7h\" (UID: \"fa668020-72be-46a7-aa53-e8249130c4f1\") " pod="openshift-ingress/e2e-worker-w5-686689-openshift-default-5fbf5787bf-jrt7h"
Oct 30 19:38:34.778669 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:38:34.778554    4892 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/fa668020-72be-46a7-aa53-e8249130c4f1-workload-socket\") pod \"e2e-worker-w5-686689-openshift-default-5fbf5787bf-jrt7h\" (UID: \"fa668020-72be-46a7-aa53-e8249130c4f1\") " pod="openshift-ingress/e2e-worker-w5-686689-openshift-default-5fbf5787bf-jrt7h"
Oct 30 19:38:34.778736 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:38:34.778705    4892 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/fa668020-72be-46a7-aa53-e8249130c4f1-istio-podinfo\") pod \"e2e-worker-w5-686689-openshift-default-5fbf5787bf-jrt7h\" (UID: \"fa668020-72be-46a7-aa53-e8249130c4f1\") " pod="openshift-ingress/e2e-worker-w5-686689-openshift-default-5fbf5787bf-jrt7h"
Oct 30 19:38:34.778821 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:38:34.778805    4892 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/fa668020-72be-46a7-aa53-e8249130c4f1-credential-socket\") pod \"e2e-worker-w5-686689-openshift-default-5fbf5787bf-jrt7h\" (UID: \"fa668020-72be-46a7-aa53-e8249130c4f1\") " pod="openshift-ingress/e2e-worker-w5-686689-openshift-default-5fbf5787bf-jrt7h"
Oct 30 19:38:34.778875 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:38:34.778835    4892 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/fa668020-72be-46a7-aa53-e8249130c4f1-istio-token\") pod \"e2e-worker-w5-686689-openshift-default-5fbf5787bf-jrt7h\" (UID: \"fa668020-72be-46a7-aa53-e8249130c4f1\") " pod="openshift-ingress/e2e-worker-w5-686689-openshift-default-5fbf5787bf-jrt7h"
Oct 30 19:38:34.778875 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:38:34.778859    4892 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/fa668020-72be-46a7-aa53-e8249130c4f1-workload-certs\") pod \"e2e-worker-w5-686689-openshift-default-5fbf5787bf-jrt7h\" (UID: \"fa668020-72be-46a7-aa53-e8249130c4f1\") " pod="openshift-ingress/e2e-worker-w5-686689-openshift-default-5fbf5787bf-jrt7h"
Oct 30 19:38:34.779015 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:38:34.778997    4892 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/fa668020-72be-46a7-aa53-e8249130c4f1-istio-data\") pod \"e2e-worker-w5-686689-openshift-default-5fbf5787bf-jrt7h\" (UID: \"fa668020-72be-46a7-aa53-e8249130c4f1\") " pod="openshift-ingress/e2e-worker-w5-686689-openshift-default-5fbf5787bf-jrt7h"
Oct 30 19:38:34.779061 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:38:34.779050    4892 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-f7478\" (UniqueName: \"kubernetes.io/projected/fa668020-72be-46a7-aa53-e8249130c4f1-kube-api-access-f7478\") pod \"e2e-worker-w5-686689-openshift-default-5fbf5787bf-jrt7h\" (UID: \"fa668020-72be-46a7-aa53-e8249130c4f1\") " pod="openshift-ingress/e2e-worker-w5-686689-openshift-default-5fbf5787bf-jrt7h"
Oct 30 19:38:34.779106 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:38:34.779091    4892 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/fa668020-72be-46a7-aa53-e8249130c4f1-istiod-ca-cert\") pod \"e2e-worker-w5-686689-openshift-default-5fbf5787bf-jrt7h\" (UID: \"fa668020-72be-46a7-aa53-e8249130c4f1\") " pod="openshift-ingress/e2e-worker-w5-686689-openshift-default-5fbf5787bf-jrt7h"
Oct 30 19:38:34.880529 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:38:34.880444    4892 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/fa668020-72be-46a7-aa53-e8249130c4f1-workload-socket\") pod \"e2e-worker-w5-686689-openshift-default-5fbf5787bf-jrt7h\" (UID: \"fa668020-72be-46a7-aa53-e8249130c4f1\") " pod="openshift-ingress/e2e-worker-w5-686689-openshift-default-5fbf5787bf-jrt7h"
Oct 30 19:38:34.880529 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:38:34.880482    4892 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/fa668020-72be-46a7-aa53-e8249130c4f1-istio-podinfo\") pod \"e2e-worker-w5-686689-openshift-default-5fbf5787bf-jrt7h\" (UID: \"fa668020-72be-46a7-aa53-e8249130c4f1\") " pod="openshift-ingress/e2e-worker-w5-686689-openshift-default-5fbf5787bf-jrt7h"
Oct 30 19:38:34.880529 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:38:34.880520    4892 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/fa668020-72be-46a7-aa53-e8249130c4f1-credential-socket\") pod \"e2e-worker-w5-686689-openshift-default-5fbf5787bf-jrt7h\" (UID: \"fa668020-72be-46a7-aa53-e8249130c4f1\") " pod="openshift-ingress/e2e-worker-w5-686689-openshift-default-5fbf5787bf-jrt7h"
Oct 30 19:38:34.880831 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:38:34.880551    4892 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/fa668020-72be-46a7-aa53-e8249130c4f1-istio-token\") pod \"e2e-worker-w5-686689-openshift-default-5fbf5787bf-jrt7h\" (UID: \"fa668020-72be-46a7-aa53-e8249130c4f1\") " pod="openshift-ingress/e2e-worker-w5-686689-openshift-default-5fbf5787bf-jrt7h"
Oct 30 19:38:34.880831 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:38:34.880581    4892 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/fa668020-72be-46a7-aa53-e8249130c4f1-workload-certs\") pod \"e2e-worker-w5-686689-openshift-default-5fbf5787bf-jrt7h\" (UID: \"fa668020-72be-46a7-aa53-e8249130c4f1\") " pod="openshift-ingress/e2e-worker-w5-686689-openshift-default-5fbf5787bf-jrt7h"
Oct 30 19:38:34.880831 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:38:34.880628    4892 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/fa668020-72be-46a7-aa53-e8249130c4f1-istio-data\") pod \"e2e-worker-w5-686689-openshift-default-5fbf5787bf-jrt7h\" (UID: \"fa668020-72be-46a7-aa53-e8249130c4f1\") " pod="openshift-ingress/e2e-worker-w5-686689-openshift-default-5fbf5787bf-jrt7h"
Oct 30 19:38:34.880831 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:38:34.880675    4892 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-f7478\" (UniqueName: \"kubernetes.io/projected/fa668020-72be-46a7-aa53-e8249130c4f1-kube-api-access-f7478\") pod \"e2e-worker-w5-686689-openshift-default-5fbf5787bf-jrt7h\" (UID: \"fa668020-72be-46a7-aa53-e8249130c4f1\") " pod="openshift-ingress/e2e-worker-w5-686689-openshift-default-5fbf5787bf-jrt7h"
Oct 30 19:38:34.880831 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:38:34.880758    4892 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/fa668020-72be-46a7-aa53-e8249130c4f1-istiod-ca-cert\") pod \"e2e-worker-w5-686689-openshift-default-5fbf5787bf-jrt7h\" (UID: \"fa668020-72be-46a7-aa53-e8249130c4f1\") " pod="openshift-ingress/e2e-worker-w5-686689-openshift-default-5fbf5787bf-jrt7h"
Oct 30 19:38:34.881141 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:38:34.880814    4892 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/fa668020-72be-46a7-aa53-e8249130c4f1-istio-envoy\") pod \"e2e-worker-w5-686689-openshift-default-5fbf5787bf-jrt7h\" (UID: \"fa668020-72be-46a7-aa53-e8249130c4f1\") " pod="openshift-ingress/e2e-worker-w5-686689-openshift-default-5fbf5787bf-jrt7h"
Oct 30 19:38:34.881141 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:38:34.880941    4892 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/fa668020-72be-46a7-aa53-e8249130c4f1-workload-socket\") pod \"e2e-worker-w5-686689-openshift-default-5fbf5787bf-jrt7h\" (UID: \"fa668020-72be-46a7-aa53-e8249130c4f1\") " pod="openshift-ingress/e2e-worker-w5-686689-openshift-default-5fbf5787bf-jrt7h"
Oct 30 19:38:34.881141 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:38:34.881000    4892 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/fa668020-72be-46a7-aa53-e8249130c4f1-istio-data\") pod \"e2e-worker-w5-686689-openshift-default-5fbf5787bf-jrt7h\" (UID: \"fa668020-72be-46a7-aa53-e8249130c4f1\") " pod="openshift-ingress/e2e-worker-w5-686689-openshift-default-5fbf5787bf-jrt7h"
Oct 30 19:38:34.881141 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:38:34.881027    4892 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/fa668020-72be-46a7-aa53-e8249130c4f1-credential-socket\") pod \"e2e-worker-w5-686689-openshift-default-5fbf5787bf-jrt7h\" (UID: \"fa668020-72be-46a7-aa53-e8249130c4f1\") " pod="openshift-ingress/e2e-worker-w5-686689-openshift-default-5fbf5787bf-jrt7h"
Oct 30 19:38:34.881335 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:38:34.881305    4892 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/fa668020-72be-46a7-aa53-e8249130c4f1-workload-certs\") pod \"e2e-worker-w5-686689-openshift-default-5fbf5787bf-jrt7h\" (UID: \"fa668020-72be-46a7-aa53-e8249130c4f1\") " pod="openshift-ingress/e2e-worker-w5-686689-openshift-default-5fbf5787bf-jrt7h"
Oct 30 19:38:34.881571 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:38:34.881516    4892 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/fa668020-72be-46a7-aa53-e8249130c4f1-istiod-ca-cert\") pod \"e2e-worker-w5-686689-openshift-default-5fbf5787bf-jrt7h\" (UID: \"fa668020-72be-46a7-aa53-e8249130c4f1\") " pod="openshift-ingress/e2e-worker-w5-686689-openshift-default-5fbf5787bf-jrt7h"
Oct 30 19:38:34.883143 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:38:34.883125    4892 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/fa668020-72be-46a7-aa53-e8249130c4f1-istio-envoy\") pod \"e2e-worker-w5-686689-openshift-default-5fbf5787bf-jrt7h\" (UID: \"fa668020-72be-46a7-aa53-e8249130c4f1\") " pod="openshift-ingress/e2e-worker-w5-686689-openshift-default-5fbf5787bf-jrt7h"
Oct 30 19:38:34.883318 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:38:34.883299    4892 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/fa668020-72be-46a7-aa53-e8249130c4f1-istio-podinfo\") pod \"e2e-worker-w5-686689-openshift-default-5fbf5787bf-jrt7h\" (UID: \"fa668020-72be-46a7-aa53-e8249130c4f1\") " pod="openshift-ingress/e2e-worker-w5-686689-openshift-default-5fbf5787bf-jrt7h"
Oct 30 19:38:34.938861 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:38:34.938836    4892 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/fa668020-72be-46a7-aa53-e8249130c4f1-istio-token\") pod \"e2e-worker-w5-686689-openshift-default-5fbf5787bf-jrt7h\" (UID: \"fa668020-72be-46a7-aa53-e8249130c4f1\") " pod="openshift-ingress/e2e-worker-w5-686689-openshift-default-5fbf5787bf-jrt7h"
Oct 30 19:38:34.939012 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:38:34.938993    4892 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-f7478\" (UniqueName: \"kubernetes.io/projected/fa668020-72be-46a7-aa53-e8249130c4f1-kube-api-access-f7478\") pod \"e2e-worker-w5-686689-openshift-default-5fbf5787bf-jrt7h\" (UID: \"fa668020-72be-46a7-aa53-e8249130c4f1\") " pod="openshift-ingress/e2e-worker-w5-686689-openshift-default-5fbf5787bf-jrt7h"
Oct 30 19:38:34.944802 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:38:34.944781    4892 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-686689-openshift-default-5fbf5787bf-jrt7h"
Oct 30 19:38:35.411786 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:38:35.411753    4892 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w5-686689-openshift-default-5fbf5787bf-jrt7h"]
Oct 30 19:38:35.413019 ip-10-0-132-82 kubenswrapper[4892]: W1030 19:38:35.412990    4892 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfa668020_72be_46a7_aa53_e8249130c4f1.slice/crio-c8f34947636c0237d3bb41844f7545ded494ee81bd036089b50da280252418b8 WatchSource:0}: Error finding container c8f34947636c0237d3bb41844f7545ded494ee81bd036089b50da280252418b8: Status 404 returned error can't find the container with id c8f34947636c0237d3bb41844f7545ded494ee81bd036089b50da280252418b8
Oct 30 19:38:35.415331 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:38:35.415298    4892 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892000Ki","pods":"250"}
Oct 30 19:38:35.415410 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:38:35.415387    4892 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892000Ki","pods":"250"}
Oct 30 19:38:35.415448 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:38:35.415429    4892 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892000Ki","pods":"250"}
Oct 30 19:38:35.817682 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:38:35.817590    4892 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-686689-openshift-default-5fbf5787bf-jrt7h" event={"ID":"fa668020-72be-46a7-aa53-e8249130c4f1","Type":"ContainerStarted","Data":"708e6d3035d20872700663b9fa3f2578cb766d96a3df70f687bb61988f9c56a7"}
Oct 30 19:38:35.817682 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:38:35.817652    4892 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-686689-openshift-default-5fbf5787bf-jrt7h" event={"ID":"fa668020-72be-46a7-aa53-e8249130c4f1","Type":"ContainerStarted","Data":"c8f34947636c0237d3bb41844f7545ded494ee81bd036089b50da280252418b8"}
Oct 30 19:38:35.945989 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:38:35.945372    4892 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w5-686689-openshift-default-5fbf5787bf-jrt7h"
Oct 30 19:38:36.948608 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:38:36.948266    4892 patch_prober.go:29] interesting pod/e2e-worker-w5-686689-openshift-default-5fbf5787bf-jrt7h container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.80:15021/healthz/ready\": dial tcp 10.132.0.80:15021: connect: connection refused" start-of-body=
Oct 30 19:38:36.948608 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:38:36.948323    4892 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-686689-openshift-default-5fbf5787bf-jrt7h" podUID="fa668020-72be-46a7-aa53-e8249130c4f1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.80:15021/healthz/ready\": dial tcp 10.132.0.80:15021: connect: connection refused"
Oct 30 19:38:37.352788 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:38:37.352672    4892 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w5-686689-openshift-default-5fbf5787bf-jrt7h" podStartSLOduration=3.352654201 podStartE2EDuration="3.352654201s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:38:34 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:38:35.9403859 +0000 UTC m=+1363.510851496" watchObservedRunningTime="2026-10-30 19:38:37.352654201 +0000 UTC m=+1364.923119801"
Oct 30 19:38:37.353440 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:38:37.353410    4892 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-296bffe5-8sgf4"]
Oct 30 19:38:37.357677 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:38:37.357656    4892 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-296bffe5-8sgf4"
Oct 30 19:38:37.400573 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:38:37.400550    4892 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-296bffe5-8sgf4"]
Oct 30 19:38:37.462986 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:38:37.462209    4892 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 30 19:38:37.462986 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:38:37.462267    4892 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 19:38:37.462986 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:38:37.462307    4892 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 30 19:38:37.462986 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:38:37.462428    4892 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-nlxp4\""
Oct 30 19:38:37.512719 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:38:37.512684    4892 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lr5x8\" (UniqueName: \"kubernetes.io/projected/aee774c4-0eaf-4fa5-bec7-5b371c039f4a-kube-api-access-lr5x8\") pod \"maas-api-revoke-sub-296bffe5-8sgf4\" (UID: \"aee774c4-0eaf-4fa5-bec7-5b371c039f4a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-296bffe5-8sgf4"
Oct 30 19:38:37.512864 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:38:37.512790    4892 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/aee774c4-0eaf-4fa5-bec7-5b371c039f4a-maas-api-xks-ca\") pod \"maas-api-revoke-sub-296bffe5-8sgf4\" (UID: \"aee774c4-0eaf-4fa5-bec7-5b371c039f4a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-296bffe5-8sgf4"
Oct 30 19:38:37.512864 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:38:37.512824    4892 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/aee774c4-0eaf-4fa5-bec7-5b371c039f4a-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-296bffe5-8sgf4\" (UID: \"aee774c4-0eaf-4fa5-bec7-5b371c039f4a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-296bffe5-8sgf4"
Oct 30 19:38:37.613663 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:38:37.613584    4892 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/aee774c4-0eaf-4fa5-bec7-5b371c039f4a-maas-api-xks-ca\") pod \"maas-api-revoke-sub-296bffe5-8sgf4\" (UID: \"aee774c4-0eaf-4fa5-bec7-5b371c039f4a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-296bffe5-8sgf4"
Oct 30 19:38:37.613663 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:38:37.613628    4892 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/aee774c4-0eaf-4fa5-bec7-5b371c039f4a-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-296bffe5-8sgf4\" (UID: \"aee774c4-0eaf-4fa5-bec7-5b371c039f4a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-296bffe5-8sgf4"
Oct 30 19:38:37.613842 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:38:37.613668    4892 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lr5x8\" (UniqueName: \"kubernetes.io/projected/aee774c4-0eaf-4fa5-bec7-5b371c039f4a-kube-api-access-lr5x8\") pod \"maas-api-revoke-sub-296bffe5-8sgf4\" (UID: \"aee774c4-0eaf-4fa5-bec7-5b371c039f4a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-296bffe5-8sgf4"
Oct 30 19:38:37.614710 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:38:37.614692    4892 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/aee774c4-0eaf-4fa5-bec7-5b371c039f4a-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-296bffe5-8sgf4\" (UID: \"aee774c4-0eaf-4fa5-bec7-5b371c039f4a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-296bffe5-8sgf4"
Oct 30 19:38:37.616483 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:38:37.616460    4892 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/aee774c4-0eaf-4fa5-bec7-5b371c039f4a-maas-api-xks-ca\") pod \"maas-api-revoke-sub-296bffe5-8sgf4\" (UID: \"aee774c4-0eaf-4fa5-bec7-5b371c039f4a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-296bffe5-8sgf4"
Oct 30 19:38:37.671473 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:38:37.670716    4892 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lr5x8\" (UniqueName: \"kubernetes.io/projected/aee774c4-0eaf-4fa5-bec7-5b371c039f4a-kube-api-access-lr5x8\") pod \"maas-api-revoke-sub-296bffe5-8sgf4\" (UID: \"aee774c4-0eaf-4fa5-bec7-5b371c039f4a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-296bffe5-8sgf4"
Oct 30 19:38:37.672516 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:38:37.672487    4892 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-296bffe5-8sgf4"
Oct 30 19:38:37.955008 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:38:37.954945    4892 patch_prober.go:29] interesting pod/e2e-worker-w5-686689-openshift-default-5fbf5787bf-jrt7h container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.80:15021/healthz/ready\": dial tcp 10.132.0.80:15021: connect: connection refused" start-of-body=
Oct 30 19:38:37.955626 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:38:37.955595    4892 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-686689-openshift-default-5fbf5787bf-jrt7h" podUID="fa668020-72be-46a7-aa53-e8249130c4f1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.80:15021/healthz/ready\": dial tcp 10.132.0.80:15021: connect: connection refused"
Oct 30 19:38:38.082072 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:38:38.082022    4892 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-296bffe5-8sgf4"]
Oct 30 19:38:38.873293 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:38:38.873157    4892 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-296bffe5-8sgf4" event={"ID":"aee774c4-0eaf-4fa5-bec7-5b371c039f4a","Type":"ContainerStarted","Data":"43bf6d0c5d9060c1c318ee33f6aee895a3b122ee22e8ba36475fc912923bc1b5"}
Oct 30 19:38:38.873293 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:38:38.873218    4892 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-296bffe5-8sgf4" event={"ID":"aee774c4-0eaf-4fa5-bec7-5b371c039f4a","Type":"ContainerStarted","Data":"94e7073c7170bba90b38b62ecacedb3c8a6c69391896d2c046c56b6d33f18240"}
Oct 30 19:38:38.952990 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:38:38.951302    4892 patch_prober.go:29] interesting pod/e2e-worker-w5-686689-openshift-default-5fbf5787bf-jrt7h container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.80:15021/healthz/ready\": dial tcp 10.132.0.80:15021: connect: connection refused" start-of-body=
Oct 30 19:38:38.952990 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:38:38.951358    4892 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-686689-openshift-default-5fbf5787bf-jrt7h" podUID="fa668020-72be-46a7-aa53-e8249130c4f1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.80:15021/healthz/ready\": dial tcp 10.132.0.80:15021: connect: connection refused"
Oct 30 19:38:38.988392 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:38:38.988328    4892 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-296bffe5-8sgf4" podStartSLOduration=1.988314211 podStartE2EDuration="1.988314211s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:38:37 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:38:38.984931362 +0000 UTC m=+1366.555396961" watchObservedRunningTime="2026-10-30 19:38:38.988314211 +0000 UTC m=+1366.558779811"
Oct 30 19:38:39.884986 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:38:39.881995    4892 generic.go:356] "Generic (PLEG): container finished" podID="aee774c4-0eaf-4fa5-bec7-5b371c039f4a" containerID="43bf6d0c5d9060c1c318ee33f6aee895a3b122ee22e8ba36475fc912923bc1b5" exitCode=0
Oct 30 19:38:39.884986 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:38:39.882047    4892 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-296bffe5-8sgf4" event={"ID":"aee774c4-0eaf-4fa5-bec7-5b371c039f4a","Type":"ContainerDied","Data":"43bf6d0c5d9060c1c318ee33f6aee895a3b122ee22e8ba36475fc912923bc1b5"}
Oct 30 19:38:39.947988 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:38:39.947384    4892 patch_prober.go:29] interesting pod/e2e-worker-w5-686689-openshift-default-5fbf5787bf-jrt7h container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.80:15021/healthz/ready\": dial tcp 10.132.0.80:15021: connect: connection refused" start-of-body=
Oct 30 19:38:39.947988 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:38:39.947443    4892 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-686689-openshift-default-5fbf5787bf-jrt7h" podUID="fa668020-72be-46a7-aa53-e8249130c4f1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.80:15021/healthz/ready\": dial tcp 10.132.0.80:15021: connect: connection refused"
Oct 30 19:38:40.424670 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:38:40.424634    4892 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e2375738-tbbvw"]
Oct 30 19:38:40.444989 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:38:40.444215    4892 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2375738-tbbvw"
Oct 30 19:38:40.491529 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:38:40.491503    4892 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e2375738-tbbvw"]
Oct 30 19:38:40.570490 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:38:40.570463    4892 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gbr6h\" (UniqueName: \"kubernetes.io/projected/06d9238c-da95-4e55-b07d-d95dae4f2b29-kube-api-access-gbr6h\") pod \"maas-api-revoke-sub-e2375738-tbbvw\" (UID: \"06d9238c-da95-4e55-b07d-d95dae4f2b29\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2375738-tbbvw"
Oct 30 19:38:40.570719 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:38:40.570704    4892 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/06d9238c-da95-4e55-b07d-d95dae4f2b29-maas-api-xks-ca\") pod \"maas-api-revoke-sub-e2375738-tbbvw\" (UID: \"06d9238c-da95-4e55-b07d-d95dae4f2b29\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2375738-tbbvw"
Oct 30 19:38:40.570807 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:38:40.570797    4892 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/06d9238c-da95-4e55-b07d-d95dae4f2b29-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-e2375738-tbbvw\" (UID: \"06d9238c-da95-4e55-b07d-d95dae4f2b29\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2375738-tbbvw"
Oct 30 19:38:40.671729 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:38:40.671691    4892 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gbr6h\" (UniqueName: \"kubernetes.io/projected/06d9238c-da95-4e55-b07d-d95dae4f2b29-kube-api-access-gbr6h\") pod \"maas-api-revoke-sub-e2375738-tbbvw\" (UID: \"06d9238c-da95-4e55-b07d-d95dae4f2b29\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2375738-tbbvw"
Oct 30 19:38:40.671937 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:38:40.671791    4892 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/06d9238c-da95-4e55-b07d-d95dae4f2b29-maas-api-xks-ca\") pod \"maas-api-revoke-sub-e2375738-tbbvw\" (UID: \"06d9238c-da95-4e55-b07d-d95dae4f2b29\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2375738-tbbvw"
Oct 30 19:38:40.671937 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:38:40.671830    4892 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/06d9238c-da95-4e55-b07d-d95dae4f2b29-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-e2375738-tbbvw\" (UID: \"06d9238c-da95-4e55-b07d-d95dae4f2b29\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2375738-tbbvw"
Oct 30 19:38:40.673446 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:38:40.673418    4892 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/06d9238c-da95-4e55-b07d-d95dae4f2b29-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-e2375738-tbbvw\" (UID: \"06d9238c-da95-4e55-b07d-d95dae4f2b29\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2375738-tbbvw"
Oct 30 19:38:40.675277 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:38:40.675233    4892 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/06d9238c-da95-4e55-b07d-d95dae4f2b29-maas-api-xks-ca\") pod \"maas-api-revoke-sub-e2375738-tbbvw\" (UID: \"06d9238c-da95-4e55-b07d-d95dae4f2b29\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2375738-tbbvw"
Oct 30 19:38:40.761654 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:38:40.761624    4892 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gbr6h\" (UniqueName: \"kubernetes.io/projected/06d9238c-da95-4e55-b07d-d95dae4f2b29-kube-api-access-gbr6h\") pod \"maas-api-revoke-sub-e2375738-tbbvw\" (UID: \"06d9238c-da95-4e55-b07d-d95dae4f2b29\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2375738-tbbvw"
Oct 30 19:38:40.770246 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:38:40.770221    4892 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2375738-tbbvw"
Oct 30 19:38:40.958048 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:38:40.953363    4892 patch_prober.go:29] interesting pod/e2e-worker-w5-686689-openshift-default-5fbf5787bf-jrt7h container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.80:15021/healthz/ready\": dial tcp 10.132.0.80:15021: connect: connection refused" start-of-body=
Oct 30 19:38:40.958048 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:38:40.953422    4892 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-686689-openshift-default-5fbf5787bf-jrt7h" podUID="fa668020-72be-46a7-aa53-e8249130c4f1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.80:15021/healthz/ready\": dial tcp 10.132.0.80:15021: connect: connection refused"
Oct 30 19:38:41.096225 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:38:41.096196    4892 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-296bffe5-8sgf4"
Oct 30 19:38:41.161991 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:38:41.161948    4892 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e2375738-tbbvw"]
Oct 30 19:38:41.163410 ip-10-0-132-82 kubenswrapper[4892]: W1030 19:38:41.163385    4892 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod06d9238c_da95_4e55_b07d_d95dae4f2b29.slice/crio-7cdcdc46b5199cdbbebf36ace603e012e5297a9491c322e23ab6ed2dbdbfcd3c WatchSource:0}: Error finding container 7cdcdc46b5199cdbbebf36ace603e012e5297a9491c322e23ab6ed2dbdbfcd3c: Status 404 returned error can't find the container with id 7cdcdc46b5199cdbbebf36ace603e012e5297a9491c322e23ab6ed2dbdbfcd3c
Oct 30 19:38:41.282043 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:38:41.280782    4892 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/aee774c4-0eaf-4fa5-bec7-5b371c039f4a-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/aee774c4-0eaf-4fa5-bec7-5b371c039f4a-maas-api-xks-ca\") pod \"aee774c4-0eaf-4fa5-bec7-5b371c039f4a\" (UID: \"aee774c4-0eaf-4fa5-bec7-5b371c039f4a\") "
Oct 30 19:38:41.282043 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:38:41.281029    4892 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/aee774c4-0eaf-4fa5-bec7-5b371c039f4a-kube-api-access-lr5x8\" (UniqueName: \"kubernetes.io/projected/aee774c4-0eaf-4fa5-bec7-5b371c039f4a-kube-api-access-lr5x8\") pod \"aee774c4-0eaf-4fa5-bec7-5b371c039f4a\" (UID: \"aee774c4-0eaf-4fa5-bec7-5b371c039f4a\") "
Oct 30 19:38:41.282043 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:38:41.281128    4892 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/aee774c4-0eaf-4fa5-bec7-5b371c039f4a-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/aee774c4-0eaf-4fa5-bec7-5b371c039f4a-maas-api-openshift-ca\") pod \"aee774c4-0eaf-4fa5-bec7-5b371c039f4a\" (UID: \"aee774c4-0eaf-4fa5-bec7-5b371c039f4a\") "
Oct 30 19:38:41.295197 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:38:41.295159    4892 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/aee774c4-0eaf-4fa5-bec7-5b371c039f4a-maas-api-openshift-ca" pod "aee774c4-0eaf-4fa5-bec7-5b371c039f4a" (UID: "aee774c4-0eaf-4fa5-bec7-5b371c039f4a"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 19:38:41.329264 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:38:41.329128    4892 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/aee774c4-0eaf-4fa5-bec7-5b371c039f4a-maas-api-xks-ca" pod "aee774c4-0eaf-4fa5-bec7-5b371c039f4a" (UID: "aee774c4-0eaf-4fa5-bec7-5b371c039f4a"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 19:38:41.329264 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:38:41.329180    4892 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/aee774c4-0eaf-4fa5-bec7-5b371c039f4a-kube-api-access-lr5x8" pod "aee774c4-0eaf-4fa5-bec7-5b371c039f4a" (UID: "aee774c4-0eaf-4fa5-bec7-5b371c039f4a"). InnerVolumeSpecName "kube-api-access-lr5x8". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:38:41.382909 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:38:41.382864    4892 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lr5x8\" (UniqueName: \"kubernetes.io/projected/aee774c4-0eaf-4fa5-bec7-5b371c039f4a-kube-api-access-lr5x8\") on node \"ip-10-0-132-82.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 19:38:41.382909 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:38:41.382898    4892 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/aee774c4-0eaf-4fa5-bec7-5b371c039f4a-maas-api-openshift-ca\") on node \"ip-10-0-132-82.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 19:38:41.382909 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:38:41.382911    4892 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/aee774c4-0eaf-4fa5-bec7-5b371c039f4a-maas-api-xks-ca\") on node \"ip-10-0-132-82.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 19:38:41.897081 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:38:41.897053    4892 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-296bffe5-8sgf4"
Oct 30 19:38:41.897505 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:38:41.897092    4892 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-296bffe5-8sgf4" event={"ID":"aee774c4-0eaf-4fa5-bec7-5b371c039f4a","Type":"ContainerDied","Data":"94e7073c7170bba90b38b62ecacedb3c8a6c69391896d2c046c56b6d33f18240"}
Oct 30 19:38:41.897505 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:38:41.897142    4892 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="94e7073c7170bba90b38b62ecacedb3c8a6c69391896d2c046c56b6d33f18240"
Oct 30 19:38:41.899002 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:38:41.898976    4892 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2375738-tbbvw" event={"ID":"06d9238c-da95-4e55-b07d-d95dae4f2b29","Type":"ContainerStarted","Data":"8c2fda42837807c71d2e1d6969eaf6df339c1781989f8c4376e2430afc419be6"}
Oct 30 19:38:41.899147 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:38:41.899018    4892 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2375738-tbbvw" event={"ID":"06d9238c-da95-4e55-b07d-d95dae4f2b29","Type":"ContainerStarted","Data":"7cdcdc46b5199cdbbebf36ace603e012e5297a9491c322e23ab6ed2dbdbfcd3c"}
Oct 30 19:38:41.945840 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:38:41.945814    4892 patch_prober.go:29] interesting pod/e2e-worker-w5-686689-openshift-default-5fbf5787bf-jrt7h container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.80:15021/healthz/ready\": dial tcp 10.132.0.80:15021: connect: connection refused" start-of-body=
Oct 30 19:38:41.945997 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:38:41.945875    4892 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-686689-openshift-default-5fbf5787bf-jrt7h" podUID="fa668020-72be-46a7-aa53-e8249130c4f1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.80:15021/healthz/ready\": dial tcp 10.132.0.80:15021: connect: connection refused"
Oct 30 19:38:42.053927 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:38:42.053852    4892 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2375738-tbbvw" podStartSLOduration=2.053827506 podStartE2EDuration="2.053827506s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:38:40 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:38:42.012522151 +0000 UTC m=+1369.582987751" watchObservedRunningTime="2026-10-30 19:38:42.053827506 +0000 UTC m=+1369.624293106"
Oct 30 19:38:42.906342 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:38:42.906310    4892 generic.go:356] "Generic (PLEG): container finished" podID="06d9238c-da95-4e55-b07d-d95dae4f2b29" containerID="8c2fda42837807c71d2e1d6969eaf6df339c1781989f8c4376e2430afc419be6" exitCode=0
Oct 30 19:38:42.906692 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:38:42.906401    4892 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2375738-tbbvw" event={"ID":"06d9238c-da95-4e55-b07d-d95dae4f2b29","Type":"ContainerDied","Data":"8c2fda42837807c71d2e1d6969eaf6df339c1781989f8c4376e2430afc419be6"}
Oct 30 19:38:42.949671 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:38:42.949643    4892 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w5-686689-openshift-default-5fbf5787bf-jrt7h"
Oct 30 19:38:42.949930 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:38:42.949908    4892 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w5-686689-openshift-default-5fbf5787bf-jrt7h"
Oct 30 19:38:42.950637 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:38:42.950622    4892 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w5-686689-openshift-default-5fbf5787bf-jrt7h"
Oct 30 19:38:44.060052 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:38:44.060029    4892 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2375738-tbbvw"
Oct 30 19:38:44.212915 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:38:44.212842    4892 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/06d9238c-da95-4e55-b07d-d95dae4f2b29-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/06d9238c-da95-4e55-b07d-d95dae4f2b29-maas-api-xks-ca\") pod \"06d9238c-da95-4e55-b07d-d95dae4f2b29\" (UID: \"06d9238c-da95-4e55-b07d-d95dae4f2b29\") "
Oct 30 19:38:44.213099 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:38:44.212917    4892 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/06d9238c-da95-4e55-b07d-d95dae4f2b29-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/06d9238c-da95-4e55-b07d-d95dae4f2b29-maas-api-openshift-ca\") pod \"06d9238c-da95-4e55-b07d-d95dae4f2b29\" (UID: \"06d9238c-da95-4e55-b07d-d95dae4f2b29\") "
Oct 30 19:38:44.213099 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:38:44.213032    4892 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/06d9238c-da95-4e55-b07d-d95dae4f2b29-kube-api-access-gbr6h\" (UniqueName: \"kubernetes.io/projected/06d9238c-da95-4e55-b07d-d95dae4f2b29-kube-api-access-gbr6h\") pod \"06d9238c-da95-4e55-b07d-d95dae4f2b29\" (UID: \"06d9238c-da95-4e55-b07d-d95dae4f2b29\") "
Oct 30 19:38:44.213336 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:38:44.213313    4892 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/06d9238c-da95-4e55-b07d-d95dae4f2b29-maas-api-openshift-ca" pod "06d9238c-da95-4e55-b07d-d95dae4f2b29" (UID: "06d9238c-da95-4e55-b07d-d95dae4f2b29"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 19:38:44.213447 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:38:44.213424    4892 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/06d9238c-da95-4e55-b07d-d95dae4f2b29-maas-api-openshift-ca\") on node \"ip-10-0-132-82.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 19:38:44.215241 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:38:44.215219    4892 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/06d9238c-da95-4e55-b07d-d95dae4f2b29-kube-api-access-gbr6h" pod "06d9238c-da95-4e55-b07d-d95dae4f2b29" (UID: "06d9238c-da95-4e55-b07d-d95dae4f2b29"). InnerVolumeSpecName "kube-api-access-gbr6h". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:38:44.215332 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:38:44.215260    4892 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/06d9238c-da95-4e55-b07d-d95dae4f2b29-maas-api-xks-ca" pod "06d9238c-da95-4e55-b07d-d95dae4f2b29" (UID: "06d9238c-da95-4e55-b07d-d95dae4f2b29"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 19:38:44.314498 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:38:44.314475    4892 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gbr6h\" (UniqueName: \"kubernetes.io/projected/06d9238c-da95-4e55-b07d-d95dae4f2b29-kube-api-access-gbr6h\") on node \"ip-10-0-132-82.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 19:38:44.314498 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:38:44.314499    4892 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/06d9238c-da95-4e55-b07d-d95dae4f2b29-maas-api-xks-ca\") on node \"ip-10-0-132-82.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 19:38:44.922619 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:38:44.922586    4892 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2375738-tbbvw"
Oct 30 19:38:44.922811 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:38:44.922623    4892 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2375738-tbbvw" event={"ID":"06d9238c-da95-4e55-b07d-d95dae4f2b29","Type":"ContainerDied","Data":"7cdcdc46b5199cdbbebf36ace603e012e5297a9491c322e23ab6ed2dbdbfcd3c"}
Oct 30 19:38:44.922811 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:38:44.922683    4892 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="7cdcdc46b5199cdbbebf36ace603e012e5297a9491c322e23ab6ed2dbdbfcd3c"
Oct 30 19:38:54.844612 ip-10-0-132-82 kubenswrapper[4892]: E1030 19:38:54.844571    4892 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-jrm87" containerName="extractor"
Oct 30 19:38:58.160158 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:38:58.160126    4892 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w1-d52df0/distinct-2-w1-d52df0-kserve-7846558464-xrkdc"]
Oct 30 19:38:58.160907 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:38:58.160745    4892 cpu_manager.go:414] "containerMap: removing container" podUID="aee774c4-0eaf-4fa5-bec7-5b371c039f4a" containerName="revoke-keys"
Oct 30 19:38:58.160907 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:38:58.160769    4892 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="aee774c4-0eaf-4fa5-bec7-5b371c039f4a" containerName="revoke-keys"
Oct 30 19:38:58.160907 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:38:58.160783    4892 cpu_manager.go:414] "containerMap: removing container" podUID="06d9238c-da95-4e55-b07d-d95dae4f2b29" containerName="revoke-keys"
Oct 30 19:38:58.160907 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:38:58.160792    4892 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="06d9238c-da95-4e55-b07d-d95dae4f2b29" containerName="revoke-keys"
Oct 30 19:38:58.161217 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:38:58.160933    4892 memory_manager.go:365] "RemoveStaleState removing state" podUID="aee774c4-0eaf-4fa5-bec7-5b371c039f4a" containerName="revoke-keys"
Oct 30 19:38:58.161217 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:38:58.160946    4892 memory_manager.go:365] "RemoveStaleState removing state" podUID="06d9238c-da95-4e55-b07d-d95dae4f2b29" containerName="revoke-keys"
Oct 30 19:38:58.165657 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:38:58.165638    4892 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w1-d52df0/distinct-2-w1-d52df0-kserve-7846558464-xrkdc"
Oct 30 19:38:58.266916 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:38:58.266889    4892 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w1-d52df0\"/\"distinct-2-w1-d52df0-kserve-self-signed-certs\""
Oct 30 19:38:58.270184 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:38:58.270164    4892 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w1-d52df0/distinct-2-w1-d52df0-kserve-7846558464-xrkdc"]
Oct 30 19:38:58.293105 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:38:58.293083    4892 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hw5fw\" (UniqueName: \"kubernetes.io/projected/9c614ab6-30e2-4fe1-8609-d5e2dadc198a-kube-api-access-hw5fw\") pod \"distinct-2-w1-d52df0-kserve-7846558464-xrkdc\" (UID: \"9c614ab6-30e2-4fe1-8609-d5e2dadc198a\") " pod="e2e-models-e2e-worker-w1-d52df0/distinct-2-w1-d52df0-kserve-7846558464-xrkdc"
Oct 30 19:38:58.293219 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:38:58.293113    4892 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/9c614ab6-30e2-4fe1-8609-d5e2dadc198a-dshm\") pod \"distinct-2-w1-d52df0-kserve-7846558464-xrkdc\" (UID: \"9c614ab6-30e2-4fe1-8609-d5e2dadc198a\") " pod="e2e-models-e2e-worker-w1-d52df0/distinct-2-w1-d52df0-kserve-7846558464-xrkdc"
Oct 30 19:38:58.293219 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:38:58.293133    4892 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9c614ab6-30e2-4fe1-8609-d5e2dadc198a-tmp-dir\") pod \"distinct-2-w1-d52df0-kserve-7846558464-xrkdc\" (UID: \"9c614ab6-30e2-4fe1-8609-d5e2dadc198a\") " pod="e2e-models-e2e-worker-w1-d52df0/distinct-2-w1-d52df0-kserve-7846558464-xrkdc"
Oct 30 19:38:58.293347 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:38:58.293280    4892 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/9c614ab6-30e2-4fe1-8609-d5e2dadc198a-model-cache\") pod \"distinct-2-w1-d52df0-kserve-7846558464-xrkdc\" (UID: \"9c614ab6-30e2-4fe1-8609-d5e2dadc198a\") " pod="e2e-models-e2e-worker-w1-d52df0/distinct-2-w1-d52df0-kserve-7846558464-xrkdc"
Oct 30 19:38:58.293401 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:38:58.293341    4892 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/9c614ab6-30e2-4fe1-8609-d5e2dadc198a-tls-certs\") pod \"distinct-2-w1-d52df0-kserve-7846558464-xrkdc\" (UID: \"9c614ab6-30e2-4fe1-8609-d5e2dadc198a\") " pod="e2e-models-e2e-worker-w1-d52df0/distinct-2-w1-d52df0-kserve-7846558464-xrkdc"
Oct 30 19:38:58.293461 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:38:58.293445    4892 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/9c614ab6-30e2-4fe1-8609-d5e2dadc198a-home\") pod \"distinct-2-w1-d52df0-kserve-7846558464-xrkdc\" (UID: \"9c614ab6-30e2-4fe1-8609-d5e2dadc198a\") " pod="e2e-models-e2e-worker-w1-d52df0/distinct-2-w1-d52df0-kserve-7846558464-xrkdc"
Oct 30 19:38:58.394578 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:38:58.394541    4892 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hw5fw\" (UniqueName: \"kubernetes.io/projected/9c614ab6-30e2-4fe1-8609-d5e2dadc198a-kube-api-access-hw5fw\") pod \"distinct-2-w1-d52df0-kserve-7846558464-xrkdc\" (UID: \"9c614ab6-30e2-4fe1-8609-d5e2dadc198a\") " pod="e2e-models-e2e-worker-w1-d52df0/distinct-2-w1-d52df0-kserve-7846558464-xrkdc"
Oct 30 19:38:58.394578 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:38:58.394576    4892 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/9c614ab6-30e2-4fe1-8609-d5e2dadc198a-dshm\") pod \"distinct-2-w1-d52df0-kserve-7846558464-xrkdc\" (UID: \"9c614ab6-30e2-4fe1-8609-d5e2dadc198a\") " pod="e2e-models-e2e-worker-w1-d52df0/distinct-2-w1-d52df0-kserve-7846558464-xrkdc"
Oct 30 19:38:58.394836 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:38:58.394596    4892 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9c614ab6-30e2-4fe1-8609-d5e2dadc198a-tmp-dir\") pod \"distinct-2-w1-d52df0-kserve-7846558464-xrkdc\" (UID: \"9c614ab6-30e2-4fe1-8609-d5e2dadc198a\") " pod="e2e-models-e2e-worker-w1-d52df0/distinct-2-w1-d52df0-kserve-7846558464-xrkdc"
Oct 30 19:38:58.394836 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:38:58.394658    4892 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/9c614ab6-30e2-4fe1-8609-d5e2dadc198a-model-cache\") pod \"distinct-2-w1-d52df0-kserve-7846558464-xrkdc\" (UID: \"9c614ab6-30e2-4fe1-8609-d5e2dadc198a\") " pod="e2e-models-e2e-worker-w1-d52df0/distinct-2-w1-d52df0-kserve-7846558464-xrkdc"
Oct 30 19:38:58.394836 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:38:58.394705    4892 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/9c614ab6-30e2-4fe1-8609-d5e2dadc198a-tls-certs\") pod \"distinct-2-w1-d52df0-kserve-7846558464-xrkdc\" (UID: \"9c614ab6-30e2-4fe1-8609-d5e2dadc198a\") " pod="e2e-models-e2e-worker-w1-d52df0/distinct-2-w1-d52df0-kserve-7846558464-xrkdc"
Oct 30 19:38:58.394836 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:38:58.394784    4892 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/9c614ab6-30e2-4fe1-8609-d5e2dadc198a-home\") pod \"distinct-2-w1-d52df0-kserve-7846558464-xrkdc\" (UID: \"9c614ab6-30e2-4fe1-8609-d5e2dadc198a\") " pod="e2e-models-e2e-worker-w1-d52df0/distinct-2-w1-d52df0-kserve-7846558464-xrkdc"
Oct 30 19:38:58.395140 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:38:58.395115    4892 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9c614ab6-30e2-4fe1-8609-d5e2dadc198a-tmp-dir\") pod \"distinct-2-w1-d52df0-kserve-7846558464-xrkdc\" (UID: \"9c614ab6-30e2-4fe1-8609-d5e2dadc198a\") " pod="e2e-models-e2e-worker-w1-d52df0/distinct-2-w1-d52df0-kserve-7846558464-xrkdc"
Oct 30 19:38:58.395206 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:38:58.395127    4892 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/9c614ab6-30e2-4fe1-8609-d5e2dadc198a-home\") pod \"distinct-2-w1-d52df0-kserve-7846558464-xrkdc\" (UID: \"9c614ab6-30e2-4fe1-8609-d5e2dadc198a\") " pod="e2e-models-e2e-worker-w1-d52df0/distinct-2-w1-d52df0-kserve-7846558464-xrkdc"
Oct 30 19:38:58.395297 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:38:58.395280    4892 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/9c614ab6-30e2-4fe1-8609-d5e2dadc198a-model-cache\") pod \"distinct-2-w1-d52df0-kserve-7846558464-xrkdc\" (UID: \"9c614ab6-30e2-4fe1-8609-d5e2dadc198a\") " pod="e2e-models-e2e-worker-w1-d52df0/distinct-2-w1-d52df0-kserve-7846558464-xrkdc"
Oct 30 19:38:58.397101 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:38:58.397071    4892 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/9c614ab6-30e2-4fe1-8609-d5e2dadc198a-dshm\") pod \"distinct-2-w1-d52df0-kserve-7846558464-xrkdc\" (UID: \"9c614ab6-30e2-4fe1-8609-d5e2dadc198a\") " pod="e2e-models-e2e-worker-w1-d52df0/distinct-2-w1-d52df0-kserve-7846558464-xrkdc"
Oct 30 19:38:58.397339 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:38:58.397320    4892 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/9c614ab6-30e2-4fe1-8609-d5e2dadc198a-tls-certs\") pod \"distinct-2-w1-d52df0-kserve-7846558464-xrkdc\" (UID: \"9c614ab6-30e2-4fe1-8609-d5e2dadc198a\") " pod="e2e-models-e2e-worker-w1-d52df0/distinct-2-w1-d52df0-kserve-7846558464-xrkdc"
Oct 30 19:38:58.451709 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:38:58.451625    4892 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hw5fw\" (UniqueName: \"kubernetes.io/projected/9c614ab6-30e2-4fe1-8609-d5e2dadc198a-kube-api-access-hw5fw\") pod \"distinct-2-w1-d52df0-kserve-7846558464-xrkdc\" (UID: \"9c614ab6-30e2-4fe1-8609-d5e2dadc198a\") " pod="e2e-models-e2e-worker-w1-d52df0/distinct-2-w1-d52df0-kserve-7846558464-xrkdc"
Oct 30 19:38:58.478770 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:38:58.478744    4892 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w1-d52df0/distinct-2-w1-d52df0-kserve-7846558464-xrkdc"
Oct 30 19:38:58.901113 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:38:58.901085    4892 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w1-d52df0/distinct-2-w1-d52df0-kserve-7846558464-xrkdc"]
Oct 30 19:38:58.902257 ip-10-0-132-82 kubenswrapper[4892]: W1030 19:38:58.902224    4892 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod9c614ab6_30e2_4fe1_8609_d5e2dadc198a.slice/crio-1bab29e42ab285badb43d510b55ed3f68d2ae2b109925a479c8d5ad9c3828d6c WatchSource:0}: Error finding container 1bab29e42ab285badb43d510b55ed3f68d2ae2b109925a479c8d5ad9c3828d6c: Status 404 returned error can't find the container with id 1bab29e42ab285badb43d510b55ed3f68d2ae2b109925a479c8d5ad9c3828d6c
Oct 30 19:38:59.023132 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:38:59.023097    4892 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w1-d52df0/distinct-2-w1-d52df0-kserve-7846558464-xrkdc" event={"ID":"9c614ab6-30e2-4fe1-8609-d5e2dadc198a","Type":"ContainerStarted","Data":"b228e7b15dedc2b43fcd3cfd5c1fadc04c9de4564d3fe81f8d34d8831242a1a9"}
Oct 30 19:38:59.023296 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:38:59.023150    4892 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w1-d52df0/distinct-2-w1-d52df0-kserve-7846558464-xrkdc" event={"ID":"9c614ab6-30e2-4fe1-8609-d5e2dadc198a","Type":"ContainerStarted","Data":"1bab29e42ab285badb43d510b55ed3f68d2ae2b109925a479c8d5ad9c3828d6c"}
Oct 30 19:38:59.139591 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:38:59.139541    4892 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w1-d52df0/distinct-2-w1-d52df0-kserve-7846558464-xrkdc" podStartSLOduration=1.139526953 podStartE2EDuration="1.139526953s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:38:58 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:38:59.135955141 +0000 UTC m=+1386.706420769" watchObservedRunningTime="2026-10-30 19:38:59.139526953 +0000 UTC m=+1386.709992552"
Oct 30 19:38:59.479623 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:38:59.479597    4892 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-d52df0/distinct-2-w1-d52df0-kserve-7846558464-xrkdc"
Oct 30 19:39:05.158143 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:39:05.158056    4892 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c4294e09-5b4hh"]
Oct 30 19:39:05.162011 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:39:05.161995    4892 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c4294e09-5b4hh"
Oct 30 19:39:05.213730 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:39:05.213711    4892 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 19:39:05.213953 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:39:05.213933    4892 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-nlxp4\""
Oct 30 19:39:05.263390 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:39:05.263343    4892 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 30 19:39:05.263390 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:39:05.263367    4892 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 30 19:39:05.267391 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:39:05.267368    4892 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c4294e09-5b4hh"]
Oct 30 19:39:05.272943 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:39:05.272922    4892 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/59b8e731-59dc-4dd9-a71c-04425e64db5b-maas-api-xks-ca\") pod \"maas-api-revoke-sub-c4294e09-5b4hh\" (UID: \"59b8e731-59dc-4dd9-a71c-04425e64db5b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c4294e09-5b4hh"
Oct 30 19:39:05.273088 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:39:05.273069    4892 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ngx4g\" (UniqueName: \"kubernetes.io/projected/59b8e731-59dc-4dd9-a71c-04425e64db5b-kube-api-access-ngx4g\") pod \"maas-api-revoke-sub-c4294e09-5b4hh\" (UID: \"59b8e731-59dc-4dd9-a71c-04425e64db5b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c4294e09-5b4hh"
Oct 30 19:39:05.273188 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:39:05.273175    4892 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/59b8e731-59dc-4dd9-a71c-04425e64db5b-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-c4294e09-5b4hh\" (UID: \"59b8e731-59dc-4dd9-a71c-04425e64db5b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c4294e09-5b4hh"
Oct 30 19:39:05.374653 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:39:05.374623    4892 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/59b8e731-59dc-4dd9-a71c-04425e64db5b-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-c4294e09-5b4hh\" (UID: \"59b8e731-59dc-4dd9-a71c-04425e64db5b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c4294e09-5b4hh"
Oct 30 19:39:05.374803 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:39:05.374692    4892 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/59b8e731-59dc-4dd9-a71c-04425e64db5b-maas-api-xks-ca\") pod \"maas-api-revoke-sub-c4294e09-5b4hh\" (UID: \"59b8e731-59dc-4dd9-a71c-04425e64db5b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c4294e09-5b4hh"
Oct 30 19:39:05.374803 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:39:05.374769    4892 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ngx4g\" (UniqueName: \"kubernetes.io/projected/59b8e731-59dc-4dd9-a71c-04425e64db5b-kube-api-access-ngx4g\") pod \"maas-api-revoke-sub-c4294e09-5b4hh\" (UID: \"59b8e731-59dc-4dd9-a71c-04425e64db5b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c4294e09-5b4hh"
Oct 30 19:39:05.375342 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:39:05.375317    4892 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/59b8e731-59dc-4dd9-a71c-04425e64db5b-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-c4294e09-5b4hh\" (UID: \"59b8e731-59dc-4dd9-a71c-04425e64db5b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c4294e09-5b4hh"
Oct 30 19:39:05.377312 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:39:05.377294    4892 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/59b8e731-59dc-4dd9-a71c-04425e64db5b-maas-api-xks-ca\") pod \"maas-api-revoke-sub-c4294e09-5b4hh\" (UID: \"59b8e731-59dc-4dd9-a71c-04425e64db5b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c4294e09-5b4hh"
Oct 30 19:39:05.430884 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:39:05.430831    4892 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ngx4g\" (UniqueName: \"kubernetes.io/projected/59b8e731-59dc-4dd9-a71c-04425e64db5b-kube-api-access-ngx4g\") pod \"maas-api-revoke-sub-c4294e09-5b4hh\" (UID: \"59b8e731-59dc-4dd9-a71c-04425e64db5b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c4294e09-5b4hh"
Oct 30 19:39:05.479670 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:39:05.479647    4892 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c4294e09-5b4hh"
Oct 30 19:39:05.856751 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:39:05.856723    4892 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c4294e09-5b4hh"]
Oct 30 19:39:05.857290 ip-10-0-132-82 kubenswrapper[4892]: W1030 19:39:05.857268    4892 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod59b8e731_59dc_4dd9_a71c_04425e64db5b.slice/crio-6387953d62e7c52358ebe424ff403039a18c95ddb8b2068452a4a509f50a7568 WatchSource:0}: Error finding container 6387953d62e7c52358ebe424ff403039a18c95ddb8b2068452a4a509f50a7568: Status 404 returned error can't find the container with id 6387953d62e7c52358ebe424ff403039a18c95ddb8b2068452a4a509f50a7568
Oct 30 19:39:06.062654 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:39:06.062620    4892 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-fsfvk"]
Oct 30 19:39:06.087787 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:39:06.087756    4892 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c4294e09-5b4hh" event={"ID":"59b8e731-59dc-4dd9-a71c-04425e64db5b","Type":"ContainerStarted","Data":"6387953d62e7c52358ebe424ff403039a18c95ddb8b2068452a4a509f50a7568"}
Oct 30 19:39:07.107384 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:39:07.107307    4892 generic.go:356] "Generic (PLEG): container finished" podID="59b8e731-59dc-4dd9-a71c-04425e64db5b" containerID="58655ebd49fd1ebfde56ed164df82683d25afb542ea68bf6d233a7587e6d15dd" exitCode=0
Oct 30 19:39:07.107865 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:39:07.107845    4892 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c4294e09-5b4hh" event={"ID":"59b8e731-59dc-4dd9-a71c-04425e64db5b","Type":"ContainerDied","Data":"58655ebd49fd1ebfde56ed164df82683d25afb542ea68bf6d233a7587e6d15dd"}
Oct 30 19:39:08.294186 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:39:08.294160    4892 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c4294e09-5b4hh"
Oct 30 19:39:08.412778 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:39:08.412750    4892 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/59b8e731-59dc-4dd9-a71c-04425e64db5b-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/59b8e731-59dc-4dd9-a71c-04425e64db5b-maas-api-openshift-ca\") pod \"59b8e731-59dc-4dd9-a71c-04425e64db5b\" (UID: \"59b8e731-59dc-4dd9-a71c-04425e64db5b\") "
Oct 30 19:39:08.412941 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:39:08.412868    4892 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/59b8e731-59dc-4dd9-a71c-04425e64db5b-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/59b8e731-59dc-4dd9-a71c-04425e64db5b-maas-api-xks-ca\") pod \"59b8e731-59dc-4dd9-a71c-04425e64db5b\" (UID: \"59b8e731-59dc-4dd9-a71c-04425e64db5b\") "
Oct 30 19:39:08.412941 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:39:08.412898    4892 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/59b8e731-59dc-4dd9-a71c-04425e64db5b-kube-api-access-ngx4g\" (UniqueName: \"kubernetes.io/projected/59b8e731-59dc-4dd9-a71c-04425e64db5b-kube-api-access-ngx4g\") pod \"59b8e731-59dc-4dd9-a71c-04425e64db5b\" (UID: \"59b8e731-59dc-4dd9-a71c-04425e64db5b\") "
Oct 30 19:39:08.413163 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:39:08.413138    4892 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/59b8e731-59dc-4dd9-a71c-04425e64db5b-maas-api-openshift-ca" pod "59b8e731-59dc-4dd9-a71c-04425e64db5b" (UID: "59b8e731-59dc-4dd9-a71c-04425e64db5b"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 19:39:08.413377 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:39:08.413353    4892 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/59b8e731-59dc-4dd9-a71c-04425e64db5b-maas-api-openshift-ca\") on node \"ip-10-0-132-82.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 19:39:08.416650 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:39:08.415324    4892 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/59b8e731-59dc-4dd9-a71c-04425e64db5b-kube-api-access-ngx4g" pod "59b8e731-59dc-4dd9-a71c-04425e64db5b" (UID: "59b8e731-59dc-4dd9-a71c-04425e64db5b"). InnerVolumeSpecName "kube-api-access-ngx4g". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:39:08.416650 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:39:08.415620    4892 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/59b8e731-59dc-4dd9-a71c-04425e64db5b-maas-api-xks-ca" pod "59b8e731-59dc-4dd9-a71c-04425e64db5b" (UID: "59b8e731-59dc-4dd9-a71c-04425e64db5b"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 19:39:08.479529 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:39:08.479499    4892 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w1-d52df0/distinct-2-w1-d52df0-kserve-7846558464-xrkdc"
Oct 30 19:39:08.491651 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:39:08.491627    4892 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w1-d52df0/distinct-2-w1-d52df0-kserve-7846558464-xrkdc"
Oct 30 19:39:08.514370 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:39:08.514348    4892 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/59b8e731-59dc-4dd9-a71c-04425e64db5b-maas-api-xks-ca\") on node \"ip-10-0-132-82.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 19:39:08.514370 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:39:08.514371    4892 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ngx4g\" (UniqueName: \"kubernetes.io/projected/59b8e731-59dc-4dd9-a71c-04425e64db5b-kube-api-access-ngx4g\") on node \"ip-10-0-132-82.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 19:39:09.128665 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:39:09.128626    4892 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c4294e09-5b4hh" event={"ID":"59b8e731-59dc-4dd9-a71c-04425e64db5b","Type":"ContainerDied","Data":"6387953d62e7c52358ebe424ff403039a18c95ddb8b2068452a4a509f50a7568"}
Oct 30 19:39:09.128846 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:39:09.128683    4892 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="6387953d62e7c52358ebe424ff403039a18c95ddb8b2068452a4a509f50a7568"
Oct 30 19:39:09.128846 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:39:09.128793    4892 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c4294e09-5b4hh"
Oct 30 19:39:09.144465 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:39:09.144441    4892 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-d52df0/distinct-2-w1-d52df0-kserve-7846558464-xrkdc"
Oct 30 19:39:11.268352 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:39:11.268316    4892 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-32855537-vqhss"]
Oct 30 19:39:11.269596 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:39:11.269565    4892 cpu_manager.go:414] "containerMap: removing container" podUID="59b8e731-59dc-4dd9-a71c-04425e64db5b" containerName="revoke-keys"
Oct 30 19:39:11.269596 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:39:11.269594    4892 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="59b8e731-59dc-4dd9-a71c-04425e64db5b" containerName="revoke-keys"
Oct 30 19:39:11.269747 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:39:11.269721    4892 memory_manager.go:365] "RemoveStaleState removing state" podUID="59b8e731-59dc-4dd9-a71c-04425e64db5b" containerName="revoke-keys"
Oct 30 19:39:11.273728 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:39:11.273709    4892 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-32855537-vqhss"
Oct 30 19:39:11.332045 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:39:11.332016    4892 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-32855537-vqhss"]
Oct 30 19:39:11.352943 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:39:11.352909    4892 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5bbe30fd-2c9a-475d-976f-77a7e8c41401-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-32855537-vqhss\" (UID: \"5bbe30fd-2c9a-475d-976f-77a7e8c41401\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-32855537-vqhss"
Oct 30 19:39:11.353099 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:39:11.352995    4892 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5bbe30fd-2c9a-475d-976f-77a7e8c41401-maas-api-xks-ca\") pod \"maas-api-revoke-sub-32855537-vqhss\" (UID: \"5bbe30fd-2c9a-475d-976f-77a7e8c41401\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-32855537-vqhss"
Oct 30 19:39:11.353151 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:39:11.353123    4892 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rpwnl\" (UniqueName: \"kubernetes.io/projected/5bbe30fd-2c9a-475d-976f-77a7e8c41401-kube-api-access-rpwnl\") pod \"maas-api-revoke-sub-32855537-vqhss\" (UID: \"5bbe30fd-2c9a-475d-976f-77a7e8c41401\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-32855537-vqhss"
Oct 30 19:39:11.374753 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:39:11.374720    4892 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 19:39:11.375040 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:39:11.375021    4892 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-nlxp4\""
Oct 30 19:39:11.375159 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:39:11.375050    4892 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 30 19:39:11.375289 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:39:11.375271    4892 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 30 19:39:11.454518 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:39:11.454492    4892 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rpwnl\" (UniqueName: \"kubernetes.io/projected/5bbe30fd-2c9a-475d-976f-77a7e8c41401-kube-api-access-rpwnl\") pod \"maas-api-revoke-sub-32855537-vqhss\" (UID: \"5bbe30fd-2c9a-475d-976f-77a7e8c41401\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-32855537-vqhss"
Oct 30 19:39:11.454641 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:39:11.454594    4892 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5bbe30fd-2c9a-475d-976f-77a7e8c41401-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-32855537-vqhss\" (UID: \"5bbe30fd-2c9a-475d-976f-77a7e8c41401\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-32855537-vqhss"
Oct 30 19:39:11.454641 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:39:11.454622    4892 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5bbe30fd-2c9a-475d-976f-77a7e8c41401-maas-api-xks-ca\") pod \"maas-api-revoke-sub-32855537-vqhss\" (UID: \"5bbe30fd-2c9a-475d-976f-77a7e8c41401\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-32855537-vqhss"
Oct 30 19:39:11.455420 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:39:11.455393    4892 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5bbe30fd-2c9a-475d-976f-77a7e8c41401-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-32855537-vqhss\" (UID: \"5bbe30fd-2c9a-475d-976f-77a7e8c41401\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-32855537-vqhss"
Oct 30 19:39:11.457294 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:39:11.457270    4892 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5bbe30fd-2c9a-475d-976f-77a7e8c41401-maas-api-xks-ca\") pod \"maas-api-revoke-sub-32855537-vqhss\" (UID: \"5bbe30fd-2c9a-475d-976f-77a7e8c41401\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-32855537-vqhss"
Oct 30 19:39:11.511871 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:39:11.511841    4892 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rpwnl\" (UniqueName: \"kubernetes.io/projected/5bbe30fd-2c9a-475d-976f-77a7e8c41401-kube-api-access-rpwnl\") pod \"maas-api-revoke-sub-32855537-vqhss\" (UID: \"5bbe30fd-2c9a-475d-976f-77a7e8c41401\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-32855537-vqhss"
Oct 30 19:39:11.589366 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:39:11.589305    4892 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-32855537-vqhss"
Oct 30 19:39:11.966536 ip-10-0-132-82 kubenswrapper[4892]: W1030 19:39:11.966509    4892 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5bbe30fd_2c9a_475d_976f_77a7e8c41401.slice/crio-dad085dbaf9de5d19647703a3b14488fed2676b2f7b055e2c3df583bcf3e91f6 WatchSource:0}: Error finding container dad085dbaf9de5d19647703a3b14488fed2676b2f7b055e2c3df583bcf3e91f6: Status 404 returned error can't find the container with id dad085dbaf9de5d19647703a3b14488fed2676b2f7b055e2c3df583bcf3e91f6
Oct 30 19:39:11.967598 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:39:11.967571    4892 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-32855537-vqhss"]
Oct 30 19:39:12.180649 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:39:12.180614    4892 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-32855537-vqhss" event={"ID":"5bbe30fd-2c9a-475d-976f-77a7e8c41401","Type":"ContainerStarted","Data":"dad085dbaf9de5d19647703a3b14488fed2676b2f7b055e2c3df583bcf3e91f6"}
Oct 30 19:39:13.188415 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:39:13.188383    4892 generic.go:356] "Generic (PLEG): container finished" podID="5bbe30fd-2c9a-475d-976f-77a7e8c41401" containerID="6adb9a485da8f09c3db161aca29c7746a2413878183d349991a17b2e2221f027" exitCode=0
Oct 30 19:39:13.188796 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:39:13.188450    4892 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-32855537-vqhss" event={"ID":"5bbe30fd-2c9a-475d-976f-77a7e8c41401","Type":"ContainerDied","Data":"6adb9a485da8f09c3db161aca29c7746a2413878183d349991a17b2e2221f027"}
Oct 30 19:39:14.378463 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:39:14.378431    4892 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-32855537-vqhss"
Oct 30 19:39:14.492250 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:39:14.492164    4892 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5bbe30fd-2c9a-475d-976f-77a7e8c41401-kube-api-access-rpwnl\" (UniqueName: \"kubernetes.io/projected/5bbe30fd-2c9a-475d-976f-77a7e8c41401-kube-api-access-rpwnl\") pod \"5bbe30fd-2c9a-475d-976f-77a7e8c41401\" (UID: \"5bbe30fd-2c9a-475d-976f-77a7e8c41401\") "
Oct 30 19:39:14.492415 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:39:14.492256    4892 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/5bbe30fd-2c9a-475d-976f-77a7e8c41401-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5bbe30fd-2c9a-475d-976f-77a7e8c41401-maas-api-openshift-ca\") pod \"5bbe30fd-2c9a-475d-976f-77a7e8c41401\" (UID: \"5bbe30fd-2c9a-475d-976f-77a7e8c41401\") "
Oct 30 19:39:14.492415 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:39:14.492347    4892 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/5bbe30fd-2c9a-475d-976f-77a7e8c41401-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5bbe30fd-2c9a-475d-976f-77a7e8c41401-maas-api-xks-ca\") pod \"5bbe30fd-2c9a-475d-976f-77a7e8c41401\" (UID: \"5bbe30fd-2c9a-475d-976f-77a7e8c41401\") "
Oct 30 19:39:14.492870 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:39:14.492623    4892 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5bbe30fd-2c9a-475d-976f-77a7e8c41401-maas-api-openshift-ca" pod "5bbe30fd-2c9a-475d-976f-77a7e8c41401" (UID: "5bbe30fd-2c9a-475d-976f-77a7e8c41401"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 19:39:14.493601 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:39:14.493571    4892 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5bbe30fd-2c9a-475d-976f-77a7e8c41401-maas-api-openshift-ca\") on node \"ip-10-0-132-82.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 19:39:14.494905 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:39:14.494884    4892 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5bbe30fd-2c9a-475d-976f-77a7e8c41401-kube-api-access-rpwnl" pod "5bbe30fd-2c9a-475d-976f-77a7e8c41401" (UID: "5bbe30fd-2c9a-475d-976f-77a7e8c41401"). InnerVolumeSpecName "kube-api-access-rpwnl". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:39:14.495145 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:39:14.495116    4892 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5bbe30fd-2c9a-475d-976f-77a7e8c41401-maas-api-xks-ca" pod "5bbe30fd-2c9a-475d-976f-77a7e8c41401" (UID: "5bbe30fd-2c9a-475d-976f-77a7e8c41401"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 19:39:14.594134 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:39:14.594103    4892 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rpwnl\" (UniqueName: \"kubernetes.io/projected/5bbe30fd-2c9a-475d-976f-77a7e8c41401-kube-api-access-rpwnl\") on node \"ip-10-0-132-82.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 19:39:14.594134 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:39:14.594131    4892 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5bbe30fd-2c9a-475d-976f-77a7e8c41401-maas-api-xks-ca\") on node \"ip-10-0-132-82.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 19:39:15.202057 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:39:15.202016    4892 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-32855537-vqhss" event={"ID":"5bbe30fd-2c9a-475d-976f-77a7e8c41401","Type":"ContainerDied","Data":"dad085dbaf9de5d19647703a3b14488fed2676b2f7b055e2c3df583bcf3e91f6"}
Oct 30 19:39:15.202057 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:39:15.202044    4892 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-32855537-vqhss"
Oct 30 19:39:15.202292 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:39:15.202094    4892 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="dad085dbaf9de5d19647703a3b14488fed2676b2f7b055e2c3df583bcf3e91f6"
Oct 30 19:39:21.950903 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:39:21.950869    4892 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-6517618f-nph8j"]
Oct 30 19:39:21.951375 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:39:21.951361    4892 cpu_manager.go:414] "containerMap: removing container" podUID="5bbe30fd-2c9a-475d-976f-77a7e8c41401" containerName="revoke-keys"
Oct 30 19:39:21.951419 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:39:21.951377    4892 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5bbe30fd-2c9a-475d-976f-77a7e8c41401" containerName="revoke-keys"
Oct 30 19:39:21.951464 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:39:21.951455    4892 memory_manager.go:365] "RemoveStaleState removing state" podUID="5bbe30fd-2c9a-475d-976f-77a7e8c41401" containerName="revoke-keys"
Oct 30 19:39:21.954873 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:39:21.954858    4892 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6517618f-nph8j"
Oct 30 19:39:22.056952 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:39:22.056911    4892 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 19:39:22.056952 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:39:22.056931    4892 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-nlxp4\""
Oct 30 19:39:22.057177 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:39:22.056956    4892 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 30 19:39:22.057177 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:39:22.057012    4892 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 30 19:39:22.060597 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:39:22.060574    4892 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ncf9l\" (UniqueName: \"kubernetes.io/projected/ad5c24d2-e4cc-4737-aab6-bf77769262e5-kube-api-access-ncf9l\") pod \"maas-api-revoke-sub-6517618f-nph8j\" (UID: \"ad5c24d2-e4cc-4737-aab6-bf77769262e5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6517618f-nph8j"
Oct 30 19:39:22.060706 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:39:22.060620    4892 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-6517618f-nph8j"]
Oct 30 19:39:22.060706 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:39:22.060637    4892 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ad5c24d2-e4cc-4737-aab6-bf77769262e5-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-6517618f-nph8j\" (UID: \"ad5c24d2-e4cc-4737-aab6-bf77769262e5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6517618f-nph8j"
Oct 30 19:39:22.060779 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:39:22.060768    4892 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ad5c24d2-e4cc-4737-aab6-bf77769262e5-maas-api-xks-ca\") pod \"maas-api-revoke-sub-6517618f-nph8j\" (UID: \"ad5c24d2-e4cc-4737-aab6-bf77769262e5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6517618f-nph8j"
Oct 30 19:39:22.161958 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:39:22.161925    4892 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ad5c24d2-e4cc-4737-aab6-bf77769262e5-maas-api-xks-ca\") pod \"maas-api-revoke-sub-6517618f-nph8j\" (UID: \"ad5c24d2-e4cc-4737-aab6-bf77769262e5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6517618f-nph8j"
Oct 30 19:39:22.162132 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:39:22.162065    4892 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ncf9l\" (UniqueName: \"kubernetes.io/projected/ad5c24d2-e4cc-4737-aab6-bf77769262e5-kube-api-access-ncf9l\") pod \"maas-api-revoke-sub-6517618f-nph8j\" (UID: \"ad5c24d2-e4cc-4737-aab6-bf77769262e5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6517618f-nph8j"
Oct 30 19:39:22.162132 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:39:22.162111    4892 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ad5c24d2-e4cc-4737-aab6-bf77769262e5-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-6517618f-nph8j\" (UID: \"ad5c24d2-e4cc-4737-aab6-bf77769262e5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6517618f-nph8j"
Oct 30 19:39:22.162663 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:39:22.162640    4892 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ad5c24d2-e4cc-4737-aab6-bf77769262e5-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-6517618f-nph8j\" (UID: \"ad5c24d2-e4cc-4737-aab6-bf77769262e5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6517618f-nph8j"
Oct 30 19:39:22.164528 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:39:22.164505    4892 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ad5c24d2-e4cc-4737-aab6-bf77769262e5-maas-api-xks-ca\") pod \"maas-api-revoke-sub-6517618f-nph8j\" (UID: \"ad5c24d2-e4cc-4737-aab6-bf77769262e5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6517618f-nph8j"
Oct 30 19:39:22.218857 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:39:22.218785    4892 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ncf9l\" (UniqueName: \"kubernetes.io/projected/ad5c24d2-e4cc-4737-aab6-bf77769262e5-kube-api-access-ncf9l\") pod \"maas-api-revoke-sub-6517618f-nph8j\" (UID: \"ad5c24d2-e4cc-4737-aab6-bf77769262e5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6517618f-nph8j"
Oct 30 19:39:22.267360 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:39:22.267329    4892 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6517618f-nph8j"
Oct 30 19:39:22.643658 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:39:22.643631    4892 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-6517618f-nph8j"]
Oct 30 19:39:22.644783 ip-10-0-132-82 kubenswrapper[4892]: W1030 19:39:22.644758    4892 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podad5c24d2_e4cc_4737_aab6_bf77769262e5.slice/crio-ab3a2e933b59b6e17946606cadaf2f49918297dbb936c1bc6c9ff87f963fa24d WatchSource:0}: Error finding container ab3a2e933b59b6e17946606cadaf2f49918297dbb936c1bc6c9ff87f963fa24d: Status 404 returned error can't find the container with id ab3a2e933b59b6e17946606cadaf2f49918297dbb936c1bc6c9ff87f963fa24d
Oct 30 19:39:23.256996 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:39:23.256860    4892 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6517618f-nph8j" event={"ID":"ad5c24d2-e4cc-4737-aab6-bf77769262e5","Type":"ContainerStarted","Data":"7806b68e79518c76f170bf51b65616bc277e2def2c60080b6501c59c4e454f07"}
Oct 30 19:39:23.256996 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:39:23.256931    4892 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6517618f-nph8j" event={"ID":"ad5c24d2-e4cc-4737-aab6-bf77769262e5","Type":"ContainerStarted","Data":"ab3a2e933b59b6e17946606cadaf2f49918297dbb936c1bc6c9ff87f963fa24d"}
Oct 30 19:39:23.372340 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:39:23.372292    4892 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6517618f-nph8j" podStartSLOduration=2.372276341 podStartE2EDuration="2.372276341s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:39:21 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:39:23.371677558 +0000 UTC m=+1410.942143158" watchObservedRunningTime="2026-10-30 19:39:23.372276341 +0000 UTC m=+1410.942741940"
Oct 30 19:39:24.266096 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:39:24.266064    4892 generic.go:356] "Generic (PLEG): container finished" podID="ad5c24d2-e4cc-4737-aab6-bf77769262e5" containerID="7806b68e79518c76f170bf51b65616bc277e2def2c60080b6501c59c4e454f07" exitCode=0
Oct 30 19:39:24.266528 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:39:24.266161    4892 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6517618f-nph8j" event={"ID":"ad5c24d2-e4cc-4737-aab6-bf77769262e5","Type":"ContainerDied","Data":"7806b68e79518c76f170bf51b65616bc277e2def2c60080b6501c59c4e454f07"}
Oct 30 19:39:24.843285 ip-10-0-132-82 kubenswrapper[4892]: E1030 19:39:24.843248    4892 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-jrm87" containerName="extractor"
Oct 30 19:39:25.425548 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:39:25.425525    4892 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6517618f-nph8j"
Oct 30 19:39:25.496109 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:39:25.496075    4892 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ad5c24d2-e4cc-4737-aab6-bf77769262e5-kube-api-access-ncf9l\" (UniqueName: \"kubernetes.io/projected/ad5c24d2-e4cc-4737-aab6-bf77769262e5-kube-api-access-ncf9l\") pod \"ad5c24d2-e4cc-4737-aab6-bf77769262e5\" (UID: \"ad5c24d2-e4cc-4737-aab6-bf77769262e5\") "
Oct 30 19:39:25.496256 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:39:25.496169    4892 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ad5c24d2-e4cc-4737-aab6-bf77769262e5-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ad5c24d2-e4cc-4737-aab6-bf77769262e5-maas-api-openshift-ca\") pod \"ad5c24d2-e4cc-4737-aab6-bf77769262e5\" (UID: \"ad5c24d2-e4cc-4737-aab6-bf77769262e5\") "
Oct 30 19:39:25.496337 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:39:25.496271    4892 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/ad5c24d2-e4cc-4737-aab6-bf77769262e5-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ad5c24d2-e4cc-4737-aab6-bf77769262e5-maas-api-xks-ca\") pod \"ad5c24d2-e4cc-4737-aab6-bf77769262e5\" (UID: \"ad5c24d2-e4cc-4737-aab6-bf77769262e5\") "
Oct 30 19:39:25.496562 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:39:25.496534    4892 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ad5c24d2-e4cc-4737-aab6-bf77769262e5-maas-api-openshift-ca" pod "ad5c24d2-e4cc-4737-aab6-bf77769262e5" (UID: "ad5c24d2-e4cc-4737-aab6-bf77769262e5"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 19:39:25.496695 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:39:25.496677    4892 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ad5c24d2-e4cc-4737-aab6-bf77769262e5-maas-api-openshift-ca\") on node \"ip-10-0-132-82.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 19:39:25.498417 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:39:25.498392    4892 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ad5c24d2-e4cc-4737-aab6-bf77769262e5-maas-api-xks-ca" pod "ad5c24d2-e4cc-4737-aab6-bf77769262e5" (UID: "ad5c24d2-e4cc-4737-aab6-bf77769262e5"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 19:39:25.498498 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:39:25.498416    4892 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ad5c24d2-e4cc-4737-aab6-bf77769262e5-kube-api-access-ncf9l" pod "ad5c24d2-e4cc-4737-aab6-bf77769262e5" (UID: "ad5c24d2-e4cc-4737-aab6-bf77769262e5"). InnerVolumeSpecName "kube-api-access-ncf9l". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:39:25.597904 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:39:25.597836    4892 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ad5c24d2-e4cc-4737-aab6-bf77769262e5-maas-api-xks-ca\") on node \"ip-10-0-132-82.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 19:39:25.597904 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:39:25.597870    4892 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ncf9l\" (UniqueName: \"kubernetes.io/projected/ad5c24d2-e4cc-4737-aab6-bf77769262e5-kube-api-access-ncf9l\") on node \"ip-10-0-132-82.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 19:39:25.897271 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:39:25.897238    4892 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w5-686689/distinct-2-w5-686689-kserve-5ff578b4cd-rkc6s"]
Oct 30 19:39:25.897733 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:39:25.897721    4892 cpu_manager.go:414] "containerMap: removing container" podUID="ad5c24d2-e4cc-4737-aab6-bf77769262e5" containerName="revoke-keys"
Oct 30 19:39:25.897778 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:39:25.897734    4892 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ad5c24d2-e4cc-4737-aab6-bf77769262e5" containerName="revoke-keys"
Oct 30 19:39:25.897813 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:39:25.897809    4892 memory_manager.go:365] "RemoveStaleState removing state" podUID="ad5c24d2-e4cc-4737-aab6-bf77769262e5" containerName="revoke-keys"
Oct 30 19:39:25.901133 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:39:25.901117    4892 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w5-686689/distinct-2-w5-686689-kserve-5ff578b4cd-rkc6s"
Oct 30 19:39:25.959469 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:39:25.959437    4892 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w5-686689/distinct-2-w5-686689-kserve-5ff578b4cd-rkc6s"]
Oct 30 19:39:26.000519 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:39:26.000489    4892 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a8829732-ac6f-4238-b8ce-fc781084668b-model-cache\") pod \"distinct-2-w5-686689-kserve-5ff578b4cd-rkc6s\" (UID: \"a8829732-ac6f-4238-b8ce-fc781084668b\") " pod="e2e-models-e2e-worker-w5-686689/distinct-2-w5-686689-kserve-5ff578b4cd-rkc6s"
Oct 30 19:39:26.000679 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:39:26.000533    4892 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a8829732-ac6f-4238-b8ce-fc781084668b-tmp-dir\") pod \"distinct-2-w5-686689-kserve-5ff578b4cd-rkc6s\" (UID: \"a8829732-ac6f-4238-b8ce-fc781084668b\") " pod="e2e-models-e2e-worker-w5-686689/distinct-2-w5-686689-kserve-5ff578b4cd-rkc6s"
Oct 30 19:39:26.000679 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:39:26.000637    4892 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-98wsx\" (UniqueName: \"kubernetes.io/projected/a8829732-ac6f-4238-b8ce-fc781084668b-kube-api-access-98wsx\") pod \"distinct-2-w5-686689-kserve-5ff578b4cd-rkc6s\" (UID: \"a8829732-ac6f-4238-b8ce-fc781084668b\") " pod="e2e-models-e2e-worker-w5-686689/distinct-2-w5-686689-kserve-5ff578b4cd-rkc6s"
Oct 30 19:39:26.000752 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:39:26.000685    4892 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a8829732-ac6f-4238-b8ce-fc781084668b-home\") pod \"distinct-2-w5-686689-kserve-5ff578b4cd-rkc6s\" (UID: \"a8829732-ac6f-4238-b8ce-fc781084668b\") " pod="e2e-models-e2e-worker-w5-686689/distinct-2-w5-686689-kserve-5ff578b4cd-rkc6s"
Oct 30 19:39:26.000800 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:39:26.000784    4892 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a8829732-ac6f-4238-b8ce-fc781084668b-tls-certs\") pod \"distinct-2-w5-686689-kserve-5ff578b4cd-rkc6s\" (UID: \"a8829732-ac6f-4238-b8ce-fc781084668b\") " pod="e2e-models-e2e-worker-w5-686689/distinct-2-w5-686689-kserve-5ff578b4cd-rkc6s"
Oct 30 19:39:26.000836 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:39:26.000816    4892 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a8829732-ac6f-4238-b8ce-fc781084668b-dshm\") pod \"distinct-2-w5-686689-kserve-5ff578b4cd-rkc6s\" (UID: \"a8829732-ac6f-4238-b8ce-fc781084668b\") " pod="e2e-models-e2e-worker-w5-686689/distinct-2-w5-686689-kserve-5ff578b4cd-rkc6s"
Oct 30 19:39:26.003946 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:39:26.003917    4892 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w5-686689\"/\"default-dockercfg-244mk\""
Oct 30 19:39:26.004090 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:39:26.003991    4892 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w5-686689\"/\"openshift-service-ca.crt\""
Oct 30 19:39:26.004090 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:39:26.004042    4892 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w5-686689\"/\"kube-root-ca.crt\""
Oct 30 19:39:26.004090 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:39:26.004056    4892 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w5-686689\"/\"distinct-2-w5-686689-kserve-self-signed-certs\""
Oct 30 19:39:26.101934 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:39:26.101899    4892 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a8829732-ac6f-4238-b8ce-fc781084668b-home\") pod \"distinct-2-w5-686689-kserve-5ff578b4cd-rkc6s\" (UID: \"a8829732-ac6f-4238-b8ce-fc781084668b\") " pod="e2e-models-e2e-worker-w5-686689/distinct-2-w5-686689-kserve-5ff578b4cd-rkc6s"
Oct 30 19:39:26.102140 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:39:26.101977    4892 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a8829732-ac6f-4238-b8ce-fc781084668b-tls-certs\") pod \"distinct-2-w5-686689-kserve-5ff578b4cd-rkc6s\" (UID: \"a8829732-ac6f-4238-b8ce-fc781084668b\") " pod="e2e-models-e2e-worker-w5-686689/distinct-2-w5-686689-kserve-5ff578b4cd-rkc6s"
Oct 30 19:39:26.102140 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:39:26.102011    4892 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a8829732-ac6f-4238-b8ce-fc781084668b-dshm\") pod \"distinct-2-w5-686689-kserve-5ff578b4cd-rkc6s\" (UID: \"a8829732-ac6f-4238-b8ce-fc781084668b\") " pod="e2e-models-e2e-worker-w5-686689/distinct-2-w5-686689-kserve-5ff578b4cd-rkc6s"
Oct 30 19:39:26.102140 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:39:26.102104    4892 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a8829732-ac6f-4238-b8ce-fc781084668b-model-cache\") pod \"distinct-2-w5-686689-kserve-5ff578b4cd-rkc6s\" (UID: \"a8829732-ac6f-4238-b8ce-fc781084668b\") " pod="e2e-models-e2e-worker-w5-686689/distinct-2-w5-686689-kserve-5ff578b4cd-rkc6s"
Oct 30 19:39:26.102311 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:39:26.102172    4892 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a8829732-ac6f-4238-b8ce-fc781084668b-tmp-dir\") pod \"distinct-2-w5-686689-kserve-5ff578b4cd-rkc6s\" (UID: \"a8829732-ac6f-4238-b8ce-fc781084668b\") " pod="e2e-models-e2e-worker-w5-686689/distinct-2-w5-686689-kserve-5ff578b4cd-rkc6s"
Oct 30 19:39:26.102311 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:39:26.102234    4892 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-98wsx\" (UniqueName: \"kubernetes.io/projected/a8829732-ac6f-4238-b8ce-fc781084668b-kube-api-access-98wsx\") pod \"distinct-2-w5-686689-kserve-5ff578b4cd-rkc6s\" (UID: \"a8829732-ac6f-4238-b8ce-fc781084668b\") " pod="e2e-models-e2e-worker-w5-686689/distinct-2-w5-686689-kserve-5ff578b4cd-rkc6s"
Oct 30 19:39:26.102412 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:39:26.102390    4892 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a8829732-ac6f-4238-b8ce-fc781084668b-home\") pod \"distinct-2-w5-686689-kserve-5ff578b4cd-rkc6s\" (UID: \"a8829732-ac6f-4238-b8ce-fc781084668b\") " pod="e2e-models-e2e-worker-w5-686689/distinct-2-w5-686689-kserve-5ff578b4cd-rkc6s"
Oct 30 19:39:26.102557 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:39:26.102532    4892 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a8829732-ac6f-4238-b8ce-fc781084668b-tmp-dir\") pod \"distinct-2-w5-686689-kserve-5ff578b4cd-rkc6s\" (UID: \"a8829732-ac6f-4238-b8ce-fc781084668b\") " pod="e2e-models-e2e-worker-w5-686689/distinct-2-w5-686689-kserve-5ff578b4cd-rkc6s"
Oct 30 19:39:26.102631 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:39:26.102577    4892 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a8829732-ac6f-4238-b8ce-fc781084668b-model-cache\") pod \"distinct-2-w5-686689-kserve-5ff578b4cd-rkc6s\" (UID: \"a8829732-ac6f-4238-b8ce-fc781084668b\") " pod="e2e-models-e2e-worker-w5-686689/distinct-2-w5-686689-kserve-5ff578b4cd-rkc6s"
Oct 30 19:39:26.104577 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:39:26.104558    4892 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a8829732-ac6f-4238-b8ce-fc781084668b-dshm\") pod \"distinct-2-w5-686689-kserve-5ff578b4cd-rkc6s\" (UID: \"a8829732-ac6f-4238-b8ce-fc781084668b\") " pod="e2e-models-e2e-worker-w5-686689/distinct-2-w5-686689-kserve-5ff578b4cd-rkc6s"
Oct 30 19:39:26.104784 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:39:26.104767    4892 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a8829732-ac6f-4238-b8ce-fc781084668b-tls-certs\") pod \"distinct-2-w5-686689-kserve-5ff578b4cd-rkc6s\" (UID: \"a8829732-ac6f-4238-b8ce-fc781084668b\") " pod="e2e-models-e2e-worker-w5-686689/distinct-2-w5-686689-kserve-5ff578b4cd-rkc6s"
Oct 30 19:39:26.173945 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:39:26.173861    4892 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-98wsx\" (UniqueName: \"kubernetes.io/projected/a8829732-ac6f-4238-b8ce-fc781084668b-kube-api-access-98wsx\") pod \"distinct-2-w5-686689-kserve-5ff578b4cd-rkc6s\" (UID: \"a8829732-ac6f-4238-b8ce-fc781084668b\") " pod="e2e-models-e2e-worker-w5-686689/distinct-2-w5-686689-kserve-5ff578b4cd-rkc6s"
Oct 30 19:39:26.214038 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:39:26.214007    4892 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w5-686689/distinct-2-w5-686689-kserve-5ff578b4cd-rkc6s"
Oct 30 19:39:26.377727 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:39:26.377387    4892 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6517618f-nph8j" event={"ID":"ad5c24d2-e4cc-4737-aab6-bf77769262e5","Type":"ContainerDied","Data":"ab3a2e933b59b6e17946606cadaf2f49918297dbb936c1bc6c9ff87f963fa24d"}
Oct 30 19:39:26.377727 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:39:26.377446    4892 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="ab3a2e933b59b6e17946606cadaf2f49918297dbb936c1bc6c9ff87f963fa24d"
Oct 30 19:39:26.377727 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:39:26.377525    4892 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6517618f-nph8j"
Oct 30 19:39:26.636361 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:39:26.636334    4892 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w5-686689/distinct-2-w5-686689-kserve-5ff578b4cd-rkc6s"]
Oct 30 19:39:26.637496 ip-10-0-132-82 kubenswrapper[4892]: W1030 19:39:26.637455    4892 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-poda8829732_ac6f_4238_b8ce_fc781084668b.slice/crio-e864449f59a7d28459aa2ba16d05d1ba99d97fc377a1190547d98610a4c76b78 WatchSource:0}: Error finding container e864449f59a7d28459aa2ba16d05d1ba99d97fc377a1190547d98610a4c76b78: Status 404 returned error can't find the container with id e864449f59a7d28459aa2ba16d05d1ba99d97fc377a1190547d98610a4c76b78
Oct 30 19:39:27.386020 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:39:27.385953    4892 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w5-686689/distinct-2-w5-686689-kserve-5ff578b4cd-rkc6s" event={"ID":"a8829732-ac6f-4238-b8ce-fc781084668b","Type":"ContainerStarted","Data":"1dbd29f80b8f4f21ed34a016fec351a1060e931fb51709446a65f40332fd4fb6"}
Oct 30 19:39:27.386020 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:39:27.386020    4892 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w5-686689/distinct-2-w5-686689-kserve-5ff578b4cd-rkc6s" event={"ID":"a8829732-ac6f-4238-b8ce-fc781084668b","Type":"ContainerStarted","Data":"e864449f59a7d28459aa2ba16d05d1ba99d97fc377a1190547d98610a4c76b78"}
Oct 30 19:39:27.501736 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:39:27.501676    4892 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w5-686689/distinct-2-w5-686689-kserve-5ff578b4cd-rkc6s" podStartSLOduration=2.501663232 podStartE2EDuration="2.501663232s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:39:25 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:39:27.499344542 +0000 UTC m=+1415.069810141" watchObservedRunningTime="2026-10-30 19:39:27.501663232 +0000 UTC m=+1415.072128831"
Oct 30 19:39:28.214757 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:39:28.214723    4892 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w5-686689/distinct-2-w5-686689-kserve-5ff578b4cd-rkc6s"
Oct 30 19:39:35.850903 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:39:35.850867    4892 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w1-d52df0-openshift-default-6d5f887b45-2f4xc"]
Oct 30 19:39:35.856042 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:39:35.856022    4892 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-d52df0-openshift-default-6d5f887b45-2f4xc"
Oct 30 19:39:35.925661 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:39:35.925627    4892 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6befc94b-40e5-42a4-adf5-392b5ecde123-workload-socket\") pod \"e2e-worker-w1-d52df0-openshift-default-6d5f887b45-2f4xc\" (UID: \"6befc94b-40e5-42a4-adf5-392b5ecde123\") " pod="openshift-ingress/e2e-worker-w1-d52df0-openshift-default-6d5f887b45-2f4xc"
Oct 30 19:39:35.925773 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:39:35.925674    4892 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6befc94b-40e5-42a4-adf5-392b5ecde123-istio-podinfo\") pod \"e2e-worker-w1-d52df0-openshift-default-6d5f887b45-2f4xc\" (UID: \"6befc94b-40e5-42a4-adf5-392b5ecde123\") " pod="openshift-ingress/e2e-worker-w1-d52df0-openshift-default-6d5f887b45-2f4xc"
Oct 30 19:39:35.925823 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:39:35.925772    4892 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6befc94b-40e5-42a4-adf5-392b5ecde123-istio-data\") pod \"e2e-worker-w1-d52df0-openshift-default-6d5f887b45-2f4xc\" (UID: \"6befc94b-40e5-42a4-adf5-392b5ecde123\") " pod="openshift-ingress/e2e-worker-w1-d52df0-openshift-default-6d5f887b45-2f4xc"
Oct 30 19:39:35.925823 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:39:35.925800    4892 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-z7f72\" (UniqueName: \"kubernetes.io/projected/6befc94b-40e5-42a4-adf5-392b5ecde123-kube-api-access-z7f72\") pod \"e2e-worker-w1-d52df0-openshift-default-6d5f887b45-2f4xc\" (UID: \"6befc94b-40e5-42a4-adf5-392b5ecde123\") " pod="openshift-ingress/e2e-worker-w1-d52df0-openshift-default-6d5f887b45-2f4xc"
Oct 30 19:39:35.925911 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:39:35.925824    4892 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6befc94b-40e5-42a4-adf5-392b5ecde123-istiod-ca-cert\") pod \"e2e-worker-w1-d52df0-openshift-default-6d5f887b45-2f4xc\" (UID: \"6befc94b-40e5-42a4-adf5-392b5ecde123\") " pod="openshift-ingress/e2e-worker-w1-d52df0-openshift-default-6d5f887b45-2f4xc"
Oct 30 19:39:35.925911 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:39:35.925905    4892 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6befc94b-40e5-42a4-adf5-392b5ecde123-istio-envoy\") pod \"e2e-worker-w1-d52df0-openshift-default-6d5f887b45-2f4xc\" (UID: \"6befc94b-40e5-42a4-adf5-392b5ecde123\") " pod="openshift-ingress/e2e-worker-w1-d52df0-openshift-default-6d5f887b45-2f4xc"
Oct 30 19:39:35.925988 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:39:35.925948    4892 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6befc94b-40e5-42a4-adf5-392b5ecde123-credential-socket\") pod \"e2e-worker-w1-d52df0-openshift-default-6d5f887b45-2f4xc\" (UID: \"6befc94b-40e5-42a4-adf5-392b5ecde123\") " pod="openshift-ingress/e2e-worker-w1-d52df0-openshift-default-6d5f887b45-2f4xc"
Oct 30 19:39:35.926030 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:39:35.926020    4892 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6befc94b-40e5-42a4-adf5-392b5ecde123-workload-certs\") pod \"e2e-worker-w1-d52df0-openshift-default-6d5f887b45-2f4xc\" (UID: \"6befc94b-40e5-42a4-adf5-392b5ecde123\") " pod="openshift-ingress/e2e-worker-w1-d52df0-openshift-default-6d5f887b45-2f4xc"
Oct 30 19:39:35.926071 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:39:35.926060    4892 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6befc94b-40e5-42a4-adf5-392b5ecde123-istio-token\") pod \"e2e-worker-w1-d52df0-openshift-default-6d5f887b45-2f4xc\" (UID: \"6befc94b-40e5-42a4-adf5-392b5ecde123\") " pod="openshift-ingress/e2e-worker-w1-d52df0-openshift-default-6d5f887b45-2f4xc"
Oct 30 19:39:35.961065 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:39:35.961001    4892 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-d52df0-openshift-default-6d5f887b45-2f4xc"]
Oct 30 19:39:36.027522 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:39:36.027493    4892 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6befc94b-40e5-42a4-adf5-392b5ecde123-workload-socket\") pod \"e2e-worker-w1-d52df0-openshift-default-6d5f887b45-2f4xc\" (UID: \"6befc94b-40e5-42a4-adf5-392b5ecde123\") " pod="openshift-ingress/e2e-worker-w1-d52df0-openshift-default-6d5f887b45-2f4xc"
Oct 30 19:39:36.027702 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:39:36.027532    4892 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6befc94b-40e5-42a4-adf5-392b5ecde123-istio-podinfo\") pod \"e2e-worker-w1-d52df0-openshift-default-6d5f887b45-2f4xc\" (UID: \"6befc94b-40e5-42a4-adf5-392b5ecde123\") " pod="openshift-ingress/e2e-worker-w1-d52df0-openshift-default-6d5f887b45-2f4xc"
Oct 30 19:39:36.027702 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:39:36.027569    4892 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6befc94b-40e5-42a4-adf5-392b5ecde123-istio-data\") pod \"e2e-worker-w1-d52df0-openshift-default-6d5f887b45-2f4xc\" (UID: \"6befc94b-40e5-42a4-adf5-392b5ecde123\") " pod="openshift-ingress/e2e-worker-w1-d52df0-openshift-default-6d5f887b45-2f4xc"
Oct 30 19:39:36.027702 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:39:36.027594    4892 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-z7f72\" (UniqueName: \"kubernetes.io/projected/6befc94b-40e5-42a4-adf5-392b5ecde123-kube-api-access-z7f72\") pod \"e2e-worker-w1-d52df0-openshift-default-6d5f887b45-2f4xc\" (UID: \"6befc94b-40e5-42a4-adf5-392b5ecde123\") " pod="openshift-ingress/e2e-worker-w1-d52df0-openshift-default-6d5f887b45-2f4xc"
Oct 30 19:39:36.027702 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:39:36.027622    4892 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6befc94b-40e5-42a4-adf5-392b5ecde123-istiod-ca-cert\") pod \"e2e-worker-w1-d52df0-openshift-default-6d5f887b45-2f4xc\" (UID: \"6befc94b-40e5-42a4-adf5-392b5ecde123\") " pod="openshift-ingress/e2e-worker-w1-d52df0-openshift-default-6d5f887b45-2f4xc"
Oct 30 19:39:36.027702 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:39:36.027667    4892 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6befc94b-40e5-42a4-adf5-392b5ecde123-istio-envoy\") pod \"e2e-worker-w1-d52df0-openshift-default-6d5f887b45-2f4xc\" (UID: \"6befc94b-40e5-42a4-adf5-392b5ecde123\") " pod="openshift-ingress/e2e-worker-w1-d52df0-openshift-default-6d5f887b45-2f4xc"
Oct 30 19:39:36.028007 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:39:36.027719    4892 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6befc94b-40e5-42a4-adf5-392b5ecde123-credential-socket\") pod \"e2e-worker-w1-d52df0-openshift-default-6d5f887b45-2f4xc\" (UID: \"6befc94b-40e5-42a4-adf5-392b5ecde123\") " pod="openshift-ingress/e2e-worker-w1-d52df0-openshift-default-6d5f887b45-2f4xc"
Oct 30 19:39:36.028007 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:39:36.027785    4892 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6befc94b-40e5-42a4-adf5-392b5ecde123-workload-certs\") pod \"e2e-worker-w1-d52df0-openshift-default-6d5f887b45-2f4xc\" (UID: \"6befc94b-40e5-42a4-adf5-392b5ecde123\") " pod="openshift-ingress/e2e-worker-w1-d52df0-openshift-default-6d5f887b45-2f4xc"
Oct 30 19:39:36.028007 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:39:36.027834    4892 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6befc94b-40e5-42a4-adf5-392b5ecde123-istio-token\") pod \"e2e-worker-w1-d52df0-openshift-default-6d5f887b45-2f4xc\" (UID: \"6befc94b-40e5-42a4-adf5-392b5ecde123\") " pod="openshift-ingress/e2e-worker-w1-d52df0-openshift-default-6d5f887b45-2f4xc"
Oct 30 19:39:36.028174 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:39:36.028003    4892 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6befc94b-40e5-42a4-adf5-392b5ecde123-workload-socket\") pod \"e2e-worker-w1-d52df0-openshift-default-6d5f887b45-2f4xc\" (UID: \"6befc94b-40e5-42a4-adf5-392b5ecde123\") " pod="openshift-ingress/e2e-worker-w1-d52df0-openshift-default-6d5f887b45-2f4xc"
Oct 30 19:39:36.028433 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:39:36.028406    4892 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6befc94b-40e5-42a4-adf5-392b5ecde123-credential-socket\") pod \"e2e-worker-w1-d52df0-openshift-default-6d5f887b45-2f4xc\" (UID: \"6befc94b-40e5-42a4-adf5-392b5ecde123\") " pod="openshift-ingress/e2e-worker-w1-d52df0-openshift-default-6d5f887b45-2f4xc"
Oct 30 19:39:36.028593 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:39:36.028574    4892 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6befc94b-40e5-42a4-adf5-392b5ecde123-istio-data\") pod \"e2e-worker-w1-d52df0-openshift-default-6d5f887b45-2f4xc\" (UID: \"6befc94b-40e5-42a4-adf5-392b5ecde123\") " pod="openshift-ingress/e2e-worker-w1-d52df0-openshift-default-6d5f887b45-2f4xc"
Oct 30 19:39:36.028678 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:39:36.028605    4892 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6befc94b-40e5-42a4-adf5-392b5ecde123-workload-certs\") pod \"e2e-worker-w1-d52df0-openshift-default-6d5f887b45-2f4xc\" (UID: \"6befc94b-40e5-42a4-adf5-392b5ecde123\") " pod="openshift-ingress/e2e-worker-w1-d52df0-openshift-default-6d5f887b45-2f4xc"
Oct 30 19:39:36.028871 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:39:36.028844    4892 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6befc94b-40e5-42a4-adf5-392b5ecde123-istiod-ca-cert\") pod \"e2e-worker-w1-d52df0-openshift-default-6d5f887b45-2f4xc\" (UID: \"6befc94b-40e5-42a4-adf5-392b5ecde123\") " pod="openshift-ingress/e2e-worker-w1-d52df0-openshift-default-6d5f887b45-2f4xc"
Oct 30 19:39:36.030144 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:39:36.030123    4892 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6befc94b-40e5-42a4-adf5-392b5ecde123-istio-envoy\") pod \"e2e-worker-w1-d52df0-openshift-default-6d5f887b45-2f4xc\" (UID: \"6befc94b-40e5-42a4-adf5-392b5ecde123\") " pod="openshift-ingress/e2e-worker-w1-d52df0-openshift-default-6d5f887b45-2f4xc"
Oct 30 19:39:36.030733 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:39:36.030710    4892 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6befc94b-40e5-42a4-adf5-392b5ecde123-istio-podinfo\") pod \"e2e-worker-w1-d52df0-openshift-default-6d5f887b45-2f4xc\" (UID: \"6befc94b-40e5-42a4-adf5-392b5ecde123\") " pod="openshift-ingress/e2e-worker-w1-d52df0-openshift-default-6d5f887b45-2f4xc"
Oct 30 19:39:36.085559 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:39:36.085537    4892 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6befc94b-40e5-42a4-adf5-392b5ecde123-istio-token\") pod \"e2e-worker-w1-d52df0-openshift-default-6d5f887b45-2f4xc\" (UID: \"6befc94b-40e5-42a4-adf5-392b5ecde123\") " pod="openshift-ingress/e2e-worker-w1-d52df0-openshift-default-6d5f887b45-2f4xc"
Oct 30 19:39:36.085658 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:39:36.085606    4892 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-z7f72\" (UniqueName: \"kubernetes.io/projected/6befc94b-40e5-42a4-adf5-392b5ecde123-kube-api-access-z7f72\") pod \"e2e-worker-w1-d52df0-openshift-default-6d5f887b45-2f4xc\" (UID: \"6befc94b-40e5-42a4-adf5-392b5ecde123\") " pod="openshift-ingress/e2e-worker-w1-d52df0-openshift-default-6d5f887b45-2f4xc"
Oct 30 19:39:36.169616 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:39:36.169561    4892 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-d52df0-openshift-default-6d5f887b45-2f4xc"
Oct 30 19:39:36.215060 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:39:36.215026    4892 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w5-686689/distinct-2-w5-686689-kserve-5ff578b4cd-rkc6s"
Oct 30 19:39:36.236004 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:39:36.235979    4892 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w5-686689/distinct-2-w5-686689-kserve-5ff578b4cd-rkc6s"
Oct 30 19:39:36.463475 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:39:36.463400    4892 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w5-686689/distinct-2-w5-686689-kserve-5ff578b4cd-rkc6s"
Oct 30 19:39:36.600936 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:39:36.600908    4892 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-d52df0-openshift-default-6d5f887b45-2f4xc"]
Oct 30 19:39:36.606420 ip-10-0-132-82 kubenswrapper[4892]: W1030 19:39:36.606389    4892 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6befc94b_40e5_42a4_adf5_392b5ecde123.slice/crio-5d2d7cb823c7697e722c326abf9879dc0a4c1c47aa81a1b2e99322bbff75354a WatchSource:0}: Error finding container 5d2d7cb823c7697e722c326abf9879dc0a4c1c47aa81a1b2e99322bbff75354a: Status 404 returned error can't find the container with id 5d2d7cb823c7697e722c326abf9879dc0a4c1c47aa81a1b2e99322bbff75354a
Oct 30 19:39:36.609084 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:39:36.608736    4892 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892000Ki","pods":"250"}
Oct 30 19:39:36.609084 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:39:36.608816    4892 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892000Ki","pods":"250"}
Oct 30 19:39:36.609084 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:39:36.608848    4892 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892000Ki","pods":"250"}
Oct 30 19:39:37.464865 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:39:37.464826    4892 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-d52df0-openshift-default-6d5f887b45-2f4xc" event={"ID":"6befc94b-40e5-42a4-adf5-392b5ecde123","Type":"ContainerStarted","Data":"46e079b04871b67635b433adadb16a2bbd7b3117558a7b1d0bb392e0be15de8c"}
Oct 30 19:39:37.465408 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:39:37.464917    4892 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-d52df0-openshift-default-6d5f887b45-2f4xc" event={"ID":"6befc94b-40e5-42a4-adf5-392b5ecde123","Type":"ContainerStarted","Data":"5d2d7cb823c7697e722c326abf9879dc0a4c1c47aa81a1b2e99322bbff75354a"}
Oct 30 19:39:37.583610 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:39:37.583557    4892 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w1-d52df0-openshift-default-6d5f887b45-2f4xc" podStartSLOduration=2.583543288 podStartE2EDuration="2.583543288s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:39:35 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:39:37.57968923 +0000 UTC m=+1425.150154829" watchObservedRunningTime="2026-10-30 19:39:37.583543288 +0000 UTC m=+1425.154008887"
Oct 30 19:39:38.169759 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:39:38.169729    4892 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w1-d52df0-openshift-default-6d5f887b45-2f4xc"
Oct 30 19:39:38.171365 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:39:38.171339    4892 patch_prober.go:29] interesting pod/e2e-worker-w1-d52df0-openshift-default-6d5f887b45-2f4xc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.88:15021/healthz/ready\": dial tcp 10.132.0.88:15021: connect: connection refused" start-of-body=
Oct 30 19:39:38.171493 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:39:38.171395    4892 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-d52df0-openshift-default-6d5f887b45-2f4xc" podUID="6befc94b-40e5-42a4-adf5-392b5ecde123" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.88:15021/healthz/ready\": dial tcp 10.132.0.88:15021: connect: connection refused"
Oct 30 19:39:39.170597 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:39:39.170568    4892 patch_prober.go:29] interesting pod/e2e-worker-w1-d52df0-openshift-default-6d5f887b45-2f4xc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.88:15021/healthz/ready\": dial tcp 10.132.0.88:15021: connect: connection refused" start-of-body=
Oct 30 19:39:39.171084 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:39:39.171058    4892 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-d52df0-openshift-default-6d5f887b45-2f4xc" podUID="6befc94b-40e5-42a4-adf5-392b5ecde123" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.88:15021/healthz/ready\": dial tcp 10.132.0.88:15021: connect: connection refused"
Oct 30 19:39:40.170487 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:39:40.170447    4892 patch_prober.go:29] interesting pod/e2e-worker-w1-d52df0-openshift-default-6d5f887b45-2f4xc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.88:15021/healthz/ready\": dial tcp 10.132.0.88:15021: connect: connection refused" start-of-body=
Oct 30 19:39:40.170690 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:39:40.170501    4892 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-d52df0-openshift-default-6d5f887b45-2f4xc" podUID="6befc94b-40e5-42a4-adf5-392b5ecde123" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.88:15021/healthz/ready\": dial tcp 10.132.0.88:15021: connect: connection refused"
Oct 30 19:39:41.170768 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:39:41.170725    4892 patch_prober.go:29] interesting pod/e2e-worker-w1-d52df0-openshift-default-6d5f887b45-2f4xc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.88:15021/healthz/ready\": dial tcp 10.132.0.88:15021: connect: connection refused" start-of-body=
Oct 30 19:39:41.171282 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:39:41.170779    4892 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-d52df0-openshift-default-6d5f887b45-2f4xc" podUID="6befc94b-40e5-42a4-adf5-392b5ecde123" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.88:15021/healthz/ready\": dial tcp 10.132.0.88:15021: connect: connection refused"
Oct 30 19:39:42.170319 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:39:42.170276    4892 patch_prober.go:29] interesting pod/e2e-worker-w1-d52df0-openshift-default-6d5f887b45-2f4xc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.88:15021/healthz/ready\": dial tcp 10.132.0.88:15021: connect: connection refused" start-of-body=
Oct 30 19:39:42.170512 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:39:42.170358    4892 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-d52df0-openshift-default-6d5f887b45-2f4xc" podUID="6befc94b-40e5-42a4-adf5-392b5ecde123" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.88:15021/healthz/ready\": dial tcp 10.132.0.88:15021: connect: connection refused"
Oct 30 19:39:43.170913 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:39:43.170880    4892 patch_prober.go:29] interesting pod/e2e-worker-w1-d52df0-openshift-default-6d5f887b45-2f4xc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.88:15021/healthz/ready\": dial tcp 10.132.0.88:15021: connect: connection refused" start-of-body=
Oct 30 19:39:43.171316 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:39:43.170935    4892 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-d52df0-openshift-default-6d5f887b45-2f4xc" podUID="6befc94b-40e5-42a4-adf5-392b5ecde123" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.88:15021/healthz/ready\": dial tcp 10.132.0.88:15021: connect: connection refused"
Oct 30 19:39:44.174682 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:39:44.174649    4892 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w1-d52df0-openshift-default-6d5f887b45-2f4xc"
Oct 30 19:39:44.175128 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:39:44.174991    4892 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w1-d52df0-openshift-default-6d5f887b45-2f4xc"
Oct 30 19:39:44.175724 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:39:44.175702    4892 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w1-d52df0-openshift-default-6d5f887b45-2f4xc"
Oct 30 19:39:49.390148 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:39:49.390113    4892 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-203da51c-5zcqh"]
Oct 30 19:39:49.394218 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:39:49.394202    4892 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-203da51c-5zcqh"
Oct 30 19:39:49.447209 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:39:49.447080    4892 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 19:39:49.447209 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:39:49.447179    4892 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 30 19:39:49.485717 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:39:49.485690    4892 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7cc551c1-9fcd-4df4-afa9-db440c768fc5-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-203da51c-5zcqh\" (UID: \"7cc551c1-9fcd-4df4-afa9-db440c768fc5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-203da51c-5zcqh"
Oct 30 19:39:49.485892 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:39:49.485840    4892 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7cc551c1-9fcd-4df4-afa9-db440c768fc5-maas-api-xks-ca\") pod \"maas-api-revoke-sub-203da51c-5zcqh\" (UID: \"7cc551c1-9fcd-4df4-afa9-db440c768fc5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-203da51c-5zcqh"
Oct 30 19:39:49.486005 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:39:49.485912    4892 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8vhmc\" (UniqueName: \"kubernetes.io/projected/7cc551c1-9fcd-4df4-afa9-db440c768fc5-kube-api-access-8vhmc\") pod \"maas-api-revoke-sub-203da51c-5zcqh\" (UID: \"7cc551c1-9fcd-4df4-afa9-db440c768fc5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-203da51c-5zcqh"
Oct 30 19:39:49.496220 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:39:49.496192    4892 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 30 19:39:49.496353 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:39:49.496246    4892 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-nlxp4\""
Oct 30 19:39:49.500310 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:39:49.500283    4892 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-203da51c-5zcqh"]
Oct 30 19:39:49.586781 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:39:49.586751    4892 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8vhmc\" (UniqueName: \"kubernetes.io/projected/7cc551c1-9fcd-4df4-afa9-db440c768fc5-kube-api-access-8vhmc\") pod \"maas-api-revoke-sub-203da51c-5zcqh\" (UID: \"7cc551c1-9fcd-4df4-afa9-db440c768fc5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-203da51c-5zcqh"
Oct 30 19:39:49.586992 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:39:49.586877    4892 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7cc551c1-9fcd-4df4-afa9-db440c768fc5-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-203da51c-5zcqh\" (UID: \"7cc551c1-9fcd-4df4-afa9-db440c768fc5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-203da51c-5zcqh"
Oct 30 19:39:49.587077 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:39:49.586990    4892 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7cc551c1-9fcd-4df4-afa9-db440c768fc5-maas-api-xks-ca\") pod \"maas-api-revoke-sub-203da51c-5zcqh\" (UID: \"7cc551c1-9fcd-4df4-afa9-db440c768fc5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-203da51c-5zcqh"
Oct 30 19:39:49.587572 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:39:49.587546    4892 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7cc551c1-9fcd-4df4-afa9-db440c768fc5-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-203da51c-5zcqh\" (UID: \"7cc551c1-9fcd-4df4-afa9-db440c768fc5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-203da51c-5zcqh"
Oct 30 19:39:49.589467 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:39:49.589444    4892 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7cc551c1-9fcd-4df4-afa9-db440c768fc5-maas-api-xks-ca\") pod \"maas-api-revoke-sub-203da51c-5zcqh\" (UID: \"7cc551c1-9fcd-4df4-afa9-db440c768fc5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-203da51c-5zcqh"
Oct 30 19:39:49.644181 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:39:49.644108    4892 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8vhmc\" (UniqueName: \"kubernetes.io/projected/7cc551c1-9fcd-4df4-afa9-db440c768fc5-kube-api-access-8vhmc\") pod \"maas-api-revoke-sub-203da51c-5zcqh\" (UID: \"7cc551c1-9fcd-4df4-afa9-db440c768fc5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-203da51c-5zcqh"
Oct 30 19:39:49.709709 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:39:49.709678    4892 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-203da51c-5zcqh"
Oct 30 19:39:50.088464 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:39:50.088438    4892 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-203da51c-5zcqh"]
Oct 30 19:39:50.089378 ip-10-0-132-82 kubenswrapper[4892]: W1030 19:39:50.089342    4892 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7cc551c1_9fcd_4df4_afa9_db440c768fc5.slice/crio-8cbfb878f114acf7ca222049e41716c65823bab38a9193c124dfed933e565861 WatchSource:0}: Error finding container 8cbfb878f114acf7ca222049e41716c65823bab38a9193c124dfed933e565861: Status 404 returned error can't find the container with id 8cbfb878f114acf7ca222049e41716c65823bab38a9193c124dfed933e565861
Oct 30 19:39:50.571052 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:39:50.571012    4892 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-203da51c-5zcqh" event={"ID":"7cc551c1-9fcd-4df4-afa9-db440c768fc5","Type":"ContainerStarted","Data":"74f9b4a40227cdd628d7f0a2a1ce04c819df41f8615bfc28d152dbb5de52b06d"}
Oct 30 19:39:50.571526 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:39:50.571063    4892 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-203da51c-5zcqh" event={"ID":"7cc551c1-9fcd-4df4-afa9-db440c768fc5","Type":"ContainerStarted","Data":"8cbfb878f114acf7ca222049e41716c65823bab38a9193c124dfed933e565861"}
Oct 30 19:39:50.683734 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:39:50.683681    4892 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-203da51c-5zcqh" podStartSLOduration=1.683667932 podStartE2EDuration="1.683667932s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:39:49 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:39:50.679873291 +0000 UTC m=+1438.250338891" watchObservedRunningTime="2026-10-30 19:39:50.683667932 +0000 UTC m=+1438.254133532"
Oct 30 19:39:51.576323 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:39:51.576289    4892 generic.go:356] "Generic (PLEG): container finished" podID="7cc551c1-9fcd-4df4-afa9-db440c768fc5" containerID="74f9b4a40227cdd628d7f0a2a1ce04c819df41f8615bfc28d152dbb5de52b06d" exitCode=0
Oct 30 19:39:51.576761 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:39:51.576385    4892 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-203da51c-5zcqh" event={"ID":"7cc551c1-9fcd-4df4-afa9-db440c768fc5","Type":"ContainerDied","Data":"74f9b4a40227cdd628d7f0a2a1ce04c819df41f8615bfc28d152dbb5de52b06d"}
Oct 30 19:39:51.661677 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:39:51.661639    4892 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-fsfvk"]
Oct 30 19:39:52.734337 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:39:52.734307    4892 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-203da51c-5zcqh"
Oct 30 19:39:52.831868 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:39:52.831834    4892 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/7cc551c1-9fcd-4df4-afa9-db440c768fc5-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7cc551c1-9fcd-4df4-afa9-db440c768fc5-maas-api-openshift-ca\") pod \"7cc551c1-9fcd-4df4-afa9-db440c768fc5\" (UID: \"7cc551c1-9fcd-4df4-afa9-db440c768fc5\") "
Oct 30 19:39:52.832071 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:39:52.831945    4892 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/7cc551c1-9fcd-4df4-afa9-db440c768fc5-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7cc551c1-9fcd-4df4-afa9-db440c768fc5-maas-api-xks-ca\") pod \"7cc551c1-9fcd-4df4-afa9-db440c768fc5\" (UID: \"7cc551c1-9fcd-4df4-afa9-db440c768fc5\") "
Oct 30 19:39:52.832071 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:39:52.831984    4892 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7cc551c1-9fcd-4df4-afa9-db440c768fc5-kube-api-access-8vhmc\" (UniqueName: \"kubernetes.io/projected/7cc551c1-9fcd-4df4-afa9-db440c768fc5-kube-api-access-8vhmc\") pod \"7cc551c1-9fcd-4df4-afa9-db440c768fc5\" (UID: \"7cc551c1-9fcd-4df4-afa9-db440c768fc5\") "
Oct 30 19:39:52.832242 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:39:52.832213    4892 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7cc551c1-9fcd-4df4-afa9-db440c768fc5-maas-api-openshift-ca" pod "7cc551c1-9fcd-4df4-afa9-db440c768fc5" (UID: "7cc551c1-9fcd-4df4-afa9-db440c768fc5"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 19:39:52.832378 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:39:52.832361    4892 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7cc551c1-9fcd-4df4-afa9-db440c768fc5-maas-api-openshift-ca\") on node \"ip-10-0-132-82.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 19:39:52.834203 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:39:52.834136    4892 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7cc551c1-9fcd-4df4-afa9-db440c768fc5-maas-api-xks-ca" pod "7cc551c1-9fcd-4df4-afa9-db440c768fc5" (UID: "7cc551c1-9fcd-4df4-afa9-db440c768fc5"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 19:39:52.834323 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:39:52.834238    4892 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7cc551c1-9fcd-4df4-afa9-db440c768fc5-kube-api-access-8vhmc" pod "7cc551c1-9fcd-4df4-afa9-db440c768fc5" (UID: "7cc551c1-9fcd-4df4-afa9-db440c768fc5"). InnerVolumeSpecName "kube-api-access-8vhmc". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:39:52.933090 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:39:52.933045    4892 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7cc551c1-9fcd-4df4-afa9-db440c768fc5-maas-api-xks-ca\") on node \"ip-10-0-132-82.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 19:39:52.933090 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:39:52.933076    4892 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8vhmc\" (UniqueName: \"kubernetes.io/projected/7cc551c1-9fcd-4df4-afa9-db440c768fc5-kube-api-access-8vhmc\") on node \"ip-10-0-132-82.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 19:39:53.594460 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:39:53.594428    4892 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-203da51c-5zcqh" event={"ID":"7cc551c1-9fcd-4df4-afa9-db440c768fc5","Type":"ContainerDied","Data":"8cbfb878f114acf7ca222049e41716c65823bab38a9193c124dfed933e565861"}
Oct 30 19:39:53.594642 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:39:53.594476    4892 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="8cbfb878f114acf7ca222049e41716c65823bab38a9193c124dfed933e565861"
Oct 30 19:39:53.594642 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:39:53.594479    4892 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-203da51c-5zcqh"
Oct 30 19:39:54.847553 ip-10-0-132-82 kubenswrapper[4892]: E1030 19:39:54.847514    4892 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-jrm87" containerName="extractor"
Oct 30 19:40:10.897249 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:40:10.897216    4892 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w4-b5178f/distinct-2-w4-b5178f-kserve-69ddf8b665-9pdth"]
Oct 30 19:40:10.897870 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:40:10.897683    4892 cpu_manager.go:414] "containerMap: removing container" podUID="7cc551c1-9fcd-4df4-afa9-db440c768fc5" containerName="revoke-keys"
Oct 30 19:40:10.897870 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:40:10.897693    4892 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7cc551c1-9fcd-4df4-afa9-db440c768fc5" containerName="revoke-keys"
Oct 30 19:40:10.897870 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:40:10.897765    4892 memory_manager.go:365] "RemoveStaleState removing state" podUID="7cc551c1-9fcd-4df4-afa9-db440c768fc5" containerName="revoke-keys"
Oct 30 19:40:10.900975 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:40:10.900937    4892 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-b5178f/distinct-2-w4-b5178f-kserve-69ddf8b665-9pdth"
Oct 30 19:40:11.003492 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:40:11.003460    4892 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w4-b5178f\"/\"kube-root-ca.crt\""
Oct 30 19:40:11.003665 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:40:11.003467    4892 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w4-b5178f\"/\"openshift-service-ca.crt\""
Oct 30 19:40:11.003665 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:40:11.003496    4892 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w4-b5178f\"/\"default-dockercfg-wljhm\""
Oct 30 19:40:11.003665 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:40:11.003552    4892 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w4-b5178f\"/\"distinct-2-w4-b5178f-kserve-self-signed-certs\""
Oct 30 19:40:11.007606 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:40:11.007586    4892 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w4-b5178f/distinct-2-w4-b5178f-kserve-69ddf8b665-9pdth"]
Oct 30 19:40:11.027041 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:40:11.027011    4892 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/fb71d961-8500-4367-b48c-88d385962bd5-dshm\") pod \"distinct-2-w4-b5178f-kserve-69ddf8b665-9pdth\" (UID: \"fb71d961-8500-4367-b48c-88d385962bd5\") " pod="e2e-models-e2e-worker-w4-b5178f/distinct-2-w4-b5178f-kserve-69ddf8b665-9pdth"
Oct 30 19:40:11.027186 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:40:11.027046    4892 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/fb71d961-8500-4367-b48c-88d385962bd5-home\") pod \"distinct-2-w4-b5178f-kserve-69ddf8b665-9pdth\" (UID: \"fb71d961-8500-4367-b48c-88d385962bd5\") " pod="e2e-models-e2e-worker-w4-b5178f/distinct-2-w4-b5178f-kserve-69ddf8b665-9pdth"
Oct 30 19:40:11.027186 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:40:11.027145    4892 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/fb71d961-8500-4367-b48c-88d385962bd5-tls-certs\") pod \"distinct-2-w4-b5178f-kserve-69ddf8b665-9pdth\" (UID: \"fb71d961-8500-4367-b48c-88d385962bd5\") " pod="e2e-models-e2e-worker-w4-b5178f/distinct-2-w4-b5178f-kserve-69ddf8b665-9pdth"
Oct 30 19:40:11.027303 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:40:11.027248    4892 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-92v49\" (UniqueName: \"kubernetes.io/projected/fb71d961-8500-4367-b48c-88d385962bd5-kube-api-access-92v49\") pod \"distinct-2-w4-b5178f-kserve-69ddf8b665-9pdth\" (UID: \"fb71d961-8500-4367-b48c-88d385962bd5\") " pod="e2e-models-e2e-worker-w4-b5178f/distinct-2-w4-b5178f-kserve-69ddf8b665-9pdth"
Oct 30 19:40:11.027303 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:40:11.027280    4892 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/fb71d961-8500-4367-b48c-88d385962bd5-tmp-dir\") pod \"distinct-2-w4-b5178f-kserve-69ddf8b665-9pdth\" (UID: \"fb71d961-8500-4367-b48c-88d385962bd5\") " pod="e2e-models-e2e-worker-w4-b5178f/distinct-2-w4-b5178f-kserve-69ddf8b665-9pdth"
Oct 30 19:40:11.027415 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:40:11.027343    4892 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/fb71d961-8500-4367-b48c-88d385962bd5-model-cache\") pod \"distinct-2-w4-b5178f-kserve-69ddf8b665-9pdth\" (UID: \"fb71d961-8500-4367-b48c-88d385962bd5\") " pod="e2e-models-e2e-worker-w4-b5178f/distinct-2-w4-b5178f-kserve-69ddf8b665-9pdth"
Oct 30 19:40:11.128856 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:40:11.128815    4892 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-92v49\" (UniqueName: \"kubernetes.io/projected/fb71d961-8500-4367-b48c-88d385962bd5-kube-api-access-92v49\") pod \"distinct-2-w4-b5178f-kserve-69ddf8b665-9pdth\" (UID: \"fb71d961-8500-4367-b48c-88d385962bd5\") " pod="e2e-models-e2e-worker-w4-b5178f/distinct-2-w4-b5178f-kserve-69ddf8b665-9pdth"
Oct 30 19:40:11.129040 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:40:11.128869    4892 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/fb71d961-8500-4367-b48c-88d385962bd5-tmp-dir\") pod \"distinct-2-w4-b5178f-kserve-69ddf8b665-9pdth\" (UID: \"fb71d961-8500-4367-b48c-88d385962bd5\") " pod="e2e-models-e2e-worker-w4-b5178f/distinct-2-w4-b5178f-kserve-69ddf8b665-9pdth"
Oct 30 19:40:11.129040 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:40:11.128932    4892 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/fb71d961-8500-4367-b48c-88d385962bd5-model-cache\") pod \"distinct-2-w4-b5178f-kserve-69ddf8b665-9pdth\" (UID: \"fb71d961-8500-4367-b48c-88d385962bd5\") " pod="e2e-models-e2e-worker-w4-b5178f/distinct-2-w4-b5178f-kserve-69ddf8b665-9pdth"
Oct 30 19:40:11.129040 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:40:11.129026    4892 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/fb71d961-8500-4367-b48c-88d385962bd5-dshm\") pod \"distinct-2-w4-b5178f-kserve-69ddf8b665-9pdth\" (UID: \"fb71d961-8500-4367-b48c-88d385962bd5\") " pod="e2e-models-e2e-worker-w4-b5178f/distinct-2-w4-b5178f-kserve-69ddf8b665-9pdth"
Oct 30 19:40:11.129219 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:40:11.129057    4892 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/fb71d961-8500-4367-b48c-88d385962bd5-home\") pod \"distinct-2-w4-b5178f-kserve-69ddf8b665-9pdth\" (UID: \"fb71d961-8500-4367-b48c-88d385962bd5\") " pod="e2e-models-e2e-worker-w4-b5178f/distinct-2-w4-b5178f-kserve-69ddf8b665-9pdth"
Oct 30 19:40:11.129219 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:40:11.129144    4892 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/fb71d961-8500-4367-b48c-88d385962bd5-tls-certs\") pod \"distinct-2-w4-b5178f-kserve-69ddf8b665-9pdth\" (UID: \"fb71d961-8500-4367-b48c-88d385962bd5\") " pod="e2e-models-e2e-worker-w4-b5178f/distinct-2-w4-b5178f-kserve-69ddf8b665-9pdth"
Oct 30 19:40:11.129539 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:40:11.129509    4892 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/fb71d961-8500-4367-b48c-88d385962bd5-model-cache\") pod \"distinct-2-w4-b5178f-kserve-69ddf8b665-9pdth\" (UID: \"fb71d961-8500-4367-b48c-88d385962bd5\") " pod="e2e-models-e2e-worker-w4-b5178f/distinct-2-w4-b5178f-kserve-69ddf8b665-9pdth"
Oct 30 19:40:11.129637 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:40:11.129577    4892 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/fb71d961-8500-4367-b48c-88d385962bd5-tmp-dir\") pod \"distinct-2-w4-b5178f-kserve-69ddf8b665-9pdth\" (UID: \"fb71d961-8500-4367-b48c-88d385962bd5\") " pod="e2e-models-e2e-worker-w4-b5178f/distinct-2-w4-b5178f-kserve-69ddf8b665-9pdth"
Oct 30 19:40:11.129791 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:40:11.129769    4892 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/fb71d961-8500-4367-b48c-88d385962bd5-home\") pod \"distinct-2-w4-b5178f-kserve-69ddf8b665-9pdth\" (UID: \"fb71d961-8500-4367-b48c-88d385962bd5\") " pod="e2e-models-e2e-worker-w4-b5178f/distinct-2-w4-b5178f-kserve-69ddf8b665-9pdth"
Oct 30 19:40:11.132019 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:40:11.131992    4892 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/fb71d961-8500-4367-b48c-88d385962bd5-tls-certs\") pod \"distinct-2-w4-b5178f-kserve-69ddf8b665-9pdth\" (UID: \"fb71d961-8500-4367-b48c-88d385962bd5\") " pod="e2e-models-e2e-worker-w4-b5178f/distinct-2-w4-b5178f-kserve-69ddf8b665-9pdth"
Oct 30 19:40:11.132225 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:40:11.132204    4892 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/fb71d961-8500-4367-b48c-88d385962bd5-dshm\") pod \"distinct-2-w4-b5178f-kserve-69ddf8b665-9pdth\" (UID: \"fb71d961-8500-4367-b48c-88d385962bd5\") " pod="e2e-models-e2e-worker-w4-b5178f/distinct-2-w4-b5178f-kserve-69ddf8b665-9pdth"
Oct 30 19:40:11.186499 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:40:11.186426    4892 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-92v49\" (UniqueName: \"kubernetes.io/projected/fb71d961-8500-4367-b48c-88d385962bd5-kube-api-access-92v49\") pod \"distinct-2-w4-b5178f-kserve-69ddf8b665-9pdth\" (UID: \"fb71d961-8500-4367-b48c-88d385962bd5\") " pod="e2e-models-e2e-worker-w4-b5178f/distinct-2-w4-b5178f-kserve-69ddf8b665-9pdth"
Oct 30 19:40:11.213756 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:40:11.213727    4892 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-b5178f/distinct-2-w4-b5178f-kserve-69ddf8b665-9pdth"
Oct 30 19:40:11.687766 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:40:11.687734    4892 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w4-b5178f/distinct-2-w4-b5178f-kserve-69ddf8b665-9pdth"]
Oct 30 19:40:11.688449 ip-10-0-132-82 kubenswrapper[4892]: W1030 19:40:11.688425    4892 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podfb71d961_8500_4367_b48c_88d385962bd5.slice/crio-db43981c0f9cbbd30fb0e757523de51563b791d2c2aa9357f01b461d9acb174c WatchSource:0}: Error finding container db43981c0f9cbbd30fb0e757523de51563b791d2c2aa9357f01b461d9acb174c: Status 404 returned error can't find the container with id db43981c0f9cbbd30fb0e757523de51563b791d2c2aa9357f01b461d9acb174c
Oct 30 19:40:11.735187 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:40:11.735145    4892 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-b5178f/distinct-2-w4-b5178f-kserve-69ddf8b665-9pdth" event={"ID":"fb71d961-8500-4367-b48c-88d385962bd5","Type":"ContainerStarted","Data":"db43981c0f9cbbd30fb0e757523de51563b791d2c2aa9357f01b461d9acb174c"}
Oct 30 19:40:12.745236 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:40:12.745198    4892 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-b5178f/distinct-2-w4-b5178f-kserve-69ddf8b665-9pdth" event={"ID":"fb71d961-8500-4367-b48c-88d385962bd5","Type":"ContainerStarted","Data":"ee63f2eb62b582c733c1bbfd46528dc7e66ac14c84946fe2fbe09a604ca1d05d"}
Oct 30 19:40:12.863024 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:40:12.862952    4892 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w4-b5178f/distinct-2-w4-b5178f-kserve-69ddf8b665-9pdth" podStartSLOduration=2.862937484 podStartE2EDuration="2.862937484s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:40:10 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:40:12.85984672 +0000 UTC m=+1460.430312354" watchObservedRunningTime="2026-10-30 19:40:12.862937484 +0000 UTC m=+1460.433403082"
Oct 30 19:40:13.214838 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:40:13.214804    4892 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-b5178f/distinct-2-w4-b5178f-kserve-69ddf8b665-9pdth"
Oct 30 19:40:13.566357 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:40:13.566282    4892 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a348f93c-jgspj"]
Oct 30 19:40:13.570165 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:40:13.570150    4892 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a348f93c-jgspj"
Oct 30 19:40:13.627379 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:40:13.627350    4892 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a348f93c-jgspj"]
Oct 30 19:40:13.661537 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:40:13.661496    4892 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4ad702ac-5c7b-4188-b918-f036e91a209f-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-a348f93c-jgspj\" (UID: \"4ad702ac-5c7b-4188-b918-f036e91a209f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a348f93c-jgspj"
Oct 30 19:40:13.661707 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:40:13.661552    4892 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4ad702ac-5c7b-4188-b918-f036e91a209f-maas-api-xks-ca\") pod \"maas-api-revoke-sub-a348f93c-jgspj\" (UID: \"4ad702ac-5c7b-4188-b918-f036e91a209f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a348f93c-jgspj"
Oct 30 19:40:13.661707 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:40:13.661580    4892 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vzd2d\" (UniqueName: \"kubernetes.io/projected/4ad702ac-5c7b-4188-b918-f036e91a209f-kube-api-access-vzd2d\") pod \"maas-api-revoke-sub-a348f93c-jgspj\" (UID: \"4ad702ac-5c7b-4188-b918-f036e91a209f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a348f93c-jgspj"
Oct 30 19:40:13.672060 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:40:13.672031    4892 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 19:40:13.672208 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:40:13.672079    4892 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 30 19:40:13.672208 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:40:13.672115    4892 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 30 19:40:13.672208 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:40:13.672155    4892 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-nlxp4\""
Oct 30 19:40:13.763175 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:40:13.763148    4892 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4ad702ac-5c7b-4188-b918-f036e91a209f-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-a348f93c-jgspj\" (UID: \"4ad702ac-5c7b-4188-b918-f036e91a209f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a348f93c-jgspj"
Oct 30 19:40:13.763653 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:40:13.763187    4892 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4ad702ac-5c7b-4188-b918-f036e91a209f-maas-api-xks-ca\") pod \"maas-api-revoke-sub-a348f93c-jgspj\" (UID: \"4ad702ac-5c7b-4188-b918-f036e91a209f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a348f93c-jgspj"
Oct 30 19:40:13.763653 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:40:13.763214    4892 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vzd2d\" (UniqueName: \"kubernetes.io/projected/4ad702ac-5c7b-4188-b918-f036e91a209f-kube-api-access-vzd2d\") pod \"maas-api-revoke-sub-a348f93c-jgspj\" (UID: \"4ad702ac-5c7b-4188-b918-f036e91a209f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a348f93c-jgspj"
Oct 30 19:40:13.763839 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:40:13.763817    4892 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4ad702ac-5c7b-4188-b918-f036e91a209f-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-a348f93c-jgspj\" (UID: \"4ad702ac-5c7b-4188-b918-f036e91a209f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a348f93c-jgspj"
Oct 30 19:40:13.765658 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:40:13.765637    4892 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4ad702ac-5c7b-4188-b918-f036e91a209f-maas-api-xks-ca\") pod \"maas-api-revoke-sub-a348f93c-jgspj\" (UID: \"4ad702ac-5c7b-4188-b918-f036e91a209f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a348f93c-jgspj"
Oct 30 19:40:13.819687 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:40:13.819621    4892 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vzd2d\" (UniqueName: \"kubernetes.io/projected/4ad702ac-5c7b-4188-b918-f036e91a209f-kube-api-access-vzd2d\") pod \"maas-api-revoke-sub-a348f93c-jgspj\" (UID: \"4ad702ac-5c7b-4188-b918-f036e91a209f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a348f93c-jgspj"
Oct 30 19:40:13.859727 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:40:13.859690    4892 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-83bd7b88-nzq6n"]
Oct 30 19:40:13.863661 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:40:13.863641    4892 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-83bd7b88-nzq6n"
Oct 30 19:40:13.884644 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:40:13.884621    4892 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a348f93c-jgspj"
Oct 30 19:40:13.921575 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:40:13.921548    4892 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-83bd7b88-nzq6n"]
Oct 30 19:40:13.966215 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:40:13.965800    4892 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vvn4f\" (UniqueName: \"kubernetes.io/projected/3f59f302-5d8d-49a6-9cb9-2975672a6a8f-kube-api-access-vvn4f\") pod \"maas-api-revoke-sub-83bd7b88-nzq6n\" (UID: \"3f59f302-5d8d-49a6-9cb9-2975672a6a8f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-83bd7b88-nzq6n"
Oct 30 19:40:13.966215 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:40:13.965993    4892 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3f59f302-5d8d-49a6-9cb9-2975672a6a8f-maas-api-xks-ca\") pod \"maas-api-revoke-sub-83bd7b88-nzq6n\" (UID: \"3f59f302-5d8d-49a6-9cb9-2975672a6a8f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-83bd7b88-nzq6n"
Oct 30 19:40:13.966215 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:40:13.966094    4892 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3f59f302-5d8d-49a6-9cb9-2975672a6a8f-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-83bd7b88-nzq6n\" (UID: \"3f59f302-5d8d-49a6-9cb9-2975672a6a8f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-83bd7b88-nzq6n"
Oct 30 19:40:14.067823 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:40:14.067781    4892 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vvn4f\" (UniqueName: \"kubernetes.io/projected/3f59f302-5d8d-49a6-9cb9-2975672a6a8f-kube-api-access-vvn4f\") pod \"maas-api-revoke-sub-83bd7b88-nzq6n\" (UID: \"3f59f302-5d8d-49a6-9cb9-2975672a6a8f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-83bd7b88-nzq6n"
Oct 30 19:40:14.068042 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:40:14.067912    4892 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3f59f302-5d8d-49a6-9cb9-2975672a6a8f-maas-api-xks-ca\") pod \"maas-api-revoke-sub-83bd7b88-nzq6n\" (UID: \"3f59f302-5d8d-49a6-9cb9-2975672a6a8f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-83bd7b88-nzq6n"
Oct 30 19:40:14.068042 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:40:14.068032    4892 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3f59f302-5d8d-49a6-9cb9-2975672a6a8f-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-83bd7b88-nzq6n\" (UID: \"3f59f302-5d8d-49a6-9cb9-2975672a6a8f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-83bd7b88-nzq6n"
Oct 30 19:40:14.068871 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:40:14.068796    4892 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3f59f302-5d8d-49a6-9cb9-2975672a6a8f-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-83bd7b88-nzq6n\" (UID: \"3f59f302-5d8d-49a6-9cb9-2975672a6a8f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-83bd7b88-nzq6n"
Oct 30 19:40:14.070540 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:40:14.070483    4892 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3f59f302-5d8d-49a6-9cb9-2975672a6a8f-maas-api-xks-ca\") pod \"maas-api-revoke-sub-83bd7b88-nzq6n\" (UID: \"3f59f302-5d8d-49a6-9cb9-2975672a6a8f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-83bd7b88-nzq6n"
Oct 30 19:40:14.124534 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:40:14.124506    4892 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vvn4f\" (UniqueName: \"kubernetes.io/projected/3f59f302-5d8d-49a6-9cb9-2975672a6a8f-kube-api-access-vvn4f\") pod \"maas-api-revoke-sub-83bd7b88-nzq6n\" (UID: \"3f59f302-5d8d-49a6-9cb9-2975672a6a8f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-83bd7b88-nzq6n"
Oct 30 19:40:14.178110 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:40:14.178076    4892 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-83bd7b88-nzq6n"
Oct 30 19:40:14.272576 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:40:14.272544    4892 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a348f93c-jgspj"]
Oct 30 19:40:14.560180 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:40:14.560153    4892 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-83bd7b88-nzq6n"]
Oct 30 19:40:14.560819 ip-10-0-132-82 kubenswrapper[4892]: W1030 19:40:14.560789    4892 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3f59f302_5d8d_49a6_9cb9_2975672a6a8f.slice/crio-f783349b44216186fb03cf6a7c09a1dcaf52a971af772fd1cce177ecc994df8d WatchSource:0}: Error finding container f783349b44216186fb03cf6a7c09a1dcaf52a971af772fd1cce177ecc994df8d: Status 404 returned error can't find the container with id f783349b44216186fb03cf6a7c09a1dcaf52a971af772fd1cce177ecc994df8d
Oct 30 19:40:14.762713 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:40:14.762676    4892 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-83bd7b88-nzq6n" event={"ID":"3f59f302-5d8d-49a6-9cb9-2975672a6a8f","Type":"ContainerStarted","Data":"f783349b44216186fb03cf6a7c09a1dcaf52a971af772fd1cce177ecc994df8d"}
Oct 30 19:40:14.764586 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:40:14.764556    4892 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a348f93c-jgspj" event={"ID":"4ad702ac-5c7b-4188-b918-f036e91a209f","Type":"ContainerStarted","Data":"d0fd81f5a45ed327e9929fc61728405a8f866e0f8524dc02ef54489ec55c3958"}
Oct 30 19:40:14.765058 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:40:14.764606    4892 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a348f93c-jgspj" event={"ID":"4ad702ac-5c7b-4188-b918-f036e91a209f","Type":"ContainerStarted","Data":"0d6dd4de3fa972175b56e0a8b9cc8d6c3fda21e2fc42f1dc5ff502ee39e744a4"}
Oct 30 19:40:14.879729 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:40:14.879670    4892 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a348f93c-jgspj" podStartSLOduration=1.879651395 podStartE2EDuration="1.879651395s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:40:13 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:40:14.877115814 +0000 UTC m=+1462.447581455" watchObservedRunningTime="2026-10-30 19:40:14.879651395 +0000 UTC m=+1462.450116994"
Oct 30 19:40:15.775633 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:40:15.775537    4892 generic.go:356] "Generic (PLEG): container finished" podID="3f59f302-5d8d-49a6-9cb9-2975672a6a8f" containerID="f9c43cddc7ff1f77c9beb4e96f00348f5744821ba2e165f98d4dac055b950571" exitCode=0
Oct 30 19:40:15.776137 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:40:15.775632    4892 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-83bd7b88-nzq6n" event={"ID":"3f59f302-5d8d-49a6-9cb9-2975672a6a8f","Type":"ContainerDied","Data":"f9c43cddc7ff1f77c9beb4e96f00348f5744821ba2e165f98d4dac055b950571"}
Oct 30 19:40:15.777398 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:40:15.777372    4892 generic.go:356] "Generic (PLEG): container finished" podID="4ad702ac-5c7b-4188-b918-f036e91a209f" containerID="d0fd81f5a45ed327e9929fc61728405a8f866e0f8524dc02ef54489ec55c3958" exitCode=0
Oct 30 19:40:15.777512 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:40:15.777449    4892 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a348f93c-jgspj" event={"ID":"4ad702ac-5c7b-4188-b918-f036e91a209f","Type":"ContainerDied","Data":"d0fd81f5a45ed327e9929fc61728405a8f866e0f8524dc02ef54489ec55c3958"}
Oct 30 19:40:16.959045 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:40:16.959021    4892 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-83bd7b88-nzq6n"
Oct 30 19:40:16.962003 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:40:16.961985    4892 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a348f93c-jgspj"
Oct 30 19:40:17.102944 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:40:17.102874    4892 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/3f59f302-5d8d-49a6-9cb9-2975672a6a8f-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3f59f302-5d8d-49a6-9cb9-2975672a6a8f-maas-api-openshift-ca\") pod \"3f59f302-5d8d-49a6-9cb9-2975672a6a8f\" (UID: \"3f59f302-5d8d-49a6-9cb9-2975672a6a8f\") "
Oct 30 19:40:17.102944 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:40:17.102927    4892 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3f59f302-5d8d-49a6-9cb9-2975672a6a8f-kube-api-access-vvn4f\" (UniqueName: \"kubernetes.io/projected/3f59f302-5d8d-49a6-9cb9-2975672a6a8f-kube-api-access-vvn4f\") pod \"3f59f302-5d8d-49a6-9cb9-2975672a6a8f\" (UID: \"3f59f302-5d8d-49a6-9cb9-2975672a6a8f\") "
Oct 30 19:40:17.103171 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:40:17.103074    4892 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/4ad702ac-5c7b-4188-b918-f036e91a209f-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4ad702ac-5c7b-4188-b918-f036e91a209f-maas-api-xks-ca\") pod \"4ad702ac-5c7b-4188-b918-f036e91a209f\" (UID: \"4ad702ac-5c7b-4188-b918-f036e91a209f\") "
Oct 30 19:40:17.103171 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:40:17.103133    4892 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/4ad702ac-5c7b-4188-b918-f036e91a209f-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4ad702ac-5c7b-4188-b918-f036e91a209f-maas-api-openshift-ca\") pod \"4ad702ac-5c7b-4188-b918-f036e91a209f\" (UID: \"4ad702ac-5c7b-4188-b918-f036e91a209f\") "
Oct 30 19:40:17.103286 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:40:17.103194    4892 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/3f59f302-5d8d-49a6-9cb9-2975672a6a8f-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3f59f302-5d8d-49a6-9cb9-2975672a6a8f-maas-api-xks-ca\") pod \"3f59f302-5d8d-49a6-9cb9-2975672a6a8f\" (UID: \"3f59f302-5d8d-49a6-9cb9-2975672a6a8f\") "
Oct 30 19:40:17.103286 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:40:17.103267    4892 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4ad702ac-5c7b-4188-b918-f036e91a209f-kube-api-access-vzd2d\" (UniqueName: \"kubernetes.io/projected/4ad702ac-5c7b-4188-b918-f036e91a209f-kube-api-access-vzd2d\") pod \"4ad702ac-5c7b-4188-b918-f036e91a209f\" (UID: \"4ad702ac-5c7b-4188-b918-f036e91a209f\") "
Oct 30 19:40:17.103789 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:40:17.103503    4892 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4ad702ac-5c7b-4188-b918-f036e91a209f-maas-api-openshift-ca" pod "4ad702ac-5c7b-4188-b918-f036e91a209f" (UID: "4ad702ac-5c7b-4188-b918-f036e91a209f"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 19:40:17.103789 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:40:17.103664    4892 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3f59f302-5d8d-49a6-9cb9-2975672a6a8f-maas-api-openshift-ca" pod "3f59f302-5d8d-49a6-9cb9-2975672a6a8f" (UID: "3f59f302-5d8d-49a6-9cb9-2975672a6a8f"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 19:40:17.104111 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:40:17.103893    4892 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3f59f302-5d8d-49a6-9cb9-2975672a6a8f-maas-api-openshift-ca\") on node \"ip-10-0-132-82.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 19:40:17.104111 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:40:17.103915    4892 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4ad702ac-5c7b-4188-b918-f036e91a209f-maas-api-openshift-ca\") on node \"ip-10-0-132-82.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 19:40:17.105509 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:40:17.105485    4892 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/4ad702ac-5c7b-4188-b918-f036e91a209f-maas-api-xks-ca" pod "4ad702ac-5c7b-4188-b918-f036e91a209f" (UID: "4ad702ac-5c7b-4188-b918-f036e91a209f"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 19:40:17.105931 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:40:17.105902    4892 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3f59f302-5d8d-49a6-9cb9-2975672a6a8f-kube-api-access-vvn4f" pod "3f59f302-5d8d-49a6-9cb9-2975672a6a8f" (UID: "3f59f302-5d8d-49a6-9cb9-2975672a6a8f"). InnerVolumeSpecName "kube-api-access-vvn4f". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:40:17.106241 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:40:17.106006    4892 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/3f59f302-5d8d-49a6-9cb9-2975672a6a8f-maas-api-xks-ca" pod "3f59f302-5d8d-49a6-9cb9-2975672a6a8f" (UID: "3f59f302-5d8d-49a6-9cb9-2975672a6a8f"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 19:40:17.106323 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:40:17.106299    4892 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4ad702ac-5c7b-4188-b918-f036e91a209f-kube-api-access-vzd2d" pod "4ad702ac-5c7b-4188-b918-f036e91a209f" (UID: "4ad702ac-5c7b-4188-b918-f036e91a209f"). InnerVolumeSpecName "kube-api-access-vzd2d". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:40:17.204498 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:40:17.204470    4892 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3f59f302-5d8d-49a6-9cb9-2975672a6a8f-maas-api-xks-ca\") on node \"ip-10-0-132-82.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 19:40:17.204498 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:40:17.204497    4892 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vzd2d\" (UniqueName: \"kubernetes.io/projected/4ad702ac-5c7b-4188-b918-f036e91a209f-kube-api-access-vzd2d\") on node \"ip-10-0-132-82.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 19:40:17.204711 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:40:17.204508    4892 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vvn4f\" (UniqueName: \"kubernetes.io/projected/3f59f302-5d8d-49a6-9cb9-2975672a6a8f-kube-api-access-vvn4f\") on node \"ip-10-0-132-82.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 19:40:17.204711 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:40:17.204517    4892 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4ad702ac-5c7b-4188-b918-f036e91a209f-maas-api-xks-ca\") on node \"ip-10-0-132-82.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 19:40:17.792214 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:40:17.792185    4892 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-83bd7b88-nzq6n"
Oct 30 19:40:17.792402 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:40:17.792184    4892 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-83bd7b88-nzq6n" event={"ID":"3f59f302-5d8d-49a6-9cb9-2975672a6a8f","Type":"ContainerDied","Data":"f783349b44216186fb03cf6a7c09a1dcaf52a971af772fd1cce177ecc994df8d"}
Oct 30 19:40:17.792402 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:40:17.792325    4892 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="f783349b44216186fb03cf6a7c09a1dcaf52a971af772fd1cce177ecc994df8d"
Oct 30 19:40:17.793671 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:40:17.793651    4892 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a348f93c-jgspj"
Oct 30 19:40:17.793795 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:40:17.793728    4892 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a348f93c-jgspj" event={"ID":"4ad702ac-5c7b-4188-b918-f036e91a209f","Type":"ContainerDied","Data":"0d6dd4de3fa972175b56e0a8b9cc8d6c3fda21e2fc42f1dc5ff502ee39e744a4"}
Oct 30 19:40:17.793795 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:40:17.793772    4892 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="0d6dd4de3fa972175b56e0a8b9cc8d6c3fda21e2fc42f1dc5ff502ee39e744a4"
Oct 30 19:40:21.214324 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:40:21.214282    4892 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w4-b5178f/distinct-2-w4-b5178f-kserve-69ddf8b665-9pdth"
Oct 30 19:40:21.230426 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:40:21.230402    4892 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w4-b5178f/distinct-2-w4-b5178f-kserve-69ddf8b665-9pdth"
Oct 30 19:40:21.834747 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:40:21.834717    4892 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-b5178f/distinct-2-w4-b5178f-kserve-69ddf8b665-9pdth"
Oct 30 19:40:24.854836 ip-10-0-132-82 kubenswrapper[4892]: E1030 19:40:24.854581    4892 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-jrm87" containerName="extractor"
Oct 30 19:40:25.063595 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:40:25.063553    4892 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-6e1ad926-4tgkm"]
Oct 30 19:40:25.080551 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:40:25.080516    4892 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-6e1ad926-4tgkm"]
Oct 30 19:40:26.573390 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:40:26.573353    4892 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="87c18cb1-32c8-4596-84cc-05375b9e760c" path="/var/lib/kubelet/pods/87c18cb1-32c8-4596-84cc-05375b9e760c/volumes"
Oct 30 19:40:28.894323 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:40:28.894293    4892 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-8a640f9d-8ddnv"]
Oct 30 19:40:28.894785 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:40:28.894770    4892 cpu_manager.go:414] "containerMap: removing container" podUID="4ad702ac-5c7b-4188-b918-f036e91a209f" containerName="revoke-keys"
Oct 30 19:40:28.894834 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:40:28.894786    4892 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4ad702ac-5c7b-4188-b918-f036e91a209f" containerName="revoke-keys"
Oct 30 19:40:28.894834 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:40:28.894811    4892 cpu_manager.go:414] "containerMap: removing container" podUID="3f59f302-5d8d-49a6-9cb9-2975672a6a8f" containerName="revoke-keys"
Oct 30 19:40:28.894834 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:40:28.894817    4892 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3f59f302-5d8d-49a6-9cb9-2975672a6a8f" containerName="revoke-keys"
Oct 30 19:40:28.894941 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:40:28.894914    4892 memory_manager.go:365] "RemoveStaleState removing state" podUID="4ad702ac-5c7b-4188-b918-f036e91a209f" containerName="revoke-keys"
Oct 30 19:40:28.894941 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:40:28.894927    4892 memory_manager.go:365] "RemoveStaleState removing state" podUID="3f59f302-5d8d-49a6-9cb9-2975672a6a8f" containerName="revoke-keys"
Oct 30 19:40:28.899475 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:40:28.899457    4892 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8a640f9d-8ddnv"
Oct 30 19:40:28.931190 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:40:28.931166    4892 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1001fd5a-3158-413d-affa-26daff184f72-maas-api-xks-ca\") pod \"maas-api-revoke-sub-8a640f9d-8ddnv\" (UID: \"1001fd5a-3158-413d-affa-26daff184f72\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8a640f9d-8ddnv"
Oct 30 19:40:28.931331 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:40:28.931227    4892 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1001fd5a-3158-413d-affa-26daff184f72-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-8a640f9d-8ddnv\" (UID: \"1001fd5a-3158-413d-affa-26daff184f72\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8a640f9d-8ddnv"
Oct 30 19:40:28.931385 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:40:28.931335    4892 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-98hcn\" (UniqueName: \"kubernetes.io/projected/1001fd5a-3158-413d-affa-26daff184f72-kube-api-access-98hcn\") pod \"maas-api-revoke-sub-8a640f9d-8ddnv\" (UID: \"1001fd5a-3158-413d-affa-26daff184f72\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8a640f9d-8ddnv"
Oct 30 19:40:28.951746 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:40:28.951725    4892 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 30 19:40:28.952111 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:40:28.952093    4892 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 30 19:40:28.955228 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:40:28.955206    4892 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-8a640f9d-8ddnv"]
Oct 30 19:40:29.003424 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:40:29.003400    4892 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-fsfvk"]
Oct 30 19:40:29.021994 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:40:29.021938    4892 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-nlxp4\""
Oct 30 19:40:29.022134 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:40:29.022089    4892 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 19:40:29.032550 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:40:29.032525    4892 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1001fd5a-3158-413d-affa-26daff184f72-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-8a640f9d-8ddnv\" (UID: \"1001fd5a-3158-413d-affa-26daff184f72\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8a640f9d-8ddnv"
Oct 30 19:40:29.032681 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:40:29.032569    4892 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-98hcn\" (UniqueName: \"kubernetes.io/projected/1001fd5a-3158-413d-affa-26daff184f72-kube-api-access-98hcn\") pod \"maas-api-revoke-sub-8a640f9d-8ddnv\" (UID: \"1001fd5a-3158-413d-affa-26daff184f72\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8a640f9d-8ddnv"
Oct 30 19:40:29.032681 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:40:29.032653    4892 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1001fd5a-3158-413d-affa-26daff184f72-maas-api-xks-ca\") pod \"maas-api-revoke-sub-8a640f9d-8ddnv\" (UID: \"1001fd5a-3158-413d-affa-26daff184f72\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8a640f9d-8ddnv"
Oct 30 19:40:29.033303 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:40:29.033278    4892 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1001fd5a-3158-413d-affa-26daff184f72-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-8a640f9d-8ddnv\" (UID: \"1001fd5a-3158-413d-affa-26daff184f72\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8a640f9d-8ddnv"
Oct 30 19:40:29.035174 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:40:29.035151    4892 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1001fd5a-3158-413d-affa-26daff184f72-maas-api-xks-ca\") pod \"maas-api-revoke-sub-8a640f9d-8ddnv\" (UID: \"1001fd5a-3158-413d-affa-26daff184f72\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8a640f9d-8ddnv"
Oct 30 19:40:29.102245 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:40:29.102222    4892 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-98hcn\" (UniqueName: \"kubernetes.io/projected/1001fd5a-3158-413d-affa-26daff184f72-kube-api-access-98hcn\") pod \"maas-api-revoke-sub-8a640f9d-8ddnv\" (UID: \"1001fd5a-3158-413d-affa-26daff184f72\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8a640f9d-8ddnv"
Oct 30 19:40:29.212473 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:40:29.212396    4892 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8a640f9d-8ddnv"
Oct 30 19:40:29.597652 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:40:29.597626    4892 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-8a640f9d-8ddnv"]
Oct 30 19:40:29.598385 ip-10-0-132-82 kubenswrapper[4892]: W1030 19:40:29.598355    4892 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1001fd5a_3158_413d_affa_26daff184f72.slice/crio-742bdd0b8cd71cd01065536b63827e0c3cbd08f157b74432a88579a993f4d4fb WatchSource:0}: Error finding container 742bdd0b8cd71cd01065536b63827e0c3cbd08f157b74432a88579a993f4d4fb: Status 404 returned error can't find the container with id 742bdd0b8cd71cd01065536b63827e0c3cbd08f157b74432a88579a993f4d4fb
Oct 30 19:40:29.896086 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:40:29.896045    4892 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8a640f9d-8ddnv" event={"ID":"1001fd5a-3158-413d-affa-26daff184f72","Type":"ContainerStarted","Data":"742bdd0b8cd71cd01065536b63827e0c3cbd08f157b74432a88579a993f4d4fb"}
Oct 30 19:40:30.904275 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:40:30.904239    4892 generic.go:356] "Generic (PLEG): container finished" podID="1001fd5a-3158-413d-affa-26daff184f72" containerID="faf912cff9432e5db97615f7ffa6751fe6473e86d1274ddb4bbccf55c8a8b3b0" exitCode=0
Oct 30 19:40:30.904655 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:40:30.904323    4892 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8a640f9d-8ddnv" event={"ID":"1001fd5a-3158-413d-affa-26daff184f72","Type":"ContainerDied","Data":"faf912cff9432e5db97615f7ffa6751fe6473e86d1274ddb4bbccf55c8a8b3b0"}
Oct 30 19:40:32.059846 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:40:32.059820    4892 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8a640f9d-8ddnv"
Oct 30 19:40:32.167078 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:40:32.166951    4892 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/1001fd5a-3158-413d-affa-26daff184f72-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1001fd5a-3158-413d-affa-26daff184f72-maas-api-openshift-ca\") pod \"1001fd5a-3158-413d-affa-26daff184f72\" (UID: \"1001fd5a-3158-413d-affa-26daff184f72\") "
Oct 30 19:40:32.167220 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:40:32.167101    4892 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1001fd5a-3158-413d-affa-26daff184f72-kube-api-access-98hcn\" (UniqueName: \"kubernetes.io/projected/1001fd5a-3158-413d-affa-26daff184f72-kube-api-access-98hcn\") pod \"1001fd5a-3158-413d-affa-26daff184f72\" (UID: \"1001fd5a-3158-413d-affa-26daff184f72\") "
Oct 30 19:40:32.167220 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:40:32.167130    4892 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/1001fd5a-3158-413d-affa-26daff184f72-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1001fd5a-3158-413d-affa-26daff184f72-maas-api-xks-ca\") pod \"1001fd5a-3158-413d-affa-26daff184f72\" (UID: \"1001fd5a-3158-413d-affa-26daff184f72\") "
Oct 30 19:40:32.167370 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:40:32.167289    4892 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1001fd5a-3158-413d-affa-26daff184f72-maas-api-openshift-ca" pod "1001fd5a-3158-413d-affa-26daff184f72" (UID: "1001fd5a-3158-413d-affa-26daff184f72"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 19:40:32.167548 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:40:32.167533    4892 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1001fd5a-3158-413d-affa-26daff184f72-maas-api-openshift-ca\") on node \"ip-10-0-132-82.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 19:40:32.169341 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:40:32.169312    4892 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/1001fd5a-3158-413d-affa-26daff184f72-maas-api-xks-ca" pod "1001fd5a-3158-413d-affa-26daff184f72" (UID: "1001fd5a-3158-413d-affa-26daff184f72"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 19:40:32.169524 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:40:32.169506    4892 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1001fd5a-3158-413d-affa-26daff184f72-kube-api-access-98hcn" pod "1001fd5a-3158-413d-affa-26daff184f72" (UID: "1001fd5a-3158-413d-affa-26daff184f72"). InnerVolumeSpecName "kube-api-access-98hcn". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:40:32.269030 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:40:32.269001    4892 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-98hcn\" (UniqueName: \"kubernetes.io/projected/1001fd5a-3158-413d-affa-26daff184f72-kube-api-access-98hcn\") on node \"ip-10-0-132-82.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 19:40:32.269030 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:40:32.269026    4892 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1001fd5a-3158-413d-affa-26daff184f72-maas-api-xks-ca\") on node \"ip-10-0-132-82.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 19:40:32.926647 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:40:32.926616    4892 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8a640f9d-8ddnv"
Oct 30 19:40:32.926831 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:40:32.926624    4892 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8a640f9d-8ddnv" event={"ID":"1001fd5a-3158-413d-affa-26daff184f72","Type":"ContainerDied","Data":"742bdd0b8cd71cd01065536b63827e0c3cbd08f157b74432a88579a993f4d4fb"}
Oct 30 19:40:32.926831 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:40:32.926775    4892 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="742bdd0b8cd71cd01065536b63827e0c3cbd08f157b74432a88579a993f4d4fb"
Oct 30 19:40:39.048169 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:40:39.048134    4892 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c08feb3f-fwb78"]
Oct 30 19:40:39.048603 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:40:39.048589    4892 cpu_manager.go:414] "containerMap: removing container" podUID="1001fd5a-3158-413d-affa-26daff184f72" containerName="revoke-keys"
Oct 30 19:40:39.048642 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:40:39.048605    4892 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1001fd5a-3158-413d-affa-26daff184f72" containerName="revoke-keys"
Oct 30 19:40:39.048723 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:40:39.048712    4892 memory_manager.go:365] "RemoveStaleState removing state" podUID="1001fd5a-3158-413d-affa-26daff184f72" containerName="revoke-keys"
Oct 30 19:40:39.052359 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:40:39.052341    4892 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c08feb3f-fwb78"
Oct 30 19:40:39.139936 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:40:39.139901    4892 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/35e6d043-b9a3-4e16-bcb3-49de88dcbe75-maas-api-xks-ca\") pod \"maas-api-revoke-sub-c08feb3f-fwb78\" (UID: \"35e6d043-b9a3-4e16-bcb3-49de88dcbe75\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c08feb3f-fwb78"
Oct 30 19:40:39.140093 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:40:39.139949    4892 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bsczq\" (UniqueName: \"kubernetes.io/projected/35e6d043-b9a3-4e16-bcb3-49de88dcbe75-kube-api-access-bsczq\") pod \"maas-api-revoke-sub-c08feb3f-fwb78\" (UID: \"35e6d043-b9a3-4e16-bcb3-49de88dcbe75\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c08feb3f-fwb78"
Oct 30 19:40:39.140137 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:40:39.140115    4892 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/35e6d043-b9a3-4e16-bcb3-49de88dcbe75-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-c08feb3f-fwb78\" (UID: \"35e6d043-b9a3-4e16-bcb3-49de88dcbe75\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c08feb3f-fwb78"
Oct 30 19:40:39.154563 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:40:39.154540    4892 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 19:40:39.154699 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:40:39.154594    4892 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-nlxp4\""
Oct 30 19:40:39.154699 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:40:39.154555    4892 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 30 19:40:39.154699 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:40:39.154616    4892 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 30 19:40:39.160232 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:40:39.160212    4892 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c08feb3f-fwb78"]
Oct 30 19:40:39.252377 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:40:39.247095    4892 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/35e6d043-b9a3-4e16-bcb3-49de88dcbe75-maas-api-xks-ca\") pod \"maas-api-revoke-sub-c08feb3f-fwb78\" (UID: \"35e6d043-b9a3-4e16-bcb3-49de88dcbe75\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c08feb3f-fwb78"
Oct 30 19:40:39.252377 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:40:39.247158    4892 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bsczq\" (UniqueName: \"kubernetes.io/projected/35e6d043-b9a3-4e16-bcb3-49de88dcbe75-kube-api-access-bsczq\") pod \"maas-api-revoke-sub-c08feb3f-fwb78\" (UID: \"35e6d043-b9a3-4e16-bcb3-49de88dcbe75\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c08feb3f-fwb78"
Oct 30 19:40:39.252377 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:40:39.247250    4892 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/35e6d043-b9a3-4e16-bcb3-49de88dcbe75-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-c08feb3f-fwb78\" (UID: \"35e6d043-b9a3-4e16-bcb3-49de88dcbe75\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c08feb3f-fwb78"
Oct 30 19:40:39.252377 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:40:39.248307    4892 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/35e6d043-b9a3-4e16-bcb3-49de88dcbe75-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-c08feb3f-fwb78\" (UID: \"35e6d043-b9a3-4e16-bcb3-49de88dcbe75\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c08feb3f-fwb78"
Oct 30 19:40:39.268837 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:40:39.268803    4892 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/35e6d043-b9a3-4e16-bcb3-49de88dcbe75-maas-api-xks-ca\") pod \"maas-api-revoke-sub-c08feb3f-fwb78\" (UID: \"35e6d043-b9a3-4e16-bcb3-49de88dcbe75\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c08feb3f-fwb78"
Oct 30 19:40:39.306018 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:40:39.305897    4892 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bsczq\" (UniqueName: \"kubernetes.io/projected/35e6d043-b9a3-4e16-bcb3-49de88dcbe75-kube-api-access-bsczq\") pod \"maas-api-revoke-sub-c08feb3f-fwb78\" (UID: \"35e6d043-b9a3-4e16-bcb3-49de88dcbe75\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c08feb3f-fwb78"
Oct 30 19:40:39.365496 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:40:39.365468    4892 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c08feb3f-fwb78"
Oct 30 19:40:39.741711 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:40:39.741682    4892 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c08feb3f-fwb78"]
Oct 30 19:40:39.742593 ip-10-0-132-82 kubenswrapper[4892]: W1030 19:40:39.742569    4892 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod35e6d043_b9a3_4e16_bcb3_49de88dcbe75.slice/crio-be8d535234c6886477743a21f7d2e8905765edc5863461c5613be4f4db845178 WatchSource:0}: Error finding container be8d535234c6886477743a21f7d2e8905765edc5863461c5613be4f4db845178: Status 404 returned error can't find the container with id be8d535234c6886477743a21f7d2e8905765edc5863461c5613be4f4db845178
Oct 30 19:40:39.984356 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:40:39.984323    4892 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c08feb3f-fwb78" event={"ID":"35e6d043-b9a3-4e16-bcb3-49de88dcbe75","Type":"ContainerStarted","Data":"be8d535234c6886477743a21f7d2e8905765edc5863461c5613be4f4db845178"}
Oct 30 19:40:40.994163 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:40:40.994127    4892 generic.go:356] "Generic (PLEG): container finished" podID="35e6d043-b9a3-4e16-bcb3-49de88dcbe75" containerID="b85fea48a051823d697f6c6ce1cf4b097cc971f1062d085c42dedb359afebfb3" exitCode=0
Oct 30 19:40:40.994589 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:40:40.994210    4892 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c08feb3f-fwb78" event={"ID":"35e6d043-b9a3-4e16-bcb3-49de88dcbe75","Type":"ContainerDied","Data":"b85fea48a051823d697f6c6ce1cf4b097cc971f1062d085c42dedb359afebfb3"}
Oct 30 19:40:42.161945 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:40:42.161916    4892 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c08feb3f-fwb78"
Oct 30 19:40:42.279260 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:40:42.279178    4892 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/35e6d043-b9a3-4e16-bcb3-49de88dcbe75-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/35e6d043-b9a3-4e16-bcb3-49de88dcbe75-maas-api-openshift-ca\") pod \"35e6d043-b9a3-4e16-bcb3-49de88dcbe75\" (UID: \"35e6d043-b9a3-4e16-bcb3-49de88dcbe75\") "
Oct 30 19:40:42.279260 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:40:42.279246    4892 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/35e6d043-b9a3-4e16-bcb3-49de88dcbe75-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/35e6d043-b9a3-4e16-bcb3-49de88dcbe75-maas-api-xks-ca\") pod \"35e6d043-b9a3-4e16-bcb3-49de88dcbe75\" (UID: \"35e6d043-b9a3-4e16-bcb3-49de88dcbe75\") "
Oct 30 19:40:42.279518 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:40:42.279386    4892 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/35e6d043-b9a3-4e16-bcb3-49de88dcbe75-kube-api-access-bsczq\" (UniqueName: \"kubernetes.io/projected/35e6d043-b9a3-4e16-bcb3-49de88dcbe75-kube-api-access-bsczq\") pod \"35e6d043-b9a3-4e16-bcb3-49de88dcbe75\" (UID: \"35e6d043-b9a3-4e16-bcb3-49de88dcbe75\") "
Oct 30 19:40:42.279644 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:40:42.279619    4892 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/35e6d043-b9a3-4e16-bcb3-49de88dcbe75-maas-api-openshift-ca" pod "35e6d043-b9a3-4e16-bcb3-49de88dcbe75" (UID: "35e6d043-b9a3-4e16-bcb3-49de88dcbe75"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 19:40:42.279823 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:40:42.279797    4892 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/35e6d043-b9a3-4e16-bcb3-49de88dcbe75-maas-api-openshift-ca\") on node \"ip-10-0-132-82.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 19:40:42.282092 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:40:42.282049    4892 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/35e6d043-b9a3-4e16-bcb3-49de88dcbe75-maas-api-xks-ca" pod "35e6d043-b9a3-4e16-bcb3-49de88dcbe75" (UID: "35e6d043-b9a3-4e16-bcb3-49de88dcbe75"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 19:40:42.282274 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:40:42.282249    4892 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/35e6d043-b9a3-4e16-bcb3-49de88dcbe75-kube-api-access-bsczq" pod "35e6d043-b9a3-4e16-bcb3-49de88dcbe75" (UID: "35e6d043-b9a3-4e16-bcb3-49de88dcbe75"). InnerVolumeSpecName "kube-api-access-bsczq". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:40:42.380717 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:40:42.380682    4892 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/35e6d043-b9a3-4e16-bcb3-49de88dcbe75-maas-api-xks-ca\") on node \"ip-10-0-132-82.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 19:40:42.380717 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:40:42.380713    4892 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bsczq\" (UniqueName: \"kubernetes.io/projected/35e6d043-b9a3-4e16-bcb3-49de88dcbe75-kube-api-access-bsczq\") on node \"ip-10-0-132-82.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 19:40:43.010650 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:40:43.010622    4892 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c08feb3f-fwb78"
Oct 30 19:40:43.010794 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:40:43.010621    4892 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c08feb3f-fwb78" event={"ID":"35e6d043-b9a3-4e16-bcb3-49de88dcbe75","Type":"ContainerDied","Data":"be8d535234c6886477743a21f7d2e8905765edc5863461c5613be4f4db845178"}
Oct 30 19:40:43.010794 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:40:43.010747    4892 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="be8d535234c6886477743a21f7d2e8905765edc5863461c5613be4f4db845178"
Oct 30 19:40:44.055021 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:40:44.054987    4892 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-be1662e4-7l2wh"]
Oct 30 19:40:44.103621 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:40:44.103597    4892 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-be1662e4-7l2wh"]
Oct 30 19:40:44.572098 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:40:44.572062    4892 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f4922dc5-e50a-4758-8098-aad71d12fa83" path="/var/lib/kubelet/pods/f4922dc5-e50a-4758-8098-aad71d12fa83/volumes"
Oct 30 19:40:46.623816 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:40:46.623784    4892 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c9e9de3e-cd94p"]
Oct 30 19:40:46.624321 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:40:46.624304    4892 cpu_manager.go:414] "containerMap: removing container" podUID="35e6d043-b9a3-4e16-bcb3-49de88dcbe75" containerName="revoke-keys"
Oct 30 19:40:46.624373 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:40:46.624324    4892 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="35e6d043-b9a3-4e16-bcb3-49de88dcbe75" containerName="revoke-keys"
Oct 30 19:40:46.624451 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:40:46.624441    4892 memory_manager.go:365] "RemoveStaleState removing state" podUID="35e6d043-b9a3-4e16-bcb3-49de88dcbe75" containerName="revoke-keys"
Oct 30 19:40:46.628180 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:40:46.628159    4892 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c9e9de3e-cd94p"
Oct 30 19:40:46.680558 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:40:46.680533    4892 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 19:40:46.680558 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:40:46.680548    4892 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 30 19:40:46.729724 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:40:46.729697    4892 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 30 19:40:46.729916 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:40:46.729768    4892 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-nlxp4\""
Oct 30 19:40:46.733933 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:40:46.733911    4892 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c9e9de3e-cd94p"]
Oct 30 19:40:46.737854 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:40:46.737833    4892 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ktxr2\" (UniqueName: \"kubernetes.io/projected/7e48e8e5-076a-4ac3-ade6-eb7091526083-kube-api-access-ktxr2\") pod \"maas-api-revoke-sub-c9e9de3e-cd94p\" (UID: \"7e48e8e5-076a-4ac3-ade6-eb7091526083\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c9e9de3e-cd94p"
Oct 30 19:40:46.737942 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:40:46.737875    4892 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7e48e8e5-076a-4ac3-ade6-eb7091526083-maas-api-xks-ca\") pod \"maas-api-revoke-sub-c9e9de3e-cd94p\" (UID: \"7e48e8e5-076a-4ac3-ade6-eb7091526083\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c9e9de3e-cd94p"
Oct 30 19:40:46.737942 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:40:46.737898    4892 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7e48e8e5-076a-4ac3-ade6-eb7091526083-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-c9e9de3e-cd94p\" (UID: \"7e48e8e5-076a-4ac3-ade6-eb7091526083\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c9e9de3e-cd94p"
Oct 30 19:40:46.842681 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:40:46.842638    4892 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ktxr2\" (UniqueName: \"kubernetes.io/projected/7e48e8e5-076a-4ac3-ade6-eb7091526083-kube-api-access-ktxr2\") pod \"maas-api-revoke-sub-c9e9de3e-cd94p\" (UID: \"7e48e8e5-076a-4ac3-ade6-eb7091526083\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c9e9de3e-cd94p"
Oct 30 19:40:46.843389 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:40:46.843361    4892 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7e48e8e5-076a-4ac3-ade6-eb7091526083-maas-api-xks-ca\") pod \"maas-api-revoke-sub-c9e9de3e-cd94p\" (UID: \"7e48e8e5-076a-4ac3-ade6-eb7091526083\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c9e9de3e-cd94p"
Oct 30 19:40:46.844200 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:40:46.844180    4892 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7e48e8e5-076a-4ac3-ade6-eb7091526083-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-c9e9de3e-cd94p\" (UID: \"7e48e8e5-076a-4ac3-ade6-eb7091526083\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c9e9de3e-cd94p"
Oct 30 19:40:46.845288 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:40:46.845072    4892 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7e48e8e5-076a-4ac3-ade6-eb7091526083-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-c9e9de3e-cd94p\" (UID: \"7e48e8e5-076a-4ac3-ade6-eb7091526083\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c9e9de3e-cd94p"
Oct 30 19:40:46.856000 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:40:46.850481    4892 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7e48e8e5-076a-4ac3-ade6-eb7091526083-maas-api-xks-ca\") pod \"maas-api-revoke-sub-c9e9de3e-cd94p\" (UID: \"7e48e8e5-076a-4ac3-ade6-eb7091526083\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c9e9de3e-cd94p"
Oct 30 19:40:46.899828 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:40:46.899795    4892 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ktxr2\" (UniqueName: \"kubernetes.io/projected/7e48e8e5-076a-4ac3-ade6-eb7091526083-kube-api-access-ktxr2\") pod \"maas-api-revoke-sub-c9e9de3e-cd94p\" (UID: \"7e48e8e5-076a-4ac3-ade6-eb7091526083\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c9e9de3e-cd94p"
Oct 30 19:40:46.943162 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:40:46.943134    4892 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c9e9de3e-cd94p"
Oct 30 19:40:47.322732 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:40:47.322699    4892 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c9e9de3e-cd94p"]
Oct 30 19:40:47.322869 ip-10-0-132-82 kubenswrapper[4892]: W1030 19:40:47.322702    4892 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7e48e8e5_076a_4ac3_ade6_eb7091526083.slice/crio-ce4bbda1452ce20ebabf9583f5fe4bdcc9446a9d0e3fa1eba82c9dcf2bed9b37 WatchSource:0}: Error finding container ce4bbda1452ce20ebabf9583f5fe4bdcc9446a9d0e3fa1eba82c9dcf2bed9b37: Status 404 returned error can't find the container with id ce4bbda1452ce20ebabf9583f5fe4bdcc9446a9d0e3fa1eba82c9dcf2bed9b37
Oct 30 19:40:48.051334 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:40:48.051251    4892 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c9e9de3e-cd94p" event={"ID":"7e48e8e5-076a-4ac3-ade6-eb7091526083","Type":"ContainerStarted","Data":"b54f48d792fedb3c7c4ae8c1ad5c8155dd52757a3ec0d2bc22a8db9e3739ff07"}
Oct 30 19:40:48.051334 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:40:48.051316    4892 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c9e9de3e-cd94p" event={"ID":"7e48e8e5-076a-4ac3-ade6-eb7091526083","Type":"ContainerStarted","Data":"ce4bbda1452ce20ebabf9583f5fe4bdcc9446a9d0e3fa1eba82c9dcf2bed9b37"}
Oct 30 19:40:48.164873 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:40:48.164810    4892 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c9e9de3e-cd94p" podStartSLOduration=2.164793422 podStartE2EDuration="2.164793422s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:40:46 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:40:48.164472318 +0000 UTC m=+1495.734937917" watchObservedRunningTime="2026-10-30 19:40:48.164793422 +0000 UTC m=+1495.735259023"
Oct 30 19:40:49.061642 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:40:49.061607    4892 generic.go:356] "Generic (PLEG): container finished" podID="7e48e8e5-076a-4ac3-ade6-eb7091526083" containerID="b54f48d792fedb3c7c4ae8c1ad5c8155dd52757a3ec0d2bc22a8db9e3739ff07" exitCode=0
Oct 30 19:40:49.062108 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:40:49.061692    4892 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c9e9de3e-cd94p" event={"ID":"7e48e8e5-076a-4ac3-ade6-eb7091526083","Type":"ContainerDied","Data":"b54f48d792fedb3c7c4ae8c1ad5c8155dd52757a3ec0d2bc22a8db9e3739ff07"}
Oct 30 19:40:50.220916 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:40:50.220888    4892 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c9e9de3e-cd94p"
Oct 30 19:40:50.384855 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:40:50.384828    4892 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/7e48e8e5-076a-4ac3-ade6-eb7091526083-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7e48e8e5-076a-4ac3-ade6-eb7091526083-maas-api-openshift-ca\") pod \"7e48e8e5-076a-4ac3-ade6-eb7091526083\" (UID: \"7e48e8e5-076a-4ac3-ade6-eb7091526083\") "
Oct 30 19:40:50.385074 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:40:50.384929    4892 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/7e48e8e5-076a-4ac3-ade6-eb7091526083-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7e48e8e5-076a-4ac3-ade6-eb7091526083-maas-api-xks-ca\") pod \"7e48e8e5-076a-4ac3-ade6-eb7091526083\" (UID: \"7e48e8e5-076a-4ac3-ade6-eb7091526083\") "
Oct 30 19:40:50.385074 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:40:50.384990    4892 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7e48e8e5-076a-4ac3-ade6-eb7091526083-kube-api-access-ktxr2\" (UniqueName: \"kubernetes.io/projected/7e48e8e5-076a-4ac3-ade6-eb7091526083-kube-api-access-ktxr2\") pod \"7e48e8e5-076a-4ac3-ade6-eb7091526083\" (UID: \"7e48e8e5-076a-4ac3-ade6-eb7091526083\") "
Oct 30 19:40:50.385245 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:40:50.385220    4892 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7e48e8e5-076a-4ac3-ade6-eb7091526083-maas-api-openshift-ca" pod "7e48e8e5-076a-4ac3-ade6-eb7091526083" (UID: "7e48e8e5-076a-4ac3-ade6-eb7091526083"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 19:40:50.385459 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:40:50.385439    4892 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7e48e8e5-076a-4ac3-ade6-eb7091526083-maas-api-openshift-ca\") on node \"ip-10-0-132-82.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 19:40:50.387122 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:40:50.387095    4892 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7e48e8e5-076a-4ac3-ade6-eb7091526083-maas-api-xks-ca" pod "7e48e8e5-076a-4ac3-ade6-eb7091526083" (UID: "7e48e8e5-076a-4ac3-ade6-eb7091526083"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 19:40:50.387329 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:40:50.387307    4892 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7e48e8e5-076a-4ac3-ade6-eb7091526083-kube-api-access-ktxr2" pod "7e48e8e5-076a-4ac3-ade6-eb7091526083" (UID: "7e48e8e5-076a-4ac3-ade6-eb7091526083"). InnerVolumeSpecName "kube-api-access-ktxr2". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:40:50.486902 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:40:50.486874    4892 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7e48e8e5-076a-4ac3-ade6-eb7091526083-maas-api-xks-ca\") on node \"ip-10-0-132-82.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 19:40:50.486902 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:40:50.486899    4892 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ktxr2\" (UniqueName: \"kubernetes.io/projected/7e48e8e5-076a-4ac3-ade6-eb7091526083-kube-api-access-ktxr2\") on node \"ip-10-0-132-82.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 19:40:51.079941 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:40:51.079911    4892 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c9e9de3e-cd94p"
Oct 30 19:40:51.079941 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:40:51.079932    4892 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c9e9de3e-cd94p" event={"ID":"7e48e8e5-076a-4ac3-ade6-eb7091526083","Type":"ContainerDied","Data":"ce4bbda1452ce20ebabf9583f5fe4bdcc9446a9d0e3fa1eba82c9dcf2bed9b37"}
Oct 30 19:40:51.080165 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:40:51.079995    4892 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="ce4bbda1452ce20ebabf9583f5fe4bdcc9446a9d0e3fa1eba82c9dcf2bed9b37"
Oct 30 19:40:53.090092 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:40:53.090066    4892 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-tg5xj_11db9cec-653f-4aee-8937-0cf5b367efac/console-operator/3.log"
Oct 30 19:40:53.112539 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:40:53.112519    4892 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-tg5xj_11db9cec-653f-4aee-8937-0cf5b367efac/console-operator/3.log"
Oct 30 19:40:54.049233 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:40:54.049193    4892 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-99f6dcfa-whqwx"]
Oct 30 19:40:54.098557 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:40:54.098524    4892 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-99f6dcfa-whqwx"]
Oct 30 19:40:54.571671 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:40:54.571633    4892 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a2107422-c567-4aad-94ca-3b8b38b4dc9d" path="/var/lib/kubelet/pods/a2107422-c567-4aad-94ca-3b8b38b4dc9d/volumes"
Oct 30 19:40:54.852029 ip-10-0-132-82 kubenswrapper[4892]: E1030 19:40:54.851934    4892 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-jrm87" containerName="extractor"
Oct 30 19:40:55.891206 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:40:55.891177    4892 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w4-b5178f-openshift-default-c8fbdff8c-fcqwt"]
Oct 30 19:40:55.891678 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:40:55.891664    4892 cpu_manager.go:414] "containerMap: removing container" podUID="7e48e8e5-076a-4ac3-ade6-eb7091526083" containerName="revoke-keys"
Oct 30 19:40:55.891735 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:40:55.891679    4892 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7e48e8e5-076a-4ac3-ade6-eb7091526083" containerName="revoke-keys"
Oct 30 19:40:55.891791 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:40:55.891780    4892 memory_manager.go:365] "RemoveStaleState removing state" podUID="7e48e8e5-076a-4ac3-ade6-eb7091526083" containerName="revoke-keys"
Oct 30 19:40:55.896143 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:40:55.896125    4892 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-b5178f-openshift-default-c8fbdff8c-fcqwt"
Oct 30 19:40:55.950361 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:40:55.950339    4892 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8ppjf\" (UniqueName: \"kubernetes.io/projected/0f1f64ea-aef7-48e0-9238-599b572a90db-kube-api-access-8ppjf\") pod \"e2e-worker-w4-b5178f-openshift-default-c8fbdff8c-fcqwt\" (UID: \"0f1f64ea-aef7-48e0-9238-599b572a90db\") " pod="openshift-ingress/e2e-worker-w4-b5178f-openshift-default-c8fbdff8c-fcqwt"
Oct 30 19:40:55.950503 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:40:55.950368    4892 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0f1f64ea-aef7-48e0-9238-599b572a90db-istiod-ca-cert\") pod \"e2e-worker-w4-b5178f-openshift-default-c8fbdff8c-fcqwt\" (UID: \"0f1f64ea-aef7-48e0-9238-599b572a90db\") " pod="openshift-ingress/e2e-worker-w4-b5178f-openshift-default-c8fbdff8c-fcqwt"
Oct 30 19:40:55.950503 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:40:55.950407    4892 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0f1f64ea-aef7-48e0-9238-599b572a90db-istio-token\") pod \"e2e-worker-w4-b5178f-openshift-default-c8fbdff8c-fcqwt\" (UID: \"0f1f64ea-aef7-48e0-9238-599b572a90db\") " pod="openshift-ingress/e2e-worker-w4-b5178f-openshift-default-c8fbdff8c-fcqwt"
Oct 30 19:40:55.950586 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:40:55.950560    4892 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0f1f64ea-aef7-48e0-9238-599b572a90db-istio-data\") pod \"e2e-worker-w4-b5178f-openshift-default-c8fbdff8c-fcqwt\" (UID: \"0f1f64ea-aef7-48e0-9238-599b572a90db\") " pod="openshift-ingress/e2e-worker-w4-b5178f-openshift-default-c8fbdff8c-fcqwt"
Oct 30 19:40:55.950623 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:40:55.950601    4892 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0f1f64ea-aef7-48e0-9238-599b572a90db-workload-certs\") pod \"e2e-worker-w4-b5178f-openshift-default-c8fbdff8c-fcqwt\" (UID: \"0f1f64ea-aef7-48e0-9238-599b572a90db\") " pod="openshift-ingress/e2e-worker-w4-b5178f-openshift-default-c8fbdff8c-fcqwt"
Oct 30 19:40:55.950679 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:40:55.950666    4892 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0f1f64ea-aef7-48e0-9238-599b572a90db-istio-envoy\") pod \"e2e-worker-w4-b5178f-openshift-default-c8fbdff8c-fcqwt\" (UID: \"0f1f64ea-aef7-48e0-9238-599b572a90db\") " pod="openshift-ingress/e2e-worker-w4-b5178f-openshift-default-c8fbdff8c-fcqwt"
Oct 30 19:40:55.950725 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:40:55.950712    4892 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0f1f64ea-aef7-48e0-9238-599b572a90db-istio-podinfo\") pod \"e2e-worker-w4-b5178f-openshift-default-c8fbdff8c-fcqwt\" (UID: \"0f1f64ea-aef7-48e0-9238-599b572a90db\") " pod="openshift-ingress/e2e-worker-w4-b5178f-openshift-default-c8fbdff8c-fcqwt"
Oct 30 19:40:55.950827 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:40:55.950812    4892 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0f1f64ea-aef7-48e0-9238-599b572a90db-workload-socket\") pod \"e2e-worker-w4-b5178f-openshift-default-c8fbdff8c-fcqwt\" (UID: \"0f1f64ea-aef7-48e0-9238-599b572a90db\") " pod="openshift-ingress/e2e-worker-w4-b5178f-openshift-default-c8fbdff8c-fcqwt"
Oct 30 19:40:55.950881 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:40:55.950864    4892 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0f1f64ea-aef7-48e0-9238-599b572a90db-credential-socket\") pod \"e2e-worker-w4-b5178f-openshift-default-c8fbdff8c-fcqwt\" (UID: \"0f1f64ea-aef7-48e0-9238-599b572a90db\") " pod="openshift-ingress/e2e-worker-w4-b5178f-openshift-default-c8fbdff8c-fcqwt"
Oct 30 19:40:55.998946 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:40:55.998919    4892 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-b5178f-openshift-default-c8fbdff8c-fcqwt"]
Oct 30 19:40:56.052048 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:40:56.052019    4892 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0f1f64ea-aef7-48e0-9238-599b572a90db-istio-data\") pod \"e2e-worker-w4-b5178f-openshift-default-c8fbdff8c-fcqwt\" (UID: \"0f1f64ea-aef7-48e0-9238-599b572a90db\") " pod="openshift-ingress/e2e-worker-w4-b5178f-openshift-default-c8fbdff8c-fcqwt"
Oct 30 19:40:56.052048 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:40:56.052051    4892 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0f1f64ea-aef7-48e0-9238-599b572a90db-workload-certs\") pod \"e2e-worker-w4-b5178f-openshift-default-c8fbdff8c-fcqwt\" (UID: \"0f1f64ea-aef7-48e0-9238-599b572a90db\") " pod="openshift-ingress/e2e-worker-w4-b5178f-openshift-default-c8fbdff8c-fcqwt"
Oct 30 19:40:56.052284 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:40:56.052073    4892 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0f1f64ea-aef7-48e0-9238-599b572a90db-istio-envoy\") pod \"e2e-worker-w4-b5178f-openshift-default-c8fbdff8c-fcqwt\" (UID: \"0f1f64ea-aef7-48e0-9238-599b572a90db\") " pod="openshift-ingress/e2e-worker-w4-b5178f-openshift-default-c8fbdff8c-fcqwt"
Oct 30 19:40:56.052284 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:40:56.052214    4892 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0f1f64ea-aef7-48e0-9238-599b572a90db-istio-podinfo\") pod \"e2e-worker-w4-b5178f-openshift-default-c8fbdff8c-fcqwt\" (UID: \"0f1f64ea-aef7-48e0-9238-599b572a90db\") " pod="openshift-ingress/e2e-worker-w4-b5178f-openshift-default-c8fbdff8c-fcqwt"
Oct 30 19:40:56.052384 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:40:56.052291    4892 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0f1f64ea-aef7-48e0-9238-599b572a90db-workload-socket\") pod \"e2e-worker-w4-b5178f-openshift-default-c8fbdff8c-fcqwt\" (UID: \"0f1f64ea-aef7-48e0-9238-599b572a90db\") " pod="openshift-ingress/e2e-worker-w4-b5178f-openshift-default-c8fbdff8c-fcqwt"
Oct 30 19:40:56.052384 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:40:56.052377    4892 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0f1f64ea-aef7-48e0-9238-599b572a90db-credential-socket\") pod \"e2e-worker-w4-b5178f-openshift-default-c8fbdff8c-fcqwt\" (UID: \"0f1f64ea-aef7-48e0-9238-599b572a90db\") " pod="openshift-ingress/e2e-worker-w4-b5178f-openshift-default-c8fbdff8c-fcqwt"
Oct 30 19:40:56.052543 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:40:56.052511    4892 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0f1f64ea-aef7-48e0-9238-599b572a90db-workload-certs\") pod \"e2e-worker-w4-b5178f-openshift-default-c8fbdff8c-fcqwt\" (UID: \"0f1f64ea-aef7-48e0-9238-599b572a90db\") " pod="openshift-ingress/e2e-worker-w4-b5178f-openshift-default-c8fbdff8c-fcqwt"
Oct 30 19:40:56.052666 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:40:56.052625    4892 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0f1f64ea-aef7-48e0-9238-599b572a90db-istio-data\") pod \"e2e-worker-w4-b5178f-openshift-default-c8fbdff8c-fcqwt\" (UID: \"0f1f64ea-aef7-48e0-9238-599b572a90db\") " pod="openshift-ingress/e2e-worker-w4-b5178f-openshift-default-c8fbdff8c-fcqwt"
Oct 30 19:40:56.052747 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:40:56.052687    4892 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0f1f64ea-aef7-48e0-9238-599b572a90db-workload-socket\") pod \"e2e-worker-w4-b5178f-openshift-default-c8fbdff8c-fcqwt\" (UID: \"0f1f64ea-aef7-48e0-9238-599b572a90db\") " pod="openshift-ingress/e2e-worker-w4-b5178f-openshift-default-c8fbdff8c-fcqwt"
Oct 30 19:40:56.052808 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:40:56.052752    4892 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8ppjf\" (UniqueName: \"kubernetes.io/projected/0f1f64ea-aef7-48e0-9238-599b572a90db-kube-api-access-8ppjf\") pod \"e2e-worker-w4-b5178f-openshift-default-c8fbdff8c-fcqwt\" (UID: \"0f1f64ea-aef7-48e0-9238-599b572a90db\") " pod="openshift-ingress/e2e-worker-w4-b5178f-openshift-default-c8fbdff8c-fcqwt"
Oct 30 19:40:56.052808 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:40:56.052759    4892 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0f1f64ea-aef7-48e0-9238-599b572a90db-credential-socket\") pod \"e2e-worker-w4-b5178f-openshift-default-c8fbdff8c-fcqwt\" (UID: \"0f1f64ea-aef7-48e0-9238-599b572a90db\") " pod="openshift-ingress/e2e-worker-w4-b5178f-openshift-default-c8fbdff8c-fcqwt"
Oct 30 19:40:56.053291 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:40:56.053269    4892 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0f1f64ea-aef7-48e0-9238-599b572a90db-istiod-ca-cert\") pod \"e2e-worker-w4-b5178f-openshift-default-c8fbdff8c-fcqwt\" (UID: \"0f1f64ea-aef7-48e0-9238-599b572a90db\") " pod="openshift-ingress/e2e-worker-w4-b5178f-openshift-default-c8fbdff8c-fcqwt"
Oct 30 19:40:56.053377 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:40:56.053348    4892 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0f1f64ea-aef7-48e0-9238-599b572a90db-istio-token\") pod \"e2e-worker-w4-b5178f-openshift-default-c8fbdff8c-fcqwt\" (UID: \"0f1f64ea-aef7-48e0-9238-599b572a90db\") " pod="openshift-ingress/e2e-worker-w4-b5178f-openshift-default-c8fbdff8c-fcqwt"
Oct 30 19:40:56.053869 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:40:56.053846    4892 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0f1f64ea-aef7-48e0-9238-599b572a90db-istiod-ca-cert\") pod \"e2e-worker-w4-b5178f-openshift-default-c8fbdff8c-fcqwt\" (UID: \"0f1f64ea-aef7-48e0-9238-599b572a90db\") " pod="openshift-ingress/e2e-worker-w4-b5178f-openshift-default-c8fbdff8c-fcqwt"
Oct 30 19:40:56.054524 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:40:56.054502    4892 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0f1f64ea-aef7-48e0-9238-599b572a90db-istio-envoy\") pod \"e2e-worker-w4-b5178f-openshift-default-c8fbdff8c-fcqwt\" (UID: \"0f1f64ea-aef7-48e0-9238-599b572a90db\") " pod="openshift-ingress/e2e-worker-w4-b5178f-openshift-default-c8fbdff8c-fcqwt"
Oct 30 19:40:56.054783 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:40:56.054767    4892 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0f1f64ea-aef7-48e0-9238-599b572a90db-istio-podinfo\") pod \"e2e-worker-w4-b5178f-openshift-default-c8fbdff8c-fcqwt\" (UID: \"0f1f64ea-aef7-48e0-9238-599b572a90db\") " pod="openshift-ingress/e2e-worker-w4-b5178f-openshift-default-c8fbdff8c-fcqwt"
Oct 30 19:40:56.109420 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:40:56.109396    4892 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0f1f64ea-aef7-48e0-9238-599b572a90db-istio-token\") pod \"e2e-worker-w4-b5178f-openshift-default-c8fbdff8c-fcqwt\" (UID: \"0f1f64ea-aef7-48e0-9238-599b572a90db\") " pod="openshift-ingress/e2e-worker-w4-b5178f-openshift-default-c8fbdff8c-fcqwt"
Oct 30 19:40:56.110017 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:40:56.109994    4892 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8ppjf\" (UniqueName: \"kubernetes.io/projected/0f1f64ea-aef7-48e0-9238-599b572a90db-kube-api-access-8ppjf\") pod \"e2e-worker-w4-b5178f-openshift-default-c8fbdff8c-fcqwt\" (UID: \"0f1f64ea-aef7-48e0-9238-599b572a90db\") " pod="openshift-ingress/e2e-worker-w4-b5178f-openshift-default-c8fbdff8c-fcqwt"
Oct 30 19:40:56.209707 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:40:56.209648    4892 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-b5178f-openshift-default-c8fbdff8c-fcqwt"
Oct 30 19:40:56.642008 ip-10-0-132-82 kubenswrapper[4892]: W1030 19:40:56.641978    4892 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0f1f64ea_aef7_48e0_9238_599b572a90db.slice/crio-5b060b066edce425a8635627f8fb3b8525a5fd0d57951c568c2eea65b226a3aa WatchSource:0}: Error finding container 5b060b066edce425a8635627f8fb3b8525a5fd0d57951c568c2eea65b226a3aa: Status 404 returned error can't find the container with id 5b060b066edce425a8635627f8fb3b8525a5fd0d57951c568c2eea65b226a3aa
Oct 30 19:40:56.642321 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:40:56.642299    4892 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-b5178f-openshift-default-c8fbdff8c-fcqwt"]
Oct 30 19:40:56.644459 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:40:56.644429    4892 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892000Ki","pods":"250"}
Oct 30 19:40:56.644559 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:40:56.644490    4892 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892000Ki","pods":"250"}
Oct 30 19:40:56.644559 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:40:56.644522    4892 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892000Ki","pods":"250"}
Oct 30 19:40:57.123881 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:40:57.123787    4892 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-b5178f-openshift-default-c8fbdff8c-fcqwt" event={"ID":"0f1f64ea-aef7-48e0-9238-599b572a90db","Type":"ContainerStarted","Data":"3013e01c0346a4a9de2b315aaacf81b493eabef65b9af5847f687f534f0b1672"}
Oct 30 19:40:57.123881 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:40:57.123852    4892 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-b5178f-openshift-default-c8fbdff8c-fcqwt" event={"ID":"0f1f64ea-aef7-48e0-9238-599b572a90db","Type":"ContainerStarted","Data":"5b060b066edce425a8635627f8fb3b8525a5fd0d57951c568c2eea65b226a3aa"}
Oct 30 19:40:57.210470 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:40:57.210435    4892 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w4-b5178f-openshift-default-c8fbdff8c-fcqwt"
Oct 30 19:40:57.212451 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:40:57.212423    4892 patch_prober.go:29] interesting pod/e2e-worker-w4-b5178f-openshift-default-c8fbdff8c-fcqwt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.96:15021/healthz/ready\": dial tcp 10.132.0.96:15021: connect: connection refused" start-of-body=
Oct 30 19:40:57.212567 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:40:57.212470    4892 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-b5178f-openshift-default-c8fbdff8c-fcqwt" podUID="0f1f64ea-aef7-48e0-9238-599b572a90db" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.96:15021/healthz/ready\": dial tcp 10.132.0.96:15021: connect: connection refused"
Oct 30 19:40:57.292343 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:40:57.292285    4892 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w4-b5178f-openshift-default-c8fbdff8c-fcqwt" podStartSLOduration=2.292270501 podStartE2EDuration="2.292270501s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:40:55 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:40:57.289049699 +0000 UTC m=+1504.859515301" watchObservedRunningTime="2026-10-30 19:40:57.292270501 +0000 UTC m=+1504.862736099"
Oct 30 19:40:57.324083 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:40:57.324051    4892 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-9d428be1-54gl7"]
Oct 30 19:40:57.329993 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:40:57.327952    4892 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9d428be1-54gl7"
Oct 30 19:40:57.371660 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:40:57.371627    4892 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-z6szj\" (UniqueName: \"kubernetes.io/projected/b225bbb1-5af0-4000-91b6-687f9f5f69f4-kube-api-access-z6szj\") pod \"maas-api-revoke-sub-9d428be1-54gl7\" (UID: \"b225bbb1-5af0-4000-91b6-687f9f5f69f4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9d428be1-54gl7"
Oct 30 19:40:57.371897 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:40:57.371681    4892 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b225bbb1-5af0-4000-91b6-687f9f5f69f4-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-9d428be1-54gl7\" (UID: \"b225bbb1-5af0-4000-91b6-687f9f5f69f4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9d428be1-54gl7"
Oct 30 19:40:57.371897 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:40:57.371698    4892 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b225bbb1-5af0-4000-91b6-687f9f5f69f4-maas-api-xks-ca\") pod \"maas-api-revoke-sub-9d428be1-54gl7\" (UID: \"b225bbb1-5af0-4000-91b6-687f9f5f69f4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9d428be1-54gl7"
Oct 30 19:40:57.380133 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:40:57.380072    4892 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 19:40:57.380237 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:40:57.380188    4892 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-nlxp4\""
Oct 30 19:40:57.385407 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:40:57.385389    4892 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 30 19:40:57.385523 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:40:57.385500    4892 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 30 19:40:57.409579 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:40:57.409558    4892 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-9d428be1-54gl7"]
Oct 30 19:40:57.473261 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:40:57.473222    4892 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-z6szj\" (UniqueName: \"kubernetes.io/projected/b225bbb1-5af0-4000-91b6-687f9f5f69f4-kube-api-access-z6szj\") pod \"maas-api-revoke-sub-9d428be1-54gl7\" (UID: \"b225bbb1-5af0-4000-91b6-687f9f5f69f4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9d428be1-54gl7"
Oct 30 19:40:57.473455 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:40:57.473299    4892 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b225bbb1-5af0-4000-91b6-687f9f5f69f4-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-9d428be1-54gl7\" (UID: \"b225bbb1-5af0-4000-91b6-687f9f5f69f4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9d428be1-54gl7"
Oct 30 19:40:57.473455 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:40:57.473327    4892 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b225bbb1-5af0-4000-91b6-687f9f5f69f4-maas-api-xks-ca\") pod \"maas-api-revoke-sub-9d428be1-54gl7\" (UID: \"b225bbb1-5af0-4000-91b6-687f9f5f69f4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9d428be1-54gl7"
Oct 30 19:40:57.474364 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:40:57.474337    4892 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b225bbb1-5af0-4000-91b6-687f9f5f69f4-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-9d428be1-54gl7\" (UID: \"b225bbb1-5af0-4000-91b6-687f9f5f69f4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9d428be1-54gl7"
Oct 30 19:40:57.477531 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:40:57.477501    4892 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b225bbb1-5af0-4000-91b6-687f9f5f69f4-maas-api-xks-ca\") pod \"maas-api-revoke-sub-9d428be1-54gl7\" (UID: \"b225bbb1-5af0-4000-91b6-687f9f5f69f4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9d428be1-54gl7"
Oct 30 19:40:57.570734 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:40:57.570448    4892 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-z6szj\" (UniqueName: \"kubernetes.io/projected/b225bbb1-5af0-4000-91b6-687f9f5f69f4-kube-api-access-z6szj\") pod \"maas-api-revoke-sub-9d428be1-54gl7\" (UID: \"b225bbb1-5af0-4000-91b6-687f9f5f69f4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9d428be1-54gl7"
Oct 30 19:40:57.657992 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:40:57.651170    4892 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9d428be1-54gl7"
Oct 30 19:40:58.058561 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:40:58.057081    4892 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-9d428be1-54gl7"]
Oct 30 19:40:58.157786 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:40:58.155924    4892 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9d428be1-54gl7" event={"ID":"b225bbb1-5af0-4000-91b6-687f9f5f69f4","Type":"ContainerStarted","Data":"71011b06c7373bd50e4f70d8ac91914d825102cce6b96bf7932a6cc94d84343e"}
Oct 30 19:40:58.210990 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:40:58.210929    4892 patch_prober.go:29] interesting pod/e2e-worker-w4-b5178f-openshift-default-c8fbdff8c-fcqwt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.96:15021/healthz/ready\": dial tcp 10.132.0.96:15021: connect: connection refused" start-of-body=
Oct 30 19:40:58.211143 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:40:58.211011    4892 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-b5178f-openshift-default-c8fbdff8c-fcqwt" podUID="0f1f64ea-aef7-48e0-9238-599b572a90db" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.96:15021/healthz/ready\": dial tcp 10.132.0.96:15021: connect: connection refused"
Oct 30 19:40:59.164887 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:40:59.164843    4892 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9d428be1-54gl7" event={"ID":"b225bbb1-5af0-4000-91b6-687f9f5f69f4","Type":"ContainerStarted","Data":"8fe8c13c1c21d567999b067fc5db434423acabd867d95ed42721a952e4a1bf0c"}
Oct 30 19:40:59.211018 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:40:59.210957    4892 patch_prober.go:29] interesting pod/e2e-worker-w4-b5178f-openshift-default-c8fbdff8c-fcqwt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.96:15021/healthz/ready\": dial tcp 10.132.0.96:15021: connect: connection refused" start-of-body=
Oct 30 19:40:59.211202 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:40:59.211033    4892 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-b5178f-openshift-default-c8fbdff8c-fcqwt" podUID="0f1f64ea-aef7-48e0-9238-599b572a90db" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.96:15021/healthz/ready\": dial tcp 10.132.0.96:15021: connect: connection refused"
Oct 30 19:41:00.173739 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:41:00.173705    4892 generic.go:356] "Generic (PLEG): container finished" podID="b225bbb1-5af0-4000-91b6-687f9f5f69f4" containerID="8fe8c13c1c21d567999b067fc5db434423acabd867d95ed42721a952e4a1bf0c" exitCode=0
Oct 30 19:41:00.174168 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:41:00.173790    4892 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9d428be1-54gl7" event={"ID":"b225bbb1-5af0-4000-91b6-687f9f5f69f4","Type":"ContainerDied","Data":"8fe8c13c1c21d567999b067fc5db434423acabd867d95ed42721a952e4a1bf0c"}
Oct 30 19:41:00.210368 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:41:00.210342    4892 patch_prober.go:29] interesting pod/e2e-worker-w4-b5178f-openshift-default-c8fbdff8c-fcqwt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.96:15021/healthz/ready\": dial tcp 10.132.0.96:15021: connect: connection refused" start-of-body=
Oct 30 19:41:00.210489 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:41:00.210407    4892 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-b5178f-openshift-default-c8fbdff8c-fcqwt" podUID="0f1f64ea-aef7-48e0-9238-599b572a90db" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.96:15021/healthz/ready\": dial tcp 10.132.0.96:15021: connect: connection refused"
Oct 30 19:41:01.211230 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:41:01.210772    4892 patch_prober.go:29] interesting pod/e2e-worker-w4-b5178f-openshift-default-c8fbdff8c-fcqwt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.96:15021/healthz/ready\": dial tcp 10.132.0.96:15021: connect: connection refused" start-of-body=
Oct 30 19:41:01.211230 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:41:01.210839    4892 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-b5178f-openshift-default-c8fbdff8c-fcqwt" podUID="0f1f64ea-aef7-48e0-9238-599b572a90db" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.96:15021/healthz/ready\": dial tcp 10.132.0.96:15021: connect: connection refused"
Oct 30 19:41:01.379927 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:41:01.379906    4892 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9d428be1-54gl7"
Oct 30 19:41:01.423751 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:41:01.423726    4892 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/b225bbb1-5af0-4000-91b6-687f9f5f69f4-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b225bbb1-5af0-4000-91b6-687f9f5f69f4-maas-api-xks-ca\") pod \"b225bbb1-5af0-4000-91b6-687f9f5f69f4\" (UID: \"b225bbb1-5af0-4000-91b6-687f9f5f69f4\") "
Oct 30 19:41:01.423879 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:41:01.423761    4892 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b225bbb1-5af0-4000-91b6-687f9f5f69f4-kube-api-access-z6szj\" (UniqueName: \"kubernetes.io/projected/b225bbb1-5af0-4000-91b6-687f9f5f69f4-kube-api-access-z6szj\") pod \"b225bbb1-5af0-4000-91b6-687f9f5f69f4\" (UID: \"b225bbb1-5af0-4000-91b6-687f9f5f69f4\") "
Oct 30 19:41:01.423879 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:41:01.423794    4892 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b225bbb1-5af0-4000-91b6-687f9f5f69f4-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b225bbb1-5af0-4000-91b6-687f9f5f69f4-maas-api-openshift-ca\") pod \"b225bbb1-5af0-4000-91b6-687f9f5f69f4\" (UID: \"b225bbb1-5af0-4000-91b6-687f9f5f69f4\") "
Oct 30 19:41:01.424254 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:41:01.424228    4892 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b225bbb1-5af0-4000-91b6-687f9f5f69f4-maas-api-openshift-ca" pod "b225bbb1-5af0-4000-91b6-687f9f5f69f4" (UID: "b225bbb1-5af0-4000-91b6-687f9f5f69f4"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 19:41:01.424389 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:41:01.424372    4892 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b225bbb1-5af0-4000-91b6-687f9f5f69f4-maas-api-openshift-ca\") on node \"ip-10-0-132-82.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 19:41:01.426042 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:41:01.426017    4892 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b225bbb1-5af0-4000-91b6-687f9f5f69f4-kube-api-access-z6szj" pod "b225bbb1-5af0-4000-91b6-687f9f5f69f4" (UID: "b225bbb1-5af0-4000-91b6-687f9f5f69f4"). InnerVolumeSpecName "kube-api-access-z6szj". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:41:01.426123 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:41:01.426091    4892 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b225bbb1-5af0-4000-91b6-687f9f5f69f4-maas-api-xks-ca" pod "b225bbb1-5af0-4000-91b6-687f9f5f69f4" (UID: "b225bbb1-5af0-4000-91b6-687f9f5f69f4"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 19:41:01.526147 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:41:01.526114    4892 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b225bbb1-5af0-4000-91b6-687f9f5f69f4-maas-api-xks-ca\") on node \"ip-10-0-132-82.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 19:41:01.526147 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:41:01.526149    4892 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-z6szj\" (UniqueName: \"kubernetes.io/projected/b225bbb1-5af0-4000-91b6-687f9f5f69f4-kube-api-access-z6szj\") on node \"ip-10-0-132-82.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 19:41:02.189369 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:41:02.189338    4892 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9d428be1-54gl7"
Oct 30 19:41:02.189369 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:41:02.189349    4892 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9d428be1-54gl7" event={"ID":"b225bbb1-5af0-4000-91b6-687f9f5f69f4","Type":"ContainerDied","Data":"71011b06c7373bd50e4f70d8ac91914d825102cce6b96bf7932a6cc94d84343e"}
Oct 30 19:41:02.189594 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:41:02.189428    4892 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="71011b06c7373bd50e4f70d8ac91914d825102cce6b96bf7932a6cc94d84343e"
Oct 30 19:41:02.210307 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:41:02.210282    4892 patch_prober.go:29] interesting pod/e2e-worker-w4-b5178f-openshift-default-c8fbdff8c-fcqwt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.96:15021/healthz/ready\": dial tcp 10.132.0.96:15021: connect: connection refused" start-of-body=
Oct 30 19:41:02.210426 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:41:02.210326    4892 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-b5178f-openshift-default-c8fbdff8c-fcqwt" podUID="0f1f64ea-aef7-48e0-9238-599b572a90db" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.96:15021/healthz/ready\": dial tcp 10.132.0.96:15021: connect: connection refused"
Oct 30 19:41:03.209954 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:41:03.209922    4892 patch_prober.go:29] interesting pod/e2e-worker-w4-b5178f-openshift-default-c8fbdff8c-fcqwt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.96:15021/healthz/ready\": dial tcp 10.132.0.96:15021: connect: connection refused" start-of-body=
Oct 30 19:41:03.210367 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:41:03.209989    4892 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-b5178f-openshift-default-c8fbdff8c-fcqwt" podUID="0f1f64ea-aef7-48e0-9238-599b572a90db" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.96:15021/healthz/ready\": dial tcp 10.132.0.96:15021: connect: connection refused"
Oct 30 19:41:04.214301 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:41:04.214272    4892 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w4-b5178f-openshift-default-c8fbdff8c-fcqwt"
Oct 30 19:41:04.214686 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:41:04.214642    4892 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w4-b5178f-openshift-default-c8fbdff8c-fcqwt"
Oct 30 19:41:04.215251 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:41:04.215232    4892 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w4-b5178f-openshift-default-c8fbdff8c-fcqwt"
Oct 30 19:41:05.800802 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:41:05.800766    4892 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w5-686689-openshift-default-5fbf5787bf-9tcts"]
Oct 30 19:41:05.801413 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:41:05.801394    4892 cpu_manager.go:414] "containerMap: removing container" podUID="b225bbb1-5af0-4000-91b6-687f9f5f69f4" containerName="revoke-keys"
Oct 30 19:41:05.801458 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:41:05.801416    4892 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b225bbb1-5af0-4000-91b6-687f9f5f69f4" containerName="revoke-keys"
Oct 30 19:41:05.801503 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:41:05.801494    4892 memory_manager.go:365] "RemoveStaleState removing state" podUID="b225bbb1-5af0-4000-91b6-687f9f5f69f4" containerName="revoke-keys"
Oct 30 19:41:05.805118 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:41:05.805097    4892 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-686689-openshift-default-5fbf5787bf-9tcts"
Oct 30 19:41:05.891660 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:41:05.891630    4892 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1354d401-f0d6-4252-82d1-47fad799e04c-istiod-ca-cert\") pod \"e2e-worker-w5-686689-openshift-default-5fbf5787bf-9tcts\" (UID: \"1354d401-f0d6-4252-82d1-47fad799e04c\") " pod="openshift-ingress/e2e-worker-w5-686689-openshift-default-5fbf5787bf-9tcts"
Oct 30 19:41:05.891842 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:41:05.891695    4892 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1354d401-f0d6-4252-82d1-47fad799e04c-istio-envoy\") pod \"e2e-worker-w5-686689-openshift-default-5fbf5787bf-9tcts\" (UID: \"1354d401-f0d6-4252-82d1-47fad799e04c\") " pod="openshift-ingress/e2e-worker-w5-686689-openshift-default-5fbf5787bf-9tcts"
Oct 30 19:41:05.891842 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:41:05.891770    4892 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1354d401-f0d6-4252-82d1-47fad799e04c-istio-data\") pod \"e2e-worker-w5-686689-openshift-default-5fbf5787bf-9tcts\" (UID: \"1354d401-f0d6-4252-82d1-47fad799e04c\") " pod="openshift-ingress/e2e-worker-w5-686689-openshift-default-5fbf5787bf-9tcts"
Oct 30 19:41:05.892004 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:41:05.891843    4892 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1354d401-f0d6-4252-82d1-47fad799e04c-credential-socket\") pod \"e2e-worker-w5-686689-openshift-default-5fbf5787bf-9tcts\" (UID: \"1354d401-f0d6-4252-82d1-47fad799e04c\") " pod="openshift-ingress/e2e-worker-w5-686689-openshift-default-5fbf5787bf-9tcts"
Oct 30 19:41:05.892004 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:41:05.891982    4892 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1354d401-f0d6-4252-82d1-47fad799e04c-workload-socket\") pod \"e2e-worker-w5-686689-openshift-default-5fbf5787bf-9tcts\" (UID: \"1354d401-f0d6-4252-82d1-47fad799e04c\") " pod="openshift-ingress/e2e-worker-w5-686689-openshift-default-5fbf5787bf-9tcts"
Oct 30 19:41:05.892121 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:41:05.892023    4892 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tnmv5\" (UniqueName: \"kubernetes.io/projected/1354d401-f0d6-4252-82d1-47fad799e04c-kube-api-access-tnmv5\") pod \"e2e-worker-w5-686689-openshift-default-5fbf5787bf-9tcts\" (UID: \"1354d401-f0d6-4252-82d1-47fad799e04c\") " pod="openshift-ingress/e2e-worker-w5-686689-openshift-default-5fbf5787bf-9tcts"
Oct 30 19:41:05.892121 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:41:05.892068    4892 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1354d401-f0d6-4252-82d1-47fad799e04c-istio-token\") pod \"e2e-worker-w5-686689-openshift-default-5fbf5787bf-9tcts\" (UID: \"1354d401-f0d6-4252-82d1-47fad799e04c\") " pod="openshift-ingress/e2e-worker-w5-686689-openshift-default-5fbf5787bf-9tcts"
Oct 30 19:41:05.892225 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:41:05.892119    4892 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1354d401-f0d6-4252-82d1-47fad799e04c-workload-certs\") pod \"e2e-worker-w5-686689-openshift-default-5fbf5787bf-9tcts\" (UID: \"1354d401-f0d6-4252-82d1-47fad799e04c\") " pod="openshift-ingress/e2e-worker-w5-686689-openshift-default-5fbf5787bf-9tcts"
Oct 30 19:41:05.892225 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:41:05.892160    4892 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1354d401-f0d6-4252-82d1-47fad799e04c-istio-podinfo\") pod \"e2e-worker-w5-686689-openshift-default-5fbf5787bf-9tcts\" (UID: \"1354d401-f0d6-4252-82d1-47fad799e04c\") " pod="openshift-ingress/e2e-worker-w5-686689-openshift-default-5fbf5787bf-9tcts"
Oct 30 19:41:05.910646 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:41:05.910617    4892 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w5-686689-openshift-default-5fbf5787bf-9tcts"]
Oct 30 19:41:05.992940 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:41:05.992910    4892 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1354d401-f0d6-4252-82d1-47fad799e04c-istio-envoy\") pod \"e2e-worker-w5-686689-openshift-default-5fbf5787bf-9tcts\" (UID: \"1354d401-f0d6-4252-82d1-47fad799e04c\") " pod="openshift-ingress/e2e-worker-w5-686689-openshift-default-5fbf5787bf-9tcts"
Oct 30 19:41:05.992940 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:41:05.992943    4892 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1354d401-f0d6-4252-82d1-47fad799e04c-istio-data\") pod \"e2e-worker-w5-686689-openshift-default-5fbf5787bf-9tcts\" (UID: \"1354d401-f0d6-4252-82d1-47fad799e04c\") " pod="openshift-ingress/e2e-worker-w5-686689-openshift-default-5fbf5787bf-9tcts"
Oct 30 19:41:05.993158 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:41:05.993050    4892 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1354d401-f0d6-4252-82d1-47fad799e04c-credential-socket\") pod \"e2e-worker-w5-686689-openshift-default-5fbf5787bf-9tcts\" (UID: \"1354d401-f0d6-4252-82d1-47fad799e04c\") " pod="openshift-ingress/e2e-worker-w5-686689-openshift-default-5fbf5787bf-9tcts"
Oct 30 19:41:05.993158 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:41:05.993141    4892 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1354d401-f0d6-4252-82d1-47fad799e04c-workload-socket\") pod \"e2e-worker-w5-686689-openshift-default-5fbf5787bf-9tcts\" (UID: \"1354d401-f0d6-4252-82d1-47fad799e04c\") " pod="openshift-ingress/e2e-worker-w5-686689-openshift-default-5fbf5787bf-9tcts"
Oct 30 19:41:05.993228 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:41:05.993185    4892 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tnmv5\" (UniqueName: \"kubernetes.io/projected/1354d401-f0d6-4252-82d1-47fad799e04c-kube-api-access-tnmv5\") pod \"e2e-worker-w5-686689-openshift-default-5fbf5787bf-9tcts\" (UID: \"1354d401-f0d6-4252-82d1-47fad799e04c\") " pod="openshift-ingress/e2e-worker-w5-686689-openshift-default-5fbf5787bf-9tcts"
Oct 30 19:41:05.993268 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:41:05.993224    4892 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1354d401-f0d6-4252-82d1-47fad799e04c-istio-token\") pod \"e2e-worker-w5-686689-openshift-default-5fbf5787bf-9tcts\" (UID: \"1354d401-f0d6-4252-82d1-47fad799e04c\") " pod="openshift-ingress/e2e-worker-w5-686689-openshift-default-5fbf5787bf-9tcts"
Oct 30 19:41:05.993315 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:41:05.993279    4892 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1354d401-f0d6-4252-82d1-47fad799e04c-workload-certs\") pod \"e2e-worker-w5-686689-openshift-default-5fbf5787bf-9tcts\" (UID: \"1354d401-f0d6-4252-82d1-47fad799e04c\") " pod="openshift-ingress/e2e-worker-w5-686689-openshift-default-5fbf5787bf-9tcts"
Oct 30 19:41:05.993369 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:41:05.993325    4892 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1354d401-f0d6-4252-82d1-47fad799e04c-istio-podinfo\") pod \"e2e-worker-w5-686689-openshift-default-5fbf5787bf-9tcts\" (UID: \"1354d401-f0d6-4252-82d1-47fad799e04c\") " pod="openshift-ingress/e2e-worker-w5-686689-openshift-default-5fbf5787bf-9tcts"
Oct 30 19:41:05.993369 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:41:05.993354    4892 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1354d401-f0d6-4252-82d1-47fad799e04c-istiod-ca-cert\") pod \"e2e-worker-w5-686689-openshift-default-5fbf5787bf-9tcts\" (UID: \"1354d401-f0d6-4252-82d1-47fad799e04c\") " pod="openshift-ingress/e2e-worker-w5-686689-openshift-default-5fbf5787bf-9tcts"
Oct 30 19:41:05.993472 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:41:05.993444    4892 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1354d401-f0d6-4252-82d1-47fad799e04c-istio-data\") pod \"e2e-worker-w5-686689-openshift-default-5fbf5787bf-9tcts\" (UID: \"1354d401-f0d6-4252-82d1-47fad799e04c\") " pod="openshift-ingress/e2e-worker-w5-686689-openshift-default-5fbf5787bf-9tcts"
Oct 30 19:41:05.993472 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:41:05.993461    4892 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1354d401-f0d6-4252-82d1-47fad799e04c-workload-socket\") pod \"e2e-worker-w5-686689-openshift-default-5fbf5787bf-9tcts\" (UID: \"1354d401-f0d6-4252-82d1-47fad799e04c\") " pod="openshift-ingress/e2e-worker-w5-686689-openshift-default-5fbf5787bf-9tcts"
Oct 30 19:41:05.993560 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:41:05.993520    4892 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1354d401-f0d6-4252-82d1-47fad799e04c-credential-socket\") pod \"e2e-worker-w5-686689-openshift-default-5fbf5787bf-9tcts\" (UID: \"1354d401-f0d6-4252-82d1-47fad799e04c\") " pod="openshift-ingress/e2e-worker-w5-686689-openshift-default-5fbf5787bf-9tcts"
Oct 30 19:41:05.993745 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:41:05.993723    4892 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1354d401-f0d6-4252-82d1-47fad799e04c-workload-certs\") pod \"e2e-worker-w5-686689-openshift-default-5fbf5787bf-9tcts\" (UID: \"1354d401-f0d6-4252-82d1-47fad799e04c\") " pod="openshift-ingress/e2e-worker-w5-686689-openshift-default-5fbf5787bf-9tcts"
Oct 30 19:41:05.994302 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:41:05.994275    4892 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1354d401-f0d6-4252-82d1-47fad799e04c-istiod-ca-cert\") pod \"e2e-worker-w5-686689-openshift-default-5fbf5787bf-9tcts\" (UID: \"1354d401-f0d6-4252-82d1-47fad799e04c\") " pod="openshift-ingress/e2e-worker-w5-686689-openshift-default-5fbf5787bf-9tcts"
Oct 30 19:41:05.995541 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:41:05.995520    4892 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1354d401-f0d6-4252-82d1-47fad799e04c-istio-envoy\") pod \"e2e-worker-w5-686689-openshift-default-5fbf5787bf-9tcts\" (UID: \"1354d401-f0d6-4252-82d1-47fad799e04c\") " pod="openshift-ingress/e2e-worker-w5-686689-openshift-default-5fbf5787bf-9tcts"
Oct 30 19:41:05.995937 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:41:05.995917    4892 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1354d401-f0d6-4252-82d1-47fad799e04c-istio-podinfo\") pod \"e2e-worker-w5-686689-openshift-default-5fbf5787bf-9tcts\" (UID: \"1354d401-f0d6-4252-82d1-47fad799e04c\") " pod="openshift-ingress/e2e-worker-w5-686689-openshift-default-5fbf5787bf-9tcts"
Oct 30 19:41:06.049706 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:41:06.049682    4892 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1354d401-f0d6-4252-82d1-47fad799e04c-istio-token\") pod \"e2e-worker-w5-686689-openshift-default-5fbf5787bf-9tcts\" (UID: \"1354d401-f0d6-4252-82d1-47fad799e04c\") " pod="openshift-ingress/e2e-worker-w5-686689-openshift-default-5fbf5787bf-9tcts"
Oct 30 19:41:06.049818 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:41:06.049796    4892 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tnmv5\" (UniqueName: \"kubernetes.io/projected/1354d401-f0d6-4252-82d1-47fad799e04c-kube-api-access-tnmv5\") pod \"e2e-worker-w5-686689-openshift-default-5fbf5787bf-9tcts\" (UID: \"1354d401-f0d6-4252-82d1-47fad799e04c\") " pod="openshift-ingress/e2e-worker-w5-686689-openshift-default-5fbf5787bf-9tcts"
Oct 30 19:41:06.118293 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:41:06.118236    4892 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-686689-openshift-default-5fbf5787bf-9tcts"
Oct 30 19:41:06.399478 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:41:06.399449    4892 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ad0fb289-f9ffv"]
Oct 30 19:41:06.406228 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:41:06.406064    4892 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ad0fb289-f9ffv"
Oct 30 19:41:06.459478 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:41:06.459303    4892 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 19:41:06.459478 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:41:06.459366    4892 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-nlxp4\""
Oct 30 19:41:06.459478 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:41:06.459445    4892 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 30 19:41:06.460018 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:41:06.459929    4892 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 30 19:41:06.463124 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:41:06.463101    4892 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ad0fb289-f9ffv"]
Oct 30 19:41:06.498828 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:41:06.498801    4892 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5702f32d-9d4a-4176-86f7-fd358beb49bd-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-ad0fb289-f9ffv\" (UID: \"5702f32d-9d4a-4176-86f7-fd358beb49bd\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ad0fb289-f9ffv"
Oct 30 19:41:06.499009 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:41:06.498872    4892 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5702f32d-9d4a-4176-86f7-fd358beb49bd-maas-api-xks-ca\") pod \"maas-api-revoke-sub-ad0fb289-f9ffv\" (UID: \"5702f32d-9d4a-4176-86f7-fd358beb49bd\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ad0fb289-f9ffv"
Oct 30 19:41:06.499009 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:41:06.499001    4892 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jdjh9\" (UniqueName: \"kubernetes.io/projected/5702f32d-9d4a-4176-86f7-fd358beb49bd-kube-api-access-jdjh9\") pod \"maas-api-revoke-sub-ad0fb289-f9ffv\" (UID: \"5702f32d-9d4a-4176-86f7-fd358beb49bd\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ad0fb289-f9ffv"
Oct 30 19:41:06.566180 ip-10-0-132-82 kubenswrapper[4892]: W1030 19:41:06.566149    4892 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1354d401_f0d6_4252_82d1_47fad799e04c.slice/crio-cd85568748275a5cc1a4e70ca5dbfd7403ab164c5e8715ac0020b2a1014973c5 WatchSource:0}: Error finding container cd85568748275a5cc1a4e70ca5dbfd7403ab164c5e8715ac0020b2a1014973c5: Status 404 returned error can't find the container with id cd85568748275a5cc1a4e70ca5dbfd7403ab164c5e8715ac0020b2a1014973c5
Oct 30 19:41:06.568998 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:41:06.568844    4892 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892000Ki","pods":"250"}
Oct 30 19:41:06.569095 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:41:06.569010    4892 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892000Ki","pods":"250"}
Oct 30 19:41:06.569095 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:41:06.569076    4892 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892000Ki","pods":"250"}
Oct 30 19:41:06.570619 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:41:06.570595    4892 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w5-686689-openshift-default-5fbf5787bf-9tcts"]
Oct 30 19:41:06.600205 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:41:06.600177    4892 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5702f32d-9d4a-4176-86f7-fd358beb49bd-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-ad0fb289-f9ffv\" (UID: \"5702f32d-9d4a-4176-86f7-fd358beb49bd\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ad0fb289-f9ffv"
Oct 30 19:41:06.600347 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:41:06.600264    4892 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5702f32d-9d4a-4176-86f7-fd358beb49bd-maas-api-xks-ca\") pod \"maas-api-revoke-sub-ad0fb289-f9ffv\" (UID: \"5702f32d-9d4a-4176-86f7-fd358beb49bd\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ad0fb289-f9ffv"
Oct 30 19:41:06.600410 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:41:06.600370    4892 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jdjh9\" (UniqueName: \"kubernetes.io/projected/5702f32d-9d4a-4176-86f7-fd358beb49bd-kube-api-access-jdjh9\") pod \"maas-api-revoke-sub-ad0fb289-f9ffv\" (UID: \"5702f32d-9d4a-4176-86f7-fd358beb49bd\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ad0fb289-f9ffv"
Oct 30 19:41:06.600762 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:41:06.600739    4892 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5702f32d-9d4a-4176-86f7-fd358beb49bd-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-ad0fb289-f9ffv\" (UID: \"5702f32d-9d4a-4176-86f7-fd358beb49bd\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ad0fb289-f9ffv"
Oct 30 19:41:06.602539 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:41:06.602515    4892 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5702f32d-9d4a-4176-86f7-fd358beb49bd-maas-api-xks-ca\") pod \"maas-api-revoke-sub-ad0fb289-f9ffv\" (UID: \"5702f32d-9d4a-4176-86f7-fd358beb49bd\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ad0fb289-f9ffv"
Oct 30 19:41:06.663942 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:41:06.663876    4892 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jdjh9\" (UniqueName: \"kubernetes.io/projected/5702f32d-9d4a-4176-86f7-fd358beb49bd-kube-api-access-jdjh9\") pod \"maas-api-revoke-sub-ad0fb289-f9ffv\" (UID: \"5702f32d-9d4a-4176-86f7-fd358beb49bd\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ad0fb289-f9ffv"
Oct 30 19:41:06.722349 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:41:06.722319    4892 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ad0fb289-f9ffv"
Oct 30 19:41:07.109410 ip-10-0-132-82 kubenswrapper[4892]: W1030 19:41:07.109376    4892 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5702f32d_9d4a_4176_86f7_fd358beb49bd.slice/crio-90010c631ee8354678d2daa645f8afc58e6af2b7862785d121b5b42fd69f0269 WatchSource:0}: Error finding container 90010c631ee8354678d2daa645f8afc58e6af2b7862785d121b5b42fd69f0269: Status 404 returned error can't find the container with id 90010c631ee8354678d2daa645f8afc58e6af2b7862785d121b5b42fd69f0269
Oct 30 19:41:07.109833 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:41:07.109643    4892 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ad0fb289-f9ffv"]
Oct 30 19:41:07.230974 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:41:07.230801    4892 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ad0fb289-f9ffv" event={"ID":"5702f32d-9d4a-4176-86f7-fd358beb49bd","Type":"ContainerStarted","Data":"90010c631ee8354678d2daa645f8afc58e6af2b7862785d121b5b42fd69f0269"}
Oct 30 19:41:07.233028 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:41:07.233000    4892 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-686689-openshift-default-5fbf5787bf-9tcts" event={"ID":"1354d401-f0d6-4252-82d1-47fad799e04c","Type":"ContainerStarted","Data":"dee7dad7ea49c8478eb999fd610e045c82b71102bbb3e4b9470719bdbd699220"}
Oct 30 19:41:07.233151 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:41:07.233054    4892 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-686689-openshift-default-5fbf5787bf-9tcts" event={"ID":"1354d401-f0d6-4252-82d1-47fad799e04c","Type":"ContainerStarted","Data":"cd85568748275a5cc1a4e70ca5dbfd7403ab164c5e8715ac0020b2a1014973c5"}
Oct 30 19:41:07.350193 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:41:07.350143    4892 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w5-686689-openshift-default-5fbf5787bf-9tcts" podStartSLOduration=2.350126614 podStartE2EDuration="2.350126614s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:41:05 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:41:07.349352579 +0000 UTC m=+1514.919818191" watchObservedRunningTime="2026-10-30 19:41:07.350126614 +0000 UTC m=+1514.920592213"
Oct 30 19:41:08.118648 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:41:08.118615    4892 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w5-686689-openshift-default-5fbf5787bf-9tcts"
Oct 30 19:41:08.120482 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:41:08.120457    4892 patch_prober.go:29] interesting pod/e2e-worker-w5-686689-openshift-default-5fbf5787bf-9tcts container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.98:15021/healthz/ready\": dial tcp 10.132.0.98:15021: connect: connection refused" start-of-body=
Oct 30 19:41:08.120593 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:41:08.120508    4892 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-686689-openshift-default-5fbf5787bf-9tcts" podUID="1354d401-f0d6-4252-82d1-47fad799e04c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.98:15021/healthz/ready\": dial tcp 10.132.0.98:15021: connect: connection refused"
Oct 30 19:41:08.265989 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:41:08.265664    4892 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ad0fb289-f9ffv" event={"ID":"5702f32d-9d4a-4176-86f7-fd358beb49bd","Type":"ContainerStarted","Data":"ea66e3dd065608341cb2d2078371492a8e29e4a87a8186af9bfbb628c054625a"}
Oct 30 19:41:08.382902 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:41:08.382792    4892 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ad0fb289-f9ffv" podStartSLOduration=2.3827771220000002 podStartE2EDuration="2.382777122s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:41:06 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:41:08.379505888 +0000 UTC m=+1515.949971490" watchObservedRunningTime="2026-10-30 19:41:08.382777122 +0000 UTC m=+1515.953242720"
Oct 30 19:41:09.068728 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:41:09.068693    4892 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-eea179bf-l894m"]
Oct 30 19:41:09.104513 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:41:09.104473    4892 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-eea179bf-l894m"]
Oct 30 19:41:09.119090 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:41:09.119057    4892 patch_prober.go:29] interesting pod/e2e-worker-w5-686689-openshift-default-5fbf5787bf-9tcts container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.98:15021/healthz/ready\": dial tcp 10.132.0.98:15021: connect: connection refused" start-of-body=
Oct 30 19:41:09.119442 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:41:09.119110    4892 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-686689-openshift-default-5fbf5787bf-9tcts" podUID="1354d401-f0d6-4252-82d1-47fad799e04c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.98:15021/healthz/ready\": dial tcp 10.132.0.98:15021: connect: connection refused"
Oct 30 19:41:09.275616 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:41:09.275576    4892 generic.go:356] "Generic (PLEG): container finished" podID="5702f32d-9d4a-4176-86f7-fd358beb49bd" containerID="ea66e3dd065608341cb2d2078371492a8e29e4a87a8186af9bfbb628c054625a" exitCode=0
Oct 30 19:41:09.275792 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:41:09.275669    4892 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ad0fb289-f9ffv" event={"ID":"5702f32d-9d4a-4176-86f7-fd358beb49bd","Type":"ContainerDied","Data":"ea66e3dd065608341cb2d2078371492a8e29e4a87a8186af9bfbb628c054625a"}
Oct 30 19:41:09.392880 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:41:09.392842    4892 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c6271da6-fkqzd"]
Oct 30 19:41:09.397040 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:41:09.397019    4892 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c6271da6-fkqzd"
Oct 30 19:41:09.472521 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:41:09.472488    4892 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c6271da6-fkqzd"]
Oct 30 19:41:09.563401 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:41:09.563374    4892 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/04f8161f-571c-4656-b577-0a30c8a52252-maas-api-xks-ca\") pod \"maas-api-revoke-sub-c6271da6-fkqzd\" (UID: \"04f8161f-571c-4656-b577-0a30c8a52252\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c6271da6-fkqzd"
Oct 30 19:41:09.563596 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:41:09.563487    4892 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-twq4d\" (UniqueName: \"kubernetes.io/projected/04f8161f-571c-4656-b577-0a30c8a52252-kube-api-access-twq4d\") pod \"maas-api-revoke-sub-c6271da6-fkqzd\" (UID: \"04f8161f-571c-4656-b577-0a30c8a52252\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c6271da6-fkqzd"
Oct 30 19:41:09.563657 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:41:09.563626    4892 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/04f8161f-571c-4656-b577-0a30c8a52252-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-c6271da6-fkqzd\" (UID: \"04f8161f-571c-4656-b577-0a30c8a52252\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c6271da6-fkqzd"
Oct 30 19:41:09.664782 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:41:09.664675    4892 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/04f8161f-571c-4656-b577-0a30c8a52252-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-c6271da6-fkqzd\" (UID: \"04f8161f-571c-4656-b577-0a30c8a52252\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c6271da6-fkqzd"
Oct 30 19:41:09.664782 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:41:09.664756    4892 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/04f8161f-571c-4656-b577-0a30c8a52252-maas-api-xks-ca\") pod \"maas-api-revoke-sub-c6271da6-fkqzd\" (UID: \"04f8161f-571c-4656-b577-0a30c8a52252\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c6271da6-fkqzd"
Oct 30 19:41:09.665040 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:41:09.664907    4892 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-twq4d\" (UniqueName: \"kubernetes.io/projected/04f8161f-571c-4656-b577-0a30c8a52252-kube-api-access-twq4d\") pod \"maas-api-revoke-sub-c6271da6-fkqzd\" (UID: \"04f8161f-571c-4656-b577-0a30c8a52252\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c6271da6-fkqzd"
Oct 30 19:41:09.665481 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:41:09.665454    4892 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/04f8161f-571c-4656-b577-0a30c8a52252-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-c6271da6-fkqzd\" (UID: \"04f8161f-571c-4656-b577-0a30c8a52252\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c6271da6-fkqzd"
Oct 30 19:41:09.667553 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:41:09.667528    4892 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/04f8161f-571c-4656-b577-0a30c8a52252-maas-api-xks-ca\") pod \"maas-api-revoke-sub-c6271da6-fkqzd\" (UID: \"04f8161f-571c-4656-b577-0a30c8a52252\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c6271da6-fkqzd"
Oct 30 19:41:09.722257 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:41:09.722227    4892 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-twq4d\" (UniqueName: \"kubernetes.io/projected/04f8161f-571c-4656-b577-0a30c8a52252-kube-api-access-twq4d\") pod \"maas-api-revoke-sub-c6271da6-fkqzd\" (UID: \"04f8161f-571c-4656-b577-0a30c8a52252\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c6271da6-fkqzd"
Oct 30 19:41:10.011882 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:41:10.011788    4892 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c6271da6-fkqzd"
Oct 30 19:41:10.119623 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:41:10.119438    4892 patch_prober.go:29] interesting pod/e2e-worker-w5-686689-openshift-default-5fbf5787bf-9tcts container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.98:15021/healthz/ready\": dial tcp 10.132.0.98:15021: connect: connection refused" start-of-body=
Oct 30 19:41:10.119623 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:41:10.119499    4892 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-686689-openshift-default-5fbf5787bf-9tcts" podUID="1354d401-f0d6-4252-82d1-47fad799e04c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.98:15021/healthz/ready\": dial tcp 10.132.0.98:15021: connect: connection refused"
Oct 30 19:41:10.425184 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:41:10.425154    4892 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c6271da6-fkqzd"]
Oct 30 19:41:10.425976 ip-10-0-132-82 kubenswrapper[4892]: W1030 19:41:10.425935    4892 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod04f8161f_571c_4656_b577_0a30c8a52252.slice/crio-686a9978ab94fb1eedfc2ed62f02624412044d0564bbbca232bc1c622105f866 WatchSource:0}: Error finding container 686a9978ab94fb1eedfc2ed62f02624412044d0564bbbca232bc1c622105f866: Status 404 returned error can't find the container with id 686a9978ab94fb1eedfc2ed62f02624412044d0564bbbca232bc1c622105f866
Oct 30 19:41:10.482461 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:41:10.482437    4892 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ad0fb289-f9ffv"
Oct 30 19:41:10.570229 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:41:10.570151    4892 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="9c5741cb-f9d5-4a5e-a121-9b689a263a74" path="/var/lib/kubelet/pods/9c5741cb-f9d5-4a5e-a121-9b689a263a74/volumes"
Oct 30 19:41:10.578556 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:41:10.578533    4892 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5702f32d-9d4a-4176-86f7-fd358beb49bd-kube-api-access-jdjh9\" (UniqueName: \"kubernetes.io/projected/5702f32d-9d4a-4176-86f7-fd358beb49bd-kube-api-access-jdjh9\") pod \"5702f32d-9d4a-4176-86f7-fd358beb49bd\" (UID: \"5702f32d-9d4a-4176-86f7-fd358beb49bd\") "
Oct 30 19:41:10.578679 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:41:10.578643    4892 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/5702f32d-9d4a-4176-86f7-fd358beb49bd-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5702f32d-9d4a-4176-86f7-fd358beb49bd-maas-api-xks-ca\") pod \"5702f32d-9d4a-4176-86f7-fd358beb49bd\" (UID: \"5702f32d-9d4a-4176-86f7-fd358beb49bd\") "
Oct 30 19:41:10.578763 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:41:10.578744    4892 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/5702f32d-9d4a-4176-86f7-fd358beb49bd-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5702f32d-9d4a-4176-86f7-fd358beb49bd-maas-api-openshift-ca\") pod \"5702f32d-9d4a-4176-86f7-fd358beb49bd\" (UID: \"5702f32d-9d4a-4176-86f7-fd358beb49bd\") "
Oct 30 19:41:10.579190 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:41:10.579146    4892 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5702f32d-9d4a-4176-86f7-fd358beb49bd-maas-api-openshift-ca" pod "5702f32d-9d4a-4176-86f7-fd358beb49bd" (UID: "5702f32d-9d4a-4176-86f7-fd358beb49bd"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 19:41:10.635801 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:41:10.635761    4892 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5702f32d-9d4a-4176-86f7-fd358beb49bd-maas-api-xks-ca" pod "5702f32d-9d4a-4176-86f7-fd358beb49bd" (UID: "5702f32d-9d4a-4176-86f7-fd358beb49bd"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 19:41:10.640047 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:41:10.636088    4892 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5702f32d-9d4a-4176-86f7-fd358beb49bd-kube-api-access-jdjh9" pod "5702f32d-9d4a-4176-86f7-fd358beb49bd" (UID: "5702f32d-9d4a-4176-86f7-fd358beb49bd"). InnerVolumeSpecName "kube-api-access-jdjh9". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:41:10.684926 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:41:10.684893    4892 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5702f32d-9d4a-4176-86f7-fd358beb49bd-maas-api-openshift-ca\") on node \"ip-10-0-132-82.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 19:41:10.685088 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:41:10.684928    4892 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jdjh9\" (UniqueName: \"kubernetes.io/projected/5702f32d-9d4a-4176-86f7-fd358beb49bd-kube-api-access-jdjh9\") on node \"ip-10-0-132-82.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 19:41:10.685088 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:41:10.684944    4892 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5702f32d-9d4a-4176-86f7-fd358beb49bd-maas-api-xks-ca\") on node \"ip-10-0-132-82.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 19:41:11.119590 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:41:11.119503    4892 patch_prober.go:29] interesting pod/e2e-worker-w5-686689-openshift-default-5fbf5787bf-9tcts container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.98:15021/healthz/ready\": dial tcp 10.132.0.98:15021: connect: connection refused" start-of-body=
Oct 30 19:41:11.119590 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:41:11.119556    4892 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-686689-openshift-default-5fbf5787bf-9tcts" podUID="1354d401-f0d6-4252-82d1-47fad799e04c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.98:15021/healthz/ready\": dial tcp 10.132.0.98:15021: connect: connection refused"
Oct 30 19:41:11.294286 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:41:11.294248    4892 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c6271da6-fkqzd" event={"ID":"04f8161f-571c-4656-b577-0a30c8a52252","Type":"ContainerStarted","Data":"76f9ef7be90050521c9d8afc8f22c2ade61432de01a3e692c2c9472b84b9b37d"}
Oct 30 19:41:11.294487 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:41:11.294317    4892 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c6271da6-fkqzd" event={"ID":"04f8161f-571c-4656-b577-0a30c8a52252","Type":"ContainerStarted","Data":"686a9978ab94fb1eedfc2ed62f02624412044d0564bbbca232bc1c622105f866"}
Oct 30 19:41:11.295674 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:41:11.295644    4892 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ad0fb289-f9ffv" event={"ID":"5702f32d-9d4a-4176-86f7-fd358beb49bd","Type":"ContainerDied","Data":"90010c631ee8354678d2daa645f8afc58e6af2b7862785d121b5b42fd69f0269"}
Oct 30 19:41:11.295674 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:41:11.295664    4892 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ad0fb289-f9ffv"
Oct 30 19:41:11.295830 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:41:11.295692    4892 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="90010c631ee8354678d2daa645f8afc58e6af2b7862785d121b5b42fd69f0269"
Oct 30 19:41:11.416481 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:41:11.416420    4892 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c6271da6-fkqzd" podStartSLOduration=2.416400176 podStartE2EDuration="2.416400176s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:41:09 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:41:11.413295331 +0000 UTC m=+1518.983760930" watchObservedRunningTime="2026-10-30 19:41:11.416400176 +0000 UTC m=+1518.986865776"
Oct 30 19:41:11.755688 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:41:11.755602    4892 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-fsfvk"]
Oct 30 19:41:12.119347 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:41:12.119264    4892 patch_prober.go:29] interesting pod/e2e-worker-w5-686689-openshift-default-5fbf5787bf-9tcts container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.98:15021/healthz/ready\": dial tcp 10.132.0.98:15021: connect: connection refused" start-of-body=
Oct 30 19:41:12.119347 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:41:12.119330    4892 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-686689-openshift-default-5fbf5787bf-9tcts" podUID="1354d401-f0d6-4252-82d1-47fad799e04c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.98:15021/healthz/ready\": dial tcp 10.132.0.98:15021: connect: connection refused"
Oct 30 19:41:12.305171 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:41:12.305133    4892 generic.go:356] "Generic (PLEG): container finished" podID="04f8161f-571c-4656-b577-0a30c8a52252" containerID="76f9ef7be90050521c9d8afc8f22c2ade61432de01a3e692c2c9472b84b9b37d" exitCode=0
Oct 30 19:41:12.305675 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:41:12.305643    4892 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c6271da6-fkqzd" event={"ID":"04f8161f-571c-4656-b577-0a30c8a52252","Type":"ContainerDied","Data":"76f9ef7be90050521c9d8afc8f22c2ade61432de01a3e692c2c9472b84b9b37d"}
Oct 30 19:41:13.095113 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:41:13.095077    4892 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2cbf0d2e-x2csg"]
Oct 30 19:41:13.105435 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:41:13.105393    4892 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2cbf0d2e-x2csg"]
Oct 30 19:41:13.119394 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:41:13.119359    4892 patch_prober.go:29] interesting pod/e2e-worker-w5-686689-openshift-default-5fbf5787bf-9tcts container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.98:15021/healthz/ready\": dial tcp 10.132.0.98:15021: connect: connection refused" start-of-body=
Oct 30 19:41:13.119630 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:41:13.119598    4892 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-686689-openshift-default-5fbf5787bf-9tcts" podUID="1354d401-f0d6-4252-82d1-47fad799e04c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.98:15021/healthz/ready\": dial tcp 10.132.0.98:15021: connect: connection refused"
Oct 30 19:41:14.298076 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:41:14.278110    4892 patch_prober.go:29] interesting pod/e2e-worker-w5-686689-openshift-default-5fbf5787bf-9tcts container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.98:15021/healthz/ready\": dial tcp 10.132.0.98:15021: connect: connection refused" start-of-body=
Oct 30 19:41:14.298076 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:41:14.278190    4892 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-686689-openshift-default-5fbf5787bf-9tcts" podUID="1354d401-f0d6-4252-82d1-47fad799e04c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.98:15021/healthz/ready\": dial tcp 10.132.0.98:15021: connect: connection refused"
Oct 30 19:41:14.423990 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:41:14.423145    4892 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c6271da6-fkqzd" event={"ID":"04f8161f-571c-4656-b577-0a30c8a52252","Type":"ContainerDied","Data":"686a9978ab94fb1eedfc2ed62f02624412044d0564bbbca232bc1c622105f866"}
Oct 30 19:41:14.423990 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:41:14.423237    4892 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="686a9978ab94fb1eedfc2ed62f02624412044d0564bbbca232bc1c622105f866"
Oct 30 19:41:14.425466 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:41:14.425158    4892 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c6271da6-fkqzd"
Oct 30 19:41:14.495183 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:41:14.483797    4892 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/04f8161f-571c-4656-b577-0a30c8a52252-kube-api-access-twq4d\" (UniqueName: \"kubernetes.io/projected/04f8161f-571c-4656-b577-0a30c8a52252-kube-api-access-twq4d\") pod \"04f8161f-571c-4656-b577-0a30c8a52252\" (UID: \"04f8161f-571c-4656-b577-0a30c8a52252\") "
Oct 30 19:41:14.495183 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:41:14.483884    4892 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/04f8161f-571c-4656-b577-0a30c8a52252-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/04f8161f-571c-4656-b577-0a30c8a52252-maas-api-openshift-ca\") pod \"04f8161f-571c-4656-b577-0a30c8a52252\" (UID: \"04f8161f-571c-4656-b577-0a30c8a52252\") "
Oct 30 19:41:14.495183 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:41:14.483926    4892 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/04f8161f-571c-4656-b577-0a30c8a52252-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/04f8161f-571c-4656-b577-0a30c8a52252-maas-api-xks-ca\") pod \"04f8161f-571c-4656-b577-0a30c8a52252\" (UID: \"04f8161f-571c-4656-b577-0a30c8a52252\") "
Oct 30 19:41:14.511999 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:41:14.507314    4892 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/04f8161f-571c-4656-b577-0a30c8a52252-maas-api-openshift-ca" pod "04f8161f-571c-4656-b577-0a30c8a52252" (UID: "04f8161f-571c-4656-b577-0a30c8a52252"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 19:41:14.515176 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:41:14.515145    4892 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/04f8161f-571c-4656-b577-0a30c8a52252-maas-api-xks-ca" pod "04f8161f-571c-4656-b577-0a30c8a52252" (UID: "04f8161f-571c-4656-b577-0a30c8a52252"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 19:41:14.523658 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:41:14.523617    4892 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/04f8161f-571c-4656-b577-0a30c8a52252-kube-api-access-twq4d" pod "04f8161f-571c-4656-b577-0a30c8a52252" (UID: "04f8161f-571c-4656-b577-0a30c8a52252"). InnerVolumeSpecName "kube-api-access-twq4d". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:41:14.527225 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:41:14.527176    4892 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-2d9861/premium-w0-2d9861-kserve-67945c6848-4n6kz" podUID="798988b2-c769-4f73-a7c8-ca15de0beef1" containerName="main" probeResult="failure" output="Get \"https://10.132.0.62:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:41:14.597986 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:41:14.592942    4892 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-twq4d\" (UniqueName: \"kubernetes.io/projected/04f8161f-571c-4656-b577-0a30c8a52252-kube-api-access-twq4d\") on node \"ip-10-0-132-82.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 19:41:14.608983 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:41:14.606077    4892 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/04f8161f-571c-4656-b577-0a30c8a52252-maas-api-openshift-ca\") on node \"ip-10-0-132-82.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 19:41:14.608983 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:41:14.606146    4892 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/04f8161f-571c-4656-b577-0a30c8a52252-maas-api-xks-ca\") on node \"ip-10-0-132-82.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 19:41:14.625364 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:41:14.625332    4892 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c9cadeb0-5278-4c8d-b5fc-9c345360ca3f" path="/var/lib/kubelet/pods/c9cadeb0-5278-4c8d-b5fc-9c345360ca3f/volumes"
Oct 30 19:41:15.273090 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:41:15.273057    4892 patch_prober.go:29] interesting pod/e2e-worker-w1-d52df0-openshift-default-6d5f887b45-2f4xc container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.88:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 30 19:41:15.273347 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:41:15.273323    4892 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w1-d52df0-openshift-default-6d5f887b45-2f4xc" podUID="6befc94b-40e5-42a4-adf5-392b5ecde123" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.88:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:41:15.431300 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:41:15.431269    4892 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c6271da6-fkqzd"
Oct 30 19:41:16.160144 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:41:16.160102    4892 patch_prober.go:29] interesting pod/e2e-worker-w5-686689-openshift-default-5fbf5787bf-9tcts container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.98:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 30 19:41:16.160334 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:41:16.160169    4892 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-686689-openshift-default-5fbf5787bf-9tcts" podUID="1354d401-f0d6-4252-82d1-47fad799e04c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.98:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:41:16.249414 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:41:16.249375    4892 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5bcd350f-qg8dm"]
Oct 30 19:41:16.250194 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:41:16.250171    4892 cpu_manager.go:414] "containerMap: removing container" podUID="5702f32d-9d4a-4176-86f7-fd358beb49bd" containerName="revoke-keys"
Oct 30 19:41:16.250194 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:41:16.250191    4892 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5702f32d-9d4a-4176-86f7-fd358beb49bd" containerName="revoke-keys"
Oct 30 19:41:16.251751 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:41:16.251726    4892 cpu_manager.go:414] "containerMap: removing container" podUID="04f8161f-571c-4656-b577-0a30c8a52252" containerName="revoke-keys"
Oct 30 19:41:16.251751 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:41:16.251750    4892 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="04f8161f-571c-4656-b577-0a30c8a52252" containerName="revoke-keys"
Oct 30 19:41:16.252000 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:41:16.251951    4892 memory_manager.go:365] "RemoveStaleState removing state" podUID="5702f32d-9d4a-4176-86f7-fd358beb49bd" containerName="revoke-keys"
Oct 30 19:41:16.252672 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:41:16.252652    4892 memory_manager.go:365] "RemoveStaleState removing state" podUID="04f8161f-571c-4656-b577-0a30c8a52252" containerName="revoke-keys"
Oct 30 19:41:16.269030 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:41:16.260227    4892 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5bcd350f-qg8dm"
Oct 30 19:41:16.304916 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:41:16.304885    4892 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5bcd350f-qg8dm"]
Oct 30 19:41:16.313568 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:41:16.313541    4892 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 30 19:41:16.326843 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:41:16.326819    4892 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 19:41:16.326957 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:41:16.326909    4892 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 30 19:41:16.327109 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:41:16.327092    4892 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-nlxp4\""
Oct 30 19:41:16.367825 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:41:16.367700    4892 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ba863759-2880-439a-baf3-e4a9ef0bdda0-maas-api-xks-ca\") pod \"maas-api-revoke-sub-5bcd350f-qg8dm\" (UID: \"ba863759-2880-439a-baf3-e4a9ef0bdda0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5bcd350f-qg8dm"
Oct 30 19:41:16.368004 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:41:16.367874    4892 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kwftr\" (UniqueName: \"kubernetes.io/projected/ba863759-2880-439a-baf3-e4a9ef0bdda0-kube-api-access-kwftr\") pod \"maas-api-revoke-sub-5bcd350f-qg8dm\" (UID: \"ba863759-2880-439a-baf3-e4a9ef0bdda0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5bcd350f-qg8dm"
Oct 30 19:41:16.368090 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:41:16.368040    4892 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ba863759-2880-439a-baf3-e4a9ef0bdda0-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-5bcd350f-qg8dm\" (UID: \"ba863759-2880-439a-baf3-e4a9ef0bdda0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5bcd350f-qg8dm"
Oct 30 19:41:16.469814 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:41:16.469727    4892 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ba863759-2880-439a-baf3-e4a9ef0bdda0-maas-api-xks-ca\") pod \"maas-api-revoke-sub-5bcd350f-qg8dm\" (UID: \"ba863759-2880-439a-baf3-e4a9ef0bdda0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5bcd350f-qg8dm"
Oct 30 19:41:16.472332 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:41:16.472308    4892 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kwftr\" (UniqueName: \"kubernetes.io/projected/ba863759-2880-439a-baf3-e4a9ef0bdda0-kube-api-access-kwftr\") pod \"maas-api-revoke-sub-5bcd350f-qg8dm\" (UID: \"ba863759-2880-439a-baf3-e4a9ef0bdda0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5bcd350f-qg8dm"
Oct 30 19:41:16.472905 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:41:16.472882    4892 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ba863759-2880-439a-baf3-e4a9ef0bdda0-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-5bcd350f-qg8dm\" (UID: \"ba863759-2880-439a-baf3-e4a9ef0bdda0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5bcd350f-qg8dm"
Oct 30 19:41:16.473714 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:41:16.473689    4892 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ba863759-2880-439a-baf3-e4a9ef0bdda0-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-5bcd350f-qg8dm\" (UID: \"ba863759-2880-439a-baf3-e4a9ef0bdda0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5bcd350f-qg8dm"
Oct 30 19:41:16.485722 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:41:16.485696    4892 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ba863759-2880-439a-baf3-e4a9ef0bdda0-maas-api-xks-ca\") pod \"maas-api-revoke-sub-5bcd350f-qg8dm\" (UID: \"ba863759-2880-439a-baf3-e4a9ef0bdda0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5bcd350f-qg8dm"
Oct 30 19:41:16.529096 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:41:16.529067    4892 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-kwftr\" (UniqueName: \"kubernetes.io/projected/ba863759-2880-439a-baf3-e4a9ef0bdda0-kube-api-access-kwftr\") pod \"maas-api-revoke-sub-5bcd350f-qg8dm\" (UID: \"ba863759-2880-439a-baf3-e4a9ef0bdda0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5bcd350f-qg8dm"
Oct 30 19:41:16.597376 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:41:16.597347    4892 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5bcd350f-qg8dm"
Oct 30 19:41:16.807935 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:41:16.807839    4892 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w5-686689-openshift-default-5fbf5787bf-9tcts"
Oct 30 19:41:16.809275 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:41:16.809124    4892 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w5-686689-openshift-default-5fbf5787bf-9tcts"
Oct 30 19:41:16.809647 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:41:16.809631    4892 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w5-686689-openshift-default-5fbf5787bf-9tcts"
Oct 30 19:41:17.041036 ip-10-0-132-82 kubenswrapper[4892]: W1030 19:41:17.041005    4892 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podba863759_2880_439a_baf3_e4a9ef0bdda0.slice/crio-841c7fb2452f87d71fb864bd3b771f2f2b5e44ca513b13672a8e9196d6240438 WatchSource:0}: Error finding container 841c7fb2452f87d71fb864bd3b771f2f2b5e44ca513b13672a8e9196d6240438: Status 404 returned error can't find the container with id 841c7fb2452f87d71fb864bd3b771f2f2b5e44ca513b13672a8e9196d6240438
Oct 30 19:41:17.041704 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:41:17.041566    4892 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5bcd350f-qg8dm"]
Oct 30 19:41:17.452598 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:41:17.452549    4892 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5bcd350f-qg8dm" event={"ID":"ba863759-2880-439a-baf3-e4a9ef0bdda0","Type":"ContainerStarted","Data":"9ed7b544758691d118d999a1c4cfb63507a634c6e25c3011fde791f7c05092d9"}
Oct 30 19:41:17.452799 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:41:17.452653    4892 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5bcd350f-qg8dm" event={"ID":"ba863759-2880-439a-baf3-e4a9ef0bdda0","Type":"ContainerStarted","Data":"841c7fb2452f87d71fb864bd3b771f2f2b5e44ca513b13672a8e9196d6240438"}
Oct 30 19:41:17.568774 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:41:17.568705    4892 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5bcd350f-qg8dm" podStartSLOduration=1.568686325 podStartE2EDuration="1.568686325s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:41:16 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:41:17.565925067 +0000 UTC m=+1525.136390667" watchObservedRunningTime="2026-10-30 19:41:17.568686325 +0000 UTC m=+1525.139151931"
Oct 30 19:41:18.461271 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:41:18.461244    4892 generic.go:356] "Generic (PLEG): container finished" podID="ba863759-2880-439a-baf3-e4a9ef0bdda0" containerID="9ed7b544758691d118d999a1c4cfb63507a634c6e25c3011fde791f7c05092d9" exitCode=0
Oct 30 19:41:18.461458 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:41:18.461334    4892 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5bcd350f-qg8dm" event={"ID":"ba863759-2880-439a-baf3-e4a9ef0bdda0","Type":"ContainerDied","Data":"9ed7b544758691d118d999a1c4cfb63507a634c6e25c3011fde791f7c05092d9"}
Oct 30 19:41:19.624910 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:41:19.624885    4892 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5bcd350f-qg8dm"
Oct 30 19:41:19.714054 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:41:19.714022    4892 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/ba863759-2880-439a-baf3-e4a9ef0bdda0-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ba863759-2880-439a-baf3-e4a9ef0bdda0-maas-api-xks-ca\") pod \"ba863759-2880-439a-baf3-e4a9ef0bdda0\" (UID: \"ba863759-2880-439a-baf3-e4a9ef0bdda0\") "
Oct 30 19:41:19.714308 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:41:19.714117    4892 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ba863759-2880-439a-baf3-e4a9ef0bdda0-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ba863759-2880-439a-baf3-e4a9ef0bdda0-maas-api-openshift-ca\") pod \"ba863759-2880-439a-baf3-e4a9ef0bdda0\" (UID: \"ba863759-2880-439a-baf3-e4a9ef0bdda0\") "
Oct 30 19:41:19.714308 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:41:19.714165    4892 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ba863759-2880-439a-baf3-e4a9ef0bdda0-kube-api-access-kwftr\" (UniqueName: \"kubernetes.io/projected/ba863759-2880-439a-baf3-e4a9ef0bdda0-kube-api-access-kwftr\") pod \"ba863759-2880-439a-baf3-e4a9ef0bdda0\" (UID: \"ba863759-2880-439a-baf3-e4a9ef0bdda0\") "
Oct 30 19:41:19.714879 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:41:19.714834    4892 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ba863759-2880-439a-baf3-e4a9ef0bdda0-maas-api-openshift-ca" pod "ba863759-2880-439a-baf3-e4a9ef0bdda0" (UID: "ba863759-2880-439a-baf3-e4a9ef0bdda0"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 19:41:19.716839 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:41:19.716813    4892 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ba863759-2880-439a-baf3-e4a9ef0bdda0-maas-api-xks-ca" pod "ba863759-2880-439a-baf3-e4a9ef0bdda0" (UID: "ba863759-2880-439a-baf3-e4a9ef0bdda0"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 19:41:19.717337 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:41:19.717315    4892 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ba863759-2880-439a-baf3-e4a9ef0bdda0-kube-api-access-kwftr" pod "ba863759-2880-439a-baf3-e4a9ef0bdda0" (UID: "ba863759-2880-439a-baf3-e4a9ef0bdda0"). InnerVolumeSpecName "kube-api-access-kwftr". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:41:19.815690 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:41:19.815655    4892 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kwftr\" (UniqueName: \"kubernetes.io/projected/ba863759-2880-439a-baf3-e4a9ef0bdda0-kube-api-access-kwftr\") on node \"ip-10-0-132-82.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 19:41:19.815690 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:41:19.815686    4892 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ba863759-2880-439a-baf3-e4a9ef0bdda0-maas-api-xks-ca\") on node \"ip-10-0-132-82.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 19:41:19.815690 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:41:19.815695    4892 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ba863759-2880-439a-baf3-e4a9ef0bdda0-maas-api-openshift-ca\") on node \"ip-10-0-132-82.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 19:41:20.485415 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:41:20.485378    4892 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5bcd350f-qg8dm" event={"ID":"ba863759-2880-439a-baf3-e4a9ef0bdda0","Type":"ContainerDied","Data":"841c7fb2452f87d71fb864bd3b771f2f2b5e44ca513b13672a8e9196d6240438"}
Oct 30 19:41:20.485685 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:41:20.485666    4892 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="841c7fb2452f87d71fb864bd3b771f2f2b5e44ca513b13672a8e9196d6240438"
Oct 30 19:41:20.485881 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:41:20.485868    4892 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5bcd350f-qg8dm"
Oct 30 19:41:21.670394 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:41:21.670364    4892 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w1-d52df0-openshift-default-6d5f887b45-lxbtl"]
Oct 30 19:41:21.670858 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:41:21.670844    4892 cpu_manager.go:414] "containerMap: removing container" podUID="ba863759-2880-439a-baf3-e4a9ef0bdda0" containerName="revoke-keys"
Oct 30 19:41:21.670902 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:41:21.670860    4892 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ba863759-2880-439a-baf3-e4a9ef0bdda0" containerName="revoke-keys"
Oct 30 19:41:21.670954 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:41:21.670944    4892 memory_manager.go:365] "RemoveStaleState removing state" podUID="ba863759-2880-439a-baf3-e4a9ef0bdda0" containerName="revoke-keys"
Oct 30 19:41:21.674547 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:41:21.674530    4892 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-d52df0-openshift-default-6d5f887b45-lxbtl"
Oct 30 19:41:21.836622 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:41:21.836583    4892 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/dc4774a0-3da8-4c65-a0b3-887a476ab465-workload-socket\") pod \"e2e-worker-w1-d52df0-openshift-default-6d5f887b45-lxbtl\" (UID: \"dc4774a0-3da8-4c65-a0b3-887a476ab465\") " pod="openshift-ingress/e2e-worker-w1-d52df0-openshift-default-6d5f887b45-lxbtl"
Oct 30 19:41:21.836781 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:41:21.836635    4892 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/dc4774a0-3da8-4c65-a0b3-887a476ab465-istio-podinfo\") pod \"e2e-worker-w1-d52df0-openshift-default-6d5f887b45-lxbtl\" (UID: \"dc4774a0-3da8-4c65-a0b3-887a476ab465\") " pod="openshift-ingress/e2e-worker-w1-d52df0-openshift-default-6d5f887b45-lxbtl"
Oct 30 19:41:21.836781 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:41:21.836771    4892 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/dc4774a0-3da8-4c65-a0b3-887a476ab465-istio-envoy\") pod \"e2e-worker-w1-d52df0-openshift-default-6d5f887b45-lxbtl\" (UID: \"dc4774a0-3da8-4c65-a0b3-887a476ab465\") " pod="openshift-ingress/e2e-worker-w1-d52df0-openshift-default-6d5f887b45-lxbtl"
Oct 30 19:41:21.836904 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:41:21.836802    4892 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/dc4774a0-3da8-4c65-a0b3-887a476ab465-credential-socket\") pod \"e2e-worker-w1-d52df0-openshift-default-6d5f887b45-lxbtl\" (UID: \"dc4774a0-3da8-4c65-a0b3-887a476ab465\") " pod="openshift-ingress/e2e-worker-w1-d52df0-openshift-default-6d5f887b45-lxbtl"
Oct 30 19:41:21.837029 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:41:21.837007    4892 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/dc4774a0-3da8-4c65-a0b3-887a476ab465-istio-token\") pod \"e2e-worker-w1-d52df0-openshift-default-6d5f887b45-lxbtl\" (UID: \"dc4774a0-3da8-4c65-a0b3-887a476ab465\") " pod="openshift-ingress/e2e-worker-w1-d52df0-openshift-default-6d5f887b45-lxbtl"
Oct 30 19:41:21.837089 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:41:21.837049    4892 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/dc4774a0-3da8-4c65-a0b3-887a476ab465-istio-data\") pod \"e2e-worker-w1-d52df0-openshift-default-6d5f887b45-lxbtl\" (UID: \"dc4774a0-3da8-4c65-a0b3-887a476ab465\") " pod="openshift-ingress/e2e-worker-w1-d52df0-openshift-default-6d5f887b45-lxbtl"
Oct 30 19:41:21.837153 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:41:21.837129    4892 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-72lrd\" (UniqueName: \"kubernetes.io/projected/dc4774a0-3da8-4c65-a0b3-887a476ab465-kube-api-access-72lrd\") pod \"e2e-worker-w1-d52df0-openshift-default-6d5f887b45-lxbtl\" (UID: \"dc4774a0-3da8-4c65-a0b3-887a476ab465\") " pod="openshift-ingress/e2e-worker-w1-d52df0-openshift-default-6d5f887b45-lxbtl"
Oct 30 19:41:21.837226 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:41:21.837206    4892 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/dc4774a0-3da8-4c65-a0b3-887a476ab465-istiod-ca-cert\") pod \"e2e-worker-w1-d52df0-openshift-default-6d5f887b45-lxbtl\" (UID: \"dc4774a0-3da8-4c65-a0b3-887a476ab465\") " pod="openshift-ingress/e2e-worker-w1-d52df0-openshift-default-6d5f887b45-lxbtl"
Oct 30 19:41:21.837283 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:41:21.837252    4892 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/dc4774a0-3da8-4c65-a0b3-887a476ab465-workload-certs\") pod \"e2e-worker-w1-d52df0-openshift-default-6d5f887b45-lxbtl\" (UID: \"dc4774a0-3da8-4c65-a0b3-887a476ab465\") " pod="openshift-ingress/e2e-worker-w1-d52df0-openshift-default-6d5f887b45-lxbtl"
Oct 30 19:41:21.879417 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:41:21.879387    4892 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-d52df0-openshift-default-6d5f887b45-lxbtl"]
Oct 30 19:41:21.938157 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:41:21.938092    4892 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/dc4774a0-3da8-4c65-a0b3-887a476ab465-istio-token\") pod \"e2e-worker-w1-d52df0-openshift-default-6d5f887b45-lxbtl\" (UID: \"dc4774a0-3da8-4c65-a0b3-887a476ab465\") " pod="openshift-ingress/e2e-worker-w1-d52df0-openshift-default-6d5f887b45-lxbtl"
Oct 30 19:41:21.938157 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:41:21.938131    4892 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/dc4774a0-3da8-4c65-a0b3-887a476ab465-istio-data\") pod \"e2e-worker-w1-d52df0-openshift-default-6d5f887b45-lxbtl\" (UID: \"dc4774a0-3da8-4c65-a0b3-887a476ab465\") " pod="openshift-ingress/e2e-worker-w1-d52df0-openshift-default-6d5f887b45-lxbtl"
Oct 30 19:41:21.938157 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:41:21.938153    4892 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-72lrd\" (UniqueName: \"kubernetes.io/projected/dc4774a0-3da8-4c65-a0b3-887a476ab465-kube-api-access-72lrd\") pod \"e2e-worker-w1-d52df0-openshift-default-6d5f887b45-lxbtl\" (UID: \"dc4774a0-3da8-4c65-a0b3-887a476ab465\") " pod="openshift-ingress/e2e-worker-w1-d52df0-openshift-default-6d5f887b45-lxbtl"
Oct 30 19:41:21.938375 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:41:21.938212    4892 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/dc4774a0-3da8-4c65-a0b3-887a476ab465-istiod-ca-cert\") pod \"e2e-worker-w1-d52df0-openshift-default-6d5f887b45-lxbtl\" (UID: \"dc4774a0-3da8-4c65-a0b3-887a476ab465\") " pod="openshift-ingress/e2e-worker-w1-d52df0-openshift-default-6d5f887b45-lxbtl"
Oct 30 19:41:21.938375 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:41:21.938236    4892 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/dc4774a0-3da8-4c65-a0b3-887a476ab465-workload-certs\") pod \"e2e-worker-w1-d52df0-openshift-default-6d5f887b45-lxbtl\" (UID: \"dc4774a0-3da8-4c65-a0b3-887a476ab465\") " pod="openshift-ingress/e2e-worker-w1-d52df0-openshift-default-6d5f887b45-lxbtl"
Oct 30 19:41:21.938375 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:41:21.938273    4892 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/dc4774a0-3da8-4c65-a0b3-887a476ab465-workload-socket\") pod \"e2e-worker-w1-d52df0-openshift-default-6d5f887b45-lxbtl\" (UID: \"dc4774a0-3da8-4c65-a0b3-887a476ab465\") " pod="openshift-ingress/e2e-worker-w1-d52df0-openshift-default-6d5f887b45-lxbtl"
Oct 30 19:41:21.938553 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:41:21.938395    4892 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/dc4774a0-3da8-4c65-a0b3-887a476ab465-istio-podinfo\") pod \"e2e-worker-w1-d52df0-openshift-default-6d5f887b45-lxbtl\" (UID: \"dc4774a0-3da8-4c65-a0b3-887a476ab465\") " pod="openshift-ingress/e2e-worker-w1-d52df0-openshift-default-6d5f887b45-lxbtl"
Oct 30 19:41:21.938553 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:41:21.938516    4892 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/dc4774a0-3da8-4c65-a0b3-887a476ab465-istio-envoy\") pod \"e2e-worker-w1-d52df0-openshift-default-6d5f887b45-lxbtl\" (UID: \"dc4774a0-3da8-4c65-a0b3-887a476ab465\") " pod="openshift-ingress/e2e-worker-w1-d52df0-openshift-default-6d5f887b45-lxbtl"
Oct 30 19:41:21.938553 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:41:21.938525    4892 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/dc4774a0-3da8-4c65-a0b3-887a476ab465-istio-data\") pod \"e2e-worker-w1-d52df0-openshift-default-6d5f887b45-lxbtl\" (UID: \"dc4774a0-3da8-4c65-a0b3-887a476ab465\") " pod="openshift-ingress/e2e-worker-w1-d52df0-openshift-default-6d5f887b45-lxbtl"
Oct 30 19:41:21.938698 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:41:21.938553    4892 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/dc4774a0-3da8-4c65-a0b3-887a476ab465-credential-socket\") pod \"e2e-worker-w1-d52df0-openshift-default-6d5f887b45-lxbtl\" (UID: \"dc4774a0-3da8-4c65-a0b3-887a476ab465\") " pod="openshift-ingress/e2e-worker-w1-d52df0-openshift-default-6d5f887b45-lxbtl"
Oct 30 19:41:21.938752 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:41:21.938695    4892 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/dc4774a0-3da8-4c65-a0b3-887a476ab465-workload-certs\") pod \"e2e-worker-w1-d52df0-openshift-default-6d5f887b45-lxbtl\" (UID: \"dc4774a0-3da8-4c65-a0b3-887a476ab465\") " pod="openshift-ingress/e2e-worker-w1-d52df0-openshift-default-6d5f887b45-lxbtl"
Oct 30 19:41:21.938752 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:41:21.938705    4892 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/dc4774a0-3da8-4c65-a0b3-887a476ab465-workload-socket\") pod \"e2e-worker-w1-d52df0-openshift-default-6d5f887b45-lxbtl\" (UID: \"dc4774a0-3da8-4c65-a0b3-887a476ab465\") " pod="openshift-ingress/e2e-worker-w1-d52df0-openshift-default-6d5f887b45-lxbtl"
Oct 30 19:41:21.938952 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:41:21.938930    4892 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/dc4774a0-3da8-4c65-a0b3-887a476ab465-credential-socket\") pod \"e2e-worker-w1-d52df0-openshift-default-6d5f887b45-lxbtl\" (UID: \"dc4774a0-3da8-4c65-a0b3-887a476ab465\") " pod="openshift-ingress/e2e-worker-w1-d52df0-openshift-default-6d5f887b45-lxbtl"
Oct 30 19:41:21.939024 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:41:21.938944    4892 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/dc4774a0-3da8-4c65-a0b3-887a476ab465-istiod-ca-cert\") pod \"e2e-worker-w1-d52df0-openshift-default-6d5f887b45-lxbtl\" (UID: \"dc4774a0-3da8-4c65-a0b3-887a476ab465\") " pod="openshift-ingress/e2e-worker-w1-d52df0-openshift-default-6d5f887b45-lxbtl"
Oct 30 19:41:21.940843 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:41:21.940824    4892 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/dc4774a0-3da8-4c65-a0b3-887a476ab465-istio-envoy\") pod \"e2e-worker-w1-d52df0-openshift-default-6d5f887b45-lxbtl\" (UID: \"dc4774a0-3da8-4c65-a0b3-887a476ab465\") " pod="openshift-ingress/e2e-worker-w1-d52df0-openshift-default-6d5f887b45-lxbtl"
Oct 30 19:41:21.941086 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:41:21.941069    4892 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/dc4774a0-3da8-4c65-a0b3-887a476ab465-istio-podinfo\") pod \"e2e-worker-w1-d52df0-openshift-default-6d5f887b45-lxbtl\" (UID: \"dc4774a0-3da8-4c65-a0b3-887a476ab465\") " pod="openshift-ingress/e2e-worker-w1-d52df0-openshift-default-6d5f887b45-lxbtl"
Oct 30 19:41:21.995507 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:41:21.995476    4892 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/dc4774a0-3da8-4c65-a0b3-887a476ab465-istio-token\") pod \"e2e-worker-w1-d52df0-openshift-default-6d5f887b45-lxbtl\" (UID: \"dc4774a0-3da8-4c65-a0b3-887a476ab465\") " pod="openshift-ingress/e2e-worker-w1-d52df0-openshift-default-6d5f887b45-lxbtl"
Oct 30 19:41:21.995602 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:41:21.995559    4892 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-72lrd\" (UniqueName: \"kubernetes.io/projected/dc4774a0-3da8-4c65-a0b3-887a476ab465-kube-api-access-72lrd\") pod \"e2e-worker-w1-d52df0-openshift-default-6d5f887b45-lxbtl\" (UID: \"dc4774a0-3da8-4c65-a0b3-887a476ab465\") " pod="openshift-ingress/e2e-worker-w1-d52df0-openshift-default-6d5f887b45-lxbtl"
Oct 30 19:41:22.287796 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:41:22.287700    4892 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-d52df0-openshift-default-6d5f887b45-lxbtl"
Oct 30 19:41:22.721333 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:41:22.721306    4892 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-d52df0-openshift-default-6d5f887b45-lxbtl"]
Oct 30 19:41:22.724518 ip-10-0-132-82 kubenswrapper[4892]: W1030 19:41:22.724486    4892 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddc4774a0_3da8_4c65_a0b3_887a476ab465.slice/crio-ff7273f384e00244298e2b9e18ca867407e0f695aca8cf48d9209ef628100391 WatchSource:0}: Error finding container ff7273f384e00244298e2b9e18ca867407e0f695aca8cf48d9209ef628100391: Status 404 returned error can't find the container with id ff7273f384e00244298e2b9e18ca867407e0f695aca8cf48d9209ef628100391
Oct 30 19:41:22.727164 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:41:22.727122    4892 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892000Ki","pods":"250"}
Oct 30 19:41:22.727278 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:41:22.727200    4892 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892000Ki","pods":"250"}
Oct 30 19:41:22.727278 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:41:22.727237    4892 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892000Ki","pods":"250"}
Oct 30 19:41:23.508600 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:41:23.508567    4892 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-d52df0-openshift-default-6d5f887b45-lxbtl" event={"ID":"dc4774a0-3da8-4c65-a0b3-887a476ab465","Type":"ContainerStarted","Data":"440d694500bb30cdd0530699bd2d413223f8c1d1f41cccf42775b61856b664e4"}
Oct 30 19:41:23.508804 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:41:23.508620    4892 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-d52df0-openshift-default-6d5f887b45-lxbtl" event={"ID":"dc4774a0-3da8-4c65-a0b3-887a476ab465","Type":"ContainerStarted","Data":"ff7273f384e00244298e2b9e18ca867407e0f695aca8cf48d9209ef628100391"}
Oct 30 19:41:23.745990 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:41:23.742563    4892 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w1-d52df0-openshift-default-6d5f887b45-lxbtl" podStartSLOduration=2.742543682 podStartE2EDuration="2.742543682s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:41:21 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:41:23.741189261 +0000 UTC m=+1531.311654862" watchObservedRunningTime="2026-10-30 19:41:23.742543682 +0000 UTC m=+1531.313009282"
Oct 30 19:41:24.065688 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:41:24.065637    4892 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f92547ee-ldpfj"]
Oct 30 19:41:24.069897 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:41:24.069873    4892 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f92547ee-ldpfj"]
Oct 30 19:41:24.288025 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:41:24.287990    4892 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w1-d52df0-openshift-default-6d5f887b45-lxbtl"
Oct 30 19:41:24.289470 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:41:24.289447    4892 patch_prober.go:29] interesting pod/e2e-worker-w1-d52df0-openshift-default-6d5f887b45-lxbtl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.102:15021/healthz/ready\": dial tcp 10.132.0.102:15021: connect: connection refused" start-of-body=
Oct 30 19:41:24.289557 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:41:24.289505    4892 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-d52df0-openshift-default-6d5f887b45-lxbtl" podUID="dc4774a0-3da8-4c65-a0b3-887a476ab465" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.102:15021/healthz/ready\": dial tcp 10.132.0.102:15021: connect: connection refused"
Oct 30 19:41:24.569107 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:41:24.569067    4892 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="01dbf08c-a038-4419-bacb-ec4e2df61a57" path="/var/lib/kubelet/pods/01dbf08c-a038-4419-bacb-ec4e2df61a57/volumes"
Oct 30 19:41:24.867206 ip-10-0-132-82 kubenswrapper[4892]: E1030 19:41:24.867118    4892 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-jrm87" containerName="extractor"
Oct 30 19:41:25.288664 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:41:25.288632    4892 patch_prober.go:29] interesting pod/e2e-worker-w1-d52df0-openshift-default-6d5f887b45-lxbtl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.102:15021/healthz/ready\": dial tcp 10.132.0.102:15021: connect: connection refused" start-of-body=
Oct 30 19:41:25.288845 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:41:25.288697    4892 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-d52df0-openshift-default-6d5f887b45-lxbtl" podUID="dc4774a0-3da8-4c65-a0b3-887a476ab465" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.102:15021/healthz/ready\": dial tcp 10.132.0.102:15021: connect: connection refused"
Oct 30 19:41:25.592417 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:41:25.592337    4892 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e4655384-zbxsh"]
Oct 30 19:41:25.597404 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:41:25.597389    4892 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e4655384-zbxsh"
Oct 30 19:41:25.666173 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:41:25.666142    4892 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e4655384-zbxsh"]
Oct 30 19:41:25.688805 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:41:25.688769    4892 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/784612e5-1d50-4667-abdb-cf3bce669205-maas-api-xks-ca\") pod \"maas-api-revoke-sub-e4655384-zbxsh\" (UID: \"784612e5-1d50-4667-abdb-cf3bce669205\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e4655384-zbxsh"
Oct 30 19:41:25.689022 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:41:25.688828    4892 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/784612e5-1d50-4667-abdb-cf3bce669205-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-e4655384-zbxsh\" (UID: \"784612e5-1d50-4667-abdb-cf3bce669205\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e4655384-zbxsh"
Oct 30 19:41:25.689165 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:41:25.689136    4892 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7k4wp\" (UniqueName: \"kubernetes.io/projected/784612e5-1d50-4667-abdb-cf3bce669205-kube-api-access-7k4wp\") pod \"maas-api-revoke-sub-e4655384-zbxsh\" (UID: \"784612e5-1d50-4667-abdb-cf3bce669205\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e4655384-zbxsh"
Oct 30 19:41:25.699925 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:41:25.699898    4892 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 19:41:25.699925 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:41:25.699924    4892 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 30 19:41:25.700131 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:41:25.699959    4892 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 30 19:41:25.700131 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:41:25.700044    4892 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-nlxp4\""
Oct 30 19:41:25.790877 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:41:25.790839    4892 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7k4wp\" (UniqueName: \"kubernetes.io/projected/784612e5-1d50-4667-abdb-cf3bce669205-kube-api-access-7k4wp\") pod \"maas-api-revoke-sub-e4655384-zbxsh\" (UID: \"784612e5-1d50-4667-abdb-cf3bce669205\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e4655384-zbxsh"
Oct 30 19:41:25.791100 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:41:25.790925    4892 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/784612e5-1d50-4667-abdb-cf3bce669205-maas-api-xks-ca\") pod \"maas-api-revoke-sub-e4655384-zbxsh\" (UID: \"784612e5-1d50-4667-abdb-cf3bce669205\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e4655384-zbxsh"
Oct 30 19:41:25.791100 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:41:25.790957    4892 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/784612e5-1d50-4667-abdb-cf3bce669205-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-e4655384-zbxsh\" (UID: \"784612e5-1d50-4667-abdb-cf3bce669205\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e4655384-zbxsh"
Oct 30 19:41:25.791625 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:41:25.791601    4892 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/784612e5-1d50-4667-abdb-cf3bce669205-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-e4655384-zbxsh\" (UID: \"784612e5-1d50-4667-abdb-cf3bce669205\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e4655384-zbxsh"
Oct 30 19:41:25.793505 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:41:25.793484    4892 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/784612e5-1d50-4667-abdb-cf3bce669205-maas-api-xks-ca\") pod \"maas-api-revoke-sub-e4655384-zbxsh\" (UID: \"784612e5-1d50-4667-abdb-cf3bce669205\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e4655384-zbxsh"
Oct 30 19:41:25.864356 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:41:25.864277    4892 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7k4wp\" (UniqueName: \"kubernetes.io/projected/784612e5-1d50-4667-abdb-cf3bce669205-kube-api-access-7k4wp\") pod \"maas-api-revoke-sub-e4655384-zbxsh\" (UID: \"784612e5-1d50-4667-abdb-cf3bce669205\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e4655384-zbxsh"
Oct 30 19:41:25.912310 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:41:25.912275    4892 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e4655384-zbxsh"
Oct 30 19:41:26.289106 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:41:26.289067    4892 patch_prober.go:29] interesting pod/e2e-worker-w1-d52df0-openshift-default-6d5f887b45-lxbtl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.102:15021/healthz/ready\": dial tcp 10.132.0.102:15021: connect: connection refused" start-of-body=
Oct 30 19:41:26.289268 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:41:26.289131    4892 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-d52df0-openshift-default-6d5f887b45-lxbtl" podUID="dc4774a0-3da8-4c65-a0b3-887a476ab465" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.102:15021/healthz/ready\": dial tcp 10.132.0.102:15021: connect: connection refused"
Oct 30 19:41:26.325601 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:41:26.325572    4892 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e4655384-zbxsh"]
Oct 30 19:41:26.326470 ip-10-0-132-82 kubenswrapper[4892]: W1030 19:41:26.326442    4892 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod784612e5_1d50_4667_abdb_cf3bce669205.slice/crio-9c5832a3c9f90bf6b40103653b0191b64584593ec83b9ef0ddb572ebea0dac57 WatchSource:0}: Error finding container 9c5832a3c9f90bf6b40103653b0191b64584593ec83b9ef0ddb572ebea0dac57: Status 404 returned error can't find the container with id 9c5832a3c9f90bf6b40103653b0191b64584593ec83b9ef0ddb572ebea0dac57
Oct 30 19:41:26.570843 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:41:26.570810    4892 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e4655384-zbxsh" event={"ID":"784612e5-1d50-4667-abdb-cf3bce669205","Type":"ContainerStarted","Data":"9c5832a3c9f90bf6b40103653b0191b64584593ec83b9ef0ddb572ebea0dac57"}
Oct 30 19:41:27.288497 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:41:27.288455    4892 patch_prober.go:29] interesting pod/e2e-worker-w1-d52df0-openshift-default-6d5f887b45-lxbtl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.102:15021/healthz/ready\": dial tcp 10.132.0.102:15021: connect: connection refused" start-of-body=
Oct 30 19:41:27.288916 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:41:27.288512    4892 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-d52df0-openshift-default-6d5f887b45-lxbtl" podUID="dc4774a0-3da8-4c65-a0b3-887a476ab465" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.102:15021/healthz/ready\": dial tcp 10.132.0.102:15021: connect: connection refused"
Oct 30 19:41:27.579421 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:41:27.579390    4892 generic.go:356] "Generic (PLEG): container finished" podID="784612e5-1d50-4667-abdb-cf3bce669205" containerID="a1a9300f93a1c00790cb4fb07f5030fda406b554b943efe483ca4378a987bd60" exitCode=0
Oct 30 19:41:27.579598 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:41:27.579478    4892 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e4655384-zbxsh" event={"ID":"784612e5-1d50-4667-abdb-cf3bce669205","Type":"ContainerDied","Data":"a1a9300f93a1c00790cb4fb07f5030fda406b554b943efe483ca4378a987bd60"}
Oct 30 19:41:28.288881 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:41:28.288839    4892 patch_prober.go:29] interesting pod/e2e-worker-w1-d52df0-openshift-default-6d5f887b45-lxbtl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.102:15021/healthz/ready\": dial tcp 10.132.0.102:15021: connect: connection refused" start-of-body=
Oct 30 19:41:28.289275 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:41:28.288905    4892 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-d52df0-openshift-default-6d5f887b45-lxbtl" podUID="dc4774a0-3da8-4c65-a0b3-887a476ab465" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.102:15021/healthz/ready\": dial tcp 10.132.0.102:15021: connect: connection refused"
Oct 30 19:41:28.776062 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:41:28.776026    4892 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e4655384-zbxsh"
Oct 30 19:41:28.828310 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:41:28.828277    4892 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/784612e5-1d50-4667-abdb-cf3bce669205-kube-api-access-7k4wp\" (UniqueName: \"kubernetes.io/projected/784612e5-1d50-4667-abdb-cf3bce669205-kube-api-access-7k4wp\") pod \"784612e5-1d50-4667-abdb-cf3bce669205\" (UID: \"784612e5-1d50-4667-abdb-cf3bce669205\") "
Oct 30 19:41:28.830656 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:41:28.830624    4892 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/784612e5-1d50-4667-abdb-cf3bce669205-kube-api-access-7k4wp" pod "784612e5-1d50-4667-abdb-cf3bce669205" (UID: "784612e5-1d50-4667-abdb-cf3bce669205"). InnerVolumeSpecName "kube-api-access-7k4wp". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:41:28.929404 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:41:28.929373    4892 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/784612e5-1d50-4667-abdb-cf3bce669205-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/784612e5-1d50-4667-abdb-cf3bce669205-maas-api-openshift-ca\") pod \"784612e5-1d50-4667-abdb-cf3bce669205\" (UID: \"784612e5-1d50-4667-abdb-cf3bce669205\") "
Oct 30 19:41:28.929735 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:41:28.929712    4892 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/784612e5-1d50-4667-abdb-cf3bce669205-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/784612e5-1d50-4667-abdb-cf3bce669205-maas-api-xks-ca\") pod \"784612e5-1d50-4667-abdb-cf3bce669205\" (UID: \"784612e5-1d50-4667-abdb-cf3bce669205\") "
Oct 30 19:41:28.930202 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:41:28.929775    4892 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/784612e5-1d50-4667-abdb-cf3bce669205-maas-api-openshift-ca" pod "784612e5-1d50-4667-abdb-cf3bce669205" (UID: "784612e5-1d50-4667-abdb-cf3bce669205"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 19:41:28.930483 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:41:28.930461    4892 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/784612e5-1d50-4667-abdb-cf3bce669205-maas-api-openshift-ca\") on node \"ip-10-0-132-82.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 19:41:28.930607 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:41:28.930590    4892 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7k4wp\" (UniqueName: \"kubernetes.io/projected/784612e5-1d50-4667-abdb-cf3bce669205-kube-api-access-7k4wp\") on node \"ip-10-0-132-82.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 19:41:28.937085 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:41:28.933007    4892 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/784612e5-1d50-4667-abdb-cf3bce669205-maas-api-xks-ca" pod "784612e5-1d50-4667-abdb-cf3bce669205" (UID: "784612e5-1d50-4667-abdb-cf3bce669205"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 19:41:29.032281 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:41:29.032241    4892 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/784612e5-1d50-4667-abdb-cf3bce669205-maas-api-xks-ca\") on node \"ip-10-0-132-82.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 19:41:29.288533 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:41:29.288498    4892 patch_prober.go:29] interesting pod/e2e-worker-w1-d52df0-openshift-default-6d5f887b45-lxbtl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.102:15021/healthz/ready\": dial tcp 10.132.0.102:15021: connect: connection refused" start-of-body=
Oct 30 19:41:29.288707 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:41:29.288563    4892 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-d52df0-openshift-default-6d5f887b45-lxbtl" podUID="dc4774a0-3da8-4c65-a0b3-887a476ab465" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.102:15021/healthz/ready\": dial tcp 10.132.0.102:15021: connect: connection refused"
Oct 30 19:41:29.594553 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:41:29.594478    4892 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e4655384-zbxsh"
Oct 30 19:41:29.594919 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:41:29.594480    4892 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e4655384-zbxsh" event={"ID":"784612e5-1d50-4667-abdb-cf3bce669205","Type":"ContainerDied","Data":"9c5832a3c9f90bf6b40103653b0191b64584593ec83b9ef0ddb572ebea0dac57"}
Oct 30 19:41:29.594919 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:41:29.594602    4892 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="9c5832a3c9f90bf6b40103653b0191b64584593ec83b9ef0ddb572ebea0dac57"
Oct 30 19:41:30.055694 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:41:30.055658    4892 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-cf3dab52-ghxkm"]
Oct 30 19:41:30.062372 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:41:30.062342    4892 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-cf3dab52-ghxkm"]
Oct 30 19:41:30.292696 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:41:30.292663    4892 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w1-d52df0-openshift-default-6d5f887b45-lxbtl"
Oct 30 19:41:30.293171 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:41:30.293143    4892 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w1-d52df0-openshift-default-6d5f887b45-lxbtl"
Oct 30 19:41:30.294788 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:41:30.294109    4892 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w1-d52df0-openshift-default-6d5f887b45-lxbtl"
Oct 30 19:41:30.573165 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:41:30.573128    4892 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="dc25af73-f73d-49c8-aa34-a03de0cd2297" path="/var/lib/kubelet/pods/dc25af73-f73d-49c8-aa34-a03de0cd2297/volumes"
Oct 30 19:41:32.050856 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:41:32.050818    4892 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-15133760-rwwjv"]
Oct 30 19:41:32.057188 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:41:32.057156    4892 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-15133760-rwwjv"]
Oct 30 19:41:32.571627 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:41:32.571586    4892 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0f728833-f24c-48d8-acd2-37e2a738ffc8" path="/var/lib/kubelet/pods/0f728833-f24c-48d8-acd2-37e2a738ffc8/volumes"
Oct 30 19:41:39.319484 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:41:39.319452    4892 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-83faf4fb-wr9j2"]
Oct 30 19:41:39.319991 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:41:39.319957    4892 cpu_manager.go:414] "containerMap: removing container" podUID="784612e5-1d50-4667-abdb-cf3bce669205" containerName="revoke-keys"
Oct 30 19:41:39.319991 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:41:39.319989    4892 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="784612e5-1d50-4667-abdb-cf3bce669205" containerName="revoke-keys"
Oct 30 19:41:39.320129 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:41:39.320119    4892 memory_manager.go:365] "RemoveStaleState removing state" podUID="784612e5-1d50-4667-abdb-cf3bce669205" containerName="revoke-keys"
Oct 30 19:41:39.325021 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:41:39.325005    4892 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-83faf4fb-wr9j2"
Oct 30 19:41:39.382046 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:41:39.382020    4892 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-83faf4fb-wr9j2"]
Oct 30 19:41:39.425660 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:41:39.425632    4892 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 19:41:39.425660 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:41:39.425633    4892 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 30 19:41:39.425829 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:41:39.425693    4892 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-nlxp4\""
Oct 30 19:41:39.426082 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:41:39.426053    4892 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 30 19:41:39.461308 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:41:39.461281    4892 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/402433ad-7d20-46f4-8ed6-c0071f0053ce-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-83faf4fb-wr9j2\" (UID: \"402433ad-7d20-46f4-8ed6-c0071f0053ce\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-83faf4fb-wr9j2"
Oct 30 19:41:39.461440 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:41:39.461410    4892 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6gldv\" (UniqueName: \"kubernetes.io/projected/402433ad-7d20-46f4-8ed6-c0071f0053ce-kube-api-access-6gldv\") pod \"maas-api-revoke-sub-83faf4fb-wr9j2\" (UID: \"402433ad-7d20-46f4-8ed6-c0071f0053ce\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-83faf4fb-wr9j2"
Oct 30 19:41:39.461491 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:41:39.461461    4892 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/402433ad-7d20-46f4-8ed6-c0071f0053ce-maas-api-xks-ca\") pod \"maas-api-revoke-sub-83faf4fb-wr9j2\" (UID: \"402433ad-7d20-46f4-8ed6-c0071f0053ce\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-83faf4fb-wr9j2"
Oct 30 19:41:39.562528 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:41:39.562504    4892 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6gldv\" (UniqueName: \"kubernetes.io/projected/402433ad-7d20-46f4-8ed6-c0071f0053ce-kube-api-access-6gldv\") pod \"maas-api-revoke-sub-83faf4fb-wr9j2\" (UID: \"402433ad-7d20-46f4-8ed6-c0071f0053ce\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-83faf4fb-wr9j2"
Oct 30 19:41:39.562681 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:41:39.562542    4892 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/402433ad-7d20-46f4-8ed6-c0071f0053ce-maas-api-xks-ca\") pod \"maas-api-revoke-sub-83faf4fb-wr9j2\" (UID: \"402433ad-7d20-46f4-8ed6-c0071f0053ce\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-83faf4fb-wr9j2"
Oct 30 19:41:39.562681 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:41:39.562670    4892 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/402433ad-7d20-46f4-8ed6-c0071f0053ce-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-83faf4fb-wr9j2\" (UID: \"402433ad-7d20-46f4-8ed6-c0071f0053ce\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-83faf4fb-wr9j2"
Oct 30 19:41:39.563527 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:41:39.563502    4892 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/402433ad-7d20-46f4-8ed6-c0071f0053ce-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-83faf4fb-wr9j2\" (UID: \"402433ad-7d20-46f4-8ed6-c0071f0053ce\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-83faf4fb-wr9j2"
Oct 30 19:41:39.565353 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:41:39.565333    4892 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/402433ad-7d20-46f4-8ed6-c0071f0053ce-maas-api-xks-ca\") pod \"maas-api-revoke-sub-83faf4fb-wr9j2\" (UID: \"402433ad-7d20-46f4-8ed6-c0071f0053ce\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-83faf4fb-wr9j2"
Oct 30 19:41:39.620668 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:41:39.620614    4892 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6gldv\" (UniqueName: \"kubernetes.io/projected/402433ad-7d20-46f4-8ed6-c0071f0053ce-kube-api-access-6gldv\") pod \"maas-api-revoke-sub-83faf4fb-wr9j2\" (UID: \"402433ad-7d20-46f4-8ed6-c0071f0053ce\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-83faf4fb-wr9j2"
Oct 30 19:41:39.639151 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:41:39.639133    4892 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-83faf4fb-wr9j2"
Oct 30 19:41:40.027550 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:41:40.027517    4892 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-83faf4fb-wr9j2"]
Oct 30 19:41:40.028125 ip-10-0-132-82 kubenswrapper[4892]: W1030 19:41:40.028101    4892 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod402433ad_7d20_46f4_8ed6_c0071f0053ce.slice/crio-924dc430f6d28d1a7aac6c822ba9507910c9fc59024e37f8b5052a4524f9f0b0 WatchSource:0}: Error finding container 924dc430f6d28d1a7aac6c822ba9507910c9fc59024e37f8b5052a4524f9f0b0: Status 404 returned error can't find the container with id 924dc430f6d28d1a7aac6c822ba9507910c9fc59024e37f8b5052a4524f9f0b0
Oct 30 19:41:40.107249 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:41:40.107214    4892 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-68c9e294-bwx5n"]
Oct 30 19:41:40.111198 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:41:40.111179    4892 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-68c9e294-bwx5n"
Oct 30 19:41:40.170602 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:41:40.170568    4892 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-68c9e294-bwx5n"]
Oct 30 19:41:40.171217 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:41:40.171187    4892 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vrfqn\" (UniqueName: \"kubernetes.io/projected/8082afb5-3a19-4038-8987-fcadf0f420b2-kube-api-access-vrfqn\") pod \"maas-api-revoke-sub-68c9e294-bwx5n\" (UID: \"8082afb5-3a19-4038-8987-fcadf0f420b2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-68c9e294-bwx5n"
Oct 30 19:41:40.171367 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:41:40.171234    4892 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8082afb5-3a19-4038-8987-fcadf0f420b2-maas-api-xks-ca\") pod \"maas-api-revoke-sub-68c9e294-bwx5n\" (UID: \"8082afb5-3a19-4038-8987-fcadf0f420b2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-68c9e294-bwx5n"
Oct 30 19:41:40.171445 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:41:40.171421    4892 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8082afb5-3a19-4038-8987-fcadf0f420b2-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-68c9e294-bwx5n\" (UID: \"8082afb5-3a19-4038-8987-fcadf0f420b2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-68c9e294-bwx5n"
Oct 30 19:41:40.272294 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:41:40.272260    4892 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8082afb5-3a19-4038-8987-fcadf0f420b2-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-68c9e294-bwx5n\" (UID: \"8082afb5-3a19-4038-8987-fcadf0f420b2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-68c9e294-bwx5n"
Oct 30 19:41:40.272456 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:41:40.272397    4892 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vrfqn\" (UniqueName: \"kubernetes.io/projected/8082afb5-3a19-4038-8987-fcadf0f420b2-kube-api-access-vrfqn\") pod \"maas-api-revoke-sub-68c9e294-bwx5n\" (UID: \"8082afb5-3a19-4038-8987-fcadf0f420b2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-68c9e294-bwx5n"
Oct 30 19:41:40.272456 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:41:40.272427    4892 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8082afb5-3a19-4038-8987-fcadf0f420b2-maas-api-xks-ca\") pod \"maas-api-revoke-sub-68c9e294-bwx5n\" (UID: \"8082afb5-3a19-4038-8987-fcadf0f420b2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-68c9e294-bwx5n"
Oct 30 19:41:40.272996 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:41:40.272942    4892 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8082afb5-3a19-4038-8987-fcadf0f420b2-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-68c9e294-bwx5n\" (UID: \"8082afb5-3a19-4038-8987-fcadf0f420b2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-68c9e294-bwx5n"
Oct 30 19:41:40.274871 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:41:40.274848    4892 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8082afb5-3a19-4038-8987-fcadf0f420b2-maas-api-xks-ca\") pod \"maas-api-revoke-sub-68c9e294-bwx5n\" (UID: \"8082afb5-3a19-4038-8987-fcadf0f420b2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-68c9e294-bwx5n"
Oct 30 19:41:40.330551 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:41:40.330527    4892 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vrfqn\" (UniqueName: \"kubernetes.io/projected/8082afb5-3a19-4038-8987-fcadf0f420b2-kube-api-access-vrfqn\") pod \"maas-api-revoke-sub-68c9e294-bwx5n\" (UID: \"8082afb5-3a19-4038-8987-fcadf0f420b2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-68c9e294-bwx5n"
Oct 30 19:41:40.424911 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:41:40.424887    4892 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-68c9e294-bwx5n"
Oct 30 19:41:40.690044 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:41:40.690007    4892 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-83faf4fb-wr9j2" event={"ID":"402433ad-7d20-46f4-8ed6-c0071f0053ce","Type":"ContainerStarted","Data":"baddb70af190f55295bf86157eeafe35e4d47dd9bfd69b16b77a34deaf496f45"}
Oct 30 19:41:40.690195 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:41:40.690067    4892 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-83faf4fb-wr9j2" event={"ID":"402433ad-7d20-46f4-8ed6-c0071f0053ce","Type":"ContainerStarted","Data":"924dc430f6d28d1a7aac6c822ba9507910c9fc59024e37f8b5052a4524f9f0b0"}
Oct 30 19:41:40.805890 ip-10-0-132-82 kubenswrapper[4892]: W1030 19:41:40.805855    4892 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8082afb5_3a19_4038_8987_fcadf0f420b2.slice/crio-7a07c09183a9870c238dfcf48918bb03a18fabd6aa18b315e50f8491938c3a01 WatchSource:0}: Error finding container 7a07c09183a9870c238dfcf48918bb03a18fabd6aa18b315e50f8491938c3a01: Status 404 returned error can't find the container with id 7a07c09183a9870c238dfcf48918bb03a18fabd6aa18b315e50f8491938c3a01
Oct 30 19:41:40.805890 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:41:40.805890    4892 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-68c9e294-bwx5n"]
Oct 30 19:41:40.851947 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:41:40.851896    4892 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-83faf4fb-wr9j2" podStartSLOduration=1.851876543 podStartE2EDuration="1.851876543s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:41:39 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:41:40.807460563 +0000 UTC m=+1548.377926351" watchObservedRunningTime="2026-10-30 19:41:40.851876543 +0000 UTC m=+1548.422342142"
Oct 30 19:41:41.718866 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:41:41.718810    4892 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-68c9e294-bwx5n" event={"ID":"8082afb5-3a19-4038-8987-fcadf0f420b2","Type":"ContainerStarted","Data":"ea31cff2af7f01d1ae598debaa94044ecd4f592d721b2f9fb1c03680fbbd0ce4"}
Oct 30 19:41:41.719280 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:41:41.718889    4892 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-68c9e294-bwx5n" event={"ID":"8082afb5-3a19-4038-8987-fcadf0f420b2","Type":"ContainerStarted","Data":"7a07c09183a9870c238dfcf48918bb03a18fabd6aa18b315e50f8491938c3a01"}
Oct 30 19:41:41.721528 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:41:41.721495    4892 generic.go:356] "Generic (PLEG): container finished" podID="402433ad-7d20-46f4-8ed6-c0071f0053ce" containerID="baddb70af190f55295bf86157eeafe35e4d47dd9bfd69b16b77a34deaf496f45" exitCode=0
Oct 30 19:41:41.721638 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:41:41.721542    4892 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-83faf4fb-wr9j2" event={"ID":"402433ad-7d20-46f4-8ed6-c0071f0053ce","Type":"ContainerDied","Data":"baddb70af190f55295bf86157eeafe35e4d47dd9bfd69b16b77a34deaf496f45"}
Oct 30 19:41:41.835829 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:41:41.835770    4892 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-68c9e294-bwx5n" podStartSLOduration=1.835752123 podStartE2EDuration="1.835752123s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:41:40 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:41:41.832696362 +0000 UTC m=+1549.403161963" watchObservedRunningTime="2026-10-30 19:41:41.835752123 +0000 UTC m=+1549.406217723"
Oct 30 19:41:42.730315 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:41:42.730279    4892 generic.go:356] "Generic (PLEG): container finished" podID="8082afb5-3a19-4038-8987-fcadf0f420b2" containerID="ea31cff2af7f01d1ae598debaa94044ecd4f592d721b2f9fb1c03680fbbd0ce4" exitCode=0
Oct 30 19:41:42.730739 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:41:42.730318    4892 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-68c9e294-bwx5n" event={"ID":"8082afb5-3a19-4038-8987-fcadf0f420b2","Type":"ContainerDied","Data":"ea31cff2af7f01d1ae598debaa94044ecd4f592d721b2f9fb1c03680fbbd0ce4"}
Oct 30 19:41:42.874666 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:41:42.874639    4892 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-83faf4fb-wr9j2"
Oct 30 19:41:43.023011 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:41:43.022903    4892 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/402433ad-7d20-46f4-8ed6-c0071f0053ce-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/402433ad-7d20-46f4-8ed6-c0071f0053ce-maas-api-xks-ca\") pod \"402433ad-7d20-46f4-8ed6-c0071f0053ce\" (UID: \"402433ad-7d20-46f4-8ed6-c0071f0053ce\") "
Oct 30 19:41:43.023181 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:41:43.023051    4892 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/402433ad-7d20-46f4-8ed6-c0071f0053ce-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/402433ad-7d20-46f4-8ed6-c0071f0053ce-maas-api-openshift-ca\") pod \"402433ad-7d20-46f4-8ed6-c0071f0053ce\" (UID: \"402433ad-7d20-46f4-8ed6-c0071f0053ce\") "
Oct 30 19:41:43.023181 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:41:43.023169    4892 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/402433ad-7d20-46f4-8ed6-c0071f0053ce-kube-api-access-6gldv\" (UniqueName: \"kubernetes.io/projected/402433ad-7d20-46f4-8ed6-c0071f0053ce-kube-api-access-6gldv\") pod \"402433ad-7d20-46f4-8ed6-c0071f0053ce\" (UID: \"402433ad-7d20-46f4-8ed6-c0071f0053ce\") "
Oct 30 19:41:43.023505 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:41:43.023481    4892 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/402433ad-7d20-46f4-8ed6-c0071f0053ce-maas-api-openshift-ca" pod "402433ad-7d20-46f4-8ed6-c0071f0053ce" (UID: "402433ad-7d20-46f4-8ed6-c0071f0053ce"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 19:41:43.025577 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:41:43.025550    4892 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/402433ad-7d20-46f4-8ed6-c0071f0053ce-kube-api-access-6gldv" pod "402433ad-7d20-46f4-8ed6-c0071f0053ce" (UID: "402433ad-7d20-46f4-8ed6-c0071f0053ce"). InnerVolumeSpecName "kube-api-access-6gldv". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:41:43.025677 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:41:43.025656    4892 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/402433ad-7d20-46f4-8ed6-c0071f0053ce-maas-api-xks-ca" pod "402433ad-7d20-46f4-8ed6-c0071f0053ce" (UID: "402433ad-7d20-46f4-8ed6-c0071f0053ce"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 19:41:43.124745 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:41:43.124716    4892 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/402433ad-7d20-46f4-8ed6-c0071f0053ce-maas-api-openshift-ca\") on node \"ip-10-0-132-82.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 19:41:43.124745 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:41:43.124741    4892 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6gldv\" (UniqueName: \"kubernetes.io/projected/402433ad-7d20-46f4-8ed6-c0071f0053ce-kube-api-access-6gldv\") on node \"ip-10-0-132-82.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 19:41:43.124900 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:41:43.124752    4892 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/402433ad-7d20-46f4-8ed6-c0071f0053ce-maas-api-xks-ca\") on node \"ip-10-0-132-82.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 19:41:43.738585 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:41:43.738558    4892 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-83faf4fb-wr9j2"
Oct 30 19:41:43.739073 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:41:43.738595    4892 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-83faf4fb-wr9j2" event={"ID":"402433ad-7d20-46f4-8ed6-c0071f0053ce","Type":"ContainerDied","Data":"924dc430f6d28d1a7aac6c822ba9507910c9fc59024e37f8b5052a4524f9f0b0"}
Oct 30 19:41:43.739073 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:41:43.738651    4892 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="924dc430f6d28d1a7aac6c822ba9507910c9fc59024e37f8b5052a4524f9f0b0"
Oct 30 19:41:43.892855 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:41:43.892834    4892 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-68c9e294-bwx5n"
Oct 30 19:41:44.033540 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:41:44.033466    4892 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/8082afb5-3a19-4038-8987-fcadf0f420b2-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8082afb5-3a19-4038-8987-fcadf0f420b2-maas-api-xks-ca\") pod \"8082afb5-3a19-4038-8987-fcadf0f420b2\" (UID: \"8082afb5-3a19-4038-8987-fcadf0f420b2\") "
Oct 30 19:41:44.033540 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:41:44.033520    4892 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/8082afb5-3a19-4038-8987-fcadf0f420b2-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8082afb5-3a19-4038-8987-fcadf0f420b2-maas-api-openshift-ca\") pod \"8082afb5-3a19-4038-8987-fcadf0f420b2\" (UID: \"8082afb5-3a19-4038-8987-fcadf0f420b2\") "
Oct 30 19:41:44.033764 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:41:44.033618    4892 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8082afb5-3a19-4038-8987-fcadf0f420b2-kube-api-access-vrfqn\" (UniqueName: \"kubernetes.io/projected/8082afb5-3a19-4038-8987-fcadf0f420b2-kube-api-access-vrfqn\") pod \"8082afb5-3a19-4038-8987-fcadf0f420b2\" (UID: \"8082afb5-3a19-4038-8987-fcadf0f420b2\") "
Oct 30 19:41:44.033983 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:41:44.033937    4892 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8082afb5-3a19-4038-8987-fcadf0f420b2-maas-api-openshift-ca" pod "8082afb5-3a19-4038-8987-fcadf0f420b2" (UID: "8082afb5-3a19-4038-8987-fcadf0f420b2"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 19:41:44.034122 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:41:44.034102    4892 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8082afb5-3a19-4038-8987-fcadf0f420b2-maas-api-openshift-ca\") on node \"ip-10-0-132-82.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 19:41:44.035781 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:41:44.035755    4892 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8082afb5-3a19-4038-8987-fcadf0f420b2-maas-api-xks-ca" pod "8082afb5-3a19-4038-8987-fcadf0f420b2" (UID: "8082afb5-3a19-4038-8987-fcadf0f420b2"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 19:41:44.036214 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:41:44.036187    4892 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8082afb5-3a19-4038-8987-fcadf0f420b2-kube-api-access-vrfqn" pod "8082afb5-3a19-4038-8987-fcadf0f420b2" (UID: "8082afb5-3a19-4038-8987-fcadf0f420b2"). InnerVolumeSpecName "kube-api-access-vrfqn". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:41:44.135761 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:41:44.135731    4892 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vrfqn\" (UniqueName: \"kubernetes.io/projected/8082afb5-3a19-4038-8987-fcadf0f420b2-kube-api-access-vrfqn\") on node \"ip-10-0-132-82.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 19:41:44.135958 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:41:44.135765    4892 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8082afb5-3a19-4038-8987-fcadf0f420b2-maas-api-xks-ca\") on node \"ip-10-0-132-82.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 19:41:44.747530 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:41:44.747497    4892 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-68c9e294-bwx5n" event={"ID":"8082afb5-3a19-4038-8987-fcadf0f420b2","Type":"ContainerDied","Data":"7a07c09183a9870c238dfcf48918bb03a18fabd6aa18b315e50f8491938c3a01"}
Oct 30 19:41:44.747530 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:41:44.747523    4892 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-68c9e294-bwx5n"
Oct 30 19:41:44.748115 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:41:44.747542    4892 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="7a07c09183a9870c238dfcf48918bb03a18fabd6aa18b315e50f8491938c3a01"
Oct 30 19:41:46.225622 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:41:46.225583    4892 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w2-a3f9b7-openshift-default-7d6d8b8fb5-m5zsm"]
Oct 30 19:41:46.226143 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:41:46.226125    4892 cpu_manager.go:414] "containerMap: removing container" podUID="8082afb5-3a19-4038-8987-fcadf0f420b2" containerName="revoke-keys"
Oct 30 19:41:46.226143 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:41:46.226139    4892 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8082afb5-3a19-4038-8987-fcadf0f420b2" containerName="revoke-keys"
Oct 30 19:41:46.226257 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:41:46.226171    4892 cpu_manager.go:414] "containerMap: removing container" podUID="402433ad-7d20-46f4-8ed6-c0071f0053ce" containerName="revoke-keys"
Oct 30 19:41:46.226257 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:41:46.226177    4892 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="402433ad-7d20-46f4-8ed6-c0071f0053ce" containerName="revoke-keys"
Oct 30 19:41:46.226352 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:41:46.226277    4892 memory_manager.go:365] "RemoveStaleState removing state" podUID="402433ad-7d20-46f4-8ed6-c0071f0053ce" containerName="revoke-keys"
Oct 30 19:41:46.226352 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:41:46.226287    4892 memory_manager.go:365] "RemoveStaleState removing state" podUID="8082afb5-3a19-4038-8987-fcadf0f420b2" containerName="revoke-keys"
Oct 30 19:41:46.230289 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:41:46.230269    4892 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-a3f9b7-openshift-default-7d6d8b8fb5-m5zsm"
Oct 30 19:41:46.362907 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:41:46.362872    4892 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/02bc8b7f-a2b1-4994-85cf-436cfc0d221c-istio-podinfo\") pod \"e2e-worker-w2-a3f9b7-openshift-default-7d6d8b8fb5-m5zsm\" (UID: \"02bc8b7f-a2b1-4994-85cf-436cfc0d221c\") " pod="openshift-ingress/e2e-worker-w2-a3f9b7-openshift-default-7d6d8b8fb5-m5zsm"
Oct 30 19:41:46.363101 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:41:46.362922    4892 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/02bc8b7f-a2b1-4994-85cf-436cfc0d221c-istiod-ca-cert\") pod \"e2e-worker-w2-a3f9b7-openshift-default-7d6d8b8fb5-m5zsm\" (UID: \"02bc8b7f-a2b1-4994-85cf-436cfc0d221c\") " pod="openshift-ingress/e2e-worker-w2-a3f9b7-openshift-default-7d6d8b8fb5-m5zsm"
Oct 30 19:41:46.363101 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:41:46.363055    4892 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/02bc8b7f-a2b1-4994-85cf-436cfc0d221c-workload-certs\") pod \"e2e-worker-w2-a3f9b7-openshift-default-7d6d8b8fb5-m5zsm\" (UID: \"02bc8b7f-a2b1-4994-85cf-436cfc0d221c\") " pod="openshift-ingress/e2e-worker-w2-a3f9b7-openshift-default-7d6d8b8fb5-m5zsm"
Oct 30 19:41:46.363101 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:41:46.363083    4892 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/02bc8b7f-a2b1-4994-85cf-436cfc0d221c-istio-data\") pod \"e2e-worker-w2-a3f9b7-openshift-default-7d6d8b8fb5-m5zsm\" (UID: \"02bc8b7f-a2b1-4994-85cf-436cfc0d221c\") " pod="openshift-ingress/e2e-worker-w2-a3f9b7-openshift-default-7d6d8b8fb5-m5zsm"
Oct 30 19:41:46.363253 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:41:46.363105    4892 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/02bc8b7f-a2b1-4994-85cf-436cfc0d221c-credential-socket\") pod \"e2e-worker-w2-a3f9b7-openshift-default-7d6d8b8fb5-m5zsm\" (UID: \"02bc8b7f-a2b1-4994-85cf-436cfc0d221c\") " pod="openshift-ingress/e2e-worker-w2-a3f9b7-openshift-default-7d6d8b8fb5-m5zsm"
Oct 30 19:41:46.363253 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:41:46.363144    4892 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/02bc8b7f-a2b1-4994-85cf-436cfc0d221c-istio-token\") pod \"e2e-worker-w2-a3f9b7-openshift-default-7d6d8b8fb5-m5zsm\" (UID: \"02bc8b7f-a2b1-4994-85cf-436cfc0d221c\") " pod="openshift-ingress/e2e-worker-w2-a3f9b7-openshift-default-7d6d8b8fb5-m5zsm"
Oct 30 19:41:46.363253 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:41:46.363189    4892 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/02bc8b7f-a2b1-4994-85cf-436cfc0d221c-workload-socket\") pod \"e2e-worker-w2-a3f9b7-openshift-default-7d6d8b8fb5-m5zsm\" (UID: \"02bc8b7f-a2b1-4994-85cf-436cfc0d221c\") " pod="openshift-ingress/e2e-worker-w2-a3f9b7-openshift-default-7d6d8b8fb5-m5zsm"
Oct 30 19:41:46.363253 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:41:46.363219    4892 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/02bc8b7f-a2b1-4994-85cf-436cfc0d221c-istio-envoy\") pod \"e2e-worker-w2-a3f9b7-openshift-default-7d6d8b8fb5-m5zsm\" (UID: \"02bc8b7f-a2b1-4994-85cf-436cfc0d221c\") " pod="openshift-ingress/e2e-worker-w2-a3f9b7-openshift-default-7d6d8b8fb5-m5zsm"
Oct 30 19:41:46.363375 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:41:46.363277    4892 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wzxs6\" (UniqueName: \"kubernetes.io/projected/02bc8b7f-a2b1-4994-85cf-436cfc0d221c-kube-api-access-wzxs6\") pod \"e2e-worker-w2-a3f9b7-openshift-default-7d6d8b8fb5-m5zsm\" (UID: \"02bc8b7f-a2b1-4994-85cf-436cfc0d221c\") " pod="openshift-ingress/e2e-worker-w2-a3f9b7-openshift-default-7d6d8b8fb5-m5zsm"
Oct 30 19:41:46.487038 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:41:46.464590    4892 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/02bc8b7f-a2b1-4994-85cf-436cfc0d221c-istio-podinfo\") pod \"e2e-worker-w2-a3f9b7-openshift-default-7d6d8b8fb5-m5zsm\" (UID: \"02bc8b7f-a2b1-4994-85cf-436cfc0d221c\") " pod="openshift-ingress/e2e-worker-w2-a3f9b7-openshift-default-7d6d8b8fb5-m5zsm"
Oct 30 19:41:46.487038 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:41:46.464640    4892 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/02bc8b7f-a2b1-4994-85cf-436cfc0d221c-istiod-ca-cert\") pod \"e2e-worker-w2-a3f9b7-openshift-default-7d6d8b8fb5-m5zsm\" (UID: \"02bc8b7f-a2b1-4994-85cf-436cfc0d221c\") " pod="openshift-ingress/e2e-worker-w2-a3f9b7-openshift-default-7d6d8b8fb5-m5zsm"
Oct 30 19:41:46.487038 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:41:46.464695    4892 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/02bc8b7f-a2b1-4994-85cf-436cfc0d221c-workload-certs\") pod \"e2e-worker-w2-a3f9b7-openshift-default-7d6d8b8fb5-m5zsm\" (UID: \"02bc8b7f-a2b1-4994-85cf-436cfc0d221c\") " pod="openshift-ingress/e2e-worker-w2-a3f9b7-openshift-default-7d6d8b8fb5-m5zsm"
Oct 30 19:41:46.487038 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:41:46.464723    4892 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/02bc8b7f-a2b1-4994-85cf-436cfc0d221c-istio-data\") pod \"e2e-worker-w2-a3f9b7-openshift-default-7d6d8b8fb5-m5zsm\" (UID: \"02bc8b7f-a2b1-4994-85cf-436cfc0d221c\") " pod="openshift-ingress/e2e-worker-w2-a3f9b7-openshift-default-7d6d8b8fb5-m5zsm"
Oct 30 19:41:46.487038 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:41:46.464754    4892 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/02bc8b7f-a2b1-4994-85cf-436cfc0d221c-credential-socket\") pod \"e2e-worker-w2-a3f9b7-openshift-default-7d6d8b8fb5-m5zsm\" (UID: \"02bc8b7f-a2b1-4994-85cf-436cfc0d221c\") " pod="openshift-ingress/e2e-worker-w2-a3f9b7-openshift-default-7d6d8b8fb5-m5zsm"
Oct 30 19:41:46.487038 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:41:46.464782    4892 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/02bc8b7f-a2b1-4994-85cf-436cfc0d221c-istio-token\") pod \"e2e-worker-w2-a3f9b7-openshift-default-7d6d8b8fb5-m5zsm\" (UID: \"02bc8b7f-a2b1-4994-85cf-436cfc0d221c\") " pod="openshift-ingress/e2e-worker-w2-a3f9b7-openshift-default-7d6d8b8fb5-m5zsm"
Oct 30 19:41:46.487038 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:41:46.464827    4892 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/02bc8b7f-a2b1-4994-85cf-436cfc0d221c-workload-socket\") pod \"e2e-worker-w2-a3f9b7-openshift-default-7d6d8b8fb5-m5zsm\" (UID: \"02bc8b7f-a2b1-4994-85cf-436cfc0d221c\") " pod="openshift-ingress/e2e-worker-w2-a3f9b7-openshift-default-7d6d8b8fb5-m5zsm"
Oct 30 19:41:46.487038 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:41:46.464864    4892 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/02bc8b7f-a2b1-4994-85cf-436cfc0d221c-istio-envoy\") pod \"e2e-worker-w2-a3f9b7-openshift-default-7d6d8b8fb5-m5zsm\" (UID: \"02bc8b7f-a2b1-4994-85cf-436cfc0d221c\") " pod="openshift-ingress/e2e-worker-w2-a3f9b7-openshift-default-7d6d8b8fb5-m5zsm"
Oct 30 19:41:46.487038 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:41:46.464932    4892 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wzxs6\" (UniqueName: \"kubernetes.io/projected/02bc8b7f-a2b1-4994-85cf-436cfc0d221c-kube-api-access-wzxs6\") pod \"e2e-worker-w2-a3f9b7-openshift-default-7d6d8b8fb5-m5zsm\" (UID: \"02bc8b7f-a2b1-4994-85cf-436cfc0d221c\") " pod="openshift-ingress/e2e-worker-w2-a3f9b7-openshift-default-7d6d8b8fb5-m5zsm"
Oct 30 19:41:46.487038 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:41:46.474633    4892 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/02bc8b7f-a2b1-4994-85cf-436cfc0d221c-istiod-ca-cert\") pod \"e2e-worker-w2-a3f9b7-openshift-default-7d6d8b8fb5-m5zsm\" (UID: \"02bc8b7f-a2b1-4994-85cf-436cfc0d221c\") " pod="openshift-ingress/e2e-worker-w2-a3f9b7-openshift-default-7d6d8b8fb5-m5zsm"
Oct 30 19:41:46.487038 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:41:46.475550    4892 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/02bc8b7f-a2b1-4994-85cf-436cfc0d221c-workload-socket\") pod \"e2e-worker-w2-a3f9b7-openshift-default-7d6d8b8fb5-m5zsm\" (UID: \"02bc8b7f-a2b1-4994-85cf-436cfc0d221c\") " pod="openshift-ingress/e2e-worker-w2-a3f9b7-openshift-default-7d6d8b8fb5-m5zsm"
Oct 30 19:41:46.487038 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:41:46.475796    4892 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/02bc8b7f-a2b1-4994-85cf-436cfc0d221c-istio-data\") pod \"e2e-worker-w2-a3f9b7-openshift-default-7d6d8b8fb5-m5zsm\" (UID: \"02bc8b7f-a2b1-4994-85cf-436cfc0d221c\") " pod="openshift-ingress/e2e-worker-w2-a3f9b7-openshift-default-7d6d8b8fb5-m5zsm"
Oct 30 19:41:46.487038 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:41:46.476001    4892 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/02bc8b7f-a2b1-4994-85cf-436cfc0d221c-workload-certs\") pod \"e2e-worker-w2-a3f9b7-openshift-default-7d6d8b8fb5-m5zsm\" (UID: \"02bc8b7f-a2b1-4994-85cf-436cfc0d221c\") " pod="openshift-ingress/e2e-worker-w2-a3f9b7-openshift-default-7d6d8b8fb5-m5zsm"
Oct 30 19:41:46.487038 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:41:46.476173    4892 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/02bc8b7f-a2b1-4994-85cf-436cfc0d221c-credential-socket\") pod \"e2e-worker-w2-a3f9b7-openshift-default-7d6d8b8fb5-m5zsm\" (UID: \"02bc8b7f-a2b1-4994-85cf-436cfc0d221c\") " pod="openshift-ingress/e2e-worker-w2-a3f9b7-openshift-default-7d6d8b8fb5-m5zsm"
Oct 30 19:41:46.490577 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:41:46.490545    4892 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/02bc8b7f-a2b1-4994-85cf-436cfc0d221c-istio-podinfo\") pod \"e2e-worker-w2-a3f9b7-openshift-default-7d6d8b8fb5-m5zsm\" (UID: \"02bc8b7f-a2b1-4994-85cf-436cfc0d221c\") " pod="openshift-ingress/e2e-worker-w2-a3f9b7-openshift-default-7d6d8b8fb5-m5zsm"
Oct 30 19:41:46.493262 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:41:46.493236    4892 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/02bc8b7f-a2b1-4994-85cf-436cfc0d221c-istio-envoy\") pod \"e2e-worker-w2-a3f9b7-openshift-default-7d6d8b8fb5-m5zsm\" (UID: \"02bc8b7f-a2b1-4994-85cf-436cfc0d221c\") " pod="openshift-ingress/e2e-worker-w2-a3f9b7-openshift-default-7d6d8b8fb5-m5zsm"
Oct 30 19:41:46.495586 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:41:46.495537    4892 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-a3f9b7-openshift-default-7d6d8b8fb5-m5zsm"]
Oct 30 19:41:46.529848 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:41:46.529814    4892 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wzxs6\" (UniqueName: \"kubernetes.io/projected/02bc8b7f-a2b1-4994-85cf-436cfc0d221c-kube-api-access-wzxs6\") pod \"e2e-worker-w2-a3f9b7-openshift-default-7d6d8b8fb5-m5zsm\" (UID: \"02bc8b7f-a2b1-4994-85cf-436cfc0d221c\") " pod="openshift-ingress/e2e-worker-w2-a3f9b7-openshift-default-7d6d8b8fb5-m5zsm"
Oct 30 19:41:46.535604 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:41:46.535582    4892 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/02bc8b7f-a2b1-4994-85cf-436cfc0d221c-istio-token\") pod \"e2e-worker-w2-a3f9b7-openshift-default-7d6d8b8fb5-m5zsm\" (UID: \"02bc8b7f-a2b1-4994-85cf-436cfc0d221c\") " pod="openshift-ingress/e2e-worker-w2-a3f9b7-openshift-default-7d6d8b8fb5-m5zsm"
Oct 30 19:41:46.543587 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:41:46.543562    4892 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-a3f9b7-openshift-default-7d6d8b8fb5-m5zsm"
Oct 30 19:41:46.976237 ip-10-0-132-82 kubenswrapper[4892]: W1030 19:41:46.976205    4892 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod02bc8b7f_a2b1_4994_85cf_436cfc0d221c.slice/crio-f21c2d24bf38805f245ee8c8db8cf43f9f7ab0437a725940104fb9157cd96e35 WatchSource:0}: Error finding container f21c2d24bf38805f245ee8c8db8cf43f9f7ab0437a725940104fb9157cd96e35: Status 404 returned error can't find the container with id f21c2d24bf38805f245ee8c8db8cf43f9f7ab0437a725940104fb9157cd96e35
Oct 30 19:41:46.978554 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:41:46.978513    4892 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892000Ki","pods":"250"}
Oct 30 19:41:46.978658 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:41:46.978600    4892 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892000Ki","pods":"250"}
Oct 30 19:41:46.978658 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:41:46.978627    4892 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892000Ki","pods":"250"}
Oct 30 19:41:46.979044 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:41:46.979019    4892 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-a3f9b7-openshift-default-7d6d8b8fb5-m5zsm"]
Oct 30 19:41:47.771551 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:41:47.771516    4892 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-a3f9b7-openshift-default-7d6d8b8fb5-m5zsm" event={"ID":"02bc8b7f-a2b1-4994-85cf-436cfc0d221c","Type":"ContainerStarted","Data":"40cfcef57396e51e49572d8e71e1b1f64c7664b57dd6596ec8842905d10b3ae0"}
Oct 30 19:41:47.771980 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:41:47.771596    4892 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-a3f9b7-openshift-default-7d6d8b8fb5-m5zsm" event={"ID":"02bc8b7f-a2b1-4994-85cf-436cfc0d221c","Type":"ContainerStarted","Data":"f21c2d24bf38805f245ee8c8db8cf43f9f7ab0437a725940104fb9157cd96e35"}
Oct 30 19:41:48.002347 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:41:48.002294    4892 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w2-a3f9b7-openshift-default-7d6d8b8fb5-m5zsm" podStartSLOduration=2.002277318 podStartE2EDuration="2.002277318s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:41:46 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:41:47.950498155 +0000 UTC m=+1555.520963791" watchObservedRunningTime="2026-10-30 19:41:48.002277318 +0000 UTC m=+1555.572742921"
Oct 30 19:41:48.546989 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:41:48.546268    4892 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w2-a3f9b7-openshift-default-7d6d8b8fb5-m5zsm"
Oct 30 19:41:48.546989 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:41:48.546384    4892 patch_prober.go:29] interesting pod/e2e-worker-w2-a3f9b7-openshift-default-7d6d8b8fb5-m5zsm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.106:15021/healthz/ready\": dial tcp 10.132.0.106:15021: connect: connection refused" start-of-body=
Oct 30 19:41:48.546989 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:41:48.546427    4892 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-a3f9b7-openshift-default-7d6d8b8fb5-m5zsm" podUID="02bc8b7f-a2b1-4994-85cf-436cfc0d221c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.106:15021/healthz/ready\": dial tcp 10.132.0.106:15021: connect: connection refused"
Oct 30 19:41:49.547293 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:41:49.547257    4892 patch_prober.go:29] interesting pod/e2e-worker-w2-a3f9b7-openshift-default-7d6d8b8fb5-m5zsm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.106:15021/healthz/ready\": dial tcp 10.132.0.106:15021: connect: connection refused" start-of-body=
Oct 30 19:41:49.547899 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:41:49.547870    4892 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-a3f9b7-openshift-default-7d6d8b8fb5-m5zsm" podUID="02bc8b7f-a2b1-4994-85cf-436cfc0d221c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.106:15021/healthz/ready\": dial tcp 10.132.0.106:15021: connect: connection refused"
Oct 30 19:41:50.055465 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:41:50.055431    4892 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-26f039ec-7kmdh"]
Oct 30 19:41:50.101485 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:41:50.101452    4892 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-26f039ec-7kmdh"]
Oct 30 19:41:50.545992 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:41:50.545259    4892 patch_prober.go:29] interesting pod/e2e-worker-w2-a3f9b7-openshift-default-7d6d8b8fb5-m5zsm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.106:15021/healthz/ready\": dial tcp 10.132.0.106:15021: connect: connection refused" start-of-body=
Oct 30 19:41:50.545992 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:41:50.545318    4892 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-a3f9b7-openshift-default-7d6d8b8fb5-m5zsm" podUID="02bc8b7f-a2b1-4994-85cf-436cfc0d221c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.106:15021/healthz/ready\": dial tcp 10.132.0.106:15021: connect: connection refused"
Oct 30 19:41:50.569710 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:41:50.569676    4892 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="79ebfc06-f72b-48af-849d-d81baf3f9765" path="/var/lib/kubelet/pods/79ebfc06-f72b-48af-849d-d81baf3f9765/volumes"
Oct 30 19:41:50.803867 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:41:50.803775    4892 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b6c91eb9-rp7vq"]
Oct 30 19:41:50.809057 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:41:50.809034    4892 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b6c91eb9-rp7vq"
Oct 30 19:41:50.811309 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:41:50.811282    4892 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fmpx6\" (UniqueName: \"kubernetes.io/projected/01f7d0ee-1cb4-4e90-8f2c-846b45c3ed68-kube-api-access-fmpx6\") pod \"maas-api-revoke-sub-b6c91eb9-rp7vq\" (UID: \"01f7d0ee-1cb4-4e90-8f2c-846b45c3ed68\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b6c91eb9-rp7vq"
Oct 30 19:41:50.811396 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:41:50.811329    4892 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/01f7d0ee-1cb4-4e90-8f2c-846b45c3ed68-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-b6c91eb9-rp7vq\" (UID: \"01f7d0ee-1cb4-4e90-8f2c-846b45c3ed68\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b6c91eb9-rp7vq"
Oct 30 19:41:50.811469 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:41:50.811447    4892 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/01f7d0ee-1cb4-4e90-8f2c-846b45c3ed68-maas-api-xks-ca\") pod \"maas-api-revoke-sub-b6c91eb9-rp7vq\" (UID: \"01f7d0ee-1cb4-4e90-8f2c-846b45c3ed68\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b6c91eb9-rp7vq"
Oct 30 19:41:50.865419 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:41:50.865391    4892 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b6c91eb9-rp7vq"]
Oct 30 19:41:50.909622 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:41:50.909572    4892 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 19:41:50.909622 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:41:50.909610    4892 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 30 19:41:50.909622 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:41:50.909593    4892 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 30 19:41:50.910355 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:41:50.910338    4892 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-nlxp4\""
Oct 30 19:41:50.912197 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:41:50.912179    4892 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/01f7d0ee-1cb4-4e90-8f2c-846b45c3ed68-maas-api-xks-ca\") pod \"maas-api-revoke-sub-b6c91eb9-rp7vq\" (UID: \"01f7d0ee-1cb4-4e90-8f2c-846b45c3ed68\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b6c91eb9-rp7vq"
Oct 30 19:41:50.912284 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:41:50.912272    4892 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fmpx6\" (UniqueName: \"kubernetes.io/projected/01f7d0ee-1cb4-4e90-8f2c-846b45c3ed68-kube-api-access-fmpx6\") pod \"maas-api-revoke-sub-b6c91eb9-rp7vq\" (UID: \"01f7d0ee-1cb4-4e90-8f2c-846b45c3ed68\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b6c91eb9-rp7vq"
Oct 30 19:41:50.912339 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:41:50.912299    4892 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/01f7d0ee-1cb4-4e90-8f2c-846b45c3ed68-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-b6c91eb9-rp7vq\" (UID: \"01f7d0ee-1cb4-4e90-8f2c-846b45c3ed68\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b6c91eb9-rp7vq"
Oct 30 19:41:50.913027 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:41:50.912935    4892 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/01f7d0ee-1cb4-4e90-8f2c-846b45c3ed68-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-b6c91eb9-rp7vq\" (UID: \"01f7d0ee-1cb4-4e90-8f2c-846b45c3ed68\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b6c91eb9-rp7vq"
Oct 30 19:41:50.914692 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:41:50.914667    4892 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/01f7d0ee-1cb4-4e90-8f2c-846b45c3ed68-maas-api-xks-ca\") pod \"maas-api-revoke-sub-b6c91eb9-rp7vq\" (UID: \"01f7d0ee-1cb4-4e90-8f2c-846b45c3ed68\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b6c91eb9-rp7vq"
Oct 30 19:41:50.968351 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:41:50.968321    4892 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fmpx6\" (UniqueName: \"kubernetes.io/projected/01f7d0ee-1cb4-4e90-8f2c-846b45c3ed68-kube-api-access-fmpx6\") pod \"maas-api-revoke-sub-b6c91eb9-rp7vq\" (UID: \"01f7d0ee-1cb4-4e90-8f2c-846b45c3ed68\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b6c91eb9-rp7vq"
Oct 30 19:41:51.124173 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:41:51.124085    4892 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b6c91eb9-rp7vq"
Oct 30 19:41:51.523661 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:41:51.523634    4892 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b6c91eb9-rp7vq"]
Oct 30 19:41:51.524051 ip-10-0-132-82 kubenswrapper[4892]: W1030 19:41:51.524021    4892 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod01f7d0ee_1cb4_4e90_8f2c_846b45c3ed68.slice/crio-730635b6f6c6acb3440d6f34bf8613c11758384ed3e41e3b4b7ed178e511128d WatchSource:0}: Error finding container 730635b6f6c6acb3440d6f34bf8613c11758384ed3e41e3b4b7ed178e511128d: Status 404 returned error can't find the container with id 730635b6f6c6acb3440d6f34bf8613c11758384ed3e41e3b4b7ed178e511128d
Oct 30 19:41:51.555284 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:41:51.554079    4892 patch_prober.go:29] interesting pod/e2e-worker-w2-a3f9b7-openshift-default-7d6d8b8fb5-m5zsm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.106:15021/healthz/ready\": dial tcp 10.132.0.106:15021: connect: connection refused" start-of-body=
Oct 30 19:41:51.555284 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:41:51.554156    4892 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-a3f9b7-openshift-default-7d6d8b8fb5-m5zsm" podUID="02bc8b7f-a2b1-4994-85cf-436cfc0d221c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.106:15021/healthz/ready\": dial tcp 10.132.0.106:15021: connect: connection refused"
Oct 30 19:41:51.804331 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:41:51.804249    4892 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b6c91eb9-rp7vq" event={"ID":"01f7d0ee-1cb4-4e90-8f2c-846b45c3ed68","Type":"ContainerStarted","Data":"730635b6f6c6acb3440d6f34bf8613c11758384ed3e41e3b4b7ed178e511128d"}
Oct 30 19:41:52.547387 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:41:52.547352    4892 patch_prober.go:29] interesting pod/e2e-worker-w2-a3f9b7-openshift-default-7d6d8b8fb5-m5zsm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.106:15021/healthz/ready\": dial tcp 10.132.0.106:15021: connect: connection refused" start-of-body=
Oct 30 19:41:52.547637 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:41:52.547610    4892 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-a3f9b7-openshift-default-7d6d8b8fb5-m5zsm" podUID="02bc8b7f-a2b1-4994-85cf-436cfc0d221c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.106:15021/healthz/ready\": dial tcp 10.132.0.106:15021: connect: connection refused"
Oct 30 19:41:52.777656 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:41:52.777626    4892 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-75b6bc4a-ffcq9"]
Oct 30 19:41:52.781584 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:41:52.781569    4892 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-75b6bc4a-ffcq9"
Oct 30 19:41:52.814149 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:41:52.814088    4892 generic.go:356] "Generic (PLEG): container finished" podID="01f7d0ee-1cb4-4e90-8f2c-846b45c3ed68" containerID="9e7c61fd70be68775d576d62909ff5f820377f78eec5bd232c62e01619567652" exitCode=0
Oct 30 19:41:52.814149 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:41:52.814124    4892 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b6c91eb9-rp7vq" event={"ID":"01f7d0ee-1cb4-4e90-8f2c-846b45c3ed68","Type":"ContainerDied","Data":"9e7c61fd70be68775d576d62909ff5f820377f78eec5bd232c62e01619567652"}
Oct 30 19:41:52.865013 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:41:52.863055    4892 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-75b6bc4a-ffcq9"]
Oct 30 19:41:52.865013 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:41:52.863343    4892 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4lk8t\" (UniqueName: \"kubernetes.io/projected/91c01247-b2cd-497d-835b-fdd57fdf08d0-kube-api-access-4lk8t\") pod \"maas-api-revoke-sub-75b6bc4a-ffcq9\" (UID: \"91c01247-b2cd-497d-835b-fdd57fdf08d0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-75b6bc4a-ffcq9"
Oct 30 19:41:52.865013 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:41:52.863556    4892 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/91c01247-b2cd-497d-835b-fdd57fdf08d0-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-75b6bc4a-ffcq9\" (UID: \"91c01247-b2cd-497d-835b-fdd57fdf08d0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-75b6bc4a-ffcq9"
Oct 30 19:41:52.865013 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:41:52.863659    4892 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/91c01247-b2cd-497d-835b-fdd57fdf08d0-maas-api-xks-ca\") pod \"maas-api-revoke-sub-75b6bc4a-ffcq9\" (UID: \"91c01247-b2cd-497d-835b-fdd57fdf08d0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-75b6bc4a-ffcq9"
Oct 30 19:41:52.956447 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:41:52.956408    4892 scope.go:122] "RemoveContainer" containerID="95b632ddf7475722c4c29c718c4e858de1f38ddb6f4db069809f0cc09b3de4f7"
Oct 30 19:41:52.965025 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:41:52.964993    4892 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/91c01247-b2cd-497d-835b-fdd57fdf08d0-maas-api-xks-ca\") pod \"maas-api-revoke-sub-75b6bc4a-ffcq9\" (UID: \"91c01247-b2cd-497d-835b-fdd57fdf08d0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-75b6bc4a-ffcq9"
Oct 30 19:41:52.965179 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:41:52.965159    4892 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4lk8t\" (UniqueName: \"kubernetes.io/projected/91c01247-b2cd-497d-835b-fdd57fdf08d0-kube-api-access-4lk8t\") pod \"maas-api-revoke-sub-75b6bc4a-ffcq9\" (UID: \"91c01247-b2cd-497d-835b-fdd57fdf08d0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-75b6bc4a-ffcq9"
Oct 30 19:41:52.965292 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:41:52.965276    4892 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/91c01247-b2cd-497d-835b-fdd57fdf08d0-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-75b6bc4a-ffcq9\" (UID: \"91c01247-b2cd-497d-835b-fdd57fdf08d0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-75b6bc4a-ffcq9"
Oct 30 19:41:52.966225 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:41:52.966198    4892 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/91c01247-b2cd-497d-835b-fdd57fdf08d0-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-75b6bc4a-ffcq9\" (UID: \"91c01247-b2cd-497d-835b-fdd57fdf08d0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-75b6bc4a-ffcq9"
Oct 30 19:41:52.967558 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:41:52.967540    4892 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/91c01247-b2cd-497d-835b-fdd57fdf08d0-maas-api-xks-ca\") pod \"maas-api-revoke-sub-75b6bc4a-ffcq9\" (UID: \"91c01247-b2cd-497d-835b-fdd57fdf08d0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-75b6bc4a-ffcq9"
Oct 30 19:41:52.969397 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:41:52.969378    4892 scope.go:122] "RemoveContainer" containerID="2ddbe404d830c1724af73dd8448c6d113b26b80efe430f8d14c4787890731bd8"
Oct 30 19:41:52.991588 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:41:52.991567    4892 scope.go:122] "RemoveContainer" containerID="17ad1117604568a4d162450050b83f2cc2b23174d347e12d57d9963279e1bbb6"
Oct 30 19:41:53.022824 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:41:53.022799    4892 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4lk8t\" (UniqueName: \"kubernetes.io/projected/91c01247-b2cd-497d-835b-fdd57fdf08d0-kube-api-access-4lk8t\") pod \"maas-api-revoke-sub-75b6bc4a-ffcq9\" (UID: \"91c01247-b2cd-497d-835b-fdd57fdf08d0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-75b6bc4a-ffcq9"
Oct 30 19:41:53.095724 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:41:53.095647    4892 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-75b6bc4a-ffcq9"
Oct 30 19:41:53.491049 ip-10-0-132-82 kubenswrapper[4892]: W1030 19:41:53.491015    4892 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod91c01247_b2cd_497d_835b_fdd57fdf08d0.slice/crio-03e760bfb53986c04f44ceacc9988c77a703234fea78f990d995ba3520110a25 WatchSource:0}: Error finding container 03e760bfb53986c04f44ceacc9988c77a703234fea78f990d995ba3520110a25: Status 404 returned error can't find the container with id 03e760bfb53986c04f44ceacc9988c77a703234fea78f990d995ba3520110a25
Oct 30 19:41:53.492514 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:41:53.492472    4892 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-75b6bc4a-ffcq9"]
Oct 30 19:41:53.544699 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:41:53.544672    4892 patch_prober.go:29] interesting pod/e2e-worker-w2-a3f9b7-openshift-default-7d6d8b8fb5-m5zsm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.106:15021/healthz/ready\": dial tcp 10.132.0.106:15021: connect: connection refused" start-of-body=
Oct 30 19:41:53.544806 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:41:53.544727    4892 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-a3f9b7-openshift-default-7d6d8b8fb5-m5zsm" podUID="02bc8b7f-a2b1-4994-85cf-436cfc0d221c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.106:15021/healthz/ready\": dial tcp 10.132.0.106:15021: connect: connection refused"
Oct 30 19:41:53.824247 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:41:53.824207    4892 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-75b6bc4a-ffcq9" event={"ID":"91c01247-b2cd-497d-835b-fdd57fdf08d0","Type":"ContainerStarted","Data":"03e760bfb53986c04f44ceacc9988c77a703234fea78f990d995ba3520110a25"}
Oct 30 19:41:53.963779 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:41:53.963750    4892 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b6c91eb9-rp7vq"
Oct 30 19:41:53.976932 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:41:53.976896    4892 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/01f7d0ee-1cb4-4e90-8f2c-846b45c3ed68-kube-api-access-fmpx6\" (UniqueName: \"kubernetes.io/projected/01f7d0ee-1cb4-4e90-8f2c-846b45c3ed68-kube-api-access-fmpx6\") pod \"01f7d0ee-1cb4-4e90-8f2c-846b45c3ed68\" (UID: \"01f7d0ee-1cb4-4e90-8f2c-846b45c3ed68\") "
Oct 30 19:41:53.977067 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:41:53.976988    4892 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/01f7d0ee-1cb4-4e90-8f2c-846b45c3ed68-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/01f7d0ee-1cb4-4e90-8f2c-846b45c3ed68-maas-api-openshift-ca\") pod \"01f7d0ee-1cb4-4e90-8f2c-846b45c3ed68\" (UID: \"01f7d0ee-1cb4-4e90-8f2c-846b45c3ed68\") "
Oct 30 19:41:53.977146 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:41:53.977071    4892 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/01f7d0ee-1cb4-4e90-8f2c-846b45c3ed68-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/01f7d0ee-1cb4-4e90-8f2c-846b45c3ed68-maas-api-xks-ca\") pod \"01f7d0ee-1cb4-4e90-8f2c-846b45c3ed68\" (UID: \"01f7d0ee-1cb4-4e90-8f2c-846b45c3ed68\") "
Oct 30 19:41:53.978383 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:41:53.978345    4892 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/01f7d0ee-1cb4-4e90-8f2c-846b45c3ed68-maas-api-openshift-ca" pod "01f7d0ee-1cb4-4e90-8f2c-846b45c3ed68" (UID: "01f7d0ee-1cb4-4e90-8f2c-846b45c3ed68"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 19:41:53.979434 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:41:53.979356    4892 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/01f7d0ee-1cb4-4e90-8f2c-846b45c3ed68-maas-api-xks-ca" pod "01f7d0ee-1cb4-4e90-8f2c-846b45c3ed68" (UID: "01f7d0ee-1cb4-4e90-8f2c-846b45c3ed68"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 19:41:53.979949 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:41:53.979927    4892 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/01f7d0ee-1cb4-4e90-8f2c-846b45c3ed68-kube-api-access-fmpx6" pod "01f7d0ee-1cb4-4e90-8f2c-846b45c3ed68" (UID: "01f7d0ee-1cb4-4e90-8f2c-846b45c3ed68"). InnerVolumeSpecName "kube-api-access-fmpx6". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:41:54.079464 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:41:54.078812    4892 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fmpx6\" (UniqueName: \"kubernetes.io/projected/01f7d0ee-1cb4-4e90-8f2c-846b45c3ed68-kube-api-access-fmpx6\") on node \"ip-10-0-132-82.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 19:41:54.079464 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:41:54.078853    4892 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/01f7d0ee-1cb4-4e90-8f2c-846b45c3ed68-maas-api-openshift-ca\") on node \"ip-10-0-132-82.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 19:41:54.079464 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:41:54.078870    4892 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/01f7d0ee-1cb4-4e90-8f2c-846b45c3ed68-maas-api-xks-ca\") on node \"ip-10-0-132-82.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 19:41:54.548419 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:41:54.548394    4892 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w2-a3f9b7-openshift-default-7d6d8b8fb5-m5zsm"
Oct 30 19:41:54.548741 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:41:54.548717    4892 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w2-a3f9b7-openshift-default-7d6d8b8fb5-m5zsm"
Oct 30 19:41:54.549372 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:41:54.549353    4892 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w2-a3f9b7-openshift-default-7d6d8b8fb5-m5zsm"
Oct 30 19:41:54.850692 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:41:54.849695    4892 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b6c91eb9-rp7vq"
Oct 30 19:41:54.850692 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:41:54.850408    4892 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b6c91eb9-rp7vq" event={"ID":"01f7d0ee-1cb4-4e90-8f2c-846b45c3ed68","Type":"ContainerDied","Data":"730635b6f6c6acb3440d6f34bf8613c11758384ed3e41e3b4b7ed178e511128d"}
Oct 30 19:41:54.850692 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:41:54.850455    4892 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="730635b6f6c6acb3440d6f34bf8613c11758384ed3e41e3b4b7ed178e511128d"
Oct 30 19:41:54.854354 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:41:54.854211    4892 generic.go:356] "Generic (PLEG): container finished" podID="91c01247-b2cd-497d-835b-fdd57fdf08d0" containerID="4a9bd6c20c8e83e9f8336d1780cb332f7c0ec994b21816fc3647492dedba1fe4" exitCode=0
Oct 30 19:41:54.855426 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:41:54.855398    4892 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-75b6bc4a-ffcq9" event={"ID":"91c01247-b2cd-497d-835b-fdd57fdf08d0","Type":"ContainerDied","Data":"4a9bd6c20c8e83e9f8336d1780cb332f7c0ec994b21816fc3647492dedba1fe4"}
Oct 30 19:41:54.889552 ip-10-0-132-82 kubenswrapper[4892]: E1030 19:41:54.889512    4892 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-jrm87" containerName="extractor"
Oct 30 19:41:55.055386 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:41:55.055351    4892 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-1ad201c0-gzpm5"]
Oct 30 19:41:55.061703 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:41:55.061673    4892 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-1ad201c0-gzpm5"]
Oct 30 19:41:55.931805 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:41:55.931768    4892 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e52034d3-rjjj5"]
Oct 30 19:41:55.932548 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:41:55.932524    4892 cpu_manager.go:414] "containerMap: removing container" podUID="01f7d0ee-1cb4-4e90-8f2c-846b45c3ed68" containerName="revoke-keys"
Oct 30 19:41:55.932637 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:41:55.932549    4892 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="01f7d0ee-1cb4-4e90-8f2c-846b45c3ed68" containerName="revoke-keys"
Oct 30 19:41:55.932713 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:41:55.932699    4892 memory_manager.go:365] "RemoveStaleState removing state" podUID="01f7d0ee-1cb4-4e90-8f2c-846b45c3ed68" containerName="revoke-keys"
Oct 30 19:41:55.936267 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:41:55.936247    4892 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e52034d3-rjjj5"
Oct 30 19:41:55.995054 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:41:55.995024    4892 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e52034d3-rjjj5"]
Oct 30 19:41:55.998321 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:41:55.998285    4892 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8a1b4524-f816-4eab-b88d-599a7689f138-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-e52034d3-rjjj5\" (UID: \"8a1b4524-f816-4eab-b88d-599a7689f138\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e52034d3-rjjj5"
Oct 30 19:41:55.998585 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:41:55.998563    4892 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8a1b4524-f816-4eab-b88d-599a7689f138-maas-api-xks-ca\") pod \"maas-api-revoke-sub-e52034d3-rjjj5\" (UID: \"8a1b4524-f816-4eab-b88d-599a7689f138\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e52034d3-rjjj5"
Oct 30 19:41:55.998739 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:41:55.998722    4892 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vpt4t\" (UniqueName: \"kubernetes.io/projected/8a1b4524-f816-4eab-b88d-599a7689f138-kube-api-access-vpt4t\") pod \"maas-api-revoke-sub-e52034d3-rjjj5\" (UID: \"8a1b4524-f816-4eab-b88d-599a7689f138\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e52034d3-rjjj5"
Oct 30 19:41:56.073369 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:41:56.073345    4892 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-75b6bc4a-ffcq9"
Oct 30 19:41:56.099468 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:41:56.099443    4892 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/91c01247-b2cd-497d-835b-fdd57fdf08d0-kube-api-access-4lk8t\" (UniqueName: \"kubernetes.io/projected/91c01247-b2cd-497d-835b-fdd57fdf08d0-kube-api-access-4lk8t\") pod \"91c01247-b2cd-497d-835b-fdd57fdf08d0\" (UID: \"91c01247-b2cd-497d-835b-fdd57fdf08d0\") "
Oct 30 19:41:56.099593 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:41:56.099523    4892 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/91c01247-b2cd-497d-835b-fdd57fdf08d0-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/91c01247-b2cd-497d-835b-fdd57fdf08d0-maas-api-openshift-ca\") pod \"91c01247-b2cd-497d-835b-fdd57fdf08d0\" (UID: \"91c01247-b2cd-497d-835b-fdd57fdf08d0\") "
Oct 30 19:41:56.099687 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:41:56.099673    4892 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8a1b4524-f816-4eab-b88d-599a7689f138-maas-api-xks-ca\") pod \"maas-api-revoke-sub-e52034d3-rjjj5\" (UID: \"8a1b4524-f816-4eab-b88d-599a7689f138\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e52034d3-rjjj5"
Oct 30 19:41:56.099728 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:41:56.099713    4892 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vpt4t\" (UniqueName: \"kubernetes.io/projected/8a1b4524-f816-4eab-b88d-599a7689f138-kube-api-access-vpt4t\") pod \"maas-api-revoke-sub-e52034d3-rjjj5\" (UID: \"8a1b4524-f816-4eab-b88d-599a7689f138\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e52034d3-rjjj5"
Oct 30 19:41:56.099905 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:41:56.099872    4892 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/91c01247-b2cd-497d-835b-fdd57fdf08d0-maas-api-openshift-ca" pod "91c01247-b2cd-497d-835b-fdd57fdf08d0" (UID: "91c01247-b2cd-497d-835b-fdd57fdf08d0"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 19:41:56.099905 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:41:56.099887    4892 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8a1b4524-f816-4eab-b88d-599a7689f138-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-e52034d3-rjjj5\" (UID: \"8a1b4524-f816-4eab-b88d-599a7689f138\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e52034d3-rjjj5"
Oct 30 19:41:56.100098 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:41:56.100075    4892 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/91c01247-b2cd-497d-835b-fdd57fdf08d0-maas-api-openshift-ca\") on node \"ip-10-0-132-82.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 19:41:56.100433 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:41:56.100414    4892 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8a1b4524-f816-4eab-b88d-599a7689f138-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-e52034d3-rjjj5\" (UID: \"8a1b4524-f816-4eab-b88d-599a7689f138\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e52034d3-rjjj5"
Oct 30 19:41:56.101751 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:41:56.101732    4892 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/91c01247-b2cd-497d-835b-fdd57fdf08d0-kube-api-access-4lk8t" pod "91c01247-b2cd-497d-835b-fdd57fdf08d0" (UID: "91c01247-b2cd-497d-835b-fdd57fdf08d0"). InnerVolumeSpecName "kube-api-access-4lk8t". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:41:56.102149 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:41:56.102134    4892 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8a1b4524-f816-4eab-b88d-599a7689f138-maas-api-xks-ca\") pod \"maas-api-revoke-sub-e52034d3-rjjj5\" (UID: \"8a1b4524-f816-4eab-b88d-599a7689f138\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e52034d3-rjjj5"
Oct 30 19:41:56.158676 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:41:56.158614    4892 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vpt4t\" (UniqueName: \"kubernetes.io/projected/8a1b4524-f816-4eab-b88d-599a7689f138-kube-api-access-vpt4t\") pod \"maas-api-revoke-sub-e52034d3-rjjj5\" (UID: \"8a1b4524-f816-4eab-b88d-599a7689f138\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e52034d3-rjjj5"
Oct 30 19:41:56.200766 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:41:56.200738    4892 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/91c01247-b2cd-497d-835b-fdd57fdf08d0-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/91c01247-b2cd-497d-835b-fdd57fdf08d0-maas-api-xks-ca\") pod \"91c01247-b2cd-497d-835b-fdd57fdf08d0\" (UID: \"91c01247-b2cd-497d-835b-fdd57fdf08d0\") "
Oct 30 19:41:56.201234 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:41:56.201212    4892 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4lk8t\" (UniqueName: \"kubernetes.io/projected/91c01247-b2cd-497d-835b-fdd57fdf08d0-kube-api-access-4lk8t\") on node \"ip-10-0-132-82.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 19:41:56.202810 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:41:56.202779    4892 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/91c01247-b2cd-497d-835b-fdd57fdf08d0-maas-api-xks-ca" pod "91c01247-b2cd-497d-835b-fdd57fdf08d0" (UID: "91c01247-b2cd-497d-835b-fdd57fdf08d0"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 19:41:56.251263 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:41:56.251236    4892 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e52034d3-rjjj5"
Oct 30 19:41:56.302322 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:41:56.302287    4892 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/91c01247-b2cd-497d-835b-fdd57fdf08d0-maas-api-xks-ca\") on node \"ip-10-0-132-82.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 19:41:56.572230 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:41:56.572153    4892 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f9aff12a-9664-4d53-ac43-4b0667eccbc8" path="/var/lib/kubelet/pods/f9aff12a-9664-4d53-ac43-4b0667eccbc8/volumes"
Oct 30 19:41:56.632243 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:41:56.632219    4892 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e52034d3-rjjj5"]
Oct 30 19:41:56.632803 ip-10-0-132-82 kubenswrapper[4892]: W1030 19:41:56.632781    4892 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8a1b4524_f816_4eab_b88d_599a7689f138.slice/crio-9a45529b5613233c1e79baaf559f83669dd83ad7149eb0de0e80a674ce572f2b WatchSource:0}: Error finding container 9a45529b5613233c1e79baaf559f83669dd83ad7149eb0de0e80a674ce572f2b: Status 404 returned error can't find the container with id 9a45529b5613233c1e79baaf559f83669dd83ad7149eb0de0e80a674ce572f2b
Oct 30 19:41:56.889605 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:41:56.889583    4892 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-75b6bc4a-ffcq9"
Oct 30 19:41:56.889752 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:41:56.889599    4892 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-75b6bc4a-ffcq9" event={"ID":"91c01247-b2cd-497d-835b-fdd57fdf08d0","Type":"ContainerDied","Data":"03e760bfb53986c04f44ceacc9988c77a703234fea78f990d995ba3520110a25"}
Oct 30 19:41:56.889752 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:41:56.889651    4892 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="03e760bfb53986c04f44ceacc9988c77a703234fea78f990d995ba3520110a25"
Oct 30 19:41:56.891006 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:41:56.890984    4892 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e52034d3-rjjj5" event={"ID":"8a1b4524-f816-4eab-b88d-599a7689f138","Type":"ContainerStarted","Data":"9a45529b5613233c1e79baaf559f83669dd83ad7149eb0de0e80a674ce572f2b"}
Oct 30 19:41:57.900807 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:41:57.900772    4892 generic.go:356] "Generic (PLEG): container finished" podID="8a1b4524-f816-4eab-b88d-599a7689f138" containerID="d2ab1aefae7164e84473c64f8603a5a085c6bca68bb389a79efee3798941d177" exitCode=0
Oct 30 19:41:57.901185 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:41:57.900864    4892 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e52034d3-rjjj5" event={"ID":"8a1b4524-f816-4eab-b88d-599a7689f138","Type":"ContainerDied","Data":"d2ab1aefae7164e84473c64f8603a5a085c6bca68bb389a79efee3798941d177"}
Oct 30 19:41:59.078574 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:41:59.078546    4892 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e52034d3-rjjj5"
Oct 30 19:41:59.141157 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:41:59.141120    4892 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/8a1b4524-f816-4eab-b88d-599a7689f138-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8a1b4524-f816-4eab-b88d-599a7689f138-maas-api-openshift-ca\") pod \"8a1b4524-f816-4eab-b88d-599a7689f138\" (UID: \"8a1b4524-f816-4eab-b88d-599a7689f138\") "
Oct 30 19:41:59.141348 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:41:59.141236    4892 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8a1b4524-f816-4eab-b88d-599a7689f138-kube-api-access-vpt4t\" (UniqueName: \"kubernetes.io/projected/8a1b4524-f816-4eab-b88d-599a7689f138-kube-api-access-vpt4t\") pod \"8a1b4524-f816-4eab-b88d-599a7689f138\" (UID: \"8a1b4524-f816-4eab-b88d-599a7689f138\") "
Oct 30 19:41:59.141348 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:41:59.141271    4892 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/8a1b4524-f816-4eab-b88d-599a7689f138-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8a1b4524-f816-4eab-b88d-599a7689f138-maas-api-xks-ca\") pod \"8a1b4524-f816-4eab-b88d-599a7689f138\" (UID: \"8a1b4524-f816-4eab-b88d-599a7689f138\") "
Oct 30 19:41:59.141503 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:41:59.141478    4892 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8a1b4524-f816-4eab-b88d-599a7689f138-maas-api-openshift-ca" pod "8a1b4524-f816-4eab-b88d-599a7689f138" (UID: "8a1b4524-f816-4eab-b88d-599a7689f138"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 19:41:59.141815 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:41:59.141794    4892 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8a1b4524-f816-4eab-b88d-599a7689f138-maas-api-openshift-ca\") on node \"ip-10-0-132-82.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 19:41:59.143878 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:41:59.143854    4892 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8a1b4524-f816-4eab-b88d-599a7689f138-kube-api-access-vpt4t" pod "8a1b4524-f816-4eab-b88d-599a7689f138" (UID: "8a1b4524-f816-4eab-b88d-599a7689f138"). InnerVolumeSpecName "kube-api-access-vpt4t". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:41:59.143951 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:41:59.143928    4892 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8a1b4524-f816-4eab-b88d-599a7689f138-maas-api-xks-ca" pod "8a1b4524-f816-4eab-b88d-599a7689f138" (UID: "8a1b4524-f816-4eab-b88d-599a7689f138"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 19:41:59.242941 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:41:59.242863    4892 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vpt4t\" (UniqueName: \"kubernetes.io/projected/8a1b4524-f816-4eab-b88d-599a7689f138-kube-api-access-vpt4t\") on node \"ip-10-0-132-82.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 19:41:59.242941 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:41:59.242901    4892 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8a1b4524-f816-4eab-b88d-599a7689f138-maas-api-xks-ca\") on node \"ip-10-0-132-82.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 19:41:59.917254 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:41:59.917226    4892 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e52034d3-rjjj5"
Oct 30 19:41:59.917521 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:41:59.917252    4892 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e52034d3-rjjj5" event={"ID":"8a1b4524-f816-4eab-b88d-599a7689f138","Type":"ContainerDied","Data":"9a45529b5613233c1e79baaf559f83669dd83ad7149eb0de0e80a674ce572f2b"}
Oct 30 19:41:59.917521 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:41:59.917306    4892 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="9a45529b5613233c1e79baaf559f83669dd83ad7149eb0de0e80a674ce572f2b"
Oct 30 19:42:01.202940 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:42:01.202866    4892 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-fsfvk"]
Oct 30 19:42:01.487772 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:42:01.487702    4892 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w0-2d9861-openshift-default-58ddf574b9-8pwn9"]
Oct 30 19:42:01.488215 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:42:01.488200    4892 cpu_manager.go:414] "containerMap: removing container" podUID="8a1b4524-f816-4eab-b88d-599a7689f138" containerName="revoke-keys"
Oct 30 19:42:01.488215 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:42:01.488215    4892 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8a1b4524-f816-4eab-b88d-599a7689f138" containerName="revoke-keys"
Oct 30 19:42:01.488452 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:42:01.488249    4892 cpu_manager.go:414] "containerMap: removing container" podUID="91c01247-b2cd-497d-835b-fdd57fdf08d0" containerName="revoke-keys"
Oct 30 19:42:01.488452 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:42:01.488254    4892 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="91c01247-b2cd-497d-835b-fdd57fdf08d0" containerName="revoke-keys"
Oct 30 19:42:01.488452 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:42:01.488332    4892 memory_manager.go:365] "RemoveStaleState removing state" podUID="8a1b4524-f816-4eab-b88d-599a7689f138" containerName="revoke-keys"
Oct 30 19:42:01.488452 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:42:01.488345    4892 memory_manager.go:365] "RemoveStaleState removing state" podUID="91c01247-b2cd-497d-835b-fdd57fdf08d0" containerName="revoke-keys"
Oct 30 19:42:01.492022 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:42:01.492006    4892 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-2d9861-openshift-default-58ddf574b9-8pwn9"
Oct 30 19:42:01.546105 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:42:01.546078    4892 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w0-2d9861-openshift-default-dockercfg-ngs2l\""
Oct 30 19:42:01.569904 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:42:01.569871    4892 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/abea5659-2a38-406f-8221-66aefbc769b7-workload-socket\") pod \"e2e-worker-w0-2d9861-openshift-default-58ddf574b9-8pwn9\" (UID: \"abea5659-2a38-406f-8221-66aefbc769b7\") " pod="openshift-ingress/e2e-worker-w0-2d9861-openshift-default-58ddf574b9-8pwn9"
Oct 30 19:42:01.570048 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:42:01.569929    4892 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/abea5659-2a38-406f-8221-66aefbc769b7-istio-envoy\") pod \"e2e-worker-w0-2d9861-openshift-default-58ddf574b9-8pwn9\" (UID: \"abea5659-2a38-406f-8221-66aefbc769b7\") " pod="openshift-ingress/e2e-worker-w0-2d9861-openshift-default-58ddf574b9-8pwn9"
Oct 30 19:42:01.570048 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:42:01.570033    4892 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/abea5659-2a38-406f-8221-66aefbc769b7-istio-data\") pod \"e2e-worker-w0-2d9861-openshift-default-58ddf574b9-8pwn9\" (UID: \"abea5659-2a38-406f-8221-66aefbc769b7\") " pod="openshift-ingress/e2e-worker-w0-2d9861-openshift-default-58ddf574b9-8pwn9"
Oct 30 19:42:01.570157 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:42:01.570147    4892 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/abea5659-2a38-406f-8221-66aefbc769b7-credential-socket\") pod \"e2e-worker-w0-2d9861-openshift-default-58ddf574b9-8pwn9\" (UID: \"abea5659-2a38-406f-8221-66aefbc769b7\") " pod="openshift-ingress/e2e-worker-w0-2d9861-openshift-default-58ddf574b9-8pwn9"
Oct 30 19:42:01.570221 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:42:01.570199    4892 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/abea5659-2a38-406f-8221-66aefbc769b7-istiod-ca-cert\") pod \"e2e-worker-w0-2d9861-openshift-default-58ddf574b9-8pwn9\" (UID: \"abea5659-2a38-406f-8221-66aefbc769b7\") " pod="openshift-ingress/e2e-worker-w0-2d9861-openshift-default-58ddf574b9-8pwn9"
Oct 30 19:42:01.570273 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:42:01.570239    4892 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fv5d6\" (UniqueName: \"kubernetes.io/projected/abea5659-2a38-406f-8221-66aefbc769b7-kube-api-access-fv5d6\") pod \"e2e-worker-w0-2d9861-openshift-default-58ddf574b9-8pwn9\" (UID: \"abea5659-2a38-406f-8221-66aefbc769b7\") " pod="openshift-ingress/e2e-worker-w0-2d9861-openshift-default-58ddf574b9-8pwn9"
Oct 30 19:42:01.570309 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:42:01.570297    4892 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/abea5659-2a38-406f-8221-66aefbc769b7-istio-podinfo\") pod \"e2e-worker-w0-2d9861-openshift-default-58ddf574b9-8pwn9\" (UID: \"abea5659-2a38-406f-8221-66aefbc769b7\") " pod="openshift-ingress/e2e-worker-w0-2d9861-openshift-default-58ddf574b9-8pwn9"
Oct 30 19:42:01.570354 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:42:01.570319    4892 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/abea5659-2a38-406f-8221-66aefbc769b7-istio-token\") pod \"e2e-worker-w0-2d9861-openshift-default-58ddf574b9-8pwn9\" (UID: \"abea5659-2a38-406f-8221-66aefbc769b7\") " pod="openshift-ingress/e2e-worker-w0-2d9861-openshift-default-58ddf574b9-8pwn9"
Oct 30 19:42:01.570399 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:42:01.570349    4892 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/abea5659-2a38-406f-8221-66aefbc769b7-workload-certs\") pod \"e2e-worker-w0-2d9861-openshift-default-58ddf574b9-8pwn9\" (UID: \"abea5659-2a38-406f-8221-66aefbc769b7\") " pod="openshift-ingress/e2e-worker-w0-2d9861-openshift-default-58ddf574b9-8pwn9"
Oct 30 19:42:01.601005 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:42:01.600660    4892 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-2d9861-openshift-default-58ddf574b9-8pwn9"]
Oct 30 19:42:01.671685 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:42:01.671645    4892 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/abea5659-2a38-406f-8221-66aefbc769b7-workload-socket\") pod \"e2e-worker-w0-2d9861-openshift-default-58ddf574b9-8pwn9\" (UID: \"abea5659-2a38-406f-8221-66aefbc769b7\") " pod="openshift-ingress/e2e-worker-w0-2d9861-openshift-default-58ddf574b9-8pwn9"
Oct 30 19:42:01.671685 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:42:01.671685    4892 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/abea5659-2a38-406f-8221-66aefbc769b7-istio-envoy\") pod \"e2e-worker-w0-2d9861-openshift-default-58ddf574b9-8pwn9\" (UID: \"abea5659-2a38-406f-8221-66aefbc769b7\") " pod="openshift-ingress/e2e-worker-w0-2d9861-openshift-default-58ddf574b9-8pwn9"
Oct 30 19:42:01.671945 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:42:01.671707    4892 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/abea5659-2a38-406f-8221-66aefbc769b7-istio-data\") pod \"e2e-worker-w0-2d9861-openshift-default-58ddf574b9-8pwn9\" (UID: \"abea5659-2a38-406f-8221-66aefbc769b7\") " pod="openshift-ingress/e2e-worker-w0-2d9861-openshift-default-58ddf574b9-8pwn9"
Oct 30 19:42:01.671945 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:42:01.671769    4892 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/abea5659-2a38-406f-8221-66aefbc769b7-credential-socket\") pod \"e2e-worker-w0-2d9861-openshift-default-58ddf574b9-8pwn9\" (UID: \"abea5659-2a38-406f-8221-66aefbc769b7\") " pod="openshift-ingress/e2e-worker-w0-2d9861-openshift-default-58ddf574b9-8pwn9"
Oct 30 19:42:01.671945 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:42:01.671801    4892 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/abea5659-2a38-406f-8221-66aefbc769b7-istiod-ca-cert\") pod \"e2e-worker-w0-2d9861-openshift-default-58ddf574b9-8pwn9\" (UID: \"abea5659-2a38-406f-8221-66aefbc769b7\") " pod="openshift-ingress/e2e-worker-w0-2d9861-openshift-default-58ddf574b9-8pwn9"
Oct 30 19:42:01.671945 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:42:01.671816    4892 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fv5d6\" (UniqueName: \"kubernetes.io/projected/abea5659-2a38-406f-8221-66aefbc769b7-kube-api-access-fv5d6\") pod \"e2e-worker-w0-2d9861-openshift-default-58ddf574b9-8pwn9\" (UID: \"abea5659-2a38-406f-8221-66aefbc769b7\") " pod="openshift-ingress/e2e-worker-w0-2d9861-openshift-default-58ddf574b9-8pwn9"
Oct 30 19:42:01.671945 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:42:01.671852    4892 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/abea5659-2a38-406f-8221-66aefbc769b7-istio-podinfo\") pod \"e2e-worker-w0-2d9861-openshift-default-58ddf574b9-8pwn9\" (UID: \"abea5659-2a38-406f-8221-66aefbc769b7\") " pod="openshift-ingress/e2e-worker-w0-2d9861-openshift-default-58ddf574b9-8pwn9"
Oct 30 19:42:01.671945 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:42:01.671878    4892 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/abea5659-2a38-406f-8221-66aefbc769b7-istio-token\") pod \"e2e-worker-w0-2d9861-openshift-default-58ddf574b9-8pwn9\" (UID: \"abea5659-2a38-406f-8221-66aefbc769b7\") " pod="openshift-ingress/e2e-worker-w0-2d9861-openshift-default-58ddf574b9-8pwn9"
Oct 30 19:42:01.671945 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:42:01.671914    4892 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/abea5659-2a38-406f-8221-66aefbc769b7-workload-certs\") pod \"e2e-worker-w0-2d9861-openshift-default-58ddf574b9-8pwn9\" (UID: \"abea5659-2a38-406f-8221-66aefbc769b7\") " pod="openshift-ingress/e2e-worker-w0-2d9861-openshift-default-58ddf574b9-8pwn9"
Oct 30 19:42:01.672320 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:42:01.672165    4892 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/abea5659-2a38-406f-8221-66aefbc769b7-workload-socket\") pod \"e2e-worker-w0-2d9861-openshift-default-58ddf574b9-8pwn9\" (UID: \"abea5659-2a38-406f-8221-66aefbc769b7\") " pod="openshift-ingress/e2e-worker-w0-2d9861-openshift-default-58ddf574b9-8pwn9"
Oct 30 19:42:01.672320 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:42:01.672219    4892 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/abea5659-2a38-406f-8221-66aefbc769b7-istio-data\") pod \"e2e-worker-w0-2d9861-openshift-default-58ddf574b9-8pwn9\" (UID: \"abea5659-2a38-406f-8221-66aefbc769b7\") " pod="openshift-ingress/e2e-worker-w0-2d9861-openshift-default-58ddf574b9-8pwn9"
Oct 30 19:42:01.672918 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:42:01.672808    4892 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/abea5659-2a38-406f-8221-66aefbc769b7-istiod-ca-cert\") pod \"e2e-worker-w0-2d9861-openshift-default-58ddf574b9-8pwn9\" (UID: \"abea5659-2a38-406f-8221-66aefbc769b7\") " pod="openshift-ingress/e2e-worker-w0-2d9861-openshift-default-58ddf574b9-8pwn9"
Oct 30 19:42:01.672918 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:42:01.672870    4892 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/abea5659-2a38-406f-8221-66aefbc769b7-credential-socket\") pod \"e2e-worker-w0-2d9861-openshift-default-58ddf574b9-8pwn9\" (UID: \"abea5659-2a38-406f-8221-66aefbc769b7\") " pod="openshift-ingress/e2e-worker-w0-2d9861-openshift-default-58ddf574b9-8pwn9"
Oct 30 19:42:01.672918 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:42:01.672914    4892 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/abea5659-2a38-406f-8221-66aefbc769b7-workload-certs\") pod \"e2e-worker-w0-2d9861-openshift-default-58ddf574b9-8pwn9\" (UID: \"abea5659-2a38-406f-8221-66aefbc769b7\") " pod="openshift-ingress/e2e-worker-w0-2d9861-openshift-default-58ddf574b9-8pwn9"
Oct 30 19:42:01.674926 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:42:01.674897    4892 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/abea5659-2a38-406f-8221-66aefbc769b7-istio-envoy\") pod \"e2e-worker-w0-2d9861-openshift-default-58ddf574b9-8pwn9\" (UID: \"abea5659-2a38-406f-8221-66aefbc769b7\") " pod="openshift-ingress/e2e-worker-w0-2d9861-openshift-default-58ddf574b9-8pwn9"
Oct 30 19:42:01.675119 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:42:01.675102    4892 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/abea5659-2a38-406f-8221-66aefbc769b7-istio-podinfo\") pod \"e2e-worker-w0-2d9861-openshift-default-58ddf574b9-8pwn9\" (UID: \"abea5659-2a38-406f-8221-66aefbc769b7\") " pod="openshift-ingress/e2e-worker-w0-2d9861-openshift-default-58ddf574b9-8pwn9"
Oct 30 19:42:01.736768 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:42:01.733696    4892 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fv5d6\" (UniqueName: \"kubernetes.io/projected/abea5659-2a38-406f-8221-66aefbc769b7-kube-api-access-fv5d6\") pod \"e2e-worker-w0-2d9861-openshift-default-58ddf574b9-8pwn9\" (UID: \"abea5659-2a38-406f-8221-66aefbc769b7\") " pod="openshift-ingress/e2e-worker-w0-2d9861-openshift-default-58ddf574b9-8pwn9"
Oct 30 19:42:01.738819 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:42:01.738735    4892 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/abea5659-2a38-406f-8221-66aefbc769b7-istio-token\") pod \"e2e-worker-w0-2d9861-openshift-default-58ddf574b9-8pwn9\" (UID: \"abea5659-2a38-406f-8221-66aefbc769b7\") " pod="openshift-ingress/e2e-worker-w0-2d9861-openshift-default-58ddf574b9-8pwn9"
Oct 30 19:42:01.813650 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:42:01.812594    4892 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-2d9861-openshift-default-58ddf574b9-8pwn9"
Oct 30 19:42:02.382015 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:42:02.380265    4892 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-2d9861-openshift-default-58ddf574b9-8pwn9"]
Oct 30 19:42:02.385499 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:42:02.384681    4892 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892000Ki","pods":"250"}
Oct 30 19:42:02.385499 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:42:02.384772    4892 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892000Ki","pods":"250"}
Oct 30 19:42:02.385499 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:42:02.384817    4892 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892000Ki","pods":"250"}
Oct 30 19:42:02.494055 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:42:02.494017    4892 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ccb90673-knzsq"]
Oct 30 19:42:02.511993 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:42:02.511376    4892 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ccb90673-knzsq"
Oct 30 19:42:02.570017 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:42:02.567031    4892 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-nlxp4\""
Oct 30 19:42:02.570017 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:42:02.567817    4892 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 30 19:42:02.570017 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:42:02.567895    4892 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 19:42:02.570017 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:42:02.568007    4892 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 30 19:42:02.614633 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:42:02.612638    4892 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6cdcce3e-9dc9-4d19-a0cf-fac60ebf288e-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-ccb90673-knzsq\" (UID: \"6cdcce3e-9dc9-4d19-a0cf-fac60ebf288e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ccb90673-knzsq"
Oct 30 19:42:02.614633 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:42:02.612776    4892 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6cdcce3e-9dc9-4d19-a0cf-fac60ebf288e-maas-api-xks-ca\") pod \"maas-api-revoke-sub-ccb90673-knzsq\" (UID: \"6cdcce3e-9dc9-4d19-a0cf-fac60ebf288e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ccb90673-knzsq"
Oct 30 19:42:02.614633 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:42:02.612926    4892 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-w2k57\" (UniqueName: \"kubernetes.io/projected/6cdcce3e-9dc9-4d19-a0cf-fac60ebf288e-kube-api-access-w2k57\") pod \"maas-api-revoke-sub-ccb90673-knzsq\" (UID: \"6cdcce3e-9dc9-4d19-a0cf-fac60ebf288e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ccb90673-knzsq"
Oct 30 19:42:02.715990 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:42:02.715415    4892 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6cdcce3e-9dc9-4d19-a0cf-fac60ebf288e-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-ccb90673-knzsq\" (UID: \"6cdcce3e-9dc9-4d19-a0cf-fac60ebf288e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ccb90673-knzsq"
Oct 30 19:42:02.715990 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:42:02.715526    4892 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6cdcce3e-9dc9-4d19-a0cf-fac60ebf288e-maas-api-xks-ca\") pod \"maas-api-revoke-sub-ccb90673-knzsq\" (UID: \"6cdcce3e-9dc9-4d19-a0cf-fac60ebf288e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ccb90673-knzsq"
Oct 30 19:42:02.715990 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:42:02.715620    4892 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-w2k57\" (UniqueName: \"kubernetes.io/projected/6cdcce3e-9dc9-4d19-a0cf-fac60ebf288e-kube-api-access-w2k57\") pod \"maas-api-revoke-sub-ccb90673-knzsq\" (UID: \"6cdcce3e-9dc9-4d19-a0cf-fac60ebf288e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ccb90673-knzsq"
Oct 30 19:42:02.720099 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:42:02.716833    4892 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6cdcce3e-9dc9-4d19-a0cf-fac60ebf288e-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-ccb90673-knzsq\" (UID: \"6cdcce3e-9dc9-4d19-a0cf-fac60ebf288e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ccb90673-knzsq"
Oct 30 19:42:02.746006 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:42:02.743441    4892 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6cdcce3e-9dc9-4d19-a0cf-fac60ebf288e-maas-api-xks-ca\") pod \"maas-api-revoke-sub-ccb90673-knzsq\" (UID: \"6cdcce3e-9dc9-4d19-a0cf-fac60ebf288e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ccb90673-knzsq"
Oct 30 19:42:02.794510 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:42:02.794380    4892 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ccb90673-knzsq"]
Oct 30 19:42:02.854990 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:42:02.854616    4892 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-w2k57\" (UniqueName: \"kubernetes.io/projected/6cdcce3e-9dc9-4d19-a0cf-fac60ebf288e-kube-api-access-w2k57\") pod \"maas-api-revoke-sub-ccb90673-knzsq\" (UID: \"6cdcce3e-9dc9-4d19-a0cf-fac60ebf288e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ccb90673-knzsq"
Oct 30 19:42:02.856011 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:42:02.855258    4892 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ccb90673-knzsq"
Oct 30 19:42:03.042005 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:42:03.039083    4892 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-2d9861-openshift-default-58ddf574b9-8pwn9" event={"ID":"abea5659-2a38-406f-8221-66aefbc769b7","Type":"ContainerStarted","Data":"44556d16a4998e8e9546aa97be4a412f85102c05ced870571e9c67dddcdf8592"}
Oct 30 19:42:04.280190 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:42:04.280139    4892 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-d52df0/distinct-2-w1-d52df0-kserve-7846558464-xrkdc" podUID="9c614ab6-30e2-4fe1-8609-d5e2dadc198a" containerName="main" probeResult="failure" output="Get \"https://10.132.0.83:8000/ready\": context deadline exceeded"
Oct 30 19:42:04.467984 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:42:04.467926    4892 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-2d9861/premium-w0-2d9861-kserve-67945c6848-4n6kz" podUID="798988b2-c769-4f73-a7c8-ca15de0beef1" containerName="main" probeResult="failure" output="Get \"https://10.132.0.62:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:42:04.498995 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:42:04.498939    4892 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-2d9861-openshift-default-58ddf574b9-8pwn9" event={"ID":"abea5659-2a38-406f-8221-66aefbc769b7","Type":"ContainerStarted","Data":"f8042e8de7eb3bfd7ddbca7161920b7d44b7965b6e8358ded7f787f0f714009b"}
Oct 30 19:42:04.539174 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:42:04.539076    4892 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-ada8d7/simulator-w3-ada8d7-kserve-7d575cfcb5-5txgn" podUID="10a5c7c8-6146-47b0-9643-bb137cf10bf5" containerName="main" probeResult="failure" output="Get \"https://10.132.0.55:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:42:04.682992 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:42:04.679039    4892 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w0-2d9861-openshift-default-58ddf574b9-8pwn9" podStartSLOduration=3.679016597 podStartE2EDuration="3.679016597s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:42:01 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:42:04.651388387 +0000 UTC m=+1572.221853987" watchObservedRunningTime="2026-10-30 19:42:04.679016597 +0000 UTC m=+1572.249482199"
Oct 30 19:42:04.731841 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:42:04.727385    4892 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-d52df0/simulator-w1-d52df0-kserve-67f74bc577-q5bjz" podUID="81b7bb84-1003-4452-b7e0-b4aaba6fefa4" containerName="main" probeResult="failure" output="Get \"https://10.132.0.57:8000/ready\": context deadline exceeded"
Oct 30 19:42:04.826046 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:42:04.820019    4892 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w0-2d9861-openshift-default-58ddf574b9-8pwn9"
Oct 30 19:42:04.826046 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:42:04.820332    4892 patch_prober.go:29] interesting pod/e2e-worker-w0-2d9861-openshift-default-58ddf574b9-8pwn9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.110:15021/healthz/ready\": dial tcp 10.132.0.110:15021: connect: connection refused" start-of-body=
Oct 30 19:42:04.826046 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:42:04.820377    4892 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-2d9861-openshift-default-58ddf574b9-8pwn9" podUID="abea5659-2a38-406f-8221-66aefbc769b7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.110:15021/healthz/ready\": dial tcp 10.132.0.110:15021: connect: connection refused"
Oct 30 19:42:04.826046 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:42:04.821069    4892 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ccb90673-knzsq"]
Oct 30 19:42:04.831994 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:42:04.829257    4892 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-b5178f/distinct-2-w4-b5178f-kserve-69ddf8b665-9pdth" podUID="fb71d961-8500-4367-b48c-88d385962bd5" containerName="main" probeResult="failure" output="Get \"https://10.132.0.90:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:42:04.852433 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:42:04.848683    4892 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-5f55cd/simulator-w6-5f55cd-kserve-5ff86b7447-h9tm7" podUID="3cb47a99-8be1-4756-b31a-43b58cc6b8dc" containerName="main" probeResult="failure" output="Get \"https://10.132.0.58:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:42:04.940191 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:42:04.936061    4892 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-a3f9b7/simulator-w2-a3f9b7-kserve-79bbc4cb5f-8m488" podUID="50fb2283-bf3f-41da-835b-f407ee1a04f3" containerName="main" probeResult="failure" output="Get \"https://10.132.0.59:8000/ready\": context deadline exceeded"
Oct 30 19:42:05.351274 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:42:05.351229    4892 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-d52df0/distinct-2-w1-d52df0-kserve-7846558464-xrkdc" podUID="9c614ab6-30e2-4fe1-8609-d5e2dadc198a" containerName="main" probeResult="failure" output="Get \"https://10.132.0.83:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:42:05.373159 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:42:05.352395    4892 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-d52df0/distinct-2-w1-d52df0-kserve-7846558464-xrkdc"
Oct 30 19:42:05.409016 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:42:05.393127    4892 patch_prober.go:29] interesting pod/e2e-worker-w4-b5178f-openshift-default-c8fbdff8c-fcqwt container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.96:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 30 19:42:05.409016 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:42:05.393189    4892 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w4-b5178f-openshift-default-c8fbdff8c-fcqwt" podUID="0f1f64ea-aef7-48e0-9238-599b572a90db" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.96:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:42:05.511050 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:42:05.502599    4892 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-2d9861/premium-w0-2d9861-kserve-67945c6848-4n6kz" podUID="798988b2-c769-4f73-a7c8-ca15de0beef1" containerName="main" probeResult="failure" output="Get \"https://10.132.0.62:8000/ready\": context deadline exceeded"
Oct 30 19:42:05.511050 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:42:05.503040    4892 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-2d9861/premium-w0-2d9861-kserve-67945c6848-4n6kz"
Oct 30 19:42:05.511050 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:42:05.505761    4892 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-686689/distinct-2-w5-686689-kserve-5ff578b4cd-rkc6s" podUID="a8829732-ac6f-4238-b8ce-fc781084668b" containerName="main" probeResult="failure" output="Get \"https://10.132.0.87:8000/ready\": context deadline exceeded"
Oct 30 19:42:05.554742 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:42:05.539999    4892 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-ada8d7/simulator-w3-ada8d7-kserve-7d575cfcb5-5txgn" podUID="10a5c7c8-6146-47b0-9643-bb137cf10bf5" containerName="main" probeResult="failure" output="Get \"https://10.132.0.55:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:42:05.556304 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:42:05.555071    4892 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-ada8d7/simulator-w3-ada8d7-kserve-7d575cfcb5-5txgn"
Oct 30 19:42:05.853050 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:42:05.851809    4892 patch_prober.go:29] interesting pod/e2e-worker-w0-2d9861-openshift-default-58ddf574b9-8pwn9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.110:15021/healthz/ready\": dial tcp 10.132.0.110:15021: connect: connection refused" start-of-body=
Oct 30 19:42:05.853050 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:42:05.851881    4892 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-2d9861-openshift-default-58ddf574b9-8pwn9" podUID="abea5659-2a38-406f-8221-66aefbc769b7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.110:15021/healthz/ready\": dial tcp 10.132.0.110:15021: connect: connection refused"
Oct 30 19:42:05.881993 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:42:05.872487    4892 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ccb90673-knzsq" event={"ID":"6cdcce3e-9dc9-4d19-a0cf-fac60ebf288e","Type":"ContainerStarted","Data":"eeb41853b054f17cddeebdad8ff9a1e61413c07b68715eaec314f5c24807b358"}
Oct 30 19:42:05.881993 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:42:05.873399    4892 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-b5178f/distinct-2-w4-b5178f-kserve-69ddf8b665-9pdth" podUID="fb71d961-8500-4367-b48c-88d385962bd5" containerName="main" probeResult="failure" output="Get \"https://10.132.0.90:8000/ready\": context deadline exceeded"
Oct 30 19:42:05.881993 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:42:05.873814    4892 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-b5178f/distinct-2-w4-b5178f-kserve-69ddf8b665-9pdth"
Oct 30 19:42:06.231907 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:42:06.231859    4892 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-5f55cd/premium-w6-5f55cd-kserve-7f785c544b-pcvhb" podUID="191593a2-e3a9-4ce4-a1c6-c089723b0237" containerName="main" probeResult="failure" output="Get \"https://10.132.0.60:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:42:06.380989 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:42:06.369053    4892 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-d52df0/distinct-2-w1-d52df0-kserve-7846558464-xrkdc" podUID="9c614ab6-30e2-4fe1-8609-d5e2dadc198a" containerName="main" probeResult="failure" output="Get \"https://10.132.0.83:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:42:06.581977 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:42:06.571062    4892 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-686689/distinct-2-w5-686689-kserve-5ff578b4cd-rkc6s" podUID="a8829732-ac6f-4238-b8ce-fc781084668b" containerName="main" probeResult="failure" output="Get \"https://10.132.0.87:8000/ready\": context deadline exceeded"
Oct 30 19:42:06.581977 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:42:06.571440    4892 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-2d9861/premium-w0-2d9861-kserve-67945c6848-4n6kz" podUID="798988b2-c769-4f73-a7c8-ca15de0beef1" containerName="main" probeResult="failure" output="Get \"https://10.132.0.62:8000/ready\": context deadline exceeded"
Oct 30 19:42:06.581977 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:42:06.572104    4892 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w5-686689/distinct-2-w5-686689-kserve-5ff578b4cd-rkc6s"
Oct 30 19:42:06.617001 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:42:06.599277    4892 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-ada8d7/simulator-w3-ada8d7-kserve-7d575cfcb5-5txgn" podUID="10a5c7c8-6146-47b0-9643-bb137cf10bf5" containerName="main" probeResult="failure" output="Get \"https://10.132.0.55:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:42:06.768993 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:42:06.759261    4892 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-d52df0/distinct-2-w1-d52df0-kserve-7846558464-xrkdc"
Oct 30 19:42:06.804992 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:42:06.770267    4892 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-d52df0/simulator-w1-d52df0-kserve-67f74bc577-q5bjz" podUID="81b7bb84-1003-4452-b7e0-b4aaba6fefa4" containerName="main" probeResult="failure" output="Get \"https://10.132.0.57:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:42:06.839618 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:42:06.839529    4892 patch_prober.go:29] interesting pod/e2e-worker-w0-2d9861-openshift-default-58ddf574b9-8pwn9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.110:15021/healthz/ready\": dial tcp 10.132.0.110:15021: connect: connection refused" start-of-body=
Oct 30 19:42:06.887078 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:42:06.887036    4892 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-2d9861-openshift-default-58ddf574b9-8pwn9" podUID="abea5659-2a38-406f-8221-66aefbc769b7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.110:15021/healthz/ready\": dial tcp 10.132.0.110:15021: connect: connection refused"
Oct 30 19:42:06.887356 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:42:06.887334    4892 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-b5178f/distinct-2-w4-b5178f-kserve-69ddf8b665-9pdth" podUID="fb71d961-8500-4367-b48c-88d385962bd5" containerName="main" probeResult="failure" output="Get \"https://10.132.0.90:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:42:06.887751 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:42:06.841066    4892 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-2d9861/premium-w0-2d9861-kserve-67945c6848-4n6kz"
Oct 30 19:42:06.887977 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:42:06.887948    4892 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-ada8d7/simulator-w3-ada8d7-kserve-7d575cfcb5-5txgn"
Oct 30 19:42:06.944985 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:42:06.928139    4892 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-a3f9b7/simulator-w2-a3f9b7-kserve-79bbc4cb5f-8m488" podUID="50fb2283-bf3f-41da-835b-f407ee1a04f3" containerName="main" probeResult="failure" output="Get \"https://10.132.0.59:8000/ready\": context deadline exceeded"
Oct 30 19:42:07.139070 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:42:07.102080    4892 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w5-686689/distinct-2-w5-686689-kserve-5ff578b4cd-rkc6s"
Oct 30 19:42:07.303992 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:42:07.265232    4892 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-5f55cd/premium-w6-5f55cd-kserve-7f785c544b-pcvhb" podUID="191593a2-e3a9-4ce4-a1c6-c089723b0237" containerName="main" probeResult="failure" output="Get \"https://10.132.0.60:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:42:07.321094 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:42:07.286903    4892 generic.go:356] "Generic (PLEG): container finished" podID="6cdcce3e-9dc9-4d19-a0cf-fac60ebf288e" containerID="4dfa5a2a4510e0889a1cbc0f9862d3e92a5a6fc1b8cb500c5b5cb7ab06052a7d" exitCode=0
Oct 30 19:42:07.321094 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:42:07.286932    4892 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ccb90673-knzsq" event={"ID":"6cdcce3e-9dc9-4d19-a0cf-fac60ebf288e","Type":"ContainerDied","Data":"4dfa5a2a4510e0889a1cbc0f9862d3e92a5a6fc1b8cb500c5b5cb7ab06052a7d"}
Oct 30 19:42:07.321094 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:42:07.307190    4892 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w6-5f55cd/premium-w6-5f55cd-kserve-7f785c544b-pcvhb"
Oct 30 19:42:07.614732 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:42:07.614193    4892 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w6-5f55cd/premium-w6-5f55cd-kserve-7f785c544b-pcvhb"
Oct 30 19:42:07.686048 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:42:07.686014    4892 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-b5178f/distinct-2-w4-b5178f-kserve-69ddf8b665-9pdth"
Oct 30 19:42:07.814709 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:42:07.814675    4892 patch_prober.go:29] interesting pod/e2e-worker-w0-2d9861-openshift-default-58ddf574b9-8pwn9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.110:15021/healthz/ready\": dial tcp 10.132.0.110:15021: connect: connection refused" start-of-body=
Oct 30 19:42:07.814890 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:42:07.814721    4892 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-2d9861-openshift-default-58ddf574b9-8pwn9" podUID="abea5659-2a38-406f-8221-66aefbc769b7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.110:15021/healthz/ready\": dial tcp 10.132.0.110:15021: connect: connection refused"
Oct 30 19:42:08.522660 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:42:08.522637    4892 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ccb90673-knzsq"
Oct 30 19:42:08.624453 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:42:08.624370    4892 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6cdcce3e-9dc9-4d19-a0cf-fac60ebf288e-kube-api-access-w2k57\" (UniqueName: \"kubernetes.io/projected/6cdcce3e-9dc9-4d19-a0cf-fac60ebf288e-kube-api-access-w2k57\") pod \"6cdcce3e-9dc9-4d19-a0cf-fac60ebf288e\" (UID: \"6cdcce3e-9dc9-4d19-a0cf-fac60ebf288e\") "
Oct 30 19:42:08.624453 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:42:08.624425    4892 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/6cdcce3e-9dc9-4d19-a0cf-fac60ebf288e-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6cdcce3e-9dc9-4d19-a0cf-fac60ebf288e-maas-api-openshift-ca\") pod \"6cdcce3e-9dc9-4d19-a0cf-fac60ebf288e\" (UID: \"6cdcce3e-9dc9-4d19-a0cf-fac60ebf288e\") "
Oct 30 19:42:08.624949 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:42:08.624475    4892 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/6cdcce3e-9dc9-4d19-a0cf-fac60ebf288e-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6cdcce3e-9dc9-4d19-a0cf-fac60ebf288e-maas-api-xks-ca\") pod \"6cdcce3e-9dc9-4d19-a0cf-fac60ebf288e\" (UID: \"6cdcce3e-9dc9-4d19-a0cf-fac60ebf288e\") "
Oct 30 19:42:08.624949 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:42:08.624784    4892 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6cdcce3e-9dc9-4d19-a0cf-fac60ebf288e-maas-api-openshift-ca" pod "6cdcce3e-9dc9-4d19-a0cf-fac60ebf288e" (UID: "6cdcce3e-9dc9-4d19-a0cf-fac60ebf288e"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 19:42:08.625111 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:42:08.624959    4892 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6cdcce3e-9dc9-4d19-a0cf-fac60ebf288e-maas-api-openshift-ca\") on node \"ip-10-0-132-82.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 19:42:08.626669 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:42:08.626627    4892 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6cdcce3e-9dc9-4d19-a0cf-fac60ebf288e-kube-api-access-w2k57" pod "6cdcce3e-9dc9-4d19-a0cf-fac60ebf288e" (UID: "6cdcce3e-9dc9-4d19-a0cf-fac60ebf288e"). InnerVolumeSpecName "kube-api-access-w2k57". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:42:08.626812 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:42:08.626787    4892 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6cdcce3e-9dc9-4d19-a0cf-fac60ebf288e-maas-api-xks-ca" pod "6cdcce3e-9dc9-4d19-a0cf-fac60ebf288e" (UID: "6cdcce3e-9dc9-4d19-a0cf-fac60ebf288e"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 19:42:08.726525 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:42:08.726493    4892 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-w2k57\" (UniqueName: \"kubernetes.io/projected/6cdcce3e-9dc9-4d19-a0cf-fac60ebf288e-kube-api-access-w2k57\") on node \"ip-10-0-132-82.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 19:42:08.726525 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:42:08.726519    4892 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6cdcce3e-9dc9-4d19-a0cf-fac60ebf288e-maas-api-xks-ca\") on node \"ip-10-0-132-82.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 19:42:08.813284 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:42:08.813251    4892 patch_prober.go:29] interesting pod/e2e-worker-w0-2d9861-openshift-default-58ddf574b9-8pwn9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.110:15021/healthz/ready\": dial tcp 10.132.0.110:15021: connect: connection refused" start-of-body=
Oct 30 19:42:08.813470 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:42:08.813306    4892 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-2d9861-openshift-default-58ddf574b9-8pwn9" podUID="abea5659-2a38-406f-8221-66aefbc769b7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.110:15021/healthz/ready\": dial tcp 10.132.0.110:15021: connect: connection refused"
Oct 30 19:42:09.321993 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:42:09.321939    4892 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ccb90673-knzsq"
Oct 30 19:42:09.322185 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:42:09.321998    4892 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ccb90673-knzsq" event={"ID":"6cdcce3e-9dc9-4d19-a0cf-fac60ebf288e","Type":"ContainerDied","Data":"eeb41853b054f17cddeebdad8ff9a1e61413c07b68715eaec314f5c24807b358"}
Oct 30 19:42:09.322185 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:42:09.322056    4892 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="eeb41853b054f17cddeebdad8ff9a1e61413c07b68715eaec314f5c24807b358"
Oct 30 19:42:09.813799 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:42:09.813765    4892 patch_prober.go:29] interesting pod/e2e-worker-w0-2d9861-openshift-default-58ddf574b9-8pwn9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.110:15021/healthz/ready\": dial tcp 10.132.0.110:15021: connect: connection refused" start-of-body=
Oct 30 19:42:09.814252 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:42:09.813829    4892 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-2d9861-openshift-default-58ddf574b9-8pwn9" podUID="abea5659-2a38-406f-8221-66aefbc769b7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.110:15021/healthz/ready\": dial tcp 10.132.0.110:15021: connect: connection refused"
Oct 30 19:42:10.009995 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:42:10.009946    4892 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b7f56eab-tg2gm"]
Oct 30 19:42:10.010534 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:42:10.010519    4892 cpu_manager.go:414] "containerMap: removing container" podUID="6cdcce3e-9dc9-4d19-a0cf-fac60ebf288e" containerName="revoke-keys"
Oct 30 19:42:10.010582 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:42:10.010536    4892 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6cdcce3e-9dc9-4d19-a0cf-fac60ebf288e" containerName="revoke-keys"
Oct 30 19:42:10.010654 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:42:10.010645    4892 memory_manager.go:365] "RemoveStaleState removing state" podUID="6cdcce3e-9dc9-4d19-a0cf-fac60ebf288e" containerName="revoke-keys"
Oct 30 19:42:10.014124 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:42:10.014109    4892 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b7f56eab-tg2gm"
Oct 30 19:42:10.066237 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:42:10.066126    4892 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 19:42:10.066237 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:42:10.066216    4892 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 30 19:42:10.066624 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:42:10.066598    4892 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-nlxp4\""
Oct 30 19:42:10.066690 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:42:10.066668    4892 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 30 19:42:10.071640 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:42:10.071612    4892 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b7f56eab-tg2gm"]
Oct 30 19:42:10.168956 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:42:10.168916    4892 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-n4zbd\" (UniqueName: \"kubernetes.io/projected/e7cd4cc4-e0fa-4b6b-8895-8be85ddb31c5-kube-api-access-n4zbd\") pod \"maas-api-revoke-sub-b7f56eab-tg2gm\" (UID: \"e7cd4cc4-e0fa-4b6b-8895-8be85ddb31c5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b7f56eab-tg2gm"
Oct 30 19:42:10.169153 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:42:10.169102    4892 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e7cd4cc4-e0fa-4b6b-8895-8be85ddb31c5-maas-api-xks-ca\") pod \"maas-api-revoke-sub-b7f56eab-tg2gm\" (UID: \"e7cd4cc4-e0fa-4b6b-8895-8be85ddb31c5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b7f56eab-tg2gm"
Oct 30 19:42:10.169280 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:42:10.169263    4892 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e7cd4cc4-e0fa-4b6b-8895-8be85ddb31c5-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-b7f56eab-tg2gm\" (UID: \"e7cd4cc4-e0fa-4b6b-8895-8be85ddb31c5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b7f56eab-tg2gm"
Oct 30 19:42:10.270099 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:42:10.270067    4892 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e7cd4cc4-e0fa-4b6b-8895-8be85ddb31c5-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-b7f56eab-tg2gm\" (UID: \"e7cd4cc4-e0fa-4b6b-8895-8be85ddb31c5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b7f56eab-tg2gm"
Oct 30 19:42:10.270311 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:42:10.270138    4892 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-n4zbd\" (UniqueName: \"kubernetes.io/projected/e7cd4cc4-e0fa-4b6b-8895-8be85ddb31c5-kube-api-access-n4zbd\") pod \"maas-api-revoke-sub-b7f56eab-tg2gm\" (UID: \"e7cd4cc4-e0fa-4b6b-8895-8be85ddb31c5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b7f56eab-tg2gm"
Oct 30 19:42:10.270311 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:42:10.270208    4892 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e7cd4cc4-e0fa-4b6b-8895-8be85ddb31c5-maas-api-xks-ca\") pod \"maas-api-revoke-sub-b7f56eab-tg2gm\" (UID: \"e7cd4cc4-e0fa-4b6b-8895-8be85ddb31c5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b7f56eab-tg2gm"
Oct 30 19:42:10.270789 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:42:10.270766    4892 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e7cd4cc4-e0fa-4b6b-8895-8be85ddb31c5-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-b7f56eab-tg2gm\" (UID: \"e7cd4cc4-e0fa-4b6b-8895-8be85ddb31c5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b7f56eab-tg2gm"
Oct 30 19:42:10.272837 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:42:10.272804    4892 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e7cd4cc4-e0fa-4b6b-8895-8be85ddb31c5-maas-api-xks-ca\") pod \"maas-api-revoke-sub-b7f56eab-tg2gm\" (UID: \"e7cd4cc4-e0fa-4b6b-8895-8be85ddb31c5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b7f56eab-tg2gm"
Oct 30 19:42:10.327211 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:42:10.327146    4892 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-n4zbd\" (UniqueName: \"kubernetes.io/projected/e7cd4cc4-e0fa-4b6b-8895-8be85ddb31c5-kube-api-access-n4zbd\") pod \"maas-api-revoke-sub-b7f56eab-tg2gm\" (UID: \"e7cd4cc4-e0fa-4b6b-8895-8be85ddb31c5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b7f56eab-tg2gm"
Oct 30 19:42:10.633389 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:42:10.633138    4892 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b7f56eab-tg2gm"
Oct 30 19:42:10.813779 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:42:10.813744    4892 patch_prober.go:29] interesting pod/e2e-worker-w0-2d9861-openshift-default-58ddf574b9-8pwn9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.110:15021/healthz/ready\": dial tcp 10.132.0.110:15021: connect: connection refused" start-of-body=
Oct 30 19:42:10.813954 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:42:10.813798    4892 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-2d9861-openshift-default-58ddf574b9-8pwn9" podUID="abea5659-2a38-406f-8221-66aefbc769b7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.110:15021/healthz/ready\": dial tcp 10.132.0.110:15021: connect: connection refused"
Oct 30 19:42:11.061488 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:42:11.061407    4892 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b7f56eab-tg2gm"]
Oct 30 19:42:11.062759 ip-10-0-132-82 kubenswrapper[4892]: W1030 19:42:11.062727    4892 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode7cd4cc4_e0fa_4b6b_8895_8be85ddb31c5.slice/crio-83c116bd2258d340ac9a439606c7e6e3587b8216dca02f4dea987b4b5856de04 WatchSource:0}: Error finding container 83c116bd2258d340ac9a439606c7e6e3587b8216dca02f4dea987b4b5856de04: Status 404 returned error can't find the container with id 83c116bd2258d340ac9a439606c7e6e3587b8216dca02f4dea987b4b5856de04
Oct 30 19:42:11.353267 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:42:11.353172    4892 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b7f56eab-tg2gm" event={"ID":"e7cd4cc4-e0fa-4b6b-8895-8be85ddb31c5","Type":"ContainerStarted","Data":"83c116bd2258d340ac9a439606c7e6e3587b8216dca02f4dea987b4b5856de04"}
Oct 30 19:42:11.821510 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:42:11.820360    4892 patch_prober.go:29] interesting pod/e2e-worker-w0-2d9861-openshift-default-58ddf574b9-8pwn9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.110:15021/healthz/ready\": dial tcp 10.132.0.110:15021: connect: connection refused" start-of-body=
Oct 30 19:42:11.821510 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:42:11.820420    4892 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-2d9861-openshift-default-58ddf574b9-8pwn9" podUID="abea5659-2a38-406f-8221-66aefbc769b7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.110:15021/healthz/ready\": dial tcp 10.132.0.110:15021: connect: connection refused"
Oct 30 19:42:12.366604 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:42:12.366563    4892 generic.go:356] "Generic (PLEG): container finished" podID="e7cd4cc4-e0fa-4b6b-8895-8be85ddb31c5" containerID="d754ba5106e4070ba151b76294bc6b69c5bb4fa6703b29c8759278e4337167e8" exitCode=0
Oct 30 19:42:12.366794 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:42:12.366712    4892 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b7f56eab-tg2gm" event={"ID":"e7cd4cc4-e0fa-4b6b-8895-8be85ddb31c5","Type":"ContainerDied","Data":"d754ba5106e4070ba151b76294bc6b69c5bb4fa6703b29c8759278e4337167e8"}
Oct 30 19:42:12.818424 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:42:12.818393    4892 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w0-2d9861-openshift-default-58ddf574b9-8pwn9"
Oct 30 19:42:12.818754 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:42:12.818731    4892 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w0-2d9861-openshift-default-58ddf574b9-8pwn9"
Oct 30 19:42:12.819369 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:42:12.819354    4892 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w0-2d9861-openshift-default-58ddf574b9-8pwn9"
Oct 30 19:42:13.529543 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:42:13.529522    4892 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b7f56eab-tg2gm"
Oct 30 19:42:13.717344 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:42:13.717317    4892 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e7cd4cc4-e0fa-4b6b-8895-8be85ddb31c5-kube-api-access-n4zbd\" (UniqueName: \"kubernetes.io/projected/e7cd4cc4-e0fa-4b6b-8895-8be85ddb31c5-kube-api-access-n4zbd\") pod \"e7cd4cc4-e0fa-4b6b-8895-8be85ddb31c5\" (UID: \"e7cd4cc4-e0fa-4b6b-8895-8be85ddb31c5\") "
Oct 30 19:42:13.717592 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:42:13.717422    4892 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e7cd4cc4-e0fa-4b6b-8895-8be85ddb31c5-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e7cd4cc4-e0fa-4b6b-8895-8be85ddb31c5-maas-api-openshift-ca\") pod \"e7cd4cc4-e0fa-4b6b-8895-8be85ddb31c5\" (UID: \"e7cd4cc4-e0fa-4b6b-8895-8be85ddb31c5\") "
Oct 30 19:42:13.717592 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:42:13.717466    4892 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/e7cd4cc4-e0fa-4b6b-8895-8be85ddb31c5-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e7cd4cc4-e0fa-4b6b-8895-8be85ddb31c5-maas-api-xks-ca\") pod \"e7cd4cc4-e0fa-4b6b-8895-8be85ddb31c5\" (UID: \"e7cd4cc4-e0fa-4b6b-8895-8be85ddb31c5\") "
Oct 30 19:42:13.717821 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:42:13.717794    4892 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e7cd4cc4-e0fa-4b6b-8895-8be85ddb31c5-maas-api-openshift-ca" pod "e7cd4cc4-e0fa-4b6b-8895-8be85ddb31c5" (UID: "e7cd4cc4-e0fa-4b6b-8895-8be85ddb31c5"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 19:42:13.719643 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:42:13.719619    4892 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e7cd4cc4-e0fa-4b6b-8895-8be85ddb31c5-kube-api-access-n4zbd" pod "e7cd4cc4-e0fa-4b6b-8895-8be85ddb31c5" (UID: "e7cd4cc4-e0fa-4b6b-8895-8be85ddb31c5"). InnerVolumeSpecName "kube-api-access-n4zbd". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:42:13.719741 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:42:13.719642    4892 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e7cd4cc4-e0fa-4b6b-8895-8be85ddb31c5-maas-api-xks-ca" pod "e7cd4cc4-e0fa-4b6b-8895-8be85ddb31c5" (UID: "e7cd4cc4-e0fa-4b6b-8895-8be85ddb31c5"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 19:42:13.819486 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:42:13.819433    4892 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-n4zbd\" (UniqueName: \"kubernetes.io/projected/e7cd4cc4-e0fa-4b6b-8895-8be85ddb31c5-kube-api-access-n4zbd\") on node \"ip-10-0-132-82.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 19:42:13.819486 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:42:13.819467    4892 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e7cd4cc4-e0fa-4b6b-8895-8be85ddb31c5-maas-api-openshift-ca\") on node \"ip-10-0-132-82.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 19:42:13.819486 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:42:13.819483    4892 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e7cd4cc4-e0fa-4b6b-8895-8be85ddb31c5-maas-api-xks-ca\") on node \"ip-10-0-132-82.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 19:42:14.384989 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:42:14.384944    4892 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b7f56eab-tg2gm"
Oct 30 19:42:14.385168 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:42:14.384939    4892 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b7f56eab-tg2gm" event={"ID":"e7cd4cc4-e0fa-4b6b-8895-8be85ddb31c5","Type":"ContainerDied","Data":"83c116bd2258d340ac9a439606c7e6e3587b8216dca02f4dea987b4b5856de04"}
Oct 30 19:42:14.385168 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:42:14.385102    4892 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="83c116bd2258d340ac9a439606c7e6e3587b8216dca02f4dea987b4b5856de04"
Oct 30 19:42:18.027100 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:42:18.027061    4892 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e426ecf3-gfxx4"]
Oct 30 19:42:18.027564 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:42:18.027551    4892 cpu_manager.go:414] "containerMap: removing container" podUID="e7cd4cc4-e0fa-4b6b-8895-8be85ddb31c5" containerName="revoke-keys"
Oct 30 19:42:18.027608 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:42:18.027564    4892 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e7cd4cc4-e0fa-4b6b-8895-8be85ddb31c5" containerName="revoke-keys"
Oct 30 19:42:18.027673 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:42:18.027664    4892 memory_manager.go:365] "RemoveStaleState removing state" podUID="e7cd4cc4-e0fa-4b6b-8895-8be85ddb31c5" containerName="revoke-keys"
Oct 30 19:42:18.032258 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:42:18.032242    4892 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e426ecf3-gfxx4"
Oct 30 19:42:18.084584 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:42:18.084557    4892 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6f5154a2-9a70-4ccc-8584-a2ce72ba6564-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-e426ecf3-gfxx4\" (UID: \"6f5154a2-9a70-4ccc-8584-a2ce72ba6564\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e426ecf3-gfxx4"
Oct 30 19:42:18.084681 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:42:18.084612    4892 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6f5154a2-9a70-4ccc-8584-a2ce72ba6564-maas-api-xks-ca\") pod \"maas-api-revoke-sub-e426ecf3-gfxx4\" (UID: \"6f5154a2-9a70-4ccc-8584-a2ce72ba6564\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e426ecf3-gfxx4"
Oct 30 19:42:18.084737 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:42:18.084717    4892 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5sbtb\" (UniqueName: \"kubernetes.io/projected/6f5154a2-9a70-4ccc-8584-a2ce72ba6564-kube-api-access-5sbtb\") pod \"maas-api-revoke-sub-e426ecf3-gfxx4\" (UID: \"6f5154a2-9a70-4ccc-8584-a2ce72ba6564\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e426ecf3-gfxx4"
Oct 30 19:42:18.136264 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:42:18.136234    4892 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 19:42:18.136413 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:42:18.136320    4892 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-nlxp4\""
Oct 30 19:42:18.136413 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:42:18.136397    4892 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 30 19:42:18.136538 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:42:18.136458    4892 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 30 19:42:18.140934 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:42:18.140910    4892 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e426ecf3-gfxx4"]
Oct 30 19:42:18.186173 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:42:18.186148    4892 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6f5154a2-9a70-4ccc-8584-a2ce72ba6564-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-e426ecf3-gfxx4\" (UID: \"6f5154a2-9a70-4ccc-8584-a2ce72ba6564\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e426ecf3-gfxx4"
Oct 30 19:42:18.186276 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:42:18.186220    4892 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6f5154a2-9a70-4ccc-8584-a2ce72ba6564-maas-api-xks-ca\") pod \"maas-api-revoke-sub-e426ecf3-gfxx4\" (UID: \"6f5154a2-9a70-4ccc-8584-a2ce72ba6564\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e426ecf3-gfxx4"
Oct 30 19:42:18.186329 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:42:18.186286    4892 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5sbtb\" (UniqueName: \"kubernetes.io/projected/6f5154a2-9a70-4ccc-8584-a2ce72ba6564-kube-api-access-5sbtb\") pod \"maas-api-revoke-sub-e426ecf3-gfxx4\" (UID: \"6f5154a2-9a70-4ccc-8584-a2ce72ba6564\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e426ecf3-gfxx4"
Oct 30 19:42:18.186827 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:42:18.186806    4892 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6f5154a2-9a70-4ccc-8584-a2ce72ba6564-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-e426ecf3-gfxx4\" (UID: \"6f5154a2-9a70-4ccc-8584-a2ce72ba6564\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e426ecf3-gfxx4"
Oct 30 19:42:18.188658 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:42:18.188640    4892 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6f5154a2-9a70-4ccc-8584-a2ce72ba6564-maas-api-xks-ca\") pod \"maas-api-revoke-sub-e426ecf3-gfxx4\" (UID: \"6f5154a2-9a70-4ccc-8584-a2ce72ba6564\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e426ecf3-gfxx4"
Oct 30 19:42:18.243000 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:42:18.242954    4892 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5sbtb\" (UniqueName: \"kubernetes.io/projected/6f5154a2-9a70-4ccc-8584-a2ce72ba6564-kube-api-access-5sbtb\") pod \"maas-api-revoke-sub-e426ecf3-gfxx4\" (UID: \"6f5154a2-9a70-4ccc-8584-a2ce72ba6564\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e426ecf3-gfxx4"
Oct 30 19:42:18.345304 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:42:18.345236    4892 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e426ecf3-gfxx4"
Oct 30 19:42:18.735003 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:42:18.733487    4892 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e426ecf3-gfxx4"]
Oct 30 19:42:18.738908 ip-10-0-132-82 kubenswrapper[4892]: W1030 19:42:18.738875    4892 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6f5154a2_9a70_4ccc_8584_a2ce72ba6564.slice/crio-6dda13bb325860d2a0cde7f4b19548ba9dbba84dafe099a09dea323a698293f7 WatchSource:0}: Error finding container 6dda13bb325860d2a0cde7f4b19548ba9dbba84dafe099a09dea323a698293f7: Status 404 returned error can't find the container with id 6dda13bb325860d2a0cde7f4b19548ba9dbba84dafe099a09dea323a698293f7
Oct 30 19:42:19.433877 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:42:19.433837    4892 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e426ecf3-gfxx4" event={"ID":"6f5154a2-9a70-4ccc-8584-a2ce72ba6564","Type":"ContainerStarted","Data":"810822b52e5f41f6c0fc2f4dc1ad71e32ba41392583e4936720f0152313d922c"}
Oct 30 19:42:19.434261 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:42:19.433900    4892 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e426ecf3-gfxx4" event={"ID":"6f5154a2-9a70-4ccc-8584-a2ce72ba6564","Type":"ContainerStarted","Data":"6dda13bb325860d2a0cde7f4b19548ba9dbba84dafe099a09dea323a698293f7"}
Oct 30 19:42:19.547585 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:42:19.547534    4892 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e426ecf3-gfxx4" podStartSLOduration=2.547520071 podStartE2EDuration="2.547520071s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:42:17 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:42:19.546384881 +0000 UTC m=+1587.116850480" watchObservedRunningTime="2026-10-30 19:42:19.547520071 +0000 UTC m=+1587.117985693"
Oct 30 19:42:20.443452 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:42:20.443411    4892 generic.go:356] "Generic (PLEG): container finished" podID="6f5154a2-9a70-4ccc-8584-a2ce72ba6564" containerID="810822b52e5f41f6c0fc2f4dc1ad71e32ba41392583e4936720f0152313d922c" exitCode=0
Oct 30 19:42:20.443834 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:42:20.443495    4892 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e426ecf3-gfxx4" event={"ID":"6f5154a2-9a70-4ccc-8584-a2ce72ba6564","Type":"ContainerDied","Data":"810822b52e5f41f6c0fc2f4dc1ad71e32ba41392583e4936720f0152313d922c"}
Oct 30 19:42:21.471620 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:42:21.471590    4892 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w5-686689-openshift-default-5fbf5787bf-7b547"]
Oct 30 19:42:21.476628 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:42:21.476610    4892 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-686689-openshift-default-5fbf5787bf-7b547"
Oct 30 19:42:21.542699 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:42:21.542663    4892 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/098876de-0b44-493b-a9da-45280afe2cdf-istio-data\") pod \"e2e-worker-w5-686689-openshift-default-5fbf5787bf-7b547\" (UID: \"098876de-0b44-493b-a9da-45280afe2cdf\") " pod="openshift-ingress/e2e-worker-w5-686689-openshift-default-5fbf5787bf-7b547"
Oct 30 19:42:21.543000 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:42:21.542955    4892 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/098876de-0b44-493b-a9da-45280afe2cdf-workload-certs\") pod \"e2e-worker-w5-686689-openshift-default-5fbf5787bf-7b547\" (UID: \"098876de-0b44-493b-a9da-45280afe2cdf\") " pod="openshift-ingress/e2e-worker-w5-686689-openshift-default-5fbf5787bf-7b547"
Oct 30 19:42:21.543157 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:42:21.543140    4892 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/098876de-0b44-493b-a9da-45280afe2cdf-istio-podinfo\") pod \"e2e-worker-w5-686689-openshift-default-5fbf5787bf-7b547\" (UID: \"098876de-0b44-493b-a9da-45280afe2cdf\") " pod="openshift-ingress/e2e-worker-w5-686689-openshift-default-5fbf5787bf-7b547"
Oct 30 19:42:21.543264 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:42:21.543248    4892 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/098876de-0b44-493b-a9da-45280afe2cdf-workload-socket\") pod \"e2e-worker-w5-686689-openshift-default-5fbf5787bf-7b547\" (UID: \"098876de-0b44-493b-a9da-45280afe2cdf\") " pod="openshift-ingress/e2e-worker-w5-686689-openshift-default-5fbf5787bf-7b547"
Oct 30 19:42:21.543374 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:42:21.543360    4892 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/098876de-0b44-493b-a9da-45280afe2cdf-istio-envoy\") pod \"e2e-worker-w5-686689-openshift-default-5fbf5787bf-7b547\" (UID: \"098876de-0b44-493b-a9da-45280afe2cdf\") " pod="openshift-ingress/e2e-worker-w5-686689-openshift-default-5fbf5787bf-7b547"
Oct 30 19:42:21.543617 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:42:21.543599    4892 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/098876de-0b44-493b-a9da-45280afe2cdf-istio-token\") pod \"e2e-worker-w5-686689-openshift-default-5fbf5787bf-7b547\" (UID: \"098876de-0b44-493b-a9da-45280afe2cdf\") " pod="openshift-ingress/e2e-worker-w5-686689-openshift-default-5fbf5787bf-7b547"
Oct 30 19:42:21.543807 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:42:21.543783    4892 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/098876de-0b44-493b-a9da-45280afe2cdf-istiod-ca-cert\") pod \"e2e-worker-w5-686689-openshift-default-5fbf5787bf-7b547\" (UID: \"098876de-0b44-493b-a9da-45280afe2cdf\") " pod="openshift-ingress/e2e-worker-w5-686689-openshift-default-5fbf5787bf-7b547"
Oct 30 19:42:21.543956 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:42:21.543942    4892 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/098876de-0b44-493b-a9da-45280afe2cdf-credential-socket\") pod \"e2e-worker-w5-686689-openshift-default-5fbf5787bf-7b547\" (UID: \"098876de-0b44-493b-a9da-45280afe2cdf\") " pod="openshift-ingress/e2e-worker-w5-686689-openshift-default-5fbf5787bf-7b547"
Oct 30 19:42:21.544233 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:42:21.544202    4892 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-h4jt6\" (UniqueName: \"kubernetes.io/projected/098876de-0b44-493b-a9da-45280afe2cdf-kube-api-access-h4jt6\") pod \"e2e-worker-w5-686689-openshift-default-5fbf5787bf-7b547\" (UID: \"098876de-0b44-493b-a9da-45280afe2cdf\") " pod="openshift-ingress/e2e-worker-w5-686689-openshift-default-5fbf5787bf-7b547"
Oct 30 19:42:21.576141 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:42:21.576111    4892 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w5-686689-openshift-default-5fbf5787bf-7b547"]
Oct 30 19:42:21.637188 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:42:21.637161    4892 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e426ecf3-gfxx4"
Oct 30 19:42:21.645345 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:42:21.645321    4892 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/098876de-0b44-493b-a9da-45280afe2cdf-istio-token\") pod \"e2e-worker-w5-686689-openshift-default-5fbf5787bf-7b547\" (UID: \"098876de-0b44-493b-a9da-45280afe2cdf\") " pod="openshift-ingress/e2e-worker-w5-686689-openshift-default-5fbf5787bf-7b547"
Oct 30 19:42:21.645485 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:42:21.645385    4892 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/098876de-0b44-493b-a9da-45280afe2cdf-istiod-ca-cert\") pod \"e2e-worker-w5-686689-openshift-default-5fbf5787bf-7b547\" (UID: \"098876de-0b44-493b-a9da-45280afe2cdf\") " pod="openshift-ingress/e2e-worker-w5-686689-openshift-default-5fbf5787bf-7b547"
Oct 30 19:42:21.645485 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:42:21.645440    4892 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/098876de-0b44-493b-a9da-45280afe2cdf-credential-socket\") pod \"e2e-worker-w5-686689-openshift-default-5fbf5787bf-7b547\" (UID: \"098876de-0b44-493b-a9da-45280afe2cdf\") " pod="openshift-ingress/e2e-worker-w5-686689-openshift-default-5fbf5787bf-7b547"
Oct 30 19:42:21.645609 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:42:21.645505    4892 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-h4jt6\" (UniqueName: \"kubernetes.io/projected/098876de-0b44-493b-a9da-45280afe2cdf-kube-api-access-h4jt6\") pod \"e2e-worker-w5-686689-openshift-default-5fbf5787bf-7b547\" (UID: \"098876de-0b44-493b-a9da-45280afe2cdf\") " pod="openshift-ingress/e2e-worker-w5-686689-openshift-default-5fbf5787bf-7b547"
Oct 30 19:42:21.645609 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:42:21.645571    4892 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/098876de-0b44-493b-a9da-45280afe2cdf-istio-data\") pod \"e2e-worker-w5-686689-openshift-default-5fbf5787bf-7b547\" (UID: \"098876de-0b44-493b-a9da-45280afe2cdf\") " pod="openshift-ingress/e2e-worker-w5-686689-openshift-default-5fbf5787bf-7b547"
Oct 30 19:42:21.645714 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:42:21.645622    4892 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/098876de-0b44-493b-a9da-45280afe2cdf-workload-certs\") pod \"e2e-worker-w5-686689-openshift-default-5fbf5787bf-7b547\" (UID: \"098876de-0b44-493b-a9da-45280afe2cdf\") " pod="openshift-ingress/e2e-worker-w5-686689-openshift-default-5fbf5787bf-7b547"
Oct 30 19:42:21.645714 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:42:21.645657    4892 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/098876de-0b44-493b-a9da-45280afe2cdf-istio-podinfo\") pod \"e2e-worker-w5-686689-openshift-default-5fbf5787bf-7b547\" (UID: \"098876de-0b44-493b-a9da-45280afe2cdf\") " pod="openshift-ingress/e2e-worker-w5-686689-openshift-default-5fbf5787bf-7b547"
Oct 30 19:42:21.645714 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:42:21.645707    4892 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/098876de-0b44-493b-a9da-45280afe2cdf-workload-socket\") pod \"e2e-worker-w5-686689-openshift-default-5fbf5787bf-7b547\" (UID: \"098876de-0b44-493b-a9da-45280afe2cdf\") " pod="openshift-ingress/e2e-worker-w5-686689-openshift-default-5fbf5787bf-7b547"
Oct 30 19:42:21.645881 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:42:21.645746    4892 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/098876de-0b44-493b-a9da-45280afe2cdf-istio-envoy\") pod \"e2e-worker-w5-686689-openshift-default-5fbf5787bf-7b547\" (UID: \"098876de-0b44-493b-a9da-45280afe2cdf\") " pod="openshift-ingress/e2e-worker-w5-686689-openshift-default-5fbf5787bf-7b547"
Oct 30 19:42:21.645881 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:42:21.645858    4892 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/098876de-0b44-493b-a9da-45280afe2cdf-credential-socket\") pod \"e2e-worker-w5-686689-openshift-default-5fbf5787bf-7b547\" (UID: \"098876de-0b44-493b-a9da-45280afe2cdf\") " pod="openshift-ingress/e2e-worker-w5-686689-openshift-default-5fbf5787bf-7b547"
Oct 30 19:42:21.646055 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:42:21.646033    4892 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/098876de-0b44-493b-a9da-45280afe2cdf-istio-data\") pod \"e2e-worker-w5-686689-openshift-default-5fbf5787bf-7b547\" (UID: \"098876de-0b44-493b-a9da-45280afe2cdf\") " pod="openshift-ingress/e2e-worker-w5-686689-openshift-default-5fbf5787bf-7b547"
Oct 30 19:42:21.646210 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:42:21.646190    4892 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/098876de-0b44-493b-a9da-45280afe2cdf-workload-certs\") pod \"e2e-worker-w5-686689-openshift-default-5fbf5787bf-7b547\" (UID: \"098876de-0b44-493b-a9da-45280afe2cdf\") " pod="openshift-ingress/e2e-worker-w5-686689-openshift-default-5fbf5787bf-7b547"
Oct 30 19:42:21.646505 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:42:21.646482    4892 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/098876de-0b44-493b-a9da-45280afe2cdf-workload-socket\") pod \"e2e-worker-w5-686689-openshift-default-5fbf5787bf-7b547\" (UID: \"098876de-0b44-493b-a9da-45280afe2cdf\") " pod="openshift-ingress/e2e-worker-w5-686689-openshift-default-5fbf5787bf-7b547"
Oct 30 19:42:21.646769 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:42:21.646555    4892 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/098876de-0b44-493b-a9da-45280afe2cdf-istiod-ca-cert\") pod \"e2e-worker-w5-686689-openshift-default-5fbf5787bf-7b547\" (UID: \"098876de-0b44-493b-a9da-45280afe2cdf\") " pod="openshift-ingress/e2e-worker-w5-686689-openshift-default-5fbf5787bf-7b547"
Oct 30 19:42:21.648224 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:42:21.648200    4892 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/098876de-0b44-493b-a9da-45280afe2cdf-istio-envoy\") pod \"e2e-worker-w5-686689-openshift-default-5fbf5787bf-7b547\" (UID: \"098876de-0b44-493b-a9da-45280afe2cdf\") " pod="openshift-ingress/e2e-worker-w5-686689-openshift-default-5fbf5787bf-7b547"
Oct 30 19:42:21.648471 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:42:21.648454    4892 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/098876de-0b44-493b-a9da-45280afe2cdf-istio-podinfo\") pod \"e2e-worker-w5-686689-openshift-default-5fbf5787bf-7b547\" (UID: \"098876de-0b44-493b-a9da-45280afe2cdf\") " pod="openshift-ingress/e2e-worker-w5-686689-openshift-default-5fbf5787bf-7b547"
Oct 30 19:42:21.702066 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:42:21.702003    4892 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/098876de-0b44-493b-a9da-45280afe2cdf-istio-token\") pod \"e2e-worker-w5-686689-openshift-default-5fbf5787bf-7b547\" (UID: \"098876de-0b44-493b-a9da-45280afe2cdf\") " pod="openshift-ingress/e2e-worker-w5-686689-openshift-default-5fbf5787bf-7b547"
Oct 30 19:42:21.702219 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:42:21.702200    4892 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-h4jt6\" (UniqueName: \"kubernetes.io/projected/098876de-0b44-493b-a9da-45280afe2cdf-kube-api-access-h4jt6\") pod \"e2e-worker-w5-686689-openshift-default-5fbf5787bf-7b547\" (UID: \"098876de-0b44-493b-a9da-45280afe2cdf\") " pod="openshift-ingress/e2e-worker-w5-686689-openshift-default-5fbf5787bf-7b547"
Oct 30 19:42:21.746905 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:42:21.746884    4892 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/6f5154a2-9a70-4ccc-8584-a2ce72ba6564-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6f5154a2-9a70-4ccc-8584-a2ce72ba6564-maas-api-openshift-ca\") pod \"6f5154a2-9a70-4ccc-8584-a2ce72ba6564\" (UID: \"6f5154a2-9a70-4ccc-8584-a2ce72ba6564\") "
Oct 30 19:42:21.747030 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:42:21.746952    4892 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/6f5154a2-9a70-4ccc-8584-a2ce72ba6564-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6f5154a2-9a70-4ccc-8584-a2ce72ba6564-maas-api-xks-ca\") pod \"6f5154a2-9a70-4ccc-8584-a2ce72ba6564\" (UID: \"6f5154a2-9a70-4ccc-8584-a2ce72ba6564\") "
Oct 30 19:42:21.747094 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:42:21.747081    4892 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6f5154a2-9a70-4ccc-8584-a2ce72ba6564-kube-api-access-5sbtb\" (UniqueName: \"kubernetes.io/projected/6f5154a2-9a70-4ccc-8584-a2ce72ba6564-kube-api-access-5sbtb\") pod \"6f5154a2-9a70-4ccc-8584-a2ce72ba6564\" (UID: \"6f5154a2-9a70-4ccc-8584-a2ce72ba6564\") "
Oct 30 19:42:21.747212 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:42:21.747192    4892 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6f5154a2-9a70-4ccc-8584-a2ce72ba6564-maas-api-openshift-ca" pod "6f5154a2-9a70-4ccc-8584-a2ce72ba6564" (UID: "6f5154a2-9a70-4ccc-8584-a2ce72ba6564"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 19:42:21.747372 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:42:21.747346    4892 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6f5154a2-9a70-4ccc-8584-a2ce72ba6564-maas-api-openshift-ca\") on node \"ip-10-0-132-82.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 19:42:21.748896 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:42:21.748869    4892 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6f5154a2-9a70-4ccc-8584-a2ce72ba6564-maas-api-xks-ca" pod "6f5154a2-9a70-4ccc-8584-a2ce72ba6564" (UID: "6f5154a2-9a70-4ccc-8584-a2ce72ba6564"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 19:42:21.749102 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:42:21.749083    4892 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6f5154a2-9a70-4ccc-8584-a2ce72ba6564-kube-api-access-5sbtb" pod "6f5154a2-9a70-4ccc-8584-a2ce72ba6564" (UID: "6f5154a2-9a70-4ccc-8584-a2ce72ba6564"). InnerVolumeSpecName "kube-api-access-5sbtb". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:42:21.792106 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:42:21.792080    4892 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-686689-openshift-default-5fbf5787bf-7b547"
Oct 30 19:42:21.853636 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:42:21.853601    4892 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5sbtb\" (UniqueName: \"kubernetes.io/projected/6f5154a2-9a70-4ccc-8584-a2ce72ba6564-kube-api-access-5sbtb\") on node \"ip-10-0-132-82.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 19:42:21.853785 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:42:21.853639    4892 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6f5154a2-9a70-4ccc-8584-a2ce72ba6564-maas-api-xks-ca\") on node \"ip-10-0-132-82.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 19:42:22.224677 ip-10-0-132-82 kubenswrapper[4892]: W1030 19:42:22.224635    4892 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod098876de_0b44_493b_a9da_45280afe2cdf.slice/crio-c56a324eb3f4c166a59bb3aa1c32c0d3a36e213fc254f9f5025a61e6454b8a55 WatchSource:0}: Error finding container c56a324eb3f4c166a59bb3aa1c32c0d3a36e213fc254f9f5025a61e6454b8a55: Status 404 returned error can't find the container with id c56a324eb3f4c166a59bb3aa1c32c0d3a36e213fc254f9f5025a61e6454b8a55
Oct 30 19:42:22.225291 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:42:22.225223    4892 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w5-686689-openshift-default-5fbf5787bf-7b547"]
Oct 30 19:42:22.227133 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:42:22.227099    4892 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892000Ki","pods":"250"}
Oct 30 19:42:22.227254 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:42:22.227180    4892 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892000Ki","pods":"250"}
Oct 30 19:42:22.227254 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:42:22.227224    4892 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892000Ki","pods":"250"}
Oct 30 19:42:22.460209 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:42:22.460174    4892 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e426ecf3-gfxx4"
Oct 30 19:42:22.460379 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:42:22.460214    4892 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e426ecf3-gfxx4" event={"ID":"6f5154a2-9a70-4ccc-8584-a2ce72ba6564","Type":"ContainerDied","Data":"6dda13bb325860d2a0cde7f4b19548ba9dbba84dafe099a09dea323a698293f7"}
Oct 30 19:42:22.460379 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:42:22.460268    4892 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="6dda13bb325860d2a0cde7f4b19548ba9dbba84dafe099a09dea323a698293f7"
Oct 30 19:42:22.474033 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:42:22.474003    4892 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-686689-openshift-default-5fbf5787bf-7b547" event={"ID":"098876de-0b44-493b-a9da-45280afe2cdf","Type":"ContainerStarted","Data":"ef2b5070ac0677e6a68b0547d89df8e41c1a7d13082ad109f63614b9a9d6c507"}
Oct 30 19:42:22.474436 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:42:22.474076    4892 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-686689-openshift-default-5fbf5787bf-7b547" event={"ID":"098876de-0b44-493b-a9da-45280afe2cdf","Type":"ContainerStarted","Data":"c56a324eb3f4c166a59bb3aa1c32c0d3a36e213fc254f9f5025a61e6454b8a55"}
Oct 30 19:42:22.643770 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:42:22.643717    4892 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w5-686689-openshift-default-5fbf5787bf-7b547" podStartSLOduration=1.643702676 podStartE2EDuration="1.643702676s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:42:21 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:42:22.640709403 +0000 UTC m=+1590.211175023" watchObservedRunningTime="2026-10-30 19:42:22.643702676 +0000 UTC m=+1590.214168274"
Oct 30 19:42:22.768414 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:42:22.768333    4892 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-10bbe06e-wwv8x"]
Oct 30 19:42:22.768847 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:42:22.768834    4892 cpu_manager.go:414] "containerMap: removing container" podUID="6f5154a2-9a70-4ccc-8584-a2ce72ba6564" containerName="revoke-keys"
Oct 30 19:42:22.768885 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:42:22.768848    4892 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6f5154a2-9a70-4ccc-8584-a2ce72ba6564" containerName="revoke-keys"
Oct 30 19:42:22.768991 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:42:22.768979    4892 memory_manager.go:365] "RemoveStaleState removing state" podUID="6f5154a2-9a70-4ccc-8584-a2ce72ba6564" containerName="revoke-keys"
Oct 30 19:42:22.772591 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:42:22.772575    4892 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-10bbe06e-wwv8x"
Oct 30 19:42:22.792321 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:42:22.792298    4892 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w5-686689-openshift-default-5fbf5787bf-7b547"
Oct 30 19:42:22.824945 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:42:22.824919    4892 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 30 19:42:22.825114 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:42:22.825031    4892 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 19:42:22.825114 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:42:22.825041    4892 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 30 19:42:22.825269 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:42:22.825252    4892 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-nlxp4\""
Oct 30 19:42:22.867148 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:42:22.866597    4892 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-10bbe06e-wwv8x"]
Oct 30 19:42:22.867148 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:42:22.866735    4892 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/da70be07-9754-47d1-95bc-a5970bf9ecae-maas-api-xks-ca\") pod \"maas-api-revoke-sub-10bbe06e-wwv8x\" (UID: \"da70be07-9754-47d1-95bc-a5970bf9ecae\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-10bbe06e-wwv8x"
Oct 30 19:42:22.867148 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:42:22.866848    4892 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/da70be07-9754-47d1-95bc-a5970bf9ecae-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-10bbe06e-wwv8x\" (UID: \"da70be07-9754-47d1-95bc-a5970bf9ecae\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-10bbe06e-wwv8x"
Oct 30 19:42:22.867148 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:42:22.866989    4892 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jv64c\" (UniqueName: \"kubernetes.io/projected/da70be07-9754-47d1-95bc-a5970bf9ecae-kube-api-access-jv64c\") pod \"maas-api-revoke-sub-10bbe06e-wwv8x\" (UID: \"da70be07-9754-47d1-95bc-a5970bf9ecae\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-10bbe06e-wwv8x"
Oct 30 19:42:22.968320 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:42:22.968284    4892 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jv64c\" (UniqueName: \"kubernetes.io/projected/da70be07-9754-47d1-95bc-a5970bf9ecae-kube-api-access-jv64c\") pod \"maas-api-revoke-sub-10bbe06e-wwv8x\" (UID: \"da70be07-9754-47d1-95bc-a5970bf9ecae\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-10bbe06e-wwv8x"
Oct 30 19:42:22.968500 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:42:22.968381    4892 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/da70be07-9754-47d1-95bc-a5970bf9ecae-maas-api-xks-ca\") pod \"maas-api-revoke-sub-10bbe06e-wwv8x\" (UID: \"da70be07-9754-47d1-95bc-a5970bf9ecae\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-10bbe06e-wwv8x"
Oct 30 19:42:22.968500 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:42:22.968448    4892 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/da70be07-9754-47d1-95bc-a5970bf9ecae-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-10bbe06e-wwv8x\" (UID: \"da70be07-9754-47d1-95bc-a5970bf9ecae\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-10bbe06e-wwv8x"
Oct 30 19:42:22.969357 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:42:22.969294    4892 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/da70be07-9754-47d1-95bc-a5970bf9ecae-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-10bbe06e-wwv8x\" (UID: \"da70be07-9754-47d1-95bc-a5970bf9ecae\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-10bbe06e-wwv8x"
Oct 30 19:42:22.975828 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:42:22.975797    4892 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/da70be07-9754-47d1-95bc-a5970bf9ecae-maas-api-xks-ca\") pod \"maas-api-revoke-sub-10bbe06e-wwv8x\" (UID: \"da70be07-9754-47d1-95bc-a5970bf9ecae\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-10bbe06e-wwv8x"
Oct 30 19:42:23.025008 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:42:23.024923    4892 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jv64c\" (UniqueName: \"kubernetes.io/projected/da70be07-9754-47d1-95bc-a5970bf9ecae-kube-api-access-jv64c\") pod \"maas-api-revoke-sub-10bbe06e-wwv8x\" (UID: \"da70be07-9754-47d1-95bc-a5970bf9ecae\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-10bbe06e-wwv8x"
Oct 30 19:42:23.086223 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:42:23.086197    4892 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-10bbe06e-wwv8x"
Oct 30 19:42:23.496284 ip-10-0-132-82 kubenswrapper[4892]: W1030 19:42:23.496256    4892 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podda70be07_9754_47d1_95bc_a5970bf9ecae.slice/crio-3a6f8fec34e6b83655ca4f9b365a91752dc9f2f22511a97b3ce7c0514df3e6e3 WatchSource:0}: Error finding container 3a6f8fec34e6b83655ca4f9b365a91752dc9f2f22511a97b3ce7c0514df3e6e3: Status 404 returned error can't find the container with id 3a6f8fec34e6b83655ca4f9b365a91752dc9f2f22511a97b3ce7c0514df3e6e3
Oct 30 19:42:23.496926 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:42:23.496877    4892 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-10bbe06e-wwv8x"]
Oct 30 19:42:23.793935 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:42:23.793897    4892 patch_prober.go:29] interesting pod/e2e-worker-w5-686689-openshift-default-5fbf5787bf-7b547 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.114:15021/healthz/ready\": dial tcp 10.132.0.114:15021: connect: connection refused" start-of-body=
Oct 30 19:42:23.794081 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:42:23.793992    4892 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-686689-openshift-default-5fbf5787bf-7b547" podUID="098876de-0b44-493b-a9da-45280afe2cdf" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.114:15021/healthz/ready\": dial tcp 10.132.0.114:15021: connect: connection refused"
Oct 30 19:42:24.490736 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:42:24.490697    4892 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-10bbe06e-wwv8x" event={"ID":"da70be07-9754-47d1-95bc-a5970bf9ecae","Type":"ContainerStarted","Data":"ac9164aca539e0063140988dfbb316258f71d0216817fee8518d905b8db147c2"}
Oct 30 19:42:24.490916 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:42:24.490759    4892 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-10bbe06e-wwv8x" event={"ID":"da70be07-9754-47d1-95bc-a5970bf9ecae","Type":"ContainerStarted","Data":"3a6f8fec34e6b83655ca4f9b365a91752dc9f2f22511a97b3ce7c0514df3e6e3"}
Oct 30 19:42:24.611626 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:42:24.611565    4892 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-10bbe06e-wwv8x" podStartSLOduration=2.611550835 podStartE2EDuration="2.611550835s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:42:22 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:42:24.607991076 +0000 UTC m=+1592.178456669" watchObservedRunningTime="2026-10-30 19:42:24.611550835 +0000 UTC m=+1592.182016433"
Oct 30 19:42:24.792946 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:42:24.792857    4892 patch_prober.go:29] interesting pod/e2e-worker-w5-686689-openshift-default-5fbf5787bf-7b547 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.114:15021/healthz/ready\": dial tcp 10.132.0.114:15021: connect: connection refused" start-of-body=
Oct 30 19:42:24.792946 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:42:24.792920    4892 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-686689-openshift-default-5fbf5787bf-7b547" podUID="098876de-0b44-493b-a9da-45280afe2cdf" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.114:15021/healthz/ready\": dial tcp 10.132.0.114:15021: connect: connection refused"
Oct 30 19:42:24.875209 ip-10-0-132-82 kubenswrapper[4892]: E1030 19:42:24.875168    4892 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-jrm87" containerName="extractor"
Oct 30 19:42:25.501228 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:42:25.501194    4892 generic.go:356] "Generic (PLEG): container finished" podID="da70be07-9754-47d1-95bc-a5970bf9ecae" containerID="ac9164aca539e0063140988dfbb316258f71d0216817fee8518d905b8db147c2" exitCode=0
Oct 30 19:42:25.501429 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:42:25.501268    4892 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-10bbe06e-wwv8x" event={"ID":"da70be07-9754-47d1-95bc-a5970bf9ecae","Type":"ContainerDied","Data":"ac9164aca539e0063140988dfbb316258f71d0216817fee8518d905b8db147c2"}
Oct 30 19:42:25.792958 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:42:25.792872    4892 patch_prober.go:29] interesting pod/e2e-worker-w5-686689-openshift-default-5fbf5787bf-7b547 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.114:15021/healthz/ready\": dial tcp 10.132.0.114:15021: connect: connection refused" start-of-body=
Oct 30 19:42:25.792958 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:42:25.792932    4892 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-686689-openshift-default-5fbf5787bf-7b547" podUID="098876de-0b44-493b-a9da-45280afe2cdf" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.114:15021/healthz/ready\": dial tcp 10.132.0.114:15021: connect: connection refused"
Oct 30 19:42:25.915138 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:42:25.915105    4892 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w4-b5178f-openshift-default-c8fbdff8c-wgm46"]
Oct 30 19:42:25.919378 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:42:25.919357    4892 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-b5178f-openshift-default-c8fbdff8c-wgm46"
Oct 30 19:42:26.004514 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:42:26.004431    4892 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4fwdg\" (UniqueName: \"kubernetes.io/projected/0bdf7217-4cc3-4710-b13e-ffcee12febf0-kube-api-access-4fwdg\") pod \"e2e-worker-w4-b5178f-openshift-default-c8fbdff8c-wgm46\" (UID: \"0bdf7217-4cc3-4710-b13e-ffcee12febf0\") " pod="openshift-ingress/e2e-worker-w4-b5178f-openshift-default-c8fbdff8c-wgm46"
Oct 30 19:42:26.004696 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:42:26.004579    4892 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0bdf7217-4cc3-4710-b13e-ffcee12febf0-istio-token\") pod \"e2e-worker-w4-b5178f-openshift-default-c8fbdff8c-wgm46\" (UID: \"0bdf7217-4cc3-4710-b13e-ffcee12febf0\") " pod="openshift-ingress/e2e-worker-w4-b5178f-openshift-default-c8fbdff8c-wgm46"
Oct 30 19:42:26.004696 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:42:26.004672    4892 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0bdf7217-4cc3-4710-b13e-ffcee12febf0-workload-socket\") pod \"e2e-worker-w4-b5178f-openshift-default-c8fbdff8c-wgm46\" (UID: \"0bdf7217-4cc3-4710-b13e-ffcee12febf0\") " pod="openshift-ingress/e2e-worker-w4-b5178f-openshift-default-c8fbdff8c-wgm46"
Oct 30 19:42:26.004804 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:42:26.004779    4892 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0bdf7217-4cc3-4710-b13e-ffcee12febf0-istio-data\") pod \"e2e-worker-w4-b5178f-openshift-default-c8fbdff8c-wgm46\" (UID: \"0bdf7217-4cc3-4710-b13e-ffcee12febf0\") " pod="openshift-ingress/e2e-worker-w4-b5178f-openshift-default-c8fbdff8c-wgm46"
Oct 30 19:42:26.004848 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:42:26.004818    4892 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0bdf7217-4cc3-4710-b13e-ffcee12febf0-istiod-ca-cert\") pod \"e2e-worker-w4-b5178f-openshift-default-c8fbdff8c-wgm46\" (UID: \"0bdf7217-4cc3-4710-b13e-ffcee12febf0\") " pod="openshift-ingress/e2e-worker-w4-b5178f-openshift-default-c8fbdff8c-wgm46"
Oct 30 19:42:26.004935 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:42:26.004921    4892 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0bdf7217-4cc3-4710-b13e-ffcee12febf0-istio-podinfo\") pod \"e2e-worker-w4-b5178f-openshift-default-c8fbdff8c-wgm46\" (UID: \"0bdf7217-4cc3-4710-b13e-ffcee12febf0\") " pod="openshift-ingress/e2e-worker-w4-b5178f-openshift-default-c8fbdff8c-wgm46"
Oct 30 19:42:26.005029 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:42:26.005015    4892 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0bdf7217-4cc3-4710-b13e-ffcee12febf0-workload-certs\") pod \"e2e-worker-w4-b5178f-openshift-default-c8fbdff8c-wgm46\" (UID: \"0bdf7217-4cc3-4710-b13e-ffcee12febf0\") " pod="openshift-ingress/e2e-worker-w4-b5178f-openshift-default-c8fbdff8c-wgm46"
Oct 30 19:42:26.005083 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:42:26.005043    4892 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0bdf7217-4cc3-4710-b13e-ffcee12febf0-istio-envoy\") pod \"e2e-worker-w4-b5178f-openshift-default-c8fbdff8c-wgm46\" (UID: \"0bdf7217-4cc3-4710-b13e-ffcee12febf0\") " pod="openshift-ingress/e2e-worker-w4-b5178f-openshift-default-c8fbdff8c-wgm46"
Oct 30 19:42:26.005083 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:42:26.005077    4892 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0bdf7217-4cc3-4710-b13e-ffcee12febf0-credential-socket\") pod \"e2e-worker-w4-b5178f-openshift-default-c8fbdff8c-wgm46\" (UID: \"0bdf7217-4cc3-4710-b13e-ffcee12febf0\") " pod="openshift-ingress/e2e-worker-w4-b5178f-openshift-default-c8fbdff8c-wgm46"
Oct 30 19:42:26.106792 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:42:26.106718    4892 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0bdf7217-4cc3-4710-b13e-ffcee12febf0-istio-podinfo\") pod \"e2e-worker-w4-b5178f-openshift-default-c8fbdff8c-wgm46\" (UID: \"0bdf7217-4cc3-4710-b13e-ffcee12febf0\") " pod="openshift-ingress/e2e-worker-w4-b5178f-openshift-default-c8fbdff8c-wgm46"
Oct 30 19:42:26.106792 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:42:26.106787    4892 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0bdf7217-4cc3-4710-b13e-ffcee12febf0-workload-certs\") pod \"e2e-worker-w4-b5178f-openshift-default-c8fbdff8c-wgm46\" (UID: \"0bdf7217-4cc3-4710-b13e-ffcee12febf0\") " pod="openshift-ingress/e2e-worker-w4-b5178f-openshift-default-c8fbdff8c-wgm46"
Oct 30 19:42:26.107015 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:42:26.106818    4892 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0bdf7217-4cc3-4710-b13e-ffcee12febf0-istio-envoy\") pod \"e2e-worker-w4-b5178f-openshift-default-c8fbdff8c-wgm46\" (UID: \"0bdf7217-4cc3-4710-b13e-ffcee12febf0\") " pod="openshift-ingress/e2e-worker-w4-b5178f-openshift-default-c8fbdff8c-wgm46"
Oct 30 19:42:26.107015 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:42:26.106859    4892 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0bdf7217-4cc3-4710-b13e-ffcee12febf0-credential-socket\") pod \"e2e-worker-w4-b5178f-openshift-default-c8fbdff8c-wgm46\" (UID: \"0bdf7217-4cc3-4710-b13e-ffcee12febf0\") " pod="openshift-ingress/e2e-worker-w4-b5178f-openshift-default-c8fbdff8c-wgm46"
Oct 30 19:42:26.107015 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:42:26.106923    4892 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4fwdg\" (UniqueName: \"kubernetes.io/projected/0bdf7217-4cc3-4710-b13e-ffcee12febf0-kube-api-access-4fwdg\") pod \"e2e-worker-w4-b5178f-openshift-default-c8fbdff8c-wgm46\" (UID: \"0bdf7217-4cc3-4710-b13e-ffcee12febf0\") " pod="openshift-ingress/e2e-worker-w4-b5178f-openshift-default-c8fbdff8c-wgm46"
Oct 30 19:42:26.107015 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:42:26.106977    4892 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0bdf7217-4cc3-4710-b13e-ffcee12febf0-istio-token\") pod \"e2e-worker-w4-b5178f-openshift-default-c8fbdff8c-wgm46\" (UID: \"0bdf7217-4cc3-4710-b13e-ffcee12febf0\") " pod="openshift-ingress/e2e-worker-w4-b5178f-openshift-default-c8fbdff8c-wgm46"
Oct 30 19:42:26.107233 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:42:26.107090    4892 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0bdf7217-4cc3-4710-b13e-ffcee12febf0-workload-socket\") pod \"e2e-worker-w4-b5178f-openshift-default-c8fbdff8c-wgm46\" (UID: \"0bdf7217-4cc3-4710-b13e-ffcee12febf0\") " pod="openshift-ingress/e2e-worker-w4-b5178f-openshift-default-c8fbdff8c-wgm46"
Oct 30 19:42:26.107233 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:42:26.107135    4892 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0bdf7217-4cc3-4710-b13e-ffcee12febf0-istio-data\") pod \"e2e-worker-w4-b5178f-openshift-default-c8fbdff8c-wgm46\" (UID: \"0bdf7217-4cc3-4710-b13e-ffcee12febf0\") " pod="openshift-ingress/e2e-worker-w4-b5178f-openshift-default-c8fbdff8c-wgm46"
Oct 30 19:42:26.107233 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:42:26.107159    4892 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0bdf7217-4cc3-4710-b13e-ffcee12febf0-istiod-ca-cert\") pod \"e2e-worker-w4-b5178f-openshift-default-c8fbdff8c-wgm46\" (UID: \"0bdf7217-4cc3-4710-b13e-ffcee12febf0\") " pod="openshift-ingress/e2e-worker-w4-b5178f-openshift-default-c8fbdff8c-wgm46"
Oct 30 19:42:26.107383 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:42:26.107363    4892 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0bdf7217-4cc3-4710-b13e-ffcee12febf0-workload-certs\") pod \"e2e-worker-w4-b5178f-openshift-default-c8fbdff8c-wgm46\" (UID: \"0bdf7217-4cc3-4710-b13e-ffcee12febf0\") " pod="openshift-ingress/e2e-worker-w4-b5178f-openshift-default-c8fbdff8c-wgm46"
Oct 30 19:42:26.107621 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:42:26.107597    4892 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0bdf7217-4cc3-4710-b13e-ffcee12febf0-istio-data\") pod \"e2e-worker-w4-b5178f-openshift-default-c8fbdff8c-wgm46\" (UID: \"0bdf7217-4cc3-4710-b13e-ffcee12febf0\") " pod="openshift-ingress/e2e-worker-w4-b5178f-openshift-default-c8fbdff8c-wgm46"
Oct 30 19:42:26.107715 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:42:26.107666    4892 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0bdf7217-4cc3-4710-b13e-ffcee12febf0-credential-socket\") pod \"e2e-worker-w4-b5178f-openshift-default-c8fbdff8c-wgm46\" (UID: \"0bdf7217-4cc3-4710-b13e-ffcee12febf0\") " pod="openshift-ingress/e2e-worker-w4-b5178f-openshift-default-c8fbdff8c-wgm46"
Oct 30 19:42:26.107896 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:42:26.107876    4892 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0bdf7217-4cc3-4710-b13e-ffcee12febf0-workload-socket\") pod \"e2e-worker-w4-b5178f-openshift-default-c8fbdff8c-wgm46\" (UID: \"0bdf7217-4cc3-4710-b13e-ffcee12febf0\") " pod="openshift-ingress/e2e-worker-w4-b5178f-openshift-default-c8fbdff8c-wgm46"
Oct 30 19:42:26.108029 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:42:26.108008    4892 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0bdf7217-4cc3-4710-b13e-ffcee12febf0-istiod-ca-cert\") pod \"e2e-worker-w4-b5178f-openshift-default-c8fbdff8c-wgm46\" (UID: \"0bdf7217-4cc3-4710-b13e-ffcee12febf0\") " pod="openshift-ingress/e2e-worker-w4-b5178f-openshift-default-c8fbdff8c-wgm46"
Oct 30 19:42:26.109929 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:42:26.109904    4892 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0bdf7217-4cc3-4710-b13e-ffcee12febf0-istio-envoy\") pod \"e2e-worker-w4-b5178f-openshift-default-c8fbdff8c-wgm46\" (UID: \"0bdf7217-4cc3-4710-b13e-ffcee12febf0\") " pod="openshift-ingress/e2e-worker-w4-b5178f-openshift-default-c8fbdff8c-wgm46"
Oct 30 19:42:26.110031 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:42:26.110004    4892 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0bdf7217-4cc3-4710-b13e-ffcee12febf0-istio-podinfo\") pod \"e2e-worker-w4-b5178f-openshift-default-c8fbdff8c-wgm46\" (UID: \"0bdf7217-4cc3-4710-b13e-ffcee12febf0\") " pod="openshift-ingress/e2e-worker-w4-b5178f-openshift-default-c8fbdff8c-wgm46"
Oct 30 19:42:26.164074 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:42:26.164036    4892 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0bdf7217-4cc3-4710-b13e-ffcee12febf0-istio-token\") pod \"e2e-worker-w4-b5178f-openshift-default-c8fbdff8c-wgm46\" (UID: \"0bdf7217-4cc3-4710-b13e-ffcee12febf0\") " pod="openshift-ingress/e2e-worker-w4-b5178f-openshift-default-c8fbdff8c-wgm46"
Oct 30 19:42:26.164213 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:42:26.164105    4892 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4fwdg\" (UniqueName: \"kubernetes.io/projected/0bdf7217-4cc3-4710-b13e-ffcee12febf0-kube-api-access-4fwdg\") pod \"e2e-worker-w4-b5178f-openshift-default-c8fbdff8c-wgm46\" (UID: \"0bdf7217-4cc3-4710-b13e-ffcee12febf0\") " pod="openshift-ingress/e2e-worker-w4-b5178f-openshift-default-c8fbdff8c-wgm46"
Oct 30 19:42:26.252990 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:42:26.231018    4892 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-b5178f-openshift-default-c8fbdff8c-wgm46"]
Oct 30 19:42:26.270259 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:42:26.270213    4892 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-b5178f-openshift-default-c8fbdff8c-wgm46"
Oct 30 19:42:26.706656 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:42:26.706630    4892 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-10bbe06e-wwv8x"
Oct 30 19:42:26.741395 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:42:26.741363    4892 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/da70be07-9754-47d1-95bc-a5970bf9ecae-kube-api-access-jv64c\" (UniqueName: \"kubernetes.io/projected/da70be07-9754-47d1-95bc-a5970bf9ecae-kube-api-access-jv64c\") pod \"da70be07-9754-47d1-95bc-a5970bf9ecae\" (UID: \"da70be07-9754-47d1-95bc-a5970bf9ecae\") "
Oct 30 19:42:26.751989 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:42:26.741683    4892 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/da70be07-9754-47d1-95bc-a5970bf9ecae-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/da70be07-9754-47d1-95bc-a5970bf9ecae-maas-api-openshift-ca\") pod \"da70be07-9754-47d1-95bc-a5970bf9ecae\" (UID: \"da70be07-9754-47d1-95bc-a5970bf9ecae\") "
Oct 30 19:42:26.752266 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:42:26.746608    4892 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/da70be07-9754-47d1-95bc-a5970bf9ecae-maas-api-openshift-ca" pod "da70be07-9754-47d1-95bc-a5970bf9ecae" (UID: "da70be07-9754-47d1-95bc-a5970bf9ecae"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 19:42:26.752388 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:42:26.752364    4892 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/da70be07-9754-47d1-95bc-a5970bf9ecae-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/da70be07-9754-47d1-95bc-a5970bf9ecae-maas-api-xks-ca\") pod \"da70be07-9754-47d1-95bc-a5970bf9ecae\" (UID: \"da70be07-9754-47d1-95bc-a5970bf9ecae\") "
Oct 30 19:42:26.752932 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:42:26.752909    4892 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/da70be07-9754-47d1-95bc-a5970bf9ecae-maas-api-openshift-ca\") on node \"ip-10-0-132-82.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 19:42:26.753442 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:42:26.753402    4892 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/da70be07-9754-47d1-95bc-a5970bf9ecae-kube-api-access-jv64c" pod "da70be07-9754-47d1-95bc-a5970bf9ecae" (UID: "da70be07-9754-47d1-95bc-a5970bf9ecae"). InnerVolumeSpecName "kube-api-access-jv64c". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:42:26.754487 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:42:26.754469    4892 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/da70be07-9754-47d1-95bc-a5970bf9ecae-maas-api-xks-ca" pod "da70be07-9754-47d1-95bc-a5970bf9ecae" (UID: "da70be07-9754-47d1-95bc-a5970bf9ecae"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 19:42:26.784873 ip-10-0-132-82 kubenswrapper[4892]: W1030 19:42:26.784850    4892 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0bdf7217_4cc3_4710_b13e_ffcee12febf0.slice/crio-0962651f2bd8b1d3e0e38fc296559bb9edaa48e70811a27a9457bf15b133f212 WatchSource:0}: Error finding container 0962651f2bd8b1d3e0e38fc296559bb9edaa48e70811a27a9457bf15b133f212: Status 404 returned error can't find the container with id 0962651f2bd8b1d3e0e38fc296559bb9edaa48e70811a27a9457bf15b133f212
Oct 30 19:42:26.787275 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:42:26.787245    4892 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892000Ki","pods":"250"}
Oct 30 19:42:26.787351 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:42:26.787314    4892 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892000Ki","pods":"250"}
Oct 30 19:42:26.787351 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:42:26.787346    4892 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892000Ki","pods":"250"}
Oct 30 19:42:26.792738 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:42:26.792718    4892 patch_prober.go:29] interesting pod/e2e-worker-w5-686689-openshift-default-5fbf5787bf-7b547 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.114:15021/healthz/ready\": dial tcp 10.132.0.114:15021: connect: connection refused" start-of-body=
Oct 30 19:42:26.792818 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:42:26.792756    4892 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-686689-openshift-default-5fbf5787bf-7b547" podUID="098876de-0b44-493b-a9da-45280afe2cdf" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.114:15021/healthz/ready\": dial tcp 10.132.0.114:15021: connect: connection refused"
Oct 30 19:42:26.812400 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:42:26.812380    4892 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-b5178f-openshift-default-c8fbdff8c-wgm46"]
Oct 30 19:42:26.857733 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:42:26.853993    4892 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jv64c\" (UniqueName: \"kubernetes.io/projected/da70be07-9754-47d1-95bc-a5970bf9ecae-kube-api-access-jv64c\") on node \"ip-10-0-132-82.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 19:42:26.857733 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:42:26.854029    4892 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/da70be07-9754-47d1-95bc-a5970bf9ecae-maas-api-xks-ca\") on node \"ip-10-0-132-82.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 19:42:27.519608 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:42:27.519564    4892 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-b5178f-openshift-default-c8fbdff8c-wgm46" event={"ID":"0bdf7217-4cc3-4710-b13e-ffcee12febf0","Type":"ContainerStarted","Data":"3dd4fa0d82cb56258cb9c53b4db22f21fb21a22a306f2079be7486ee58300540"}
Oct 30 19:42:27.519810 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:42:27.519630    4892 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-b5178f-openshift-default-c8fbdff8c-wgm46" event={"ID":"0bdf7217-4cc3-4710-b13e-ffcee12febf0","Type":"ContainerStarted","Data":"0962651f2bd8b1d3e0e38fc296559bb9edaa48e70811a27a9457bf15b133f212"}
Oct 30 19:42:27.521010 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:42:27.520990    4892 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-10bbe06e-wwv8x"
Oct 30 19:42:27.521100 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:42:27.521021    4892 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-10bbe06e-wwv8x" event={"ID":"da70be07-9754-47d1-95bc-a5970bf9ecae","Type":"ContainerDied","Data":"3a6f8fec34e6b83655ca4f9b365a91752dc9f2f22511a97b3ce7c0514df3e6e3"}
Oct 30 19:42:27.521100 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:42:27.521065    4892 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="3a6f8fec34e6b83655ca4f9b365a91752dc9f2f22511a97b3ce7c0514df3e6e3"
Oct 30 19:42:27.663176 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:42:27.663107    4892 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w4-b5178f-openshift-default-c8fbdff8c-wgm46" podStartSLOduration=2.6630887850000002 podStartE2EDuration="2.663088785s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:42:25 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:42:27.644934629 +0000 UTC m=+1595.215400229" watchObservedRunningTime="2026-10-30 19:42:27.663088785 +0000 UTC m=+1595.233554386"
Oct 30 19:42:27.793162 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:42:27.793083    4892 patch_prober.go:29] interesting pod/e2e-worker-w5-686689-openshift-default-5fbf5787bf-7b547 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.114:15021/healthz/ready\": dial tcp 10.132.0.114:15021: connect: connection refused" start-of-body=
Oct 30 19:42:27.793162 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:42:27.793133    4892 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-686689-openshift-default-5fbf5787bf-7b547" podUID="098876de-0b44-493b-a9da-45280afe2cdf" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.114:15021/healthz/ready\": dial tcp 10.132.0.114:15021: connect: connection refused"
Oct 30 19:42:28.271083 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:42:28.271046    4892 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w4-b5178f-openshift-default-c8fbdff8c-wgm46"
Oct 30 19:42:28.273020 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:42:28.272955    4892 patch_prober.go:29] interesting pod/e2e-worker-w4-b5178f-openshift-default-c8fbdff8c-wgm46 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.116:15021/healthz/ready\": dial tcp 10.132.0.116:15021: connect: connection refused" start-of-body=
Oct 30 19:42:28.273147 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:42:28.273049    4892 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-b5178f-openshift-default-c8fbdff8c-wgm46" podUID="0bdf7217-4cc3-4710-b13e-ffcee12febf0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.116:15021/healthz/ready\": dial tcp 10.132.0.116:15021: connect: connection refused"
Oct 30 19:42:28.792565 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:42:28.792529    4892 patch_prober.go:29] interesting pod/e2e-worker-w5-686689-openshift-default-5fbf5787bf-7b547 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.114:15021/healthz/ready\": dial tcp 10.132.0.114:15021: connect: connection refused" start-of-body=
Oct 30 19:42:28.792753 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:42:28.792602    4892 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-686689-openshift-default-5fbf5787bf-7b547" podUID="098876de-0b44-493b-a9da-45280afe2cdf" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.114:15021/healthz/ready\": dial tcp 10.132.0.114:15021: connect: connection refused"
Oct 30 19:42:29.271454 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:42:29.271421    4892 patch_prober.go:29] interesting pod/e2e-worker-w4-b5178f-openshift-default-c8fbdff8c-wgm46 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.116:15021/healthz/ready\": dial tcp 10.132.0.116:15021: connect: connection refused" start-of-body=
Oct 30 19:42:29.272013 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:42:29.271951    4892 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-b5178f-openshift-default-c8fbdff8c-wgm46" podUID="0bdf7217-4cc3-4710-b13e-ffcee12febf0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.116:15021/healthz/ready\": dial tcp 10.132.0.116:15021: connect: connection refused"
Oct 30 19:42:30.014455 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:42:30.014406    4892 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w5-686689-openshift-default-5fbf5787bf-7b547"
Oct 30 19:42:30.015031 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:42:30.015013    4892 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w5-686689-openshift-default-5fbf5787bf-7b547"
Oct 30 19:42:30.016045 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:42:30.016026    4892 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w5-686689-openshift-default-5fbf5787bf-7b547"
Oct 30 19:42:30.271560 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:42:30.271477    4892 patch_prober.go:29] interesting pod/e2e-worker-w4-b5178f-openshift-default-c8fbdff8c-wgm46 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.116:15021/healthz/ready\": dial tcp 10.132.0.116:15021: connect: connection refused" start-of-body=
Oct 30 19:42:30.271560 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:42:30.271540    4892 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-b5178f-openshift-default-c8fbdff8c-wgm46" podUID="0bdf7217-4cc3-4710-b13e-ffcee12febf0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.116:15021/healthz/ready\": dial tcp 10.132.0.116:15021: connect: connection refused"
Oct 30 19:42:31.272983 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:42:31.272360    4892 patch_prober.go:29] interesting pod/e2e-worker-w4-b5178f-openshift-default-c8fbdff8c-wgm46 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.116:15021/healthz/ready\": dial tcp 10.132.0.116:15021: connect: connection refused" start-of-body=
Oct 30 19:42:31.272983 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:42:31.272420    4892 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-b5178f-openshift-default-c8fbdff8c-wgm46" podUID="0bdf7217-4cc3-4710-b13e-ffcee12febf0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.116:15021/healthz/ready\": dial tcp 10.132.0.116:15021: connect: connection refused"
Oct 30 19:42:32.271000 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:42:32.270935    4892 patch_prober.go:29] interesting pod/e2e-worker-w4-b5178f-openshift-default-c8fbdff8c-wgm46 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.116:15021/healthz/ready\": dial tcp 10.132.0.116:15021: connect: connection refused" start-of-body=
Oct 30 19:42:32.271267 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:42:32.271229    4892 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-b5178f-openshift-default-c8fbdff8c-wgm46" podUID="0bdf7217-4cc3-4710-b13e-ffcee12febf0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.116:15021/healthz/ready\": dial tcp 10.132.0.116:15021: connect: connection refused"
Oct 30 19:42:33.270883 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:42:33.270845    4892 patch_prober.go:29] interesting pod/e2e-worker-w4-b5178f-openshift-default-c8fbdff8c-wgm46 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.116:15021/healthz/ready\": dial tcp 10.132.0.116:15021: connect: connection refused" start-of-body=
Oct 30 19:42:33.271373 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:42:33.270903    4892 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-b5178f-openshift-default-c8fbdff8c-wgm46" podUID="0bdf7217-4cc3-4710-b13e-ffcee12febf0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.116:15021/healthz/ready\": dial tcp 10.132.0.116:15021: connect: connection refused"
Oct 30 19:42:33.482841 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:42:33.482797    4892 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5c8ee9bf-j4wgj"]
Oct 30 19:42:33.483718 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:42:33.483696    4892 cpu_manager.go:414] "containerMap: removing container" podUID="da70be07-9754-47d1-95bc-a5970bf9ecae" containerName="revoke-keys"
Oct 30 19:42:33.483768 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:42:33.483723    4892 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="da70be07-9754-47d1-95bc-a5970bf9ecae" containerName="revoke-keys"
Oct 30 19:42:33.483877 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:42:33.483864    4892 memory_manager.go:365] "RemoveStaleState removing state" podUID="da70be07-9754-47d1-95bc-a5970bf9ecae" containerName="revoke-keys"
Oct 30 19:42:33.489358 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:42:33.489327    4892 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5c8ee9bf-j4wgj"
Oct 30 19:42:33.542315 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:42:33.542248    4892 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5c8ee9bf-j4wgj"]
Oct 30 19:42:33.585347 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:42:33.585304    4892 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 30 19:42:33.585347 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:42:33.585334    4892 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 19:42:33.585586 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:42:33.585373    4892 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 30 19:42:33.585586 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:42:33.585415    4892 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-nlxp4\""
Oct 30 19:42:33.590085 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:42:33.590057    4892 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7291702b-9653-4a58-a2c2-9c67afd8ed0a-maas-api-xks-ca\") pod \"maas-api-revoke-sub-5c8ee9bf-j4wgj\" (UID: \"7291702b-9653-4a58-a2c2-9c67afd8ed0a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5c8ee9bf-j4wgj"
Oct 30 19:42:33.590231 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:42:33.590117    4892 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7291702b-9653-4a58-a2c2-9c67afd8ed0a-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-5c8ee9bf-j4wgj\" (UID: \"7291702b-9653-4a58-a2c2-9c67afd8ed0a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5c8ee9bf-j4wgj"
Oct 30 19:42:33.590295 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:42:33.590273    4892 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qz5tj\" (UniqueName: \"kubernetes.io/projected/7291702b-9653-4a58-a2c2-9c67afd8ed0a-kube-api-access-qz5tj\") pod \"maas-api-revoke-sub-5c8ee9bf-j4wgj\" (UID: \"7291702b-9653-4a58-a2c2-9c67afd8ed0a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5c8ee9bf-j4wgj"
Oct 30 19:42:33.691373 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:42:33.691339    4892 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qz5tj\" (UniqueName: \"kubernetes.io/projected/7291702b-9653-4a58-a2c2-9c67afd8ed0a-kube-api-access-qz5tj\") pod \"maas-api-revoke-sub-5c8ee9bf-j4wgj\" (UID: \"7291702b-9653-4a58-a2c2-9c67afd8ed0a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5c8ee9bf-j4wgj"
Oct 30 19:42:33.691546 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:42:33.691440    4892 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7291702b-9653-4a58-a2c2-9c67afd8ed0a-maas-api-xks-ca\") pod \"maas-api-revoke-sub-5c8ee9bf-j4wgj\" (UID: \"7291702b-9653-4a58-a2c2-9c67afd8ed0a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5c8ee9bf-j4wgj"
Oct 30 19:42:33.691546 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:42:33.691472    4892 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7291702b-9653-4a58-a2c2-9c67afd8ed0a-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-5c8ee9bf-j4wgj\" (UID: \"7291702b-9653-4a58-a2c2-9c67afd8ed0a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5c8ee9bf-j4wgj"
Oct 30 19:42:33.692131 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:42:33.692110    4892 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7291702b-9653-4a58-a2c2-9c67afd8ed0a-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-5c8ee9bf-j4wgj\" (UID: \"7291702b-9653-4a58-a2c2-9c67afd8ed0a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5c8ee9bf-j4wgj"
Oct 30 19:42:33.694148 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:42:33.694120    4892 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7291702b-9653-4a58-a2c2-9c67afd8ed0a-maas-api-xks-ca\") pod \"maas-api-revoke-sub-5c8ee9bf-j4wgj\" (UID: \"7291702b-9653-4a58-a2c2-9c67afd8ed0a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5c8ee9bf-j4wgj"
Oct 30 19:42:33.749206 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:42:33.749164    4892 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qz5tj\" (UniqueName: \"kubernetes.io/projected/7291702b-9653-4a58-a2c2-9c67afd8ed0a-kube-api-access-qz5tj\") pod \"maas-api-revoke-sub-5c8ee9bf-j4wgj\" (UID: \"7291702b-9653-4a58-a2c2-9c67afd8ed0a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5c8ee9bf-j4wgj"
Oct 30 19:42:33.803143 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:42:33.803068    4892 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5c8ee9bf-j4wgj"
Oct 30 19:42:34.199290 ip-10-0-132-82 kubenswrapper[4892]: W1030 19:42:34.199262    4892 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7291702b_9653_4a58_a2c2_9c67afd8ed0a.slice/crio-2c5b948fea2d6f35153bd9f3817ff937ffc292a2cf4f465b1b37c02261e0aec8 WatchSource:0}: Error finding container 2c5b948fea2d6f35153bd9f3817ff937ffc292a2cf4f465b1b37c02261e0aec8: Status 404 returned error can't find the container with id 2c5b948fea2d6f35153bd9f3817ff937ffc292a2cf4f465b1b37c02261e0aec8
Oct 30 19:42:34.202276 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:42:34.202250    4892 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5c8ee9bf-j4wgj"]
Oct 30 19:42:34.271443 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:42:34.271409    4892 patch_prober.go:29] interesting pod/e2e-worker-w4-b5178f-openshift-default-c8fbdff8c-wgm46 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.116:15021/healthz/ready\": dial tcp 10.132.0.116:15021: connect: connection refused" start-of-body=
Oct 30 19:42:34.271769 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:42:34.271481    4892 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-b5178f-openshift-default-c8fbdff8c-wgm46" podUID="0bdf7217-4cc3-4710-b13e-ffcee12febf0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.116:15021/healthz/ready\": dial tcp 10.132.0.116:15021: connect: connection refused"
Oct 30 19:42:34.677352 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:42:34.677313    4892 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5c8ee9bf-j4wgj" event={"ID":"7291702b-9653-4a58-a2c2-9c67afd8ed0a","Type":"ContainerStarted","Data":"2c5b948fea2d6f35153bd9f3817ff937ffc292a2cf4f465b1b37c02261e0aec8"}
Oct 30 19:42:35.276803 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:42:35.276775    4892 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w4-b5178f-openshift-default-c8fbdff8c-wgm46"
Oct 30 19:42:35.277306 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:42:35.277286    4892 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w4-b5178f-openshift-default-c8fbdff8c-wgm46"
Oct 30 19:42:35.278048 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:42:35.278029    4892 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w4-b5178f-openshift-default-c8fbdff8c-wgm46"
Oct 30 19:42:35.692861 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:42:35.692809    4892 generic.go:356] "Generic (PLEG): container finished" podID="7291702b-9653-4a58-a2c2-9c67afd8ed0a" containerID="f0b5d9951889dbe6bc52d2bb56f2024fcf194176c7e04edbd4ef96c6edcb262d" exitCode=0
Oct 30 19:42:35.694668 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:42:35.694637    4892 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5c8ee9bf-j4wgj" event={"ID":"7291702b-9653-4a58-a2c2-9c67afd8ed0a","Type":"ContainerDied","Data":"f0b5d9951889dbe6bc52d2bb56f2024fcf194176c7e04edbd4ef96c6edcb262d"}
Oct 30 19:42:36.979008 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:42:36.978520    4892 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f64d0dcb-2whv4"]
Oct 30 19:42:37.010558 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:42:37.010471    4892 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f64d0dcb-2whv4"
Oct 30 19:42:37.047376 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:42:37.047344    4892 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f64d0dcb-2whv4"]
Oct 30 19:42:37.226992 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:42:37.204703    4892 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6350b173-e45f-43ef-8ec6-ee217764b552-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-f64d0dcb-2whv4\" (UID: \"6350b173-e45f-43ef-8ec6-ee217764b552\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f64d0dcb-2whv4"
Oct 30 19:42:37.279053 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:42:37.227661    4892 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-49l7d\" (UniqueName: \"kubernetes.io/projected/6350b173-e45f-43ef-8ec6-ee217764b552-kube-api-access-49l7d\") pod \"maas-api-revoke-sub-f64d0dcb-2whv4\" (UID: \"6350b173-e45f-43ef-8ec6-ee217764b552\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f64d0dcb-2whv4"
Oct 30 19:42:37.279624 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:42:37.279592    4892 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6350b173-e45f-43ef-8ec6-ee217764b552-maas-api-xks-ca\") pod \"maas-api-revoke-sub-f64d0dcb-2whv4\" (UID: \"6350b173-e45f-43ef-8ec6-ee217764b552\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f64d0dcb-2whv4"
Oct 30 19:42:37.385990 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:42:37.382690    4892 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6350b173-e45f-43ef-8ec6-ee217764b552-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-f64d0dcb-2whv4\" (UID: \"6350b173-e45f-43ef-8ec6-ee217764b552\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f64d0dcb-2whv4"
Oct 30 19:42:37.446991 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:42:37.446207    4892 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-49l7d\" (UniqueName: \"kubernetes.io/projected/6350b173-e45f-43ef-8ec6-ee217764b552-kube-api-access-49l7d\") pod \"maas-api-revoke-sub-f64d0dcb-2whv4\" (UID: \"6350b173-e45f-43ef-8ec6-ee217764b552\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f64d0dcb-2whv4"
Oct 30 19:42:37.446991 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:42:37.446305    4892 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6350b173-e45f-43ef-8ec6-ee217764b552-maas-api-xks-ca\") pod \"maas-api-revoke-sub-f64d0dcb-2whv4\" (UID: \"6350b173-e45f-43ef-8ec6-ee217764b552\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f64d0dcb-2whv4"
Oct 30 19:42:37.451013 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:42:37.390808    4892 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6350b173-e45f-43ef-8ec6-ee217764b552-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-f64d0dcb-2whv4\" (UID: \"6350b173-e45f-43ef-8ec6-ee217764b552\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f64d0dcb-2whv4"
Oct 30 19:42:37.701581 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:42:37.700348    4892 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-d52df0/simulator-w1-d52df0-kserve-67f74bc577-q5bjz" podUID="81b7bb84-1003-4452-b7e0-b4aaba6fefa4" containerName="main" probeResult="failure" output="Get \"https://10.132.0.57:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:42:37.782997 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:42:37.777509    4892 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6350b173-e45f-43ef-8ec6-ee217764b552-maas-api-xks-ca\") pod \"maas-api-revoke-sub-f64d0dcb-2whv4\" (UID: \"6350b173-e45f-43ef-8ec6-ee217764b552\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f64d0dcb-2whv4"
Oct 30 19:42:37.850000 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:42:37.847362    4892 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-2d9861/premium-w0-2d9861-kserve-67945c6848-4n6kz" podUID="798988b2-c769-4f73-a7c8-ca15de0beef1" containerName="main" probeResult="failure" output="Get \"https://10.132.0.62:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:42:37.985018 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:42:37.952147    4892 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-a3f9b7/simulator-w2-a3f9b7-kserve-79bbc4cb5f-8m488" podUID="50fb2283-bf3f-41da-835b-f407ee1a04f3" containerName="main" probeResult="failure" output="Get \"https://10.132.0.59:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:42:38.021637 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:42:38.021603    4892 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-49l7d\" (UniqueName: \"kubernetes.io/projected/6350b173-e45f-43ef-8ec6-ee217764b552-kube-api-access-49l7d\") pod \"maas-api-revoke-sub-f64d0dcb-2whv4\" (UID: \"6350b173-e45f-43ef-8ec6-ee217764b552\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f64d0dcb-2whv4"
Oct 30 19:42:38.072613 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:42:38.072246    4892 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5c8ee9bf-j4wgj"
Oct 30 19:42:38.107627 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:42:38.107585    4892 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-686689/distinct-2-w5-686689-kserve-5ff578b4cd-rkc6s" podUID="a8829732-ac6f-4238-b8ce-fc781084668b" containerName="main" probeResult="failure" output="Get \"https://10.132.0.87:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:42:38.148949 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:42:38.148821    4892 patch_prober.go:29] interesting pod/e2e-worker-w2-a3f9b7-openshift-default-7d6d8b8fb5-sfm8n container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.71:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 30 19:42:38.148949 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:42:38.148884    4892 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w2-a3f9b7-openshift-default-7d6d8b8fb5-sfm8n" podUID="1c3346aa-a989-495e-9d54-66c8ec632d91" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.71:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:42:38.225308 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:42:38.224147    4892 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7291702b-9653-4a58-a2c2-9c67afd8ed0a-kube-api-access-qz5tj\" (UniqueName: \"kubernetes.io/projected/7291702b-9653-4a58-a2c2-9c67afd8ed0a-kube-api-access-qz5tj\") pod \"7291702b-9653-4a58-a2c2-9c67afd8ed0a\" (UID: \"7291702b-9653-4a58-a2c2-9c67afd8ed0a\") "
Oct 30 19:42:38.225308 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:42:38.224226    4892 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/7291702b-9653-4a58-a2c2-9c67afd8ed0a-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7291702b-9653-4a58-a2c2-9c67afd8ed0a-maas-api-openshift-ca\") pod \"7291702b-9653-4a58-a2c2-9c67afd8ed0a\" (UID: \"7291702b-9653-4a58-a2c2-9c67afd8ed0a\") "
Oct 30 19:42:38.225308 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:42:38.224383    4892 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/7291702b-9653-4a58-a2c2-9c67afd8ed0a-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7291702b-9653-4a58-a2c2-9c67afd8ed0a-maas-api-xks-ca\") pod \"7291702b-9653-4a58-a2c2-9c67afd8ed0a\" (UID: \"7291702b-9653-4a58-a2c2-9c67afd8ed0a\") "
Oct 30 19:42:38.233138 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:42:38.233083    4892 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7291702b-9653-4a58-a2c2-9c67afd8ed0a-maas-api-openshift-ca" pod "7291702b-9653-4a58-a2c2-9c67afd8ed0a" (UID: "7291702b-9653-4a58-a2c2-9c67afd8ed0a"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 19:42:38.246740 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:42:38.246678    4892 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7291702b-9653-4a58-a2c2-9c67afd8ed0a-kube-api-access-qz5tj" pod "7291702b-9653-4a58-a2c2-9c67afd8ed0a" (UID: "7291702b-9653-4a58-a2c2-9c67afd8ed0a"). InnerVolumeSpecName "kube-api-access-qz5tj". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:42:38.254522 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:42:38.254465    4892 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7291702b-9653-4a58-a2c2-9c67afd8ed0a-maas-api-xks-ca" pod "7291702b-9653-4a58-a2c2-9c67afd8ed0a" (UID: "7291702b-9653-4a58-a2c2-9c67afd8ed0a"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 19:42:38.272509 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:42:38.272186    4892 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f64d0dcb-2whv4"
Oct 30 19:42:38.401069 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:42:38.380443    4892 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-d52df0/distinct-2-w1-d52df0-kserve-7846558464-xrkdc" podUID="9c614ab6-30e2-4fe1-8609-d5e2dadc198a" containerName="main" probeResult="failure" output="Get \"https://10.132.0.83:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:42:38.402948 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:42:38.402846    4892 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7291702b-9653-4a58-a2c2-9c67afd8ed0a-maas-api-xks-ca\") on node \"ip-10-0-132-82.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 19:42:38.402948 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:42:38.402884    4892 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qz5tj\" (UniqueName: \"kubernetes.io/projected/7291702b-9653-4a58-a2c2-9c67afd8ed0a-kube-api-access-qz5tj\") on node \"ip-10-0-132-82.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 19:42:38.402948 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:42:38.402901    4892 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7291702b-9653-4a58-a2c2-9c67afd8ed0a-maas-api-openshift-ca\") on node \"ip-10-0-132-82.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 19:42:38.427027 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:42:38.420852    4892 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w2-a3f9b7/simulator-w2-a3f9b7-kserve-79bbc4cb5f-8m488" podUID="50fb2283-bf3f-41da-835b-f407ee1a04f3" containerName="main" probeResult="failure" output="Get \"https://10.132.0.59:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:42:38.654213 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:42:38.651045    4892 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-5f55cd/premium-w6-5f55cd-kserve-7f785c544b-pcvhb" podUID="191593a2-e3a9-4ce4-a1c6-c089723b0237" containerName="main" probeResult="failure" output="Get \"https://10.132.0.60:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:42:38.687085 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:42:38.687034    4892 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-b5178f/distinct-2-w4-b5178f-kserve-69ddf8b665-9pdth" podUID="fb71d961-8500-4367-b48c-88d385962bd5" containerName="main" probeResult="failure" output="Get \"https://10.132.0.90:8000/ready\": context deadline exceeded"
Oct 30 19:42:38.727001 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:42:38.725076    4892 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-d52df0/simulator-w1-d52df0-kserve-67f74bc577-q5bjz" podUID="81b7bb84-1003-4452-b7e0-b4aaba6fefa4" containerName="main" probeResult="failure" output="Get \"https://10.132.0.57:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:42:38.727001 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:42:38.726693    4892 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-d52df0/simulator-w1-d52df0-kserve-67f74bc577-q5bjz"
Oct 30 19:42:38.829048 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:42:38.828293    4892 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-5f55cd/simulator-w6-5f55cd-kserve-5ff86b7447-h9tm7" podUID="3cb47a99-8be1-4756-b31a-43b58cc6b8dc" containerName="main" probeResult="failure" output="Get \"https://10.132.0.58:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:42:38.852659 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:42:38.848618    4892 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-2d9861/premium-w0-2d9861-kserve-67945c6848-4n6kz" podUID="798988b2-c769-4f73-a7c8-ca15de0beef1" containerName="main" probeResult="failure" output="Get \"https://10.132.0.62:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:42:38.852659 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:42:38.849402    4892 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-ada8d7/simulator-w3-ada8d7-kserve-7d575cfcb5-5txgn" podUID="10a5c7c8-6146-47b0-9643-bb137cf10bf5" containerName="main" probeResult="failure" output="Get \"https://10.132.0.55:8000/ready\": context deadline exceeded"
Oct 30 19:42:38.852659 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:42:38.851443    4892 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-2d9861/premium-w0-2d9861-kserve-67945c6848-4n6kz"
Oct 30 19:42:38.963019 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:42:38.959238    4892 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-a3f9b7/simulator-w2-a3f9b7-kserve-79bbc4cb5f-8m488" podUID="50fb2283-bf3f-41da-835b-f407ee1a04f3" containerName="main" probeResult="failure" output="Get \"https://10.132.0.59:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:42:38.963019 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:42:38.959660    4892 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-a3f9b7/simulator-w2-a3f9b7-kserve-79bbc4cb5f-8m488"
Oct 30 19:42:39.022077 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:42:38.912340    4892 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w6-5f55cd/premium-w6-5f55cd-kserve-7f785c544b-pcvhb" podUID="191593a2-e3a9-4ce4-a1c6-c089723b0237" containerName="main" probeResult="failure" output="Get \"https://10.132.0.60:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:42:39.164014 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:42:39.145449    4892 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-686689/distinct-2-w5-686689-kserve-5ff578b4cd-rkc6s" podUID="a8829732-ac6f-4238-b8ce-fc781084668b" containerName="main" probeResult="failure" output="Get \"https://10.132.0.87:8000/ready\": context deadline exceeded"
Oct 30 19:42:39.164704 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:42:39.164673    4892 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w5-686689/distinct-2-w5-686689-kserve-5ff578b4cd-rkc6s"
Oct 30 19:42:39.274698 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:42:39.274669    4892 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5c8ee9bf-j4wgj"
Oct 30 19:42:39.310471 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:42:39.303686    4892 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5c8ee9bf-j4wgj" event={"ID":"7291702b-9653-4a58-a2c2-9c67afd8ed0a","Type":"ContainerDied","Data":"2c5b948fea2d6f35153bd9f3817ff937ffc292a2cf4f465b1b37c02261e0aec8"}
Oct 30 19:42:39.310471 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:42:39.303758    4892 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="2c5b948fea2d6f35153bd9f3817ff937ffc292a2cf4f465b1b37c02261e0aec8"
Oct 30 19:42:39.418777 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:42:39.418680    4892 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-d52df0/distinct-2-w1-d52df0-kserve-7846558464-xrkdc" podUID="9c614ab6-30e2-4fe1-8609-d5e2dadc198a" containerName="main" probeResult="failure" output="Get \"https://10.132.0.83:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:42:39.484991 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:42:39.484141    4892 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w1-d52df0/distinct-2-w1-d52df0-kserve-7846558464-xrkdc" podUID="9c614ab6-30e2-4fe1-8609-d5e2dadc198a" containerName="main" probeResult="failure" output="Get \"https://10.132.0.83:8000/health\": context deadline exceeded"
Oct 30 19:42:39.484991 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:42:39.484410    4892 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-d52df0/distinct-2-w1-d52df0-kserve-7846558464-xrkdc"
Oct 30 19:42:39.875040 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:42:39.797395    4892 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-b5178f/distinct-2-w4-b5178f-kserve-69ddf8b665-9pdth" podUID="fb71d961-8500-4367-b48c-88d385962bd5" containerName="main" probeResult="failure" output="Get \"https://10.132.0.90:8000/ready\": context deadline exceeded"
Oct 30 19:42:39.875040 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:42:39.797892    4892 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-b5178f/distinct-2-w4-b5178f-kserve-69ddf8b665-9pdth"
Oct 30 19:42:39.875040 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:42:39.854081    4892 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-5f55cd/simulator-w6-5f55cd-kserve-5ff86b7447-h9tm7" podUID="3cb47a99-8be1-4756-b31a-43b58cc6b8dc" containerName="main" probeResult="failure" output="Get \"https://10.132.0.58:8000/ready\": context deadline exceeded"
Oct 30 19:42:39.875040 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:42:39.854533    4892 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w6-5f55cd/simulator-w6-5f55cd-kserve-5ff86b7447-h9tm7"
Oct 30 19:42:39.875040 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:42:39.855449    4892 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-d52df0/simulator-w1-d52df0-kserve-67f74bc577-q5bjz" podUID="81b7bb84-1003-4452-b7e0-b4aaba6fefa4" containerName="main" probeResult="failure" output="Get \"https://10.132.0.57:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:42:39.875040 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:42:39.856192    4892 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-ada8d7/simulator-w3-ada8d7-kserve-7d575cfcb5-5txgn" podUID="10a5c7c8-6146-47b0-9643-bb137cf10bf5" containerName="main" probeResult="failure" output="Get \"https://10.132.0.55:8000/ready\": context deadline exceeded"
Oct 30 19:42:39.875040 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:42:39.856487    4892 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-5f55cd/premium-w6-5f55cd-kserve-7f785c544b-pcvhb" podUID="191593a2-e3a9-4ce4-a1c6-c089723b0237" containerName="main" probeResult="failure" output="Get \"https://10.132.0.60:8000/ready\": context deadline exceeded"
Oct 30 19:42:39.875040 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:42:39.856744    4892 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-2d9861/premium-w0-2d9861-kserve-67945c6848-4n6kz" podUID="798988b2-c769-4f73-a7c8-ca15de0beef1" containerName="main" probeResult="failure" output="Get \"https://10.132.0.62:8000/ready\": context deadline exceeded"
Oct 30 19:42:39.875040 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:42:39.873168    4892 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-ada8d7/simulator-w3-ada8d7-kserve-7d575cfcb5-5txgn"
Oct 30 19:42:39.875040 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:42:39.873263    4892 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w6-5f55cd/premium-w6-5f55cd-kserve-7f785c544b-pcvhb"
Oct 30 19:42:39.981163 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:42:39.981119    4892 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-a3f9b7/simulator-w2-a3f9b7-kserve-79bbc4cb5f-8m488" podUID="50fb2283-bf3f-41da-835b-f407ee1a04f3" containerName="main" probeResult="failure" output="Get \"https://10.132.0.59:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:42:40.179991 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:42:40.175909    4892 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-686689/distinct-2-w5-686689-kserve-5ff578b4cd-rkc6s" podUID="a8829732-ac6f-4238-b8ce-fc781084668b" containerName="main" probeResult="failure" output="Get \"https://10.132.0.87:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:42:40.319764 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:42:40.319729    4892 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w5-686689/distinct-2-w5-686689-kserve-5ff578b4cd-rkc6s"
Oct 30 19:42:40.522173 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:42:40.517911    4892 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-d52df0/distinct-2-w1-d52df0-kserve-7846558464-xrkdc" podUID="9c614ab6-30e2-4fe1-8609-d5e2dadc198a" containerName="main" probeResult="failure" output="Get \"https://10.132.0.83:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:42:40.522173 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:42:40.518342    4892 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w1-d52df0/simulator-w1-d52df0-kserve-67f74bc577-q5bjz" podUID="81b7bb84-1003-4452-b7e0-b4aaba6fefa4" containerName="main" probeResult="failure" output="Get \"https://10.132.0.57:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:42:40.809649 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:42:40.809534    4892 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-b5178f/distinct-2-w4-b5178f-kserve-69ddf8b665-9pdth" podUID="fb71d961-8500-4367-b48c-88d385962bd5" containerName="main" probeResult="failure" output="Get \"https://10.132.0.90:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:42:40.869001 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:42:40.856566    4892 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-d52df0/simulator-w1-d52df0-kserve-67f74bc577-q5bjz" podUID="81b7bb84-1003-4452-b7e0-b4aaba6fefa4" containerName="main" probeResult="failure" output="Get \"https://10.132.0.57:8000/ready\": context deadline exceeded"
Oct 30 19:42:40.972225 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:42:40.972170    4892 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-2d9861/premium-w0-2d9861-kserve-67945c6848-4n6kz" podUID="798988b2-c769-4f73-a7c8-ca15de0beef1" containerName="main" probeResult="failure" output="Get \"https://10.132.0.62:8000/ready\": context deadline exceeded"
Oct 30 19:42:41.033991 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:42:41.033606    4892 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-5f55cd/simulator-w6-5f55cd-kserve-5ff86b7447-h9tm7" podUID="3cb47a99-8be1-4756-b31a-43b58cc6b8dc" containerName="main" probeResult="failure" output="Get \"https://10.132.0.58:8000/ready\": context deadline exceeded"
Oct 30 19:42:41.066040 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:42:40.856522    4892 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-ada8d7/simulator-w3-ada8d7-kserve-7d575cfcb5-5txgn" podUID="10a5c7c8-6146-47b0-9643-bb137cf10bf5" containerName="main" probeResult="failure" output="Get \"https://10.132.0.55:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:42:41.087604 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:42:41.087566    4892 patch_prober.go:29] interesting pod/e2e-worker-w2-a3f9b7-openshift-default-7d6d8b8fb5-m5zsm container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.106:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 30 19:42:41.087876 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:42:41.087842    4892 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w2-a3f9b7-openshift-default-7d6d8b8fb5-m5zsm" podUID="02bc8b7f-a2b1-4994-85cf-436cfc0d221c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.106:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:42:41.088751 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:42:41.088721    4892 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-5f55cd/premium-w6-5f55cd-kserve-7f785c544b-pcvhb" podUID="191593a2-e3a9-4ce4-a1c6-c089723b0237" containerName="main" probeResult="failure" output="Get \"https://10.132.0.60:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:42:41.133665 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:42:41.133617    4892 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-a3f9b7/simulator-w2-a3f9b7-kserve-79bbc4cb5f-8m488" podUID="50fb2283-bf3f-41da-835b-f407ee1a04f3" containerName="main" probeResult="failure" output="Get \"https://10.132.0.59:8000/ready\": context deadline exceeded"
Oct 30 19:42:41.273072 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:42:41.263752    4892 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f64d0dcb-2whv4" event={"ID":"6350b173-e45f-43ef-8ec6-ee217764b552","Type":"ContainerStarted","Data":"21faf5a97c1f248cacc9955d09b86b825d1e26b1ceee19fb7138dda45cd2ab26"}
Oct 30 19:42:41.439448 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:42:41.439414    4892 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f64d0dcb-2whv4"]
Oct 30 19:42:41.522186 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:42:41.522143    4892 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-d52df0/distinct-2-w1-d52df0-kserve-7846558464-xrkdc" podUID="9c614ab6-30e2-4fe1-8609-d5e2dadc198a" containerName="main" probeResult="failure" output="Get \"https://10.132.0.83:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:42:41.701782 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:42:41.701742    4892 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c24a2487-rpvhb"]
Oct 30 19:42:41.714364 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:42:41.714294    4892 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c24a2487-rpvhb"]
Oct 30 19:42:41.729990 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:42:41.729869    4892 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w0-2d9861/premium-w0-2d9861-kserve-67945c6848-4n6kz" podUID="798988b2-c769-4f73-a7c8-ca15de0beef1" containerName="main" probeResult="failure" output="Get \"https://10.132.0.62:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:42:41.822811 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:42:41.821599    4892 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-b5178f/distinct-2-w4-b5178f-kserve-69ddf8b665-9pdth" podUID="fb71d961-8500-4367-b48c-88d385962bd5" containerName="main" probeResult="failure" output="Get \"https://10.132.0.90:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:42:41.833283 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:42:41.825201    4892 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-a3f9b7/simulator-w2-a3f9b7-kserve-79bbc4cb5f-8m488"
Oct 30 19:42:41.859103 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:42:41.859050    4892 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-d52df0/simulator-w1-d52df0-kserve-67f74bc577-q5bjz" podUID="81b7bb84-1003-4452-b7e0-b4aaba6fefa4" containerName="main" probeResult="failure" output="Get \"https://10.132.0.57:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:42:41.916544 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:42:41.914554    4892 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-ada8d7/simulator-w3-ada8d7-kserve-7d575cfcb5-5txgn"
Oct 30 19:42:41.968909 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:42:41.968029    4892 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-b5178f/distinct-2-w4-b5178f-kserve-69ddf8b665-9pdth"
Oct 30 19:42:41.990244 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:42:41.988263    4892 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-d52df0/simulator-w1-d52df0-kserve-67f74bc577-q5bjz"
Oct 30 19:42:41.990244 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:42:41.989446    4892 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-2d9861/premium-w0-2d9861-kserve-67945c6848-4n6kz" podUID="798988b2-c769-4f73-a7c8-ca15de0beef1" containerName="main" probeResult="failure" output="Get \"https://10.132.0.62:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:42:41.994042 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:42:41.993016    4892 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w6-5f55cd/simulator-w6-5f55cd-kserve-5ff86b7447-h9tm7"
Oct 30 19:42:41.994042 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:42:41.993790    4892 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w6-5f55cd/premium-w6-5f55cd-kserve-7f785c544b-pcvhb"
Oct 30 19:42:41.999670 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:42:41.997750    4892 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-d52df0/distinct-2-w1-d52df0-kserve-7846558464-xrkdc"
Oct 30 19:42:42.007185 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:42:42.007143    4892 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-2d9861/premium-w0-2d9861-kserve-67945c6848-4n6kz"
Oct 30 19:42:42.269797 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:42:42.269696    4892 generic.go:356] "Generic (PLEG): container finished" podID="6350b173-e45f-43ef-8ec6-ee217764b552" containerID="d7d4927145a878ada4803a8ab02820b95d3237cc5c16601b2037826b0317c5a4" exitCode=0
Oct 30 19:42:42.269797 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:42:42.269754    4892 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f64d0dcb-2whv4" event={"ID":"6350b173-e45f-43ef-8ec6-ee217764b552","Type":"ContainerDied","Data":"d7d4927145a878ada4803a8ab02820b95d3237cc5c16601b2037826b0317c5a4"}
Oct 30 19:42:42.574925 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:42:42.574817    4892 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c55219bc-f5e4-4a58-8c3a-51485d360d49" path="/var/lib/kubelet/pods/c55219bc-f5e4-4a58-8c3a-51485d360d49/volumes"
Oct 30 19:42:43.471717 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:42:43.471694    4892 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f64d0dcb-2whv4"
Oct 30 19:42:43.502603 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:42:43.502574    4892 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/6350b173-e45f-43ef-8ec6-ee217764b552-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6350b173-e45f-43ef-8ec6-ee217764b552-maas-api-openshift-ca\") pod \"6350b173-e45f-43ef-8ec6-ee217764b552\" (UID: \"6350b173-e45f-43ef-8ec6-ee217764b552\") "
Oct 30 19:42:43.502747 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:42:43.502726    4892 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/6350b173-e45f-43ef-8ec6-ee217764b552-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6350b173-e45f-43ef-8ec6-ee217764b552-maas-api-xks-ca\") pod \"6350b173-e45f-43ef-8ec6-ee217764b552\" (UID: \"6350b173-e45f-43ef-8ec6-ee217764b552\") "
Oct 30 19:42:43.502858 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:42:43.502841    4892 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6350b173-e45f-43ef-8ec6-ee217764b552-kube-api-access-49l7d\" (UniqueName: \"kubernetes.io/projected/6350b173-e45f-43ef-8ec6-ee217764b552-kube-api-access-49l7d\") pod \"6350b173-e45f-43ef-8ec6-ee217764b552\" (UID: \"6350b173-e45f-43ef-8ec6-ee217764b552\") "
Oct 30 19:42:43.503005 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:42:43.502984    4892 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6350b173-e45f-43ef-8ec6-ee217764b552-maas-api-openshift-ca" pod "6350b173-e45f-43ef-8ec6-ee217764b552" (UID: "6350b173-e45f-43ef-8ec6-ee217764b552"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 19:42:43.503307 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:42:43.503276    4892 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6350b173-e45f-43ef-8ec6-ee217764b552-maas-api-openshift-ca\") on node \"ip-10-0-132-82.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 19:42:43.505291 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:42:43.505262    4892 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6350b173-e45f-43ef-8ec6-ee217764b552-maas-api-xks-ca" pod "6350b173-e45f-43ef-8ec6-ee217764b552" (UID: "6350b173-e45f-43ef-8ec6-ee217764b552"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 19:42:43.505384 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:42:43.505262    4892 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6350b173-e45f-43ef-8ec6-ee217764b552-kube-api-access-49l7d" pod "6350b173-e45f-43ef-8ec6-ee217764b552" (UID: "6350b173-e45f-43ef-8ec6-ee217764b552"). InnerVolumeSpecName "kube-api-access-49l7d". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:42:43.606690 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:42:43.606587    4892 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6350b173-e45f-43ef-8ec6-ee217764b552-maas-api-xks-ca\") on node \"ip-10-0-132-82.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 19:42:43.606690 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:42:43.606629    4892 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-49l7d\" (UniqueName: \"kubernetes.io/projected/6350b173-e45f-43ef-8ec6-ee217764b552-kube-api-access-49l7d\") on node \"ip-10-0-132-82.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 19:42:44.288655 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:42:44.288624    4892 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f64d0dcb-2whv4"
Oct 30 19:42:44.288655 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:42:44.288634    4892 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f64d0dcb-2whv4" event={"ID":"6350b173-e45f-43ef-8ec6-ee217764b552","Type":"ContainerDied","Data":"21faf5a97c1f248cacc9955d09b86b825d1e26b1ceee19fb7138dda45cd2ab26"}
Oct 30 19:42:44.288876 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:42:44.288685    4892 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="21faf5a97c1f248cacc9955d09b86b825d1e26b1ceee19fb7138dda45cd2ab26"
Oct 30 19:42:45.195574 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:42:45.195541    4892 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-9a514ead-qjp69"]
Oct 30 19:42:45.196306 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:42:45.196284    4892 cpu_manager.go:414] "containerMap: removing container" podUID="7291702b-9653-4a58-a2c2-9c67afd8ed0a" containerName="revoke-keys"
Oct 30 19:42:45.196397 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:42:45.196306    4892 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7291702b-9653-4a58-a2c2-9c67afd8ed0a" containerName="revoke-keys"
Oct 30 19:42:45.196397 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:42:45.196374    4892 cpu_manager.go:414] "containerMap: removing container" podUID="6350b173-e45f-43ef-8ec6-ee217764b552" containerName="revoke-keys"
Oct 30 19:42:45.196397 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:42:45.196382    4892 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6350b173-e45f-43ef-8ec6-ee217764b552" containerName="revoke-keys"
Oct 30 19:42:45.196559 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:42:45.196515    4892 memory_manager.go:365] "RemoveStaleState removing state" podUID="6350b173-e45f-43ef-8ec6-ee217764b552" containerName="revoke-keys"
Oct 30 19:42:45.196559 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:42:45.196531    4892 memory_manager.go:365] "RemoveStaleState removing state" podUID="7291702b-9653-4a58-a2c2-9c67afd8ed0a" containerName="revoke-keys"
Oct 30 19:42:45.200164 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:42:45.200143    4892 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9a514ead-qjp69"
Oct 30 19:42:45.224113 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:42:45.224088    4892 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/29c286de-ac5b-4dea-ba6c-565c16549eb9-maas-api-xks-ca\") pod \"maas-api-revoke-sub-9a514ead-qjp69\" (UID: \"29c286de-ac5b-4dea-ba6c-565c16549eb9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9a514ead-qjp69"
Oct 30 19:42:45.224254 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:42:45.224232    4892 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dm47z\" (UniqueName: \"kubernetes.io/projected/29c286de-ac5b-4dea-ba6c-565c16549eb9-kube-api-access-dm47z\") pod \"maas-api-revoke-sub-9a514ead-qjp69\" (UID: \"29c286de-ac5b-4dea-ba6c-565c16549eb9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9a514ead-qjp69"
Oct 30 19:42:45.224310 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:42:45.224270    4892 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/29c286de-ac5b-4dea-ba6c-565c16549eb9-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-9a514ead-qjp69\" (UID: \"29c286de-ac5b-4dea-ba6c-565c16549eb9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9a514ead-qjp69"
Oct 30 19:42:45.264131 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:42:45.264095    4892 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-9a514ead-qjp69"]
Oct 30 19:42:45.299758 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:42:45.299719    4892 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 19:42:45.299924 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:42:45.299801    4892 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-nlxp4\""
Oct 30 19:42:45.299924 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:42:45.299894    4892 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 30 19:42:45.301150 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:42:45.301128    4892 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 30 19:42:45.326062 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:42:45.326038    4892 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/29c286de-ac5b-4dea-ba6c-565c16549eb9-maas-api-xks-ca\") pod \"maas-api-revoke-sub-9a514ead-qjp69\" (UID: \"29c286de-ac5b-4dea-ba6c-565c16549eb9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9a514ead-qjp69"
Oct 30 19:42:45.326164 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:42:45.326124    4892 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dm47z\" (UniqueName: \"kubernetes.io/projected/29c286de-ac5b-4dea-ba6c-565c16549eb9-kube-api-access-dm47z\") pod \"maas-api-revoke-sub-9a514ead-qjp69\" (UID: \"29c286de-ac5b-4dea-ba6c-565c16549eb9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9a514ead-qjp69"
Oct 30 19:42:45.326164 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:42:45.326145    4892 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/29c286de-ac5b-4dea-ba6c-565c16549eb9-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-9a514ead-qjp69\" (UID: \"29c286de-ac5b-4dea-ba6c-565c16549eb9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9a514ead-qjp69"
Oct 30 19:42:45.326795 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:42:45.326709    4892 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/29c286de-ac5b-4dea-ba6c-565c16549eb9-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-9a514ead-qjp69\" (UID: \"29c286de-ac5b-4dea-ba6c-565c16549eb9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9a514ead-qjp69"
Oct 30 19:42:45.328833 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:42:45.328797    4892 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/29c286de-ac5b-4dea-ba6c-565c16549eb9-maas-api-xks-ca\") pod \"maas-api-revoke-sub-9a514ead-qjp69\" (UID: \"29c286de-ac5b-4dea-ba6c-565c16549eb9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9a514ead-qjp69"
Oct 30 19:42:45.383298 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:42:45.383267    4892 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dm47z\" (UniqueName: \"kubernetes.io/projected/29c286de-ac5b-4dea-ba6c-565c16549eb9-kube-api-access-dm47z\") pod \"maas-api-revoke-sub-9a514ead-qjp69\" (UID: \"29c286de-ac5b-4dea-ba6c-565c16549eb9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9a514ead-qjp69"
Oct 30 19:42:45.514234 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:42:45.514172    4892 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9a514ead-qjp69"
Oct 30 19:42:45.897251 ip-10-0-132-82 kubenswrapper[4892]: W1030 19:42:45.897220    4892 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod29c286de_ac5b_4dea_ba6c_565c16549eb9.slice/crio-c9a36100faf92eaf3d926821936517eeb73e912cea0ba6a05da57c9ef109d3ba WatchSource:0}: Error finding container c9a36100faf92eaf3d926821936517eeb73e912cea0ba6a05da57c9ef109d3ba: Status 404 returned error can't find the container with id c9a36100faf92eaf3d926821936517eeb73e912cea0ba6a05da57c9ef109d3ba
Oct 30 19:42:45.897407 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:42:45.897346    4892 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-9a514ead-qjp69"]
Oct 30 19:42:46.307127 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:42:46.307093    4892 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9a514ead-qjp69" event={"ID":"29c286de-ac5b-4dea-ba6c-565c16549eb9","Type":"ContainerStarted","Data":"dab73619cd618c9218820cd33f34d457cd7d234c2bcc6032495817dc06b615ea"}
Oct 30 19:42:46.307516 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:42:46.307142    4892 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9a514ead-qjp69" event={"ID":"29c286de-ac5b-4dea-ba6c-565c16549eb9","Type":"ContainerStarted","Data":"c9a36100faf92eaf3d926821936517eeb73e912cea0ba6a05da57c9ef109d3ba"}
Oct 30 19:42:46.421994 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:42:46.421921    4892 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9a514ead-qjp69" podStartSLOduration=1.421908278 podStartE2EDuration="1.421908278s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:42:45 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:42:46.419834157 +0000 UTC m=+1613.990299746" watchObservedRunningTime="2026-10-30 19:42:46.421908278 +0000 UTC m=+1613.992373928"
Oct 30 19:42:46.672215 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:42:46.672173    4892 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-7xzjb"]
Oct 30 19:42:46.672398 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:42:46.672368    4892 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-7xzjb" podUID="cbfb2293-8265-4bf9-bcab-68351ec3b9ce" containerName="istio-proxy" containerID="cri-o://1be086ce255b0acc339d50ed3248afe67be3c89237a38b74d3b4dd0fd96fc436" gracePeriod=30
Oct 30 19:42:47.326508 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:42:47.326467    4892 generic.go:356] "Generic (PLEG): container finished" podID="29c286de-ac5b-4dea-ba6c-565c16549eb9" containerID="dab73619cd618c9218820cd33f34d457cd7d234c2bcc6032495817dc06b615ea" exitCode=0
Oct 30 19:42:47.326894 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:42:47.326550    4892 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9a514ead-qjp69" event={"ID":"29c286de-ac5b-4dea-ba6c-565c16549eb9","Type":"ContainerDied","Data":"dab73619cd618c9218820cd33f34d457cd7d234c2bcc6032495817dc06b615ea"}
Oct 30 19:42:48.479150 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:42:48.479113    4892 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9a514ead-qjp69"
Oct 30 19:42:48.569850 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:42:48.569821    4892 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/29c286de-ac5b-4dea-ba6c-565c16549eb9-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/29c286de-ac5b-4dea-ba6c-565c16549eb9-maas-api-openshift-ca\") pod \"29c286de-ac5b-4dea-ba6c-565c16549eb9\" (UID: \"29c286de-ac5b-4dea-ba6c-565c16549eb9\") "
Oct 30 19:42:48.570012 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:42:48.569866    4892 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/29c286de-ac5b-4dea-ba6c-565c16549eb9-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/29c286de-ac5b-4dea-ba6c-565c16549eb9-maas-api-xks-ca\") pod \"29c286de-ac5b-4dea-ba6c-565c16549eb9\" (UID: \"29c286de-ac5b-4dea-ba6c-565c16549eb9\") "
Oct 30 19:42:48.570012 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:42:48.570005    4892 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/29c286de-ac5b-4dea-ba6c-565c16549eb9-kube-api-access-dm47z\" (UniqueName: \"kubernetes.io/projected/29c286de-ac5b-4dea-ba6c-565c16549eb9-kube-api-access-dm47z\") pod \"29c286de-ac5b-4dea-ba6c-565c16549eb9\" (UID: \"29c286de-ac5b-4dea-ba6c-565c16549eb9\") "
Oct 30 19:42:48.570187 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:42:48.570166    4892 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/29c286de-ac5b-4dea-ba6c-565c16549eb9-maas-api-openshift-ca" pod "29c286de-ac5b-4dea-ba6c-565c16549eb9" (UID: "29c286de-ac5b-4dea-ba6c-565c16549eb9"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 19:42:48.570501 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:42:48.570478    4892 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/29c286de-ac5b-4dea-ba6c-565c16549eb9-maas-api-openshift-ca\") on node \"ip-10-0-132-82.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 19:42:48.572122 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:42:48.572094    4892 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/29c286de-ac5b-4dea-ba6c-565c16549eb9-maas-api-xks-ca" pod "29c286de-ac5b-4dea-ba6c-565c16549eb9" (UID: "29c286de-ac5b-4dea-ba6c-565c16549eb9"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 19:42:48.572199 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:42:48.572163    4892 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/29c286de-ac5b-4dea-ba6c-565c16549eb9-kube-api-access-dm47z" pod "29c286de-ac5b-4dea-ba6c-565c16549eb9" (UID: "29c286de-ac5b-4dea-ba6c-565c16549eb9"). InnerVolumeSpecName "kube-api-access-dm47z". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:42:48.671236 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:42:48.671207    4892 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dm47z\" (UniqueName: \"kubernetes.io/projected/29c286de-ac5b-4dea-ba6c-565c16549eb9-kube-api-access-dm47z\") on node \"ip-10-0-132-82.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 19:42:48.671236 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:42:48.671234    4892 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/29c286de-ac5b-4dea-ba6c-565c16549eb9-maas-api-xks-ca\") on node \"ip-10-0-132-82.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 19:42:49.345283 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:42:49.345255    4892 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9a514ead-qjp69" event={"ID":"29c286de-ac5b-4dea-ba6c-565c16549eb9","Type":"ContainerDied","Data":"c9a36100faf92eaf3d926821936517eeb73e912cea0ba6a05da57c9ef109d3ba"}
Oct 30 19:42:49.345458 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:42:49.345301    4892 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="c9a36100faf92eaf3d926821936517eeb73e912cea0ba6a05da57c9ef109d3ba"
Oct 30 19:42:49.345458 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:42:49.345306    4892 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9a514ead-qjp69"
Oct 30 19:42:50.074461 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:42:50.074427    4892 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6db586dd6c-7xzjb container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Oct 30 19:42:50.074922 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:42:50.074498    4892 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-7xzjb" podUID="cbfb2293-8265-4bf9-bcab-68351ec3b9ce" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Oct 30 19:42:51.740796 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:42:51.740765    4892 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w5-686689-openshift-default-5fbf5787bf-nsql4"]
Oct 30 19:42:51.741477 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:42:51.741458    4892 cpu_manager.go:414] "containerMap: removing container" podUID="29c286de-ac5b-4dea-ba6c-565c16549eb9" containerName="revoke-keys"
Oct 30 19:42:51.741532 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:42:51.741480    4892 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="29c286de-ac5b-4dea-ba6c-565c16549eb9" containerName="revoke-keys"
Oct 30 19:42:51.741592 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:42:51.741583    4892 memory_manager.go:365] "RemoveStaleState removing state" podUID="29c286de-ac5b-4dea-ba6c-565c16549eb9" containerName="revoke-keys"
Oct 30 19:42:51.745115 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:42:51.745100    4892 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-686689-openshift-default-5fbf5787bf-nsql4"
Oct 30 19:42:51.808797 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:42:51.808769    4892 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/dffc80d1-bd47-4493-8cd9-87530034d485-workload-socket\") pod \"e2e-worker-w5-686689-openshift-default-5fbf5787bf-nsql4\" (UID: \"dffc80d1-bd47-4493-8cd9-87530034d485\") " pod="openshift-ingress/e2e-worker-w5-686689-openshift-default-5fbf5787bf-nsql4"
Oct 30 19:42:51.808940 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:42:51.808801    4892 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/dffc80d1-bd47-4493-8cd9-87530034d485-credential-socket\") pod \"e2e-worker-w5-686689-openshift-default-5fbf5787bf-nsql4\" (UID: \"dffc80d1-bd47-4493-8cd9-87530034d485\") " pod="openshift-ingress/e2e-worker-w5-686689-openshift-default-5fbf5787bf-nsql4"
Oct 30 19:42:51.808940 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:42:51.808902    4892 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/dffc80d1-bd47-4493-8cd9-87530034d485-istio-envoy\") pod \"e2e-worker-w5-686689-openshift-default-5fbf5787bf-nsql4\" (UID: \"dffc80d1-bd47-4493-8cd9-87530034d485\") " pod="openshift-ingress/e2e-worker-w5-686689-openshift-default-5fbf5787bf-nsql4"
Oct 30 19:42:51.809145 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:42:51.809122    4892 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/dffc80d1-bd47-4493-8cd9-87530034d485-istio-data\") pod \"e2e-worker-w5-686689-openshift-default-5fbf5787bf-nsql4\" (UID: \"dffc80d1-bd47-4493-8cd9-87530034d485\") " pod="openshift-ingress/e2e-worker-w5-686689-openshift-default-5fbf5787bf-nsql4"
Oct 30 19:42:51.809209 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:42:51.809160    4892 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/dffc80d1-bd47-4493-8cd9-87530034d485-istio-podinfo\") pod \"e2e-worker-w5-686689-openshift-default-5fbf5787bf-nsql4\" (UID: \"dffc80d1-bd47-4493-8cd9-87530034d485\") " pod="openshift-ingress/e2e-worker-w5-686689-openshift-default-5fbf5787bf-nsql4"
Oct 30 19:42:51.809268 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:42:51.809239    4892 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6c89v\" (UniqueName: \"kubernetes.io/projected/dffc80d1-bd47-4493-8cd9-87530034d485-kube-api-access-6c89v\") pod \"e2e-worker-w5-686689-openshift-default-5fbf5787bf-nsql4\" (UID: \"dffc80d1-bd47-4493-8cd9-87530034d485\") " pod="openshift-ingress/e2e-worker-w5-686689-openshift-default-5fbf5787bf-nsql4"
Oct 30 19:42:51.809318 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:42:51.809278    4892 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/dffc80d1-bd47-4493-8cd9-87530034d485-workload-certs\") pod \"e2e-worker-w5-686689-openshift-default-5fbf5787bf-nsql4\" (UID: \"dffc80d1-bd47-4493-8cd9-87530034d485\") " pod="openshift-ingress/e2e-worker-w5-686689-openshift-default-5fbf5787bf-nsql4"
Oct 30 19:42:51.809360 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:42:51.809338    4892 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/dffc80d1-bd47-4493-8cd9-87530034d485-istio-token\") pod \"e2e-worker-w5-686689-openshift-default-5fbf5787bf-nsql4\" (UID: \"dffc80d1-bd47-4493-8cd9-87530034d485\") " pod="openshift-ingress/e2e-worker-w5-686689-openshift-default-5fbf5787bf-nsql4"
Oct 30 19:42:51.809401 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:42:51.809379    4892 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/dffc80d1-bd47-4493-8cd9-87530034d485-istiod-ca-cert\") pod \"e2e-worker-w5-686689-openshift-default-5fbf5787bf-nsql4\" (UID: \"dffc80d1-bd47-4493-8cd9-87530034d485\") " pod="openshift-ingress/e2e-worker-w5-686689-openshift-default-5fbf5787bf-nsql4"
Oct 30 19:42:51.910583 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:42:51.910551    4892 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6c89v\" (UniqueName: \"kubernetes.io/projected/dffc80d1-bd47-4493-8cd9-87530034d485-kube-api-access-6c89v\") pod \"e2e-worker-w5-686689-openshift-default-5fbf5787bf-nsql4\" (UID: \"dffc80d1-bd47-4493-8cd9-87530034d485\") " pod="openshift-ingress/e2e-worker-w5-686689-openshift-default-5fbf5787bf-nsql4"
Oct 30 19:42:51.910752 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:42:51.910599    4892 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/dffc80d1-bd47-4493-8cd9-87530034d485-workload-certs\") pod \"e2e-worker-w5-686689-openshift-default-5fbf5787bf-nsql4\" (UID: \"dffc80d1-bd47-4493-8cd9-87530034d485\") " pod="openshift-ingress/e2e-worker-w5-686689-openshift-default-5fbf5787bf-nsql4"
Oct 30 19:42:51.910752 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:42:51.910636    4892 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/dffc80d1-bd47-4493-8cd9-87530034d485-istio-token\") pod \"e2e-worker-w5-686689-openshift-default-5fbf5787bf-nsql4\" (UID: \"dffc80d1-bd47-4493-8cd9-87530034d485\") " pod="openshift-ingress/e2e-worker-w5-686689-openshift-default-5fbf5787bf-nsql4"
Oct 30 19:42:51.910752 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:42:51.910654    4892 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/dffc80d1-bd47-4493-8cd9-87530034d485-istiod-ca-cert\") pod \"e2e-worker-w5-686689-openshift-default-5fbf5787bf-nsql4\" (UID: \"dffc80d1-bd47-4493-8cd9-87530034d485\") " pod="openshift-ingress/e2e-worker-w5-686689-openshift-default-5fbf5787bf-nsql4"
Oct 30 19:42:51.910752 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:42:51.910676    4892 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/dffc80d1-bd47-4493-8cd9-87530034d485-workload-socket\") pod \"e2e-worker-w5-686689-openshift-default-5fbf5787bf-nsql4\" (UID: \"dffc80d1-bd47-4493-8cd9-87530034d485\") " pod="openshift-ingress/e2e-worker-w5-686689-openshift-default-5fbf5787bf-nsql4"
Oct 30 19:42:51.910752 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:42:51.910690    4892 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/dffc80d1-bd47-4493-8cd9-87530034d485-credential-socket\") pod \"e2e-worker-w5-686689-openshift-default-5fbf5787bf-nsql4\" (UID: \"dffc80d1-bd47-4493-8cd9-87530034d485\") " pod="openshift-ingress/e2e-worker-w5-686689-openshift-default-5fbf5787bf-nsql4"
Oct 30 19:42:51.911067 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:42:51.910757    4892 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/dffc80d1-bd47-4493-8cd9-87530034d485-istio-envoy\") pod \"e2e-worker-w5-686689-openshift-default-5fbf5787bf-nsql4\" (UID: \"dffc80d1-bd47-4493-8cd9-87530034d485\") " pod="openshift-ingress/e2e-worker-w5-686689-openshift-default-5fbf5787bf-nsql4"
Oct 30 19:42:51.911067 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:42:51.910863    4892 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/dffc80d1-bd47-4493-8cd9-87530034d485-istio-data\") pod \"e2e-worker-w5-686689-openshift-default-5fbf5787bf-nsql4\" (UID: \"dffc80d1-bd47-4493-8cd9-87530034d485\") " pod="openshift-ingress/e2e-worker-w5-686689-openshift-default-5fbf5787bf-nsql4"
Oct 30 19:42:51.911067 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:42:51.910877    4892 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/dffc80d1-bd47-4493-8cd9-87530034d485-istio-podinfo\") pod \"e2e-worker-w5-686689-openshift-default-5fbf5787bf-nsql4\" (UID: \"dffc80d1-bd47-4493-8cd9-87530034d485\") " pod="openshift-ingress/e2e-worker-w5-686689-openshift-default-5fbf5787bf-nsql4"
Oct 30 19:42:51.911232 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:42:51.911068    4892 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/dffc80d1-bd47-4493-8cd9-87530034d485-workload-certs\") pod \"e2e-worker-w5-686689-openshift-default-5fbf5787bf-nsql4\" (UID: \"dffc80d1-bd47-4493-8cd9-87530034d485\") " pod="openshift-ingress/e2e-worker-w5-686689-openshift-default-5fbf5787bf-nsql4"
Oct 30 19:42:51.911286 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:42:51.911230    4892 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/dffc80d1-bd47-4493-8cd9-87530034d485-workload-socket\") pod \"e2e-worker-w5-686689-openshift-default-5fbf5787bf-nsql4\" (UID: \"dffc80d1-bd47-4493-8cd9-87530034d485\") " pod="openshift-ingress/e2e-worker-w5-686689-openshift-default-5fbf5787bf-nsql4"
Oct 30 19:42:51.911445 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:42:51.911415    4892 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/dffc80d1-bd47-4493-8cd9-87530034d485-istio-data\") pod \"e2e-worker-w5-686689-openshift-default-5fbf5787bf-nsql4\" (UID: \"dffc80d1-bd47-4493-8cd9-87530034d485\") " pod="openshift-ingress/e2e-worker-w5-686689-openshift-default-5fbf5787bf-nsql4"
Oct 30 19:42:51.911717 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:42:51.911692    4892 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/dffc80d1-bd47-4493-8cd9-87530034d485-credential-socket\") pod \"e2e-worker-w5-686689-openshift-default-5fbf5787bf-nsql4\" (UID: \"dffc80d1-bd47-4493-8cd9-87530034d485\") " pod="openshift-ingress/e2e-worker-w5-686689-openshift-default-5fbf5787bf-nsql4"
Oct 30 19:42:51.912054 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:42:51.912025    4892 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/dffc80d1-bd47-4493-8cd9-87530034d485-istiod-ca-cert\") pod \"e2e-worker-w5-686689-openshift-default-5fbf5787bf-nsql4\" (UID: \"dffc80d1-bd47-4493-8cd9-87530034d485\") " pod="openshift-ingress/e2e-worker-w5-686689-openshift-default-5fbf5787bf-nsql4"
Oct 30 19:42:51.914756 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:42:51.914707    4892 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/dffc80d1-bd47-4493-8cd9-87530034d485-istio-envoy\") pod \"e2e-worker-w5-686689-openshift-default-5fbf5787bf-nsql4\" (UID: \"dffc80d1-bd47-4493-8cd9-87530034d485\") " pod="openshift-ingress/e2e-worker-w5-686689-openshift-default-5fbf5787bf-nsql4"
Oct 30 19:42:51.914843 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:42:51.914756    4892 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/dffc80d1-bd47-4493-8cd9-87530034d485-istio-podinfo\") pod \"e2e-worker-w5-686689-openshift-default-5fbf5787bf-nsql4\" (UID: \"dffc80d1-bd47-4493-8cd9-87530034d485\") " pod="openshift-ingress/e2e-worker-w5-686689-openshift-default-5fbf5787bf-nsql4"
Oct 30 19:42:51.958829 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:42:51.958801    4892 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w5-686689-openshift-default-5fbf5787bf-nsql4"]
Oct 30 19:42:51.965355 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:42:51.965335    4892 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-7xzjb"
Oct 30 19:42:52.001831 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:42:52.001228    4892 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6c89v\" (UniqueName: \"kubernetes.io/projected/dffc80d1-bd47-4493-8cd9-87530034d485-kube-api-access-6c89v\") pod \"e2e-worker-w5-686689-openshift-default-5fbf5787bf-nsql4\" (UID: \"dffc80d1-bd47-4493-8cd9-87530034d485\") " pod="openshift-ingress/e2e-worker-w5-686689-openshift-default-5fbf5787bf-nsql4"
Oct 30 19:42:52.020027 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:42:52.017844    4892 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/cbfb2293-8265-4bf9-bcab-68351ec3b9ce-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/cbfb2293-8265-4bf9-bcab-68351ec3b9ce-istio-envoy\") pod \"cbfb2293-8265-4bf9-bcab-68351ec3b9ce\" (UID: \"cbfb2293-8265-4bf9-bcab-68351ec3b9ce\") "
Oct 30 19:42:52.033030 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:42:52.020859    4892 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/cbfb2293-8265-4bf9-bcab-68351ec3b9ce-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/cbfb2293-8265-4bf9-bcab-68351ec3b9ce-workload-certs\") pod \"cbfb2293-8265-4bf9-bcab-68351ec3b9ce\" (UID: \"cbfb2293-8265-4bf9-bcab-68351ec3b9ce\") "
Oct 30 19:42:52.033030 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:42:52.020994    4892 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/cbfb2293-8265-4bf9-bcab-68351ec3b9ce-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/cbfb2293-8265-4bf9-bcab-68351ec3b9ce-istio-data\") pod \"cbfb2293-8265-4bf9-bcab-68351ec3b9ce\" (UID: \"cbfb2293-8265-4bf9-bcab-68351ec3b9ce\") "
Oct 30 19:42:52.033030 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:42:52.021088    4892 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/cbfb2293-8265-4bf9-bcab-68351ec3b9ce-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/cbfb2293-8265-4bf9-bcab-68351ec3b9ce-workload-socket\") pod \"cbfb2293-8265-4bf9-bcab-68351ec3b9ce\" (UID: \"cbfb2293-8265-4bf9-bcab-68351ec3b9ce\") "
Oct 30 19:42:52.033030 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:42:52.021119    4892 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/cbfb2293-8265-4bf9-bcab-68351ec3b9ce-kube-api-access-rv7sf\" (UniqueName: \"kubernetes.io/projected/cbfb2293-8265-4bf9-bcab-68351ec3b9ce-kube-api-access-rv7sf\") pod \"cbfb2293-8265-4bf9-bcab-68351ec3b9ce\" (UID: \"cbfb2293-8265-4bf9-bcab-68351ec3b9ce\") "
Oct 30 19:42:52.033030 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:42:52.021199    4892 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/cbfb2293-8265-4bf9-bcab-68351ec3b9ce-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/cbfb2293-8265-4bf9-bcab-68351ec3b9ce-credential-socket\") pod \"cbfb2293-8265-4bf9-bcab-68351ec3b9ce\" (UID: \"cbfb2293-8265-4bf9-bcab-68351ec3b9ce\") "
Oct 30 19:42:52.033030 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:42:52.021250    4892 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/cbfb2293-8265-4bf9-bcab-68351ec3b9ce-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/cbfb2293-8265-4bf9-bcab-68351ec3b9ce-istiod-ca-cert\") pod \"cbfb2293-8265-4bf9-bcab-68351ec3b9ce\" (UID: \"cbfb2293-8265-4bf9-bcab-68351ec3b9ce\") "
Oct 30 19:42:52.033030 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:42:52.021282    4892 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/cbfb2293-8265-4bf9-bcab-68351ec3b9ce-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/cbfb2293-8265-4bf9-bcab-68351ec3b9ce-istio-podinfo\") pod \"cbfb2293-8265-4bf9-bcab-68351ec3b9ce\" (UID: \"cbfb2293-8265-4bf9-bcab-68351ec3b9ce\") "
Oct 30 19:42:52.033030 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:42:52.021377    4892 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/cbfb2293-8265-4bf9-bcab-68351ec3b9ce-istio-token\" (UniqueName: \"kubernetes.io/projected/cbfb2293-8265-4bf9-bcab-68351ec3b9ce-istio-token\") pod \"cbfb2293-8265-4bf9-bcab-68351ec3b9ce\" (UID: \"cbfb2293-8265-4bf9-bcab-68351ec3b9ce\") "
Oct 30 19:42:52.033030 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:42:52.023857    4892 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cbfb2293-8265-4bf9-bcab-68351ec3b9ce-workload-socket" pod "cbfb2293-8265-4bf9-bcab-68351ec3b9ce" (UID: "cbfb2293-8265-4bf9-bcab-68351ec3b9ce"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:42:52.033030 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:42:52.024127    4892 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cbfb2293-8265-4bf9-bcab-68351ec3b9ce-credential-socket" pod "cbfb2293-8265-4bf9-bcab-68351ec3b9ce" (UID: "cbfb2293-8265-4bf9-bcab-68351ec3b9ce"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:42:52.033030 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:42:52.024369    4892 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cbfb2293-8265-4bf9-bcab-68351ec3b9ce-workload-certs" pod "cbfb2293-8265-4bf9-bcab-68351ec3b9ce" (UID: "cbfb2293-8265-4bf9-bcab-68351ec3b9ce"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:42:52.033030 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:42:52.025009    4892 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cbfb2293-8265-4bf9-bcab-68351ec3b9ce-istio-data" pod "cbfb2293-8265-4bf9-bcab-68351ec3b9ce" (UID: "cbfb2293-8265-4bf9-bcab-68351ec3b9ce"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:42:52.033030 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:42:52.025211    4892 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/cbfb2293-8265-4bf9-bcab-68351ec3b9ce-istiod-ca-cert" pod "cbfb2293-8265-4bf9-bcab-68351ec3b9ce" (UID: "cbfb2293-8265-4bf9-bcab-68351ec3b9ce"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 19:42:52.033030 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:42:52.025680    4892 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/dffc80d1-bd47-4493-8cd9-87530034d485-istio-token\") pod \"e2e-worker-w5-686689-openshift-default-5fbf5787bf-nsql4\" (UID: \"dffc80d1-bd47-4493-8cd9-87530034d485\") " pod="openshift-ingress/e2e-worker-w5-686689-openshift-default-5fbf5787bf-nsql4"
Oct 30 19:42:52.033030 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:42:52.026013    4892 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/cbfb2293-8265-4bf9-bcab-68351ec3b9ce-workload-certs\") on node \"ip-10-0-132-82.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 19:42:52.033030 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:42:52.026034    4892 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/cbfb2293-8265-4bf9-bcab-68351ec3b9ce-istio-data\") on node \"ip-10-0-132-82.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 19:42:52.033030 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:42:52.026048    4892 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/cbfb2293-8265-4bf9-bcab-68351ec3b9ce-workload-socket\") on node \"ip-10-0-132-82.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 19:42:52.033030 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:42:52.026062    4892 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/cbfb2293-8265-4bf9-bcab-68351ec3b9ce-credential-socket\") on node \"ip-10-0-132-82.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 19:42:52.033030 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:42:52.026076    4892 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/cbfb2293-8265-4bf9-bcab-68351ec3b9ce-istiod-ca-cert\") on node \"ip-10-0-132-82.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 19:42:52.066640 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:42:52.057266    4892 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/cbfb2293-8265-4bf9-bcab-68351ec3b9ce-kube-api-access-rv7sf" pod "cbfb2293-8265-4bf9-bcab-68351ec3b9ce" (UID: "cbfb2293-8265-4bf9-bcab-68351ec3b9ce"). InnerVolumeSpecName "kube-api-access-rv7sf". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:42:52.066640 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:42:52.058224    4892 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/cbfb2293-8265-4bf9-bcab-68351ec3b9ce-istio-token" pod "cbfb2293-8265-4bf9-bcab-68351ec3b9ce" (UID: "cbfb2293-8265-4bf9-bcab-68351ec3b9ce"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:42:52.074825 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:42:52.070628    4892 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cbfb2293-8265-4bf9-bcab-68351ec3b9ce-istio-envoy" pod "cbfb2293-8265-4bf9-bcab-68351ec3b9ce" (UID: "cbfb2293-8265-4bf9-bcab-68351ec3b9ce"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:42:52.074825 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:42:52.070933    4892 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-686689-openshift-default-5fbf5787bf-nsql4"
Oct 30 19:42:52.074825 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:42:52.072776    4892 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/cbfb2293-8265-4bf9-bcab-68351ec3b9ce-istio-podinfo" pod "cbfb2293-8265-4bf9-bcab-68351ec3b9ce" (UID: "cbfb2293-8265-4bf9-bcab-68351ec3b9ce"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 19:42:52.076741 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:42:52.076693    4892 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a5b01630-p8ch7"]
Oct 30 19:42:52.081837 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:42:52.081815    4892 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a5b01630-p8ch7"]
Oct 30 19:42:52.127864 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:42:52.127766    4892 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rv7sf\" (UniqueName: \"kubernetes.io/projected/cbfb2293-8265-4bf9-bcab-68351ec3b9ce-kube-api-access-rv7sf\") on node \"ip-10-0-132-82.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 19:42:52.127864 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:42:52.127799    4892 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/cbfb2293-8265-4bf9-bcab-68351ec3b9ce-istio-podinfo\") on node \"ip-10-0-132-82.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 19:42:52.127864 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:42:52.127824    4892 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/cbfb2293-8265-4bf9-bcab-68351ec3b9ce-istio-token\") on node \"ip-10-0-132-82.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 19:42:52.127864 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:42:52.127840    4892 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/cbfb2293-8265-4bf9-bcab-68351ec3b9ce-istio-envoy\") on node \"ip-10-0-132-82.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 19:42:52.369560 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:42:52.369485    4892 generic.go:356] "Generic (PLEG): container finished" podID="cbfb2293-8265-4bf9-bcab-68351ec3b9ce" containerID="1be086ce255b0acc339d50ed3248afe67be3c89237a38b74d3b4dd0fd96fc436" exitCode=0
Oct 30 19:42:52.369560 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:42:52.369537    4892 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-7xzjb"
Oct 30 19:42:52.369737 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:42:52.369564    4892 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-7xzjb" event={"ID":"cbfb2293-8265-4bf9-bcab-68351ec3b9ce","Type":"ContainerDied","Data":"1be086ce255b0acc339d50ed3248afe67be3c89237a38b74d3b4dd0fd96fc436"}
Oct 30 19:42:52.369737 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:42:52.369616    4892 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-7xzjb" event={"ID":"cbfb2293-8265-4bf9-bcab-68351ec3b9ce","Type":"ContainerDied","Data":"a837a2f159ef3e2a3b7d2869fe808b2ab4403344e740e176dddaf49d17d7675c"}
Oct 30 19:42:52.369737 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:42:52.369638    4892 scope.go:122] "RemoveContainer" containerID="1be086ce255b0acc339d50ed3248afe67be3c89237a38b74d3b4dd0fd96fc436"
Oct 30 19:42:52.380880 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:42:52.380861    4892 scope.go:122] "RemoveContainer" containerID="1be086ce255b0acc339d50ed3248afe67be3c89237a38b74d3b4dd0fd96fc436"
Oct 30 19:42:52.381209 ip-10-0-132-82 kubenswrapper[4892]: E1030 19:42:52.381184    4892 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1be086ce255b0acc339d50ed3248afe67be3c89237a38b74d3b4dd0fd96fc436\": container with ID starting with 1be086ce255b0acc339d50ed3248afe67be3c89237a38b74d3b4dd0fd96fc436 not found: ID does not exist" containerID="1be086ce255b0acc339d50ed3248afe67be3c89237a38b74d3b4dd0fd96fc436"
Oct 30 19:42:52.381295 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:42:52.381222    4892 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1be086ce255b0acc339d50ed3248afe67be3c89237a38b74d3b4dd0fd96fc436"} err="failed to get container status \"1be086ce255b0acc339d50ed3248afe67be3c89237a38b74d3b4dd0fd96fc436\": rpc error: code = NotFound desc = could not find container \"1be086ce255b0acc339d50ed3248afe67be3c89237a38b74d3b4dd0fd96fc436\": container with ID starting with 1be086ce255b0acc339d50ed3248afe67be3c89237a38b74d3b4dd0fd96fc436 not found: ID does not exist"
Oct 30 19:42:52.531605 ip-10-0-132-82 kubenswrapper[4892]: W1030 19:42:52.531578    4892 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddffc80d1_bd47_4493_8cd9_87530034d485.slice/crio-cdfbbc85084b57edab9bf71c947c433a2b6793b930e52c0c4a0a0310f5235c69 WatchSource:0}: Error finding container cdfbbc85084b57edab9bf71c947c433a2b6793b930e52c0c4a0a0310f5235c69: Status 404 returned error can't find the container with id cdfbbc85084b57edab9bf71c947c433a2b6793b930e52c0c4a0a0310f5235c69
Oct 30 19:42:52.532547 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:42:52.532520    4892 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w5-686689-openshift-default-5fbf5787bf-nsql4"]
Oct 30 19:42:52.533822 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:42:52.533794    4892 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892000Ki","pods":"250"}
Oct 30 19:42:52.533877 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:42:52.533865    4892 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892000Ki","pods":"250"}
Oct 30 19:42:52.533918 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:42:52.533908    4892 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892000Ki","pods":"250"}
Oct 30 19:42:52.549282 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:42:52.549259    4892 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-7xzjb"]
Oct 30 19:42:52.561264 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:42:52.561240    4892 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-7xzjb"]
Oct 30 19:42:52.576780 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:42:52.576756    4892 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="cbfb2293-8265-4bf9-bcab-68351ec3b9ce" path="/var/lib/kubelet/pods/cbfb2293-8265-4bf9-bcab-68351ec3b9ce/volumes"
Oct 30 19:42:52.577221 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:42:52.577205    4892 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ee5f11d0-f573-4583-93af-1a7ec44d49ea" path="/var/lib/kubelet/pods/ee5f11d0-f573-4583-93af-1a7ec44d49ea/volumes"
Oct 30 19:42:52.664634 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:42:52.664601    4892 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w6-5f55cd-openshift-default-9dc497cb5-q2fbg"]
Oct 30 19:42:52.665672 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:42:52.665648    4892 cpu_manager.go:414] "containerMap: removing container" podUID="cbfb2293-8265-4bf9-bcab-68351ec3b9ce" containerName="istio-proxy"
Oct 30 19:42:52.665779 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:42:52.665673    4892 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="cbfb2293-8265-4bf9-bcab-68351ec3b9ce" containerName="istio-proxy"
Oct 30 19:42:52.665849 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:42:52.665837    4892 memory_manager.go:365] "RemoveStaleState removing state" podUID="cbfb2293-8265-4bf9-bcab-68351ec3b9ce" containerName="istio-proxy"
Oct 30 19:42:52.670126 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:42:52.670104    4892 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-5f55cd-openshift-default-9dc497cb5-q2fbg"
Oct 30 19:42:52.739438 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:42:52.732845    4892 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w6-5f55cd-openshift-default-dockercfg-wh68v\""
Oct 30 19:42:52.742540 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:42:52.741539    4892 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/fac4fe86-a38e-434e-bfc1-4c3dbceffd27-istio-data\") pod \"e2e-worker-w6-5f55cd-openshift-default-9dc497cb5-q2fbg\" (UID: \"fac4fe86-a38e-434e-bfc1-4c3dbceffd27\") " pod="openshift-ingress/e2e-worker-w6-5f55cd-openshift-default-9dc497cb5-q2fbg"
Oct 30 19:42:52.743803 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:42:52.743781    4892 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/fac4fe86-a38e-434e-bfc1-4c3dbceffd27-credential-socket\") pod \"e2e-worker-w6-5f55cd-openshift-default-9dc497cb5-q2fbg\" (UID: \"fac4fe86-a38e-434e-bfc1-4c3dbceffd27\") " pod="openshift-ingress/e2e-worker-w6-5f55cd-openshift-default-9dc497cb5-q2fbg"
Oct 30 19:42:52.744035 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:42:52.744016    4892 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/fac4fe86-a38e-434e-bfc1-4c3dbceffd27-istio-envoy\") pod \"e2e-worker-w6-5f55cd-openshift-default-9dc497cb5-q2fbg\" (UID: \"fac4fe86-a38e-434e-bfc1-4c3dbceffd27\") " pod="openshift-ingress/e2e-worker-w6-5f55cd-openshift-default-9dc497cb5-q2fbg"
Oct 30 19:42:52.744659 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:42:52.744638    4892 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vv9kc\" (UniqueName: \"kubernetes.io/projected/fac4fe86-a38e-434e-bfc1-4c3dbceffd27-kube-api-access-vv9kc\") pod \"e2e-worker-w6-5f55cd-openshift-default-9dc497cb5-q2fbg\" (UID: \"fac4fe86-a38e-434e-bfc1-4c3dbceffd27\") " pod="openshift-ingress/e2e-worker-w6-5f55cd-openshift-default-9dc497cb5-q2fbg"
Oct 30 19:42:52.744816 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:42:52.744801    4892 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/fac4fe86-a38e-434e-bfc1-4c3dbceffd27-workload-certs\") pod \"e2e-worker-w6-5f55cd-openshift-default-9dc497cb5-q2fbg\" (UID: \"fac4fe86-a38e-434e-bfc1-4c3dbceffd27\") " pod="openshift-ingress/e2e-worker-w6-5f55cd-openshift-default-9dc497cb5-q2fbg"
Oct 30 19:42:52.745045 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:42:52.745029    4892 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/fac4fe86-a38e-434e-bfc1-4c3dbceffd27-istio-token\") pod \"e2e-worker-w6-5f55cd-openshift-default-9dc497cb5-q2fbg\" (UID: \"fac4fe86-a38e-434e-bfc1-4c3dbceffd27\") " pod="openshift-ingress/e2e-worker-w6-5f55cd-openshift-default-9dc497cb5-q2fbg"
Oct 30 19:42:52.745202 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:42:52.745188    4892 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/fac4fe86-a38e-434e-bfc1-4c3dbceffd27-workload-socket\") pod \"e2e-worker-w6-5f55cd-openshift-default-9dc497cb5-q2fbg\" (UID: \"fac4fe86-a38e-434e-bfc1-4c3dbceffd27\") " pod="openshift-ingress/e2e-worker-w6-5f55cd-openshift-default-9dc497cb5-q2fbg"
Oct 30 19:42:52.745489 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:42:52.745473    4892 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/fac4fe86-a38e-434e-bfc1-4c3dbceffd27-istio-podinfo\") pod \"e2e-worker-w6-5f55cd-openshift-default-9dc497cb5-q2fbg\" (UID: \"fac4fe86-a38e-434e-bfc1-4c3dbceffd27\") " pod="openshift-ingress/e2e-worker-w6-5f55cd-openshift-default-9dc497cb5-q2fbg"
Oct 30 19:42:52.745855 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:42:52.745838    4892 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/fac4fe86-a38e-434e-bfc1-4c3dbceffd27-istiod-ca-cert\") pod \"e2e-worker-w6-5f55cd-openshift-default-9dc497cb5-q2fbg\" (UID: \"fac4fe86-a38e-434e-bfc1-4c3dbceffd27\") " pod="openshift-ingress/e2e-worker-w6-5f55cd-openshift-default-9dc497cb5-q2fbg"
Oct 30 19:42:52.746127 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:42:52.745798    4892 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w6-5f55cd-openshift-default-9dc497cb5-q2fbg"]
Oct 30 19:42:52.847180 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:42:52.847144    4892 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/fac4fe86-a38e-434e-bfc1-4c3dbceffd27-istio-podinfo\") pod \"e2e-worker-w6-5f55cd-openshift-default-9dc497cb5-q2fbg\" (UID: \"fac4fe86-a38e-434e-bfc1-4c3dbceffd27\") " pod="openshift-ingress/e2e-worker-w6-5f55cd-openshift-default-9dc497cb5-q2fbg"
Oct 30 19:42:52.847380 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:42:52.847190    4892 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/fac4fe86-a38e-434e-bfc1-4c3dbceffd27-istiod-ca-cert\") pod \"e2e-worker-w6-5f55cd-openshift-default-9dc497cb5-q2fbg\" (UID: \"fac4fe86-a38e-434e-bfc1-4c3dbceffd27\") " pod="openshift-ingress/e2e-worker-w6-5f55cd-openshift-default-9dc497cb5-q2fbg"
Oct 30 19:42:52.847380 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:42:52.847275    4892 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/fac4fe86-a38e-434e-bfc1-4c3dbceffd27-istio-data\") pod \"e2e-worker-w6-5f55cd-openshift-default-9dc497cb5-q2fbg\" (UID: \"fac4fe86-a38e-434e-bfc1-4c3dbceffd27\") " pod="openshift-ingress/e2e-worker-w6-5f55cd-openshift-default-9dc497cb5-q2fbg"
Oct 30 19:42:52.847380 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:42:52.847354    4892 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/fac4fe86-a38e-434e-bfc1-4c3dbceffd27-credential-socket\") pod \"e2e-worker-w6-5f55cd-openshift-default-9dc497cb5-q2fbg\" (UID: \"fac4fe86-a38e-434e-bfc1-4c3dbceffd27\") " pod="openshift-ingress/e2e-worker-w6-5f55cd-openshift-default-9dc497cb5-q2fbg"
Oct 30 19:42:52.847380 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:42:52.847379    4892 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/fac4fe86-a38e-434e-bfc1-4c3dbceffd27-istio-envoy\") pod \"e2e-worker-w6-5f55cd-openshift-default-9dc497cb5-q2fbg\" (UID: \"fac4fe86-a38e-434e-bfc1-4c3dbceffd27\") " pod="openshift-ingress/e2e-worker-w6-5f55cd-openshift-default-9dc497cb5-q2fbg"
Oct 30 19:42:52.847621 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:42:52.847405    4892 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vv9kc\" (UniqueName: \"kubernetes.io/projected/fac4fe86-a38e-434e-bfc1-4c3dbceffd27-kube-api-access-vv9kc\") pod \"e2e-worker-w6-5f55cd-openshift-default-9dc497cb5-q2fbg\" (UID: \"fac4fe86-a38e-434e-bfc1-4c3dbceffd27\") " pod="openshift-ingress/e2e-worker-w6-5f55cd-openshift-default-9dc497cb5-q2fbg"
Oct 30 19:42:52.847621 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:42:52.847438    4892 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/fac4fe86-a38e-434e-bfc1-4c3dbceffd27-workload-certs\") pod \"e2e-worker-w6-5f55cd-openshift-default-9dc497cb5-q2fbg\" (UID: \"fac4fe86-a38e-434e-bfc1-4c3dbceffd27\") " pod="openshift-ingress/e2e-worker-w6-5f55cd-openshift-default-9dc497cb5-q2fbg"
Oct 30 19:42:52.847621 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:42:52.847463    4892 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/fac4fe86-a38e-434e-bfc1-4c3dbceffd27-istio-token\") pod \"e2e-worker-w6-5f55cd-openshift-default-9dc497cb5-q2fbg\" (UID: \"fac4fe86-a38e-434e-bfc1-4c3dbceffd27\") " pod="openshift-ingress/e2e-worker-w6-5f55cd-openshift-default-9dc497cb5-q2fbg"
Oct 30 19:42:52.847621 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:42:52.847488    4892 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/fac4fe86-a38e-434e-bfc1-4c3dbceffd27-workload-socket\") pod \"e2e-worker-w6-5f55cd-openshift-default-9dc497cb5-q2fbg\" (UID: \"fac4fe86-a38e-434e-bfc1-4c3dbceffd27\") " pod="openshift-ingress/e2e-worker-w6-5f55cd-openshift-default-9dc497cb5-q2fbg"
Oct 30 19:42:52.847851 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:42:52.847758    4892 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/fac4fe86-a38e-434e-bfc1-4c3dbceffd27-istio-data\") pod \"e2e-worker-w6-5f55cd-openshift-default-9dc497cb5-q2fbg\" (UID: \"fac4fe86-a38e-434e-bfc1-4c3dbceffd27\") " pod="openshift-ingress/e2e-worker-w6-5f55cd-openshift-default-9dc497cb5-q2fbg"
Oct 30 19:42:52.847851 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:42:52.847827    4892 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/fac4fe86-a38e-434e-bfc1-4c3dbceffd27-credential-socket\") pod \"e2e-worker-w6-5f55cd-openshift-default-9dc497cb5-q2fbg\" (UID: \"fac4fe86-a38e-434e-bfc1-4c3dbceffd27\") " pod="openshift-ingress/e2e-worker-w6-5f55cd-openshift-default-9dc497cb5-q2fbg"
Oct 30 19:42:52.848026 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:42:52.847994    4892 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/fac4fe86-a38e-434e-bfc1-4c3dbceffd27-workload-certs\") pod \"e2e-worker-w6-5f55cd-openshift-default-9dc497cb5-q2fbg\" (UID: \"fac4fe86-a38e-434e-bfc1-4c3dbceffd27\") " pod="openshift-ingress/e2e-worker-w6-5f55cd-openshift-default-9dc497cb5-q2fbg"
Oct 30 19:42:52.848144 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:42:52.848122    4892 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/fac4fe86-a38e-434e-bfc1-4c3dbceffd27-istiod-ca-cert\") pod \"e2e-worker-w6-5f55cd-openshift-default-9dc497cb5-q2fbg\" (UID: \"fac4fe86-a38e-434e-bfc1-4c3dbceffd27\") " pod="openshift-ingress/e2e-worker-w6-5f55cd-openshift-default-9dc497cb5-q2fbg"
Oct 30 19:42:52.848617 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:42:52.848592    4892 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/fac4fe86-a38e-434e-bfc1-4c3dbceffd27-workload-socket\") pod \"e2e-worker-w6-5f55cd-openshift-default-9dc497cb5-q2fbg\" (UID: \"fac4fe86-a38e-434e-bfc1-4c3dbceffd27\") " pod="openshift-ingress/e2e-worker-w6-5f55cd-openshift-default-9dc497cb5-q2fbg"
Oct 30 19:42:52.850316 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:42:52.850276    4892 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/fac4fe86-a38e-434e-bfc1-4c3dbceffd27-istio-envoy\") pod \"e2e-worker-w6-5f55cd-openshift-default-9dc497cb5-q2fbg\" (UID: \"fac4fe86-a38e-434e-bfc1-4c3dbceffd27\") " pod="openshift-ingress/e2e-worker-w6-5f55cd-openshift-default-9dc497cb5-q2fbg"
Oct 30 19:42:52.850396 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:42:52.850292    4892 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/fac4fe86-a38e-434e-bfc1-4c3dbceffd27-istio-podinfo\") pod \"e2e-worker-w6-5f55cd-openshift-default-9dc497cb5-q2fbg\" (UID: \"fac4fe86-a38e-434e-bfc1-4c3dbceffd27\") " pod="openshift-ingress/e2e-worker-w6-5f55cd-openshift-default-9dc497cb5-q2fbg"
Oct 30 19:42:52.904889 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:42:52.904858    4892 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vv9kc\" (UniqueName: \"kubernetes.io/projected/fac4fe86-a38e-434e-bfc1-4c3dbceffd27-kube-api-access-vv9kc\") pod \"e2e-worker-w6-5f55cd-openshift-default-9dc497cb5-q2fbg\" (UID: \"fac4fe86-a38e-434e-bfc1-4c3dbceffd27\") " pod="openshift-ingress/e2e-worker-w6-5f55cd-openshift-default-9dc497cb5-q2fbg"
Oct 30 19:42:52.905110 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:42:52.905090    4892 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/fac4fe86-a38e-434e-bfc1-4c3dbceffd27-istio-token\") pod \"e2e-worker-w6-5f55cd-openshift-default-9dc497cb5-q2fbg\" (UID: \"fac4fe86-a38e-434e-bfc1-4c3dbceffd27\") " pod="openshift-ingress/e2e-worker-w6-5f55cd-openshift-default-9dc497cb5-q2fbg"
Oct 30 19:42:52.989338 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:42:52.989252    4892 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-5f55cd-openshift-default-9dc497cb5-q2fbg"
Oct 30 19:42:53.067807 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:42:53.062410    4892 scope.go:122] "RemoveContainer" containerID="3190864233b07b72014fee8365fbcee348c60add7739c13f42a6127c662dda91"
Oct 30 19:42:53.084065 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:42:53.084046    4892 scope.go:122] "RemoveContainer" containerID="93be56df23857f3096cc7aadba9a1d6221683899fc0d3cf07cade71a8f99d832"
Oct 30 19:42:53.103415 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:42:53.103392    4892 scope.go:122] "RemoveContainer" containerID="f0364ff0cfe6520dce00be5b77b09e5b0f6ac68d4cac81f0bf1eb7d3ecd0b820"
Oct 30 19:42:53.116794 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:42:53.116694    4892 scope.go:122] "RemoveContainer" containerID="4e7988d1b2868d1314e3ff433a0991b33726e9bd5a37b988bcb217de1a3d3c5d"
Oct 30 19:42:53.134790 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:42:53.134767    4892 scope.go:122] "RemoveContainer" containerID="fbb40fdacbc99b04faeb2b4df8b21e3a8e820389a34e98a87d07f6bc13f75398"
Oct 30 19:42:53.168175 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:42:53.168148    4892 scope.go:122] "RemoveContainer" containerID="3dc82df4c54ac64bceccf03c8b67e3113c865dcfdd564eaf679f747e6d89fde7"
Oct 30 19:42:53.178331 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:42:53.178302    4892 scope.go:122] "RemoveContainer" containerID="4e0bf3926c794ba22485d31a5ecfbda3b9e8741ea12e1d4fb4a821873eb480e3"
Oct 30 19:42:53.378243 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:42:53.378208    4892 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-686689-openshift-default-5fbf5787bf-nsql4" event={"ID":"dffc80d1-bd47-4493-8cd9-87530034d485","Type":"ContainerStarted","Data":"997549f2768f85294570e473388022acab09b95ff50b2d4b15b7a7f3bd5503aa"}
Oct 30 19:42:53.378428 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:42:53.378270    4892 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-686689-openshift-default-5fbf5787bf-nsql4" event={"ID":"dffc80d1-bd47-4493-8cd9-87530034d485","Type":"ContainerStarted","Data":"cdfbbc85084b57edab9bf71c947c433a2b6793b930e52c0c4a0a0310f5235c69"}
Oct 30 19:42:53.454325 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:42:53.454251    4892 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w6-5f55cd-openshift-default-9dc497cb5-q2fbg"]
Oct 30 19:42:53.457399 ip-10-0-132-82 kubenswrapper[4892]: W1030 19:42:53.457366    4892 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfac4fe86_a38e_434e_bfc1_4c3dbceffd27.slice/crio-7e1583030fce73aed574c4372208fad061d5c3d3bdc799ff9927177febbf4382 WatchSource:0}: Error finding container 7e1583030fce73aed574c4372208fad061d5c3d3bdc799ff9927177febbf4382: Status 404 returned error can't find the container with id 7e1583030fce73aed574c4372208fad061d5c3d3bdc799ff9927177febbf4382
Oct 30 19:42:53.460301 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:42:53.460266    4892 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892000Ki","pods":"250"}
Oct 30 19:42:53.460393 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:42:53.460334    4892 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892000Ki","pods":"250"}
Oct 30 19:42:53.460393 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:42:53.460385    4892 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892000Ki","pods":"250"}
Oct 30 19:42:53.501223 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:42:53.499141    4892 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-981da6fa-ff42h"]
Oct 30 19:42:53.502254 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:42:53.502215    4892 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w5-686689-openshift-default-5fbf5787bf-nsql4" podStartSLOduration=2.502199429 podStartE2EDuration="2.502199429s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:42:51 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:42:53.498255894 +0000 UTC m=+1621.068721504" watchObservedRunningTime="2026-10-30 19:42:53.502199429 +0000 UTC m=+1621.072665028"
Oct 30 19:42:53.506562 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:42:53.506484    4892 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-981da6fa-ff42h"
Oct 30 19:42:53.559409 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:42:53.559158    4892 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 30 19:42:53.559409 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:42:53.559228    4892 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 30 19:42:53.559409 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:42:53.559279    4892 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 19:42:53.559409 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:42:53.559297    4892 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-nlxp4\""
Oct 30 19:42:53.564102 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:42:53.564076    4892 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/dfe23e43-6b96-4557-90f0-10f4bf7866e7-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-981da6fa-ff42h\" (UID: \"dfe23e43-6b96-4557-90f0-10f4bf7866e7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-981da6fa-ff42h"
Oct 30 19:42:53.564514 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:42:53.564165    4892 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-981da6fa-ff42h"]
Oct 30 19:42:53.564514 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:42:53.564193    4892 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-58t8r\" (UniqueName: \"kubernetes.io/projected/dfe23e43-6b96-4557-90f0-10f4bf7866e7-kube-api-access-58t8r\") pod \"maas-api-revoke-sub-981da6fa-ff42h\" (UID: \"dfe23e43-6b96-4557-90f0-10f4bf7866e7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-981da6fa-ff42h"
Oct 30 19:42:53.564514 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:42:53.564356    4892 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/dfe23e43-6b96-4557-90f0-10f4bf7866e7-maas-api-xks-ca\") pod \"maas-api-revoke-sub-981da6fa-ff42h\" (UID: \"dfe23e43-6b96-4557-90f0-10f4bf7866e7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-981da6fa-ff42h"
Oct 30 19:42:53.666345 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:42:53.666306    4892 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/dfe23e43-6b96-4557-90f0-10f4bf7866e7-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-981da6fa-ff42h\" (UID: \"dfe23e43-6b96-4557-90f0-10f4bf7866e7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-981da6fa-ff42h"
Oct 30 19:42:53.667017 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:42:53.666989    4892 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-58t8r\" (UniqueName: \"kubernetes.io/projected/dfe23e43-6b96-4557-90f0-10f4bf7866e7-kube-api-access-58t8r\") pod \"maas-api-revoke-sub-981da6fa-ff42h\" (UID: \"dfe23e43-6b96-4557-90f0-10f4bf7866e7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-981da6fa-ff42h"
Oct 30 19:42:53.667105 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:42:53.667008    4892 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/dfe23e43-6b96-4557-90f0-10f4bf7866e7-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-981da6fa-ff42h\" (UID: \"dfe23e43-6b96-4557-90f0-10f4bf7866e7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-981da6fa-ff42h"
Oct 30 19:42:53.667453 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:42:53.667280    4892 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/dfe23e43-6b96-4557-90f0-10f4bf7866e7-maas-api-xks-ca\") pod \"maas-api-revoke-sub-981da6fa-ff42h\" (UID: \"dfe23e43-6b96-4557-90f0-10f4bf7866e7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-981da6fa-ff42h"
Oct 30 19:42:53.670029 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:42:53.670004    4892 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/dfe23e43-6b96-4557-90f0-10f4bf7866e7-maas-api-xks-ca\") pod \"maas-api-revoke-sub-981da6fa-ff42h\" (UID: \"dfe23e43-6b96-4557-90f0-10f4bf7866e7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-981da6fa-ff42h"
Oct 30 19:42:53.726101 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:42:53.726074    4892 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-58t8r\" (UniqueName: \"kubernetes.io/projected/dfe23e43-6b96-4557-90f0-10f4bf7866e7-kube-api-access-58t8r\") pod \"maas-api-revoke-sub-981da6fa-ff42h\" (UID: \"dfe23e43-6b96-4557-90f0-10f4bf7866e7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-981da6fa-ff42h"
Oct 30 19:42:53.839259 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:42:53.839167    4892 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-981da6fa-ff42h"
Oct 30 19:42:54.079649 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:42:54.079621    4892 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w5-686689-openshift-default-5fbf5787bf-nsql4"
Oct 30 19:42:54.080522 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:42:54.080497    4892 patch_prober.go:29] interesting pod/e2e-worker-w5-686689-openshift-default-5fbf5787bf-nsql4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.120:15021/healthz/ready\": dial tcp 10.132.0.120:15021: connect: connection refused" start-of-body=
Oct 30 19:42:54.080577 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:42:54.080551    4892 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-686689-openshift-default-5fbf5787bf-nsql4" podUID="dffc80d1-bd47-4493-8cd9-87530034d485" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.120:15021/healthz/ready\": dial tcp 10.132.0.120:15021: connect: connection refused"
Oct 30 19:42:54.288204 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:42:54.287186    4892 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-981da6fa-ff42h"]
Oct 30 19:42:54.288204 ip-10-0-132-82 kubenswrapper[4892]: W1030 19:42:54.287512    4892 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddfe23e43_6b96_4557_90f0_10f4bf7866e7.slice/crio-b026ed0ea5ecd44161823ecf16421ee4fb50f9c9ddc9550c946e86726cfe386a WatchSource:0}: Error finding container b026ed0ea5ecd44161823ecf16421ee4fb50f9c9ddc9550c946e86726cfe386a: Status 404 returned error can't find the container with id b026ed0ea5ecd44161823ecf16421ee4fb50f9c9ddc9550c946e86726cfe386a
Oct 30 19:42:54.395410 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:42:54.395366    4892 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-981da6fa-ff42h" event={"ID":"dfe23e43-6b96-4557-90f0-10f4bf7866e7","Type":"ContainerStarted","Data":"b026ed0ea5ecd44161823ecf16421ee4fb50f9c9ddc9550c946e86726cfe386a"}
Oct 30 19:42:54.398621 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:42:54.398593    4892 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-5f55cd-openshift-default-9dc497cb5-q2fbg" event={"ID":"fac4fe86-a38e-434e-bfc1-4c3dbceffd27","Type":"ContainerStarted","Data":"e3df77051ab71dd38f8567e41d2b95dcd600ea2f8b9dbe7bc16540bfdc010985"}
Oct 30 19:42:54.398837 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:42:54.398819    4892 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-5f55cd-openshift-default-9dc497cb5-q2fbg" event={"ID":"fac4fe86-a38e-434e-bfc1-4c3dbceffd27","Type":"ContainerStarted","Data":"7e1583030fce73aed574c4372208fad061d5c3d3bdc799ff9927177febbf4382"}
Oct 30 19:42:54.517419 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:42:54.517367    4892 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w6-5f55cd-openshift-default-9dc497cb5-q2fbg" podStartSLOduration=2.517352698 podStartE2EDuration="2.517352698s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:42:52 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:42:54.515082879 +0000 UTC m=+1622.085548477" watchObservedRunningTime="2026-10-30 19:42:54.517352698 +0000 UTC m=+1622.087818297"
Oct 30 19:42:54.900410 ip-10-0-132-82 kubenswrapper[4892]: E1030 19:42:54.900362    4892 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-jrm87" containerName="extractor"
Oct 30 19:42:54.989704 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:42:54.989664    4892 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w6-5f55cd-openshift-default-9dc497cb5-q2fbg"
Oct 30 19:42:54.991627 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:42:54.991594    4892 patch_prober.go:29] interesting pod/e2e-worker-w6-5f55cd-openshift-default-9dc497cb5-q2fbg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.121:15021/healthz/ready\": dial tcp 10.132.0.121:15021: connect: connection refused" start-of-body=
Oct 30 19:42:54.991758 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:42:54.991658    4892 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-5f55cd-openshift-default-9dc497cb5-q2fbg" podUID="fac4fe86-a38e-434e-bfc1-4c3dbceffd27" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.121:15021/healthz/ready\": dial tcp 10.132.0.121:15021: connect: connection refused"
Oct 30 19:42:55.078407 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:42:55.078366    4892 patch_prober.go:29] interesting pod/e2e-worker-w5-686689-openshift-default-5fbf5787bf-nsql4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.120:15021/healthz/ready\": dial tcp 10.132.0.120:15021: connect: connection refused" start-of-body=
Oct 30 19:42:55.078695 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:42:55.078664    4892 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-686689-openshift-default-5fbf5787bf-nsql4" podUID="dffc80d1-bd47-4493-8cd9-87530034d485" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.120:15021/healthz/ready\": dial tcp 10.132.0.120:15021: connect: connection refused"
Oct 30 19:42:55.406923 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:42:55.406890    4892 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-981da6fa-ff42h" event={"ID":"dfe23e43-6b96-4557-90f0-10f4bf7866e7","Type":"ContainerStarted","Data":"5abf0c86a1d954512e2f7f1807817cd053d2062b9c7efc8ed2964e380e8d8c11"}
Oct 30 19:42:55.551990 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:42:55.551501    4892 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-981da6fa-ff42h" podStartSLOduration=2.551481137 podStartE2EDuration="2.551481137s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:42:53 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:42:55.545166159 +0000 UTC m=+1623.115631759" watchObservedRunningTime="2026-10-30 19:42:55.551481137 +0000 UTC m=+1623.121946738"
Oct 30 19:42:55.902912 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:42:55.902879    4892 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-90599b65-8lzkg"]
Oct 30 19:42:55.907257 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:42:55.907235    4892 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-90599b65-8lzkg"
Oct 30 19:42:55.985992 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:42:55.985550    4892 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-90599b65-8lzkg"]
Oct 30 19:42:55.994373 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:42:55.994062    4892 patch_prober.go:29] interesting pod/e2e-worker-w6-5f55cd-openshift-default-9dc497cb5-q2fbg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.121:15021/healthz/ready\": dial tcp 10.132.0.121:15021: connect: connection refused" start-of-body=
Oct 30 19:42:55.994373 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:42:55.994115    4892 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-5f55cd-openshift-default-9dc497cb5-q2fbg" podUID="fac4fe86-a38e-434e-bfc1-4c3dbceffd27" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.121:15021/healthz/ready\": dial tcp 10.132.0.121:15021: connect: connection refused"
Oct 30 19:42:56.012658 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:42:56.012626    4892 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2409c047-ff6e-47a7-ac84-b05f825dac37-maas-api-xks-ca\") pod \"maas-api-revoke-sub-90599b65-8lzkg\" (UID: \"2409c047-ff6e-47a7-ac84-b05f825dac37\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-90599b65-8lzkg"
Oct 30 19:42:56.012807 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:42:56.012672    4892 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8kvjj\" (UniqueName: \"kubernetes.io/projected/2409c047-ff6e-47a7-ac84-b05f825dac37-kube-api-access-8kvjj\") pod \"maas-api-revoke-sub-90599b65-8lzkg\" (UID: \"2409c047-ff6e-47a7-ac84-b05f825dac37\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-90599b65-8lzkg"
Oct 30 19:42:56.012807 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:42:56.012748    4892 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2409c047-ff6e-47a7-ac84-b05f825dac37-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-90599b65-8lzkg\" (UID: \"2409c047-ff6e-47a7-ac84-b05f825dac37\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-90599b65-8lzkg"
Oct 30 19:42:56.078992 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:42:56.078936    4892 patch_prober.go:29] interesting pod/e2e-worker-w5-686689-openshift-default-5fbf5787bf-nsql4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.120:15021/healthz/ready\": dial tcp 10.132.0.120:15021: connect: connection refused" start-of-body=
Oct 30 19:42:56.079174 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:42:56.079016    4892 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-686689-openshift-default-5fbf5787bf-nsql4" podUID="dffc80d1-bd47-4493-8cd9-87530034d485" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.120:15021/healthz/ready\": dial tcp 10.132.0.120:15021: connect: connection refused"
Oct 30 19:42:56.114056 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:42:56.114020    4892 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2409c047-ff6e-47a7-ac84-b05f825dac37-maas-api-xks-ca\") pod \"maas-api-revoke-sub-90599b65-8lzkg\" (UID: \"2409c047-ff6e-47a7-ac84-b05f825dac37\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-90599b65-8lzkg"
Oct 30 19:42:56.114223 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:42:56.114072    4892 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8kvjj\" (UniqueName: \"kubernetes.io/projected/2409c047-ff6e-47a7-ac84-b05f825dac37-kube-api-access-8kvjj\") pod \"maas-api-revoke-sub-90599b65-8lzkg\" (UID: \"2409c047-ff6e-47a7-ac84-b05f825dac37\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-90599b65-8lzkg"
Oct 30 19:42:56.114223 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:42:56.114203    4892 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2409c047-ff6e-47a7-ac84-b05f825dac37-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-90599b65-8lzkg\" (UID: \"2409c047-ff6e-47a7-ac84-b05f825dac37\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-90599b65-8lzkg"
Oct 30 19:42:56.115102 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:42:56.115072    4892 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2409c047-ff6e-47a7-ac84-b05f825dac37-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-90599b65-8lzkg\" (UID: \"2409c047-ff6e-47a7-ac84-b05f825dac37\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-90599b65-8lzkg"
Oct 30 19:42:56.117105 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:42:56.117077    4892 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2409c047-ff6e-47a7-ac84-b05f825dac37-maas-api-xks-ca\") pod \"maas-api-revoke-sub-90599b65-8lzkg\" (UID: \"2409c047-ff6e-47a7-ac84-b05f825dac37\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-90599b65-8lzkg"
Oct 30 19:42:56.180157 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:42:56.180082    4892 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8kvjj\" (UniqueName: \"kubernetes.io/projected/2409c047-ff6e-47a7-ac84-b05f825dac37-kube-api-access-8kvjj\") pod \"maas-api-revoke-sub-90599b65-8lzkg\" (UID: \"2409c047-ff6e-47a7-ac84-b05f825dac37\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-90599b65-8lzkg"
Oct 30 19:42:56.221022 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:42:56.220993    4892 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-90599b65-8lzkg"
Oct 30 19:42:56.416335 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:42:56.416304    4892 generic.go:356] "Generic (PLEG): container finished" podID="dfe23e43-6b96-4557-90f0-10f4bf7866e7" containerID="5abf0c86a1d954512e2f7f1807817cd053d2062b9c7efc8ed2964e380e8d8c11" exitCode=0
Oct 30 19:42:56.417518 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:42:56.417489    4892 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-981da6fa-ff42h" event={"ID":"dfe23e43-6b96-4557-90f0-10f4bf7866e7","Type":"ContainerDied","Data":"5abf0c86a1d954512e2f7f1807817cd053d2062b9c7efc8ed2964e380e8d8c11"}
Oct 30 19:42:56.683598 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:42:56.683566    4892 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-90599b65-8lzkg"]
Oct 30 19:42:56.687058 ip-10-0-132-82 kubenswrapper[4892]: W1030 19:42:56.687022    4892 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2409c047_ff6e_47a7_ac84_b05f825dac37.slice/crio-c6366368cd7eb64f4468a7daa7080b994287f2eb0ab7972d3b0d643848caacad WatchSource:0}: Error finding container c6366368cd7eb64f4468a7daa7080b994287f2eb0ab7972d3b0d643848caacad: Status 404 returned error can't find the container with id c6366368cd7eb64f4468a7daa7080b994287f2eb0ab7972d3b0d643848caacad
Oct 30 19:42:57.026047 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:42:57.022039    4892 patch_prober.go:29] interesting pod/e2e-worker-w6-5f55cd-openshift-default-9dc497cb5-q2fbg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.121:15021/healthz/ready\": dial tcp 10.132.0.121:15021: connect: connection refused" start-of-body=
Oct 30 19:42:57.026047 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:42:57.022100    4892 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-5f55cd-openshift-default-9dc497cb5-q2fbg" podUID="fac4fe86-a38e-434e-bfc1-4c3dbceffd27" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.121:15021/healthz/ready\": dial tcp 10.132.0.121:15021: connect: connection refused"
Oct 30 19:42:57.085269 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:42:57.084101    4892 patch_prober.go:29] interesting pod/e2e-worker-w5-686689-openshift-default-5fbf5787bf-nsql4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.120:15021/healthz/ready\": dial tcp 10.132.0.120:15021: connect: connection refused" start-of-body=
Oct 30 19:42:57.085269 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:42:57.084148    4892 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-686689-openshift-default-5fbf5787bf-nsql4" podUID="dffc80d1-bd47-4493-8cd9-87530034d485" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.120:15021/healthz/ready\": dial tcp 10.132.0.120:15021: connect: connection refused"
Oct 30 19:42:57.448000 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:42:57.447403    4892 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-90599b65-8lzkg" event={"ID":"2409c047-ff6e-47a7-ac84-b05f825dac37","Type":"ContainerStarted","Data":"a9cf9716c9ffd9533e0d5be12ef76b0ae3a9f1d8cfcfadbf908c24401af1867c"}
Oct 30 19:42:57.448000 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:42:57.447510    4892 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-90599b65-8lzkg" event={"ID":"2409c047-ff6e-47a7-ac84-b05f825dac37","Type":"ContainerStarted","Data":"c6366368cd7eb64f4468a7daa7080b994287f2eb0ab7972d3b0d643848caacad"}
Oct 30 19:42:57.601643 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:42:57.600383    4892 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-90599b65-8lzkg" podStartSLOduration=2.600367548 podStartE2EDuration="2.600367548s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:42:55 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:42:57.594605485 +0000 UTC m=+1625.165071085" watchObservedRunningTime="2026-10-30 19:42:57.600367548 +0000 UTC m=+1625.170833154"
Oct 30 19:42:57.811633 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:42:57.811601    4892 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-981da6fa-ff42h"
Oct 30 19:42:57.895256 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:42:57.895223    4892 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/dfe23e43-6b96-4557-90f0-10f4bf7866e7-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/dfe23e43-6b96-4557-90f0-10f4bf7866e7-maas-api-openshift-ca\") pod \"dfe23e43-6b96-4557-90f0-10f4bf7866e7\" (UID: \"dfe23e43-6b96-4557-90f0-10f4bf7866e7\") "
Oct 30 19:42:57.895428 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:42:57.895308    4892 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/dfe23e43-6b96-4557-90f0-10f4bf7866e7-kube-api-access-58t8r\" (UniqueName: \"kubernetes.io/projected/dfe23e43-6b96-4557-90f0-10f4bf7866e7-kube-api-access-58t8r\") pod \"dfe23e43-6b96-4557-90f0-10f4bf7866e7\" (UID: \"dfe23e43-6b96-4557-90f0-10f4bf7866e7\") "
Oct 30 19:42:57.895428 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:42:57.895371    4892 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/dfe23e43-6b96-4557-90f0-10f4bf7866e7-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/dfe23e43-6b96-4557-90f0-10f4bf7866e7-maas-api-xks-ca\") pod \"dfe23e43-6b96-4557-90f0-10f4bf7866e7\" (UID: \"dfe23e43-6b96-4557-90f0-10f4bf7866e7\") "
Oct 30 19:42:57.896656 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:42:57.896624    4892 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/dfe23e43-6b96-4557-90f0-10f4bf7866e7-maas-api-openshift-ca" pod "dfe23e43-6b96-4557-90f0-10f4bf7866e7" (UID: "dfe23e43-6b96-4557-90f0-10f4bf7866e7"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 19:42:57.898912 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:42:57.898882    4892 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/dfe23e43-6b96-4557-90f0-10f4bf7866e7-kube-api-access-58t8r" pod "dfe23e43-6b96-4557-90f0-10f4bf7866e7" (UID: "dfe23e43-6b96-4557-90f0-10f4bf7866e7"). InnerVolumeSpecName "kube-api-access-58t8r". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:42:57.899064 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:42:57.898892    4892 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/dfe23e43-6b96-4557-90f0-10f4bf7866e7-maas-api-xks-ca" pod "dfe23e43-6b96-4557-90f0-10f4bf7866e7" (UID: "dfe23e43-6b96-4557-90f0-10f4bf7866e7"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 19:42:57.992709 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:42:57.992675    4892 patch_prober.go:29] interesting pod/e2e-worker-w6-5f55cd-openshift-default-9dc497cb5-q2fbg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.121:15021/healthz/ready\": dial tcp 10.132.0.121:15021: connect: connection refused" start-of-body=
Oct 30 19:42:57.993021 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:42:57.992942    4892 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-5f55cd-openshift-default-9dc497cb5-q2fbg" podUID="fac4fe86-a38e-434e-bfc1-4c3dbceffd27" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.121:15021/healthz/ready\": dial tcp 10.132.0.121:15021: connect: connection refused"
Oct 30 19:42:57.997196 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:42:57.997058    4892 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-58t8r\" (UniqueName: \"kubernetes.io/projected/dfe23e43-6b96-4557-90f0-10f4bf7866e7-kube-api-access-58t8r\") on node \"ip-10-0-132-82.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 19:42:57.997196 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:42:57.997090    4892 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/dfe23e43-6b96-4557-90f0-10f4bf7866e7-maas-api-xks-ca\") on node \"ip-10-0-132-82.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 19:42:57.997196 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:42:57.997105    4892 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/dfe23e43-6b96-4557-90f0-10f4bf7866e7-maas-api-openshift-ca\") on node \"ip-10-0-132-82.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 19:42:58.073556 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:42:58.073360    4892 patch_prober.go:29] interesting pod/e2e-worker-w5-686689-openshift-default-5fbf5787bf-nsql4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.120:15021/healthz/ready\": dial tcp 10.132.0.120:15021: connect: connection refused" start-of-body=
Oct 30 19:42:58.073556 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:42:58.073423    4892 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-686689-openshift-default-5fbf5787bf-nsql4" podUID="dffc80d1-bd47-4493-8cd9-87530034d485" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.120:15021/healthz/ready\": dial tcp 10.132.0.120:15021: connect: connection refused"
Oct 30 19:42:58.489267 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:42:58.488605    4892 generic.go:356] "Generic (PLEG): container finished" podID="2409c047-ff6e-47a7-ac84-b05f825dac37" containerID="a9cf9716c9ffd9533e0d5be12ef76b0ae3a9f1d8cfcfadbf908c24401af1867c" exitCode=0
Oct 30 19:42:58.489267 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:42:58.488732    4892 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-90599b65-8lzkg" event={"ID":"2409c047-ff6e-47a7-ac84-b05f825dac37","Type":"ContainerDied","Data":"a9cf9716c9ffd9533e0d5be12ef76b0ae3a9f1d8cfcfadbf908c24401af1867c"}
Oct 30 19:42:58.491153 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:42:58.491050    4892 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-981da6fa-ff42h"
Oct 30 19:42:58.491153 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:42:58.491079    4892 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-981da6fa-ff42h" event={"ID":"dfe23e43-6b96-4557-90f0-10f4bf7866e7","Type":"ContainerDied","Data":"b026ed0ea5ecd44161823ecf16421ee4fb50f9c9ddc9550c946e86726cfe386a"}
Oct 30 19:42:58.491153 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:42:58.491124    4892 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="b026ed0ea5ecd44161823ecf16421ee4fb50f9c9ddc9550c946e86726cfe386a"
Oct 30 19:42:58.990153 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:42:58.990115    4892 patch_prober.go:29] interesting pod/e2e-worker-w6-5f55cd-openshift-default-9dc497cb5-q2fbg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.121:15021/healthz/ready\": dial tcp 10.132.0.121:15021: connect: connection refused" start-of-body=
Oct 30 19:42:58.990309 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:42:58.990176    4892 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-5f55cd-openshift-default-9dc497cb5-q2fbg" podUID="fac4fe86-a38e-434e-bfc1-4c3dbceffd27" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.121:15021/healthz/ready\": dial tcp 10.132.0.121:15021: connect: connection refused"
Oct 30 19:42:59.072812 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:42:59.072777    4892 patch_prober.go:29] interesting pod/e2e-worker-w5-686689-openshift-default-5fbf5787bf-nsql4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.120:15021/healthz/ready\": dial tcp 10.132.0.120:15021: connect: connection refused" start-of-body=
Oct 30 19:42:59.073006 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:42:59.072842    4892 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-686689-openshift-default-5fbf5787bf-nsql4" podUID="dffc80d1-bd47-4493-8cd9-87530034d485" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.120:15021/healthz/ready\": dial tcp 10.132.0.120:15021: connect: connection refused"
Oct 30 19:42:59.692274 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:42:59.692242    4892 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-90599b65-8lzkg"
Oct 30 19:42:59.820258 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:42:59.820172    4892 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/2409c047-ff6e-47a7-ac84-b05f825dac37-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2409c047-ff6e-47a7-ac84-b05f825dac37-maas-api-xks-ca\") pod \"2409c047-ff6e-47a7-ac84-b05f825dac37\" (UID: \"2409c047-ff6e-47a7-ac84-b05f825dac37\") "
Oct 30 19:42:59.820752 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:42:59.820726    4892 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/2409c047-ff6e-47a7-ac84-b05f825dac37-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2409c047-ff6e-47a7-ac84-b05f825dac37-maas-api-openshift-ca\") pod \"2409c047-ff6e-47a7-ac84-b05f825dac37\" (UID: \"2409c047-ff6e-47a7-ac84-b05f825dac37\") "
Oct 30 19:42:59.821014 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:42:59.820995    4892 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2409c047-ff6e-47a7-ac84-b05f825dac37-kube-api-access-8kvjj\" (UniqueName: \"kubernetes.io/projected/2409c047-ff6e-47a7-ac84-b05f825dac37-kube-api-access-8kvjj\") pod \"2409c047-ff6e-47a7-ac84-b05f825dac37\" (UID: \"2409c047-ff6e-47a7-ac84-b05f825dac37\") "
Oct 30 19:42:59.822084 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:42:59.822060    4892 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2409c047-ff6e-47a7-ac84-b05f825dac37-maas-api-openshift-ca" pod "2409c047-ff6e-47a7-ac84-b05f825dac37" (UID: "2409c047-ff6e-47a7-ac84-b05f825dac37"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 19:42:59.822565 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:42:59.822546    4892 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2409c047-ff6e-47a7-ac84-b05f825dac37-maas-api-openshift-ca\") on node \"ip-10-0-132-82.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 19:42:59.848992 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:42:59.847093    4892 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2409c047-ff6e-47a7-ac84-b05f825dac37-maas-api-xks-ca" pod "2409c047-ff6e-47a7-ac84-b05f825dac37" (UID: "2409c047-ff6e-47a7-ac84-b05f825dac37"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 19:42:59.849282 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:42:59.849243    4892 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2409c047-ff6e-47a7-ac84-b05f825dac37-kube-api-access-8kvjj" pod "2409c047-ff6e-47a7-ac84-b05f825dac37" (UID: "2409c047-ff6e-47a7-ac84-b05f825dac37"). InnerVolumeSpecName "kube-api-access-8kvjj". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:42:59.924669 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:42:59.924633    4892 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2409c047-ff6e-47a7-ac84-b05f825dac37-maas-api-xks-ca\") on node \"ip-10-0-132-82.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 19:42:59.924669 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:42:59.924673    4892 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8kvjj\" (UniqueName: \"kubernetes.io/projected/2409c047-ff6e-47a7-ac84-b05f825dac37-kube-api-access-8kvjj\") on node \"ip-10-0-132-82.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 19:42:59.990623 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:42:59.990579    4892 patch_prober.go:29] interesting pod/e2e-worker-w6-5f55cd-openshift-default-9dc497cb5-q2fbg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.121:15021/healthz/ready\": dial tcp 10.132.0.121:15021: connect: connection refused" start-of-body=
Oct 30 19:42:59.990818 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:42:59.990651    4892 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-5f55cd-openshift-default-9dc497cb5-q2fbg" podUID="fac4fe86-a38e-434e-bfc1-4c3dbceffd27" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.121:15021/healthz/ready\": dial tcp 10.132.0.121:15021: connect: connection refused"
Oct 30 19:43:00.192726 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:43:00.192692    4892 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w5-686689-openshift-default-5fbf5787bf-nsql4"
Oct 30 19:43:00.193427 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:43:00.193397    4892 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w5-686689-openshift-default-5fbf5787bf-nsql4"
Oct 30 19:43:00.194242 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:43:00.194215    4892 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w5-686689-openshift-default-5fbf5787bf-nsql4"
Oct 30 19:43:00.511804 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:43:00.511722    4892 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-90599b65-8lzkg" event={"ID":"2409c047-ff6e-47a7-ac84-b05f825dac37","Type":"ContainerDied","Data":"c6366368cd7eb64f4468a7daa7080b994287f2eb0ab7972d3b0d643848caacad"}
Oct 30 19:43:00.511804 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:43:00.511755    4892 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-90599b65-8lzkg"
Oct 30 19:43:00.511804 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:43:00.511785    4892 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="c6366368cd7eb64f4468a7daa7080b994287f2eb0ab7972d3b0d643848caacad"
Oct 30 19:43:01.174054 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:43:01.174022    4892 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w6-5f55cd-openshift-default-9dc497cb5-q2fbg"
Oct 30 19:43:01.174858 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:43:01.174836    4892 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w6-5f55cd-openshift-default-9dc497cb5-q2fbg"
Oct 30 19:43:01.175719 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:43:01.175695    4892 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w6-5f55cd-openshift-default-9dc497cb5-q2fbg"
Oct 30 19:43:01.822627 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:43:01.822596    4892 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w2-a3f9b7-openshift-default-7d6d8b8fb5-5vgch"]
Oct 30 19:43:01.823385 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:43:01.823369    4892 cpu_manager.go:414] "containerMap: removing container" podUID="2409c047-ff6e-47a7-ac84-b05f825dac37" containerName="revoke-keys"
Oct 30 19:43:01.823433 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:43:01.823388    4892 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2409c047-ff6e-47a7-ac84-b05f825dac37" containerName="revoke-keys"
Oct 30 19:43:01.823464 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:43:01.823437    4892 cpu_manager.go:414] "containerMap: removing container" podUID="dfe23e43-6b96-4557-90f0-10f4bf7866e7" containerName="revoke-keys"
Oct 30 19:43:01.823464 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:43:01.823446    4892 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="dfe23e43-6b96-4557-90f0-10f4bf7866e7" containerName="revoke-keys"
Oct 30 19:43:01.823581 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:43:01.823570    4892 memory_manager.go:365] "RemoveStaleState removing state" podUID="2409c047-ff6e-47a7-ac84-b05f825dac37" containerName="revoke-keys"
Oct 30 19:43:01.823635 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:43:01.823591    4892 memory_manager.go:365] "RemoveStaleState removing state" podUID="dfe23e43-6b96-4557-90f0-10f4bf7866e7" containerName="revoke-keys"
Oct 30 19:43:01.828084 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:43:01.828060    4892 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-a3f9b7-openshift-default-7d6d8b8fb5-5vgch"
Oct 30 19:43:01.924719 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:43:01.924684    4892 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-a3f9b7-openshift-default-7d6d8b8fb5-5vgch"]
Oct 30 19:43:01.950322 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:43:01.950287    4892 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/78ea3297-af25-446c-b7d0-3da5ae88dd15-istio-token\") pod \"e2e-worker-w2-a3f9b7-openshift-default-7d6d8b8fb5-5vgch\" (UID: \"78ea3297-af25-446c-b7d0-3da5ae88dd15\") " pod="openshift-ingress/e2e-worker-w2-a3f9b7-openshift-default-7d6d8b8fb5-5vgch"
Oct 30 19:43:01.950558 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:43:01.950453    4892 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-r7qbx\" (UniqueName: \"kubernetes.io/projected/78ea3297-af25-446c-b7d0-3da5ae88dd15-kube-api-access-r7qbx\") pod \"e2e-worker-w2-a3f9b7-openshift-default-7d6d8b8fb5-5vgch\" (UID: \"78ea3297-af25-446c-b7d0-3da5ae88dd15\") " pod="openshift-ingress/e2e-worker-w2-a3f9b7-openshift-default-7d6d8b8fb5-5vgch"
Oct 30 19:43:01.950558 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:43:01.950481    4892 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/78ea3297-af25-446c-b7d0-3da5ae88dd15-istio-envoy\") pod \"e2e-worker-w2-a3f9b7-openshift-default-7d6d8b8fb5-5vgch\" (UID: \"78ea3297-af25-446c-b7d0-3da5ae88dd15\") " pod="openshift-ingress/e2e-worker-w2-a3f9b7-openshift-default-7d6d8b8fb5-5vgch"
Oct 30 19:43:01.950558 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:43:01.950524    4892 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/78ea3297-af25-446c-b7d0-3da5ae88dd15-credential-socket\") pod \"e2e-worker-w2-a3f9b7-openshift-default-7d6d8b8fb5-5vgch\" (UID: \"78ea3297-af25-446c-b7d0-3da5ae88dd15\") " pod="openshift-ingress/e2e-worker-w2-a3f9b7-openshift-default-7d6d8b8fb5-5vgch"
Oct 30 19:43:01.950558 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:43:01.950539    4892 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/78ea3297-af25-446c-b7d0-3da5ae88dd15-istio-data\") pod \"e2e-worker-w2-a3f9b7-openshift-default-7d6d8b8fb5-5vgch\" (UID: \"78ea3297-af25-446c-b7d0-3da5ae88dd15\") " pod="openshift-ingress/e2e-worker-w2-a3f9b7-openshift-default-7d6d8b8fb5-5vgch"
Oct 30 19:43:01.950771 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:43:01.950576    4892 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/78ea3297-af25-446c-b7d0-3da5ae88dd15-istio-podinfo\") pod \"e2e-worker-w2-a3f9b7-openshift-default-7d6d8b8fb5-5vgch\" (UID: \"78ea3297-af25-446c-b7d0-3da5ae88dd15\") " pod="openshift-ingress/e2e-worker-w2-a3f9b7-openshift-default-7d6d8b8fb5-5vgch"
Oct 30 19:43:01.950771 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:43:01.950636    4892 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/78ea3297-af25-446c-b7d0-3da5ae88dd15-istiod-ca-cert\") pod \"e2e-worker-w2-a3f9b7-openshift-default-7d6d8b8fb5-5vgch\" (UID: \"78ea3297-af25-446c-b7d0-3da5ae88dd15\") " pod="openshift-ingress/e2e-worker-w2-a3f9b7-openshift-default-7d6d8b8fb5-5vgch"
Oct 30 19:43:01.950771 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:43:01.950669    4892 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/78ea3297-af25-446c-b7d0-3da5ae88dd15-workload-socket\") pod \"e2e-worker-w2-a3f9b7-openshift-default-7d6d8b8fb5-5vgch\" (UID: \"78ea3297-af25-446c-b7d0-3da5ae88dd15\") " pod="openshift-ingress/e2e-worker-w2-a3f9b7-openshift-default-7d6d8b8fb5-5vgch"
Oct 30 19:43:01.950771 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:43:01.950706    4892 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/78ea3297-af25-446c-b7d0-3da5ae88dd15-workload-certs\") pod \"e2e-worker-w2-a3f9b7-openshift-default-7d6d8b8fb5-5vgch\" (UID: \"78ea3297-af25-446c-b7d0-3da5ae88dd15\") " pod="openshift-ingress/e2e-worker-w2-a3f9b7-openshift-default-7d6d8b8fb5-5vgch"
Oct 30 19:43:02.052312 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:43:02.052275    4892 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-r7qbx\" (UniqueName: \"kubernetes.io/projected/78ea3297-af25-446c-b7d0-3da5ae88dd15-kube-api-access-r7qbx\") pod \"e2e-worker-w2-a3f9b7-openshift-default-7d6d8b8fb5-5vgch\" (UID: \"78ea3297-af25-446c-b7d0-3da5ae88dd15\") " pod="openshift-ingress/e2e-worker-w2-a3f9b7-openshift-default-7d6d8b8fb5-5vgch"
Oct 30 19:43:02.052312 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:43:02.052319    4892 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/78ea3297-af25-446c-b7d0-3da5ae88dd15-istio-envoy\") pod \"e2e-worker-w2-a3f9b7-openshift-default-7d6d8b8fb5-5vgch\" (UID: \"78ea3297-af25-446c-b7d0-3da5ae88dd15\") " pod="openshift-ingress/e2e-worker-w2-a3f9b7-openshift-default-7d6d8b8fb5-5vgch"
Oct 30 19:43:02.052694 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:43:02.052367    4892 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/78ea3297-af25-446c-b7d0-3da5ae88dd15-credential-socket\") pod \"e2e-worker-w2-a3f9b7-openshift-default-7d6d8b8fb5-5vgch\" (UID: \"78ea3297-af25-446c-b7d0-3da5ae88dd15\") " pod="openshift-ingress/e2e-worker-w2-a3f9b7-openshift-default-7d6d8b8fb5-5vgch"
Oct 30 19:43:02.052694 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:43:02.052389    4892 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/78ea3297-af25-446c-b7d0-3da5ae88dd15-istio-data\") pod \"e2e-worker-w2-a3f9b7-openshift-default-7d6d8b8fb5-5vgch\" (UID: \"78ea3297-af25-446c-b7d0-3da5ae88dd15\") " pod="openshift-ingress/e2e-worker-w2-a3f9b7-openshift-default-7d6d8b8fb5-5vgch"
Oct 30 19:43:02.052694 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:43:02.052416    4892 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/78ea3297-af25-446c-b7d0-3da5ae88dd15-istio-podinfo\") pod \"e2e-worker-w2-a3f9b7-openshift-default-7d6d8b8fb5-5vgch\" (UID: \"78ea3297-af25-446c-b7d0-3da5ae88dd15\") " pod="openshift-ingress/e2e-worker-w2-a3f9b7-openshift-default-7d6d8b8fb5-5vgch"
Oct 30 19:43:02.052694 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:43:02.052483    4892 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/78ea3297-af25-446c-b7d0-3da5ae88dd15-istiod-ca-cert\") pod \"e2e-worker-w2-a3f9b7-openshift-default-7d6d8b8fb5-5vgch\" (UID: \"78ea3297-af25-446c-b7d0-3da5ae88dd15\") " pod="openshift-ingress/e2e-worker-w2-a3f9b7-openshift-default-7d6d8b8fb5-5vgch"
Oct 30 19:43:02.052694 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:43:02.052517    4892 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/78ea3297-af25-446c-b7d0-3da5ae88dd15-workload-socket\") pod \"e2e-worker-w2-a3f9b7-openshift-default-7d6d8b8fb5-5vgch\" (UID: \"78ea3297-af25-446c-b7d0-3da5ae88dd15\") " pod="openshift-ingress/e2e-worker-w2-a3f9b7-openshift-default-7d6d8b8fb5-5vgch"
Oct 30 19:43:02.052694 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:43:02.052541    4892 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/78ea3297-af25-446c-b7d0-3da5ae88dd15-workload-certs\") pod \"e2e-worker-w2-a3f9b7-openshift-default-7d6d8b8fb5-5vgch\" (UID: \"78ea3297-af25-446c-b7d0-3da5ae88dd15\") " pod="openshift-ingress/e2e-worker-w2-a3f9b7-openshift-default-7d6d8b8fb5-5vgch"
Oct 30 19:43:02.052694 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:43:02.052659    4892 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/78ea3297-af25-446c-b7d0-3da5ae88dd15-istio-token\") pod \"e2e-worker-w2-a3f9b7-openshift-default-7d6d8b8fb5-5vgch\" (UID: \"78ea3297-af25-446c-b7d0-3da5ae88dd15\") " pod="openshift-ingress/e2e-worker-w2-a3f9b7-openshift-default-7d6d8b8fb5-5vgch"
Oct 30 19:43:02.053546 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:43:02.052891    4892 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/78ea3297-af25-446c-b7d0-3da5ae88dd15-istio-data\") pod \"e2e-worker-w2-a3f9b7-openshift-default-7d6d8b8fb5-5vgch\" (UID: \"78ea3297-af25-446c-b7d0-3da5ae88dd15\") " pod="openshift-ingress/e2e-worker-w2-a3f9b7-openshift-default-7d6d8b8fb5-5vgch"
Oct 30 19:43:02.053546 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:43:02.053176    4892 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/78ea3297-af25-446c-b7d0-3da5ae88dd15-workload-socket\") pod \"e2e-worker-w2-a3f9b7-openshift-default-7d6d8b8fb5-5vgch\" (UID: \"78ea3297-af25-446c-b7d0-3da5ae88dd15\") " pod="openshift-ingress/e2e-worker-w2-a3f9b7-openshift-default-7d6d8b8fb5-5vgch"
Oct 30 19:43:02.053546 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:43:02.053382    4892 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/78ea3297-af25-446c-b7d0-3da5ae88dd15-istiod-ca-cert\") pod \"e2e-worker-w2-a3f9b7-openshift-default-7d6d8b8fb5-5vgch\" (UID: \"78ea3297-af25-446c-b7d0-3da5ae88dd15\") " pod="openshift-ingress/e2e-worker-w2-a3f9b7-openshift-default-7d6d8b8fb5-5vgch"
Oct 30 19:43:02.053712 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:43:02.053616    4892 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/78ea3297-af25-446c-b7d0-3da5ae88dd15-credential-socket\") pod \"e2e-worker-w2-a3f9b7-openshift-default-7d6d8b8fb5-5vgch\" (UID: \"78ea3297-af25-446c-b7d0-3da5ae88dd15\") " pod="openshift-ingress/e2e-worker-w2-a3f9b7-openshift-default-7d6d8b8fb5-5vgch"
Oct 30 19:43:02.053712 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:43:02.053686    4892 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/78ea3297-af25-446c-b7d0-3da5ae88dd15-workload-certs\") pod \"e2e-worker-w2-a3f9b7-openshift-default-7d6d8b8fb5-5vgch\" (UID: \"78ea3297-af25-446c-b7d0-3da5ae88dd15\") " pod="openshift-ingress/e2e-worker-w2-a3f9b7-openshift-default-7d6d8b8fb5-5vgch"
Oct 30 19:43:02.056891 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:43:02.055830    4892 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/78ea3297-af25-446c-b7d0-3da5ae88dd15-istio-envoy\") pod \"e2e-worker-w2-a3f9b7-openshift-default-7d6d8b8fb5-5vgch\" (UID: \"78ea3297-af25-446c-b7d0-3da5ae88dd15\") " pod="openshift-ingress/e2e-worker-w2-a3f9b7-openshift-default-7d6d8b8fb5-5vgch"
Oct 30 19:43:02.056891 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:43:02.056032    4892 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/78ea3297-af25-446c-b7d0-3da5ae88dd15-istio-podinfo\") pod \"e2e-worker-w2-a3f9b7-openshift-default-7d6d8b8fb5-5vgch\" (UID: \"78ea3297-af25-446c-b7d0-3da5ae88dd15\") " pod="openshift-ingress/e2e-worker-w2-a3f9b7-openshift-default-7d6d8b8fb5-5vgch"
Oct 30 19:43:02.114868 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:43:02.114802    4892 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-r7qbx\" (UniqueName: \"kubernetes.io/projected/78ea3297-af25-446c-b7d0-3da5ae88dd15-kube-api-access-r7qbx\") pod \"e2e-worker-w2-a3f9b7-openshift-default-7d6d8b8fb5-5vgch\" (UID: \"78ea3297-af25-446c-b7d0-3da5ae88dd15\") " pod="openshift-ingress/e2e-worker-w2-a3f9b7-openshift-default-7d6d8b8fb5-5vgch"
Oct 30 19:43:02.122913 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:43:02.122888    4892 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/78ea3297-af25-446c-b7d0-3da5ae88dd15-istio-token\") pod \"e2e-worker-w2-a3f9b7-openshift-default-7d6d8b8fb5-5vgch\" (UID: \"78ea3297-af25-446c-b7d0-3da5ae88dd15\") " pod="openshift-ingress/e2e-worker-w2-a3f9b7-openshift-default-7d6d8b8fb5-5vgch"
Oct 30 19:43:02.144769 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:43:02.144745    4892 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-a3f9b7-openshift-default-7d6d8b8fb5-5vgch"
Oct 30 19:43:02.584631 ip-10-0-132-82 kubenswrapper[4892]: W1030 19:43:02.584601    4892 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod78ea3297_af25_446c_b7d0_3da5ae88dd15.slice/crio-e33f3e6d2da0ee55a7d1eef0cff4fb158fdf52c67915d5bbb592670f152f4c5d WatchSource:0}: Error finding container e33f3e6d2da0ee55a7d1eef0cff4fb158fdf52c67915d5bbb592670f152f4c5d: Status 404 returned error can't find the container with id e33f3e6d2da0ee55a7d1eef0cff4fb158fdf52c67915d5bbb592670f152f4c5d
Oct 30 19:43:02.585509 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:43:02.585487    4892 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-a3f9b7-openshift-default-7d6d8b8fb5-5vgch"]
Oct 30 19:43:02.586911 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:43:02.586880    4892 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892000Ki","pods":"250"}
Oct 30 19:43:02.587019 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:43:02.586943    4892 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892000Ki","pods":"250"}
Oct 30 19:43:02.587019 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:43:02.586992    4892 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892000Ki","pods":"250"}
Oct 30 19:43:03.563908 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:43:03.563867    4892 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-a3f9b7-openshift-default-7d6d8b8fb5-5vgch" event={"ID":"78ea3297-af25-446c-b7d0-3da5ae88dd15","Type":"ContainerStarted","Data":"31f71be1a3f974094d49502d6d3ef62771bc9911e1fee1d0ae54798ae5f87cde"}
Oct 30 19:43:03.564146 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:43:03.563939    4892 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-a3f9b7-openshift-default-7d6d8b8fb5-5vgch" event={"ID":"78ea3297-af25-446c-b7d0-3da5ae88dd15","Type":"ContainerStarted","Data":"e33f3e6d2da0ee55a7d1eef0cff4fb158fdf52c67915d5bbb592670f152f4c5d"}
Oct 30 19:43:03.689506 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:43:03.689430    4892 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w2-a3f9b7-openshift-default-7d6d8b8fb5-5vgch" podStartSLOduration=2.689411602 podStartE2EDuration="2.689411602s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:43:01 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:43:03.679900498 +0000 UTC m=+1631.250366096" watchObservedRunningTime="2026-10-30 19:43:03.689411602 +0000 UTC m=+1631.259877205"
Oct 30 19:43:04.147988 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:43:04.145339    4892 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w2-a3f9b7-openshift-default-7d6d8b8fb5-5vgch"
Oct 30 19:43:04.158990 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:43:04.156110    4892 patch_prober.go:29] interesting pod/e2e-worker-w2-a3f9b7-openshift-default-7d6d8b8fb5-5vgch container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.124:15021/healthz/ready\": dial tcp 10.132.0.124:15021: connect: connection refused" start-of-body=
Oct 30 19:43:04.158990 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:43:04.156192    4892 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-a3f9b7-openshift-default-7d6d8b8fb5-5vgch" podUID="78ea3297-af25-446c-b7d0-3da5ae88dd15" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.124:15021/healthz/ready\": dial tcp 10.132.0.124:15021: connect: connection refused"
Oct 30 19:43:04.212841 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:43:04.212810    4892 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-454a9e34-dtlzm"]
Oct 30 19:43:04.217145 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:43:04.217125    4892 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-454a9e34-dtlzm"
Oct 30 19:43:04.285497 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:43:04.285461    4892 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/90541e2c-7c60-4919-830e-73341e6c4cfe-maas-api-xks-ca\") pod \"maas-api-revoke-sub-454a9e34-dtlzm\" (UID: \"90541e2c-7c60-4919-830e-73341e6c4cfe\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-454a9e34-dtlzm"
Oct 30 19:43:04.285670 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:43:04.285596    4892 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nwxq6\" (UniqueName: \"kubernetes.io/projected/90541e2c-7c60-4919-830e-73341e6c4cfe-kube-api-access-nwxq6\") pod \"maas-api-revoke-sub-454a9e34-dtlzm\" (UID: \"90541e2c-7c60-4919-830e-73341e6c4cfe\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-454a9e34-dtlzm"
Oct 30 19:43:04.285670 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:43:04.285657    4892 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/90541e2c-7c60-4919-830e-73341e6c4cfe-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-454a9e34-dtlzm\" (UID: \"90541e2c-7c60-4919-830e-73341e6c4cfe\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-454a9e34-dtlzm"
Oct 30 19:43:04.314000 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:43:04.313939    4892 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 30 19:43:04.319297 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:43:04.319276    4892 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-454a9e34-dtlzm"]
Oct 30 19:43:04.319821 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:43:04.319795    4892 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 19:43:04.319914 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:43:04.319858    4892 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-nlxp4\""
Oct 30 19:43:04.319914 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:43:04.319874    4892 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 30 19:43:04.387269 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:43:04.387237    4892 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nwxq6\" (UniqueName: \"kubernetes.io/projected/90541e2c-7c60-4919-830e-73341e6c4cfe-kube-api-access-nwxq6\") pod \"maas-api-revoke-sub-454a9e34-dtlzm\" (UID: \"90541e2c-7c60-4919-830e-73341e6c4cfe\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-454a9e34-dtlzm"
Oct 30 19:43:04.387437 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:43:04.387303    4892 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/90541e2c-7c60-4919-830e-73341e6c4cfe-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-454a9e34-dtlzm\" (UID: \"90541e2c-7c60-4919-830e-73341e6c4cfe\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-454a9e34-dtlzm"
Oct 30 19:43:04.387437 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:43:04.387361    4892 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/90541e2c-7c60-4919-830e-73341e6c4cfe-maas-api-xks-ca\") pod \"maas-api-revoke-sub-454a9e34-dtlzm\" (UID: \"90541e2c-7c60-4919-830e-73341e6c4cfe\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-454a9e34-dtlzm"
Oct 30 19:43:04.388117 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:43:04.388087    4892 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/90541e2c-7c60-4919-830e-73341e6c4cfe-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-454a9e34-dtlzm\" (UID: \"90541e2c-7c60-4919-830e-73341e6c4cfe\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-454a9e34-dtlzm"
Oct 30 19:43:04.389947 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:43:04.389926    4892 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/90541e2c-7c60-4919-830e-73341e6c4cfe-maas-api-xks-ca\") pod \"maas-api-revoke-sub-454a9e34-dtlzm\" (UID: \"90541e2c-7c60-4919-830e-73341e6c4cfe\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-454a9e34-dtlzm"
Oct 30 19:43:04.455047 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:43:04.454011    4892 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nwxq6\" (UniqueName: \"kubernetes.io/projected/90541e2c-7c60-4919-830e-73341e6c4cfe-kube-api-access-nwxq6\") pod \"maas-api-revoke-sub-454a9e34-dtlzm\" (UID: \"90541e2c-7c60-4919-830e-73341e6c4cfe\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-454a9e34-dtlzm"
Oct 30 19:43:04.547989 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:43:04.546197    4892 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-454a9e34-dtlzm"
Oct 30 19:43:04.967823 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:43:04.967787    4892 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-454a9e34-dtlzm"]
Oct 30 19:43:04.969036 ip-10-0-132-82 kubenswrapper[4892]: W1030 19:43:04.969008    4892 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod90541e2c_7c60_4919_830e_73341e6c4cfe.slice/crio-36082c404785d27ccc35431452cbabb7cfca56fff96df2c85239393304700b17 WatchSource:0}: Error finding container 36082c404785d27ccc35431452cbabb7cfca56fff96df2c85239393304700b17: Status 404 returned error can't find the container with id 36082c404785d27ccc35431452cbabb7cfca56fff96df2c85239393304700b17
Oct 30 19:43:05.158990 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:43:05.148504    4892 patch_prober.go:29] interesting pod/e2e-worker-w2-a3f9b7-openshift-default-7d6d8b8fb5-5vgch container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.124:15021/healthz/ready\": dial tcp 10.132.0.124:15021: connect: connection refused" start-of-body=
Oct 30 19:43:05.158990 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:43:05.148559    4892 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-a3f9b7-openshift-default-7d6d8b8fb5-5vgch" podUID="78ea3297-af25-446c-b7d0-3da5ae88dd15" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.124:15021/healthz/ready\": dial tcp 10.132.0.124:15021: connect: connection refused"
Oct 30 19:43:05.583853 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:43:05.583805    4892 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-454a9e34-dtlzm" event={"ID":"90541e2c-7c60-4919-830e-73341e6c4cfe","Type":"ContainerStarted","Data":"1b500499c8fafcb5e4fcde940952876bf77ddfc4d09e1fb4d2a9009c7f3579c0"}
Oct 30 19:43:05.584081 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:43:05.583874    4892 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-454a9e34-dtlzm" event={"ID":"90541e2c-7c60-4919-830e-73341e6c4cfe","Type":"ContainerStarted","Data":"36082c404785d27ccc35431452cbabb7cfca56fff96df2c85239393304700b17"}
Oct 30 19:43:05.701266 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:43:05.701214    4892 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-454a9e34-dtlzm" podStartSLOduration=1.70119958 podStartE2EDuration="1.70119958s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:43:04 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:43:05.697138939 +0000 UTC m=+1633.267604540" watchObservedRunningTime="2026-10-30 19:43:05.70119958 +0000 UTC m=+1633.271665196"
Oct 30 19:43:06.146989 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:43:06.146291    4892 patch_prober.go:29] interesting pod/e2e-worker-w2-a3f9b7-openshift-default-7d6d8b8fb5-5vgch container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.124:15021/healthz/ready\": dial tcp 10.132.0.124:15021: connect: connection refused" start-of-body=
Oct 30 19:43:06.146989 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:43:06.146362    4892 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-a3f9b7-openshift-default-7d6d8b8fb5-5vgch" podUID="78ea3297-af25-446c-b7d0-3da5ae88dd15" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.124:15021/healthz/ready\": dial tcp 10.132.0.124:15021: connect: connection refused"
Oct 30 19:43:06.832989 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:43:06.821638    4892 generic.go:356] "Generic (PLEG): container finished" podID="90541e2c-7c60-4919-830e-73341e6c4cfe" containerID="1b500499c8fafcb5e4fcde940952876bf77ddfc4d09e1fb4d2a9009c7f3579c0" exitCode=0
Oct 30 19:43:06.832989 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:43:06.821803    4892 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-454a9e34-dtlzm" event={"ID":"90541e2c-7c60-4919-830e-73341e6c4cfe","Type":"ContainerDied","Data":"1b500499c8fafcb5e4fcde940952876bf77ddfc4d09e1fb4d2a9009c7f3579c0"}
Oct 30 19:43:07.164105 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:43:07.163357    4892 patch_prober.go:29] interesting pod/e2e-worker-w2-a3f9b7-openshift-default-7d6d8b8fb5-5vgch container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.124:15021/healthz/ready\": dial tcp 10.132.0.124:15021: connect: connection refused" start-of-body=
Oct 30 19:43:07.164105 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:43:07.163417    4892 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-a3f9b7-openshift-default-7d6d8b8fb5-5vgch" podUID="78ea3297-af25-446c-b7d0-3da5ae88dd15" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.124:15021/healthz/ready\": dial tcp 10.132.0.124:15021: connect: connection refused"
Oct 30 19:43:08.131050 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:43:08.127001    4892 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-454a9e34-dtlzm"
Oct 30 19:43:08.145869 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:43:08.145367    4892 patch_prober.go:29] interesting pod/e2e-worker-w2-a3f9b7-openshift-default-7d6d8b8fb5-5vgch container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.124:15021/healthz/ready\": dial tcp 10.132.0.124:15021: connect: connection refused" start-of-body=
Oct 30 19:43:08.145869 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:43:08.145423    4892 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-a3f9b7-openshift-default-7d6d8b8fb5-5vgch" podUID="78ea3297-af25-446c-b7d0-3da5ae88dd15" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.124:15021/healthz/ready\": dial tcp 10.132.0.124:15021: connect: connection refused"
Oct 30 19:43:08.252992 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:43:08.252515    4892 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/90541e2c-7c60-4919-830e-73341e6c4cfe-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/90541e2c-7c60-4919-830e-73341e6c4cfe-maas-api-openshift-ca\") pod \"90541e2c-7c60-4919-830e-73341e6c4cfe\" (UID: \"90541e2c-7c60-4919-830e-73341e6c4cfe\") "
Oct 30 19:43:08.252992 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:43:08.252645    4892 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/90541e2c-7c60-4919-830e-73341e6c4cfe-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/90541e2c-7c60-4919-830e-73341e6c4cfe-maas-api-xks-ca\") pod \"90541e2c-7c60-4919-830e-73341e6c4cfe\" (UID: \"90541e2c-7c60-4919-830e-73341e6c4cfe\") "
Oct 30 19:43:08.252992 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:43:08.252680    4892 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/90541e2c-7c60-4919-830e-73341e6c4cfe-kube-api-access-nwxq6\" (UniqueName: \"kubernetes.io/projected/90541e2c-7c60-4919-830e-73341e6c4cfe-kube-api-access-nwxq6\") pod \"90541e2c-7c60-4919-830e-73341e6c4cfe\" (UID: \"90541e2c-7c60-4919-830e-73341e6c4cfe\") "
Oct 30 19:43:08.268491 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:43:08.266742    4892 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/90541e2c-7c60-4919-830e-73341e6c4cfe-kube-api-access-nwxq6" pod "90541e2c-7c60-4919-830e-73341e6c4cfe" (UID: "90541e2c-7c60-4919-830e-73341e6c4cfe"). InnerVolumeSpecName "kube-api-access-nwxq6". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:43:08.268491 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:43:08.267896    4892 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/90541e2c-7c60-4919-830e-73341e6c4cfe-maas-api-openshift-ca" pod "90541e2c-7c60-4919-830e-73341e6c4cfe" (UID: "90541e2c-7c60-4919-830e-73341e6c4cfe"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 19:43:08.268491 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:43:08.268385    4892 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/90541e2c-7c60-4919-830e-73341e6c4cfe-maas-api-xks-ca" pod "90541e2c-7c60-4919-830e-73341e6c4cfe" (UID: "90541e2c-7c60-4919-830e-73341e6c4cfe"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 19:43:08.359635 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:43:08.359277    4892 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/90541e2c-7c60-4919-830e-73341e6c4cfe-maas-api-openshift-ca\") on node \"ip-10-0-132-82.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 19:43:08.359635 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:43:08.359316    4892 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/90541e2c-7c60-4919-830e-73341e6c4cfe-maas-api-xks-ca\") on node \"ip-10-0-132-82.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 19:43:08.359635 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:43:08.359333    4892 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nwxq6\" (UniqueName: \"kubernetes.io/projected/90541e2c-7c60-4919-830e-73341e6c4cfe-kube-api-access-nwxq6\") on node \"ip-10-0-132-82.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 19:43:08.894550 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:43:08.894518    4892 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-454a9e34-dtlzm" event={"ID":"90541e2c-7c60-4919-830e-73341e6c4cfe","Type":"ContainerDied","Data":"36082c404785d27ccc35431452cbabb7cfca56fff96df2c85239393304700b17"}
Oct 30 19:43:08.894729 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:43:08.894572    4892 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="36082c404785d27ccc35431452cbabb7cfca56fff96df2c85239393304700b17"
Oct 30 19:43:08.894729 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:43:08.894630    4892 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-454a9e34-dtlzm"
Oct 30 19:43:09.145696 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:43:09.145616    4892 patch_prober.go:29] interesting pod/e2e-worker-w2-a3f9b7-openshift-default-7d6d8b8fb5-5vgch container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.124:15021/healthz/ready\": dial tcp 10.132.0.124:15021: connect: connection refused" start-of-body=
Oct 30 19:43:09.145914 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:43:09.145889    4892 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-a3f9b7-openshift-default-7d6d8b8fb5-5vgch" podUID="78ea3297-af25-446c-b7d0-3da5ae88dd15" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.124:15021/healthz/ready\": dial tcp 10.132.0.124:15021: connect: connection refused"
Oct 30 19:43:09.317422 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:43:09.317394    4892 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-fsfvk"]
Oct 30 19:43:11.207099 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:43:11.207059    4892 patch_prober.go:29] interesting pod/e2e-worker-w2-a3f9b7-openshift-default-7d6d8b8fb5-5vgch container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.124:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 30 19:43:11.207716 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:43:11.207681    4892 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-a3f9b7-openshift-default-7d6d8b8fb5-5vgch" podUID="78ea3297-af25-446c-b7d0-3da5ae88dd15" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.124:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:43:11.681998 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:43:11.666366    4892 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w2-a3f9b7-openshift-default-7d6d8b8fb5-5vgch"
Oct 30 19:43:11.681998 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:43:11.667857    4892 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w2-a3f9b7-openshift-default-7d6d8b8fb5-5vgch"
Oct 30 19:43:11.682593 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:43:11.682535    4892 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w2-a3f9b7-openshift-default-7d6d8b8fb5-5vgch"
Oct 30 19:43:11.862008 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:43:11.861947    4892 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-2d9861/premium-w0-2d9861-kserve-67945c6848-4n6kz" podUID="798988b2-c769-4f73-a7c8-ca15de0beef1" containerName="main" probeResult="failure" output="Get \"https://10.132.0.62:8000/ready\": context deadline exceeded"
Oct 30 19:43:11.888106 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:43:11.888064    4892 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-b5178f/distinct-2-w4-b5178f-kserve-69ddf8b665-9pdth" podUID="fb71d961-8500-4367-b48c-88d385962bd5" containerName="main" probeResult="failure" output="Get \"https://10.132.0.90:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:43:12.026047 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:43:12.015057    4892 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-5f55cd/simulator-w6-5f55cd-kserve-5ff86b7447-h9tm7" podUID="3cb47a99-8be1-4756-b31a-43b58cc6b8dc" containerName="main" probeResult="failure" output="Get \"https://10.132.0.58:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:43:12.089885 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:43:12.089540    4892 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-ada8d7/simulator-w3-ada8d7-kserve-7d575cfcb5-5txgn" podUID="10a5c7c8-6146-47b0-9643-bb137cf10bf5" containerName="main" probeResult="failure" output="Get \"https://10.132.0.55:8000/ready\": context deadline exceeded"
Oct 30 19:43:12.251152 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:43:12.251101    4892 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w4-b5178f/distinct-2-w4-b5178f-kserve-69ddf8b665-9pdth" podUID="fb71d961-8500-4367-b48c-88d385962bd5" containerName="main" probeResult="failure" output="Get \"https://10.132.0.90:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:43:12.293210 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:43:12.293056    4892 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w3-ada8d7/simulator-w3-ada8d7-kserve-7d575cfcb5-5txgn" podUID="10a5c7c8-6146-47b0-9643-bb137cf10bf5" containerName="main" probeResult="failure" output="Get \"https://10.132.0.55:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:43:12.896122 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:43:12.896053    4892 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-b5178f/distinct-2-w4-b5178f-kserve-69ddf8b665-9pdth" podUID="fb71d961-8500-4367-b48c-88d385962bd5" containerName="main" probeResult="failure" output="Get \"https://10.132.0.90:8000/ready\": context deadline exceeded"
Oct 30 19:43:12.897997 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:43:12.896465    4892 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-b5178f/distinct-2-w4-b5178f-kserve-69ddf8b665-9pdth"
Oct 30 19:43:13.098991 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:43:13.080091    4892 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-5f55cd/simulator-w6-5f55cd-kserve-5ff86b7447-h9tm7" podUID="3cb47a99-8be1-4756-b31a-43b58cc6b8dc" containerName="main" probeResult="failure" output="Get \"https://10.132.0.58:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:43:13.098991 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:43:13.080848    4892 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w6-5f55cd/simulator-w6-5f55cd-kserve-5ff86b7447-h9tm7"
Oct 30 19:43:13.127871 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:43:13.099383    4892 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-ada8d7/simulator-w3-ada8d7-kserve-7d575cfcb5-5txgn" podUID="10a5c7c8-6146-47b0-9643-bb137cf10bf5" containerName="main" probeResult="failure" output="Get \"https://10.132.0.55:8000/ready\": context deadline exceeded"
Oct 30 19:43:13.127871 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:43:13.099757    4892 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-5f55cd/premium-w6-5f55cd-kserve-7f785c544b-pcvhb" podUID="191593a2-e3a9-4ce4-a1c6-c089723b0237" containerName="main" probeResult="failure" output="Get \"https://10.132.0.60:8000/ready\": context deadline exceeded"
Oct 30 19:43:13.127871 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:43:13.100113    4892 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-ada8d7/simulator-w3-ada8d7-kserve-7d575cfcb5-5txgn"
Oct 30 19:43:13.285066 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:43:13.279618    4892 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-ada8d7/simulator-w3-ada8d7-kserve-7d575cfcb5-5txgn"
Oct 30 19:43:13.329642 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:43:13.329618    4892 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-b5178f/distinct-2-w4-b5178f-kserve-69ddf8b665-9pdth"
Oct 30 19:43:13.517516 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:43:13.516439    4892 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w6-5f55cd/simulator-w6-5f55cd-kserve-5ff86b7447-h9tm7"
Oct 30 19:43:15.346118 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:43:15.346086    4892 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ff93af8f-7pjx4"]
Oct 30 19:43:15.346635 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:43:15.346621    4892 cpu_manager.go:414] "containerMap: removing container" podUID="90541e2c-7c60-4919-830e-73341e6c4cfe" containerName="revoke-keys"
Oct 30 19:43:15.346678 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:43:15.346635    4892 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="90541e2c-7c60-4919-830e-73341e6c4cfe" containerName="revoke-keys"
Oct 30 19:43:15.346743 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:43:15.346734    4892 memory_manager.go:365] "RemoveStaleState removing state" podUID="90541e2c-7c60-4919-830e-73341e6c4cfe" containerName="revoke-keys"
Oct 30 19:43:15.350230 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:43:15.350214    4892 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ff93af8f-7pjx4"
Oct 30 19:43:15.408107 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:43:15.408076    4892 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ff93af8f-7pjx4"]
Oct 30 19:43:15.450204 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:43:15.450161    4892 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 19:43:15.450204 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:43:15.450161    4892 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 30 19:43:15.450442 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:43:15.450322    4892 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-nlxp4\""
Oct 30 19:43:15.451290 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:43:15.451271    4892 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 30 19:43:15.485174 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:43:15.485148    4892 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/398bf4dc-e72f-46de-93fe-32dfeb6574d3-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-ff93af8f-7pjx4\" (UID: \"398bf4dc-e72f-46de-93fe-32dfeb6574d3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ff93af8f-7pjx4"
Oct 30 19:43:15.485319 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:43:15.485203    4892 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/398bf4dc-e72f-46de-93fe-32dfeb6574d3-maas-api-xks-ca\") pod \"maas-api-revoke-sub-ff93af8f-7pjx4\" (UID: \"398bf4dc-e72f-46de-93fe-32dfeb6574d3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ff93af8f-7pjx4"
Oct 30 19:43:15.485319 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:43:15.485234    4892 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sxvpl\" (UniqueName: \"kubernetes.io/projected/398bf4dc-e72f-46de-93fe-32dfeb6574d3-kube-api-access-sxvpl\") pod \"maas-api-revoke-sub-ff93af8f-7pjx4\" (UID: \"398bf4dc-e72f-46de-93fe-32dfeb6574d3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ff93af8f-7pjx4"
Oct 30 19:43:15.586740 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:43:15.586702    4892 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/398bf4dc-e72f-46de-93fe-32dfeb6574d3-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-ff93af8f-7pjx4\" (UID: \"398bf4dc-e72f-46de-93fe-32dfeb6574d3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ff93af8f-7pjx4"
Oct 30 19:43:15.586919 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:43:15.586759    4892 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/398bf4dc-e72f-46de-93fe-32dfeb6574d3-maas-api-xks-ca\") pod \"maas-api-revoke-sub-ff93af8f-7pjx4\" (UID: \"398bf4dc-e72f-46de-93fe-32dfeb6574d3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ff93af8f-7pjx4"
Oct 30 19:43:15.586919 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:43:15.586791    4892 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sxvpl\" (UniqueName: \"kubernetes.io/projected/398bf4dc-e72f-46de-93fe-32dfeb6574d3-kube-api-access-sxvpl\") pod \"maas-api-revoke-sub-ff93af8f-7pjx4\" (UID: \"398bf4dc-e72f-46de-93fe-32dfeb6574d3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ff93af8f-7pjx4"
Oct 30 19:43:15.587340 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:43:15.587314    4892 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/398bf4dc-e72f-46de-93fe-32dfeb6574d3-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-ff93af8f-7pjx4\" (UID: \"398bf4dc-e72f-46de-93fe-32dfeb6574d3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ff93af8f-7pjx4"
Oct 30 19:43:15.589353 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:43:15.589326    4892 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/398bf4dc-e72f-46de-93fe-32dfeb6574d3-maas-api-xks-ca\") pod \"maas-api-revoke-sub-ff93af8f-7pjx4\" (UID: \"398bf4dc-e72f-46de-93fe-32dfeb6574d3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ff93af8f-7pjx4"
Oct 30 19:43:15.643792 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:43:15.643761    4892 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-sxvpl\" (UniqueName: \"kubernetes.io/projected/398bf4dc-e72f-46de-93fe-32dfeb6574d3-kube-api-access-sxvpl\") pod \"maas-api-revoke-sub-ff93af8f-7pjx4\" (UID: \"398bf4dc-e72f-46de-93fe-32dfeb6574d3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ff93af8f-7pjx4"
Oct 30 19:43:15.664575 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:43:15.664552    4892 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ff93af8f-7pjx4"
Oct 30 19:43:16.043997 ip-10-0-132-82 kubenswrapper[4892]: W1030 19:43:16.043304    4892 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod398bf4dc_e72f_46de_93fe_32dfeb6574d3.slice/crio-39a7881238db437f8d956df406eb033331f4cb69072cf576f905d8d7a9302577 WatchSource:0}: Error finding container 39a7881238db437f8d956df406eb033331f4cb69072cf576f905d8d7a9302577: Status 404 returned error can't find the container with id 39a7881238db437f8d956df406eb033331f4cb69072cf576f905d8d7a9302577
Oct 30 19:43:16.046202 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:43:16.046173    4892 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ff93af8f-7pjx4"]
Oct 30 19:43:16.392526 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:43:16.392495    4892 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ff93af8f-7pjx4" event={"ID":"398bf4dc-e72f-46de-93fe-32dfeb6574d3","Type":"ContainerStarted","Data":"1e4bbc8e4a1be1435698519557f20d55c89fe7141636220fec506120abd765c9"}
Oct 30 19:43:16.392912 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:43:16.392548    4892 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ff93af8f-7pjx4" event={"ID":"398bf4dc-e72f-46de-93fe-32dfeb6574d3","Type":"ContainerStarted","Data":"39a7881238db437f8d956df406eb033331f4cb69072cf576f905d8d7a9302577"}
Oct 30 19:43:16.506880 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:43:16.506809    4892 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ff93af8f-7pjx4" podStartSLOduration=1.506794011 podStartE2EDuration="1.506794011s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:43:15 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:43:16.504798155 +0000 UTC m=+1644.075263754" watchObservedRunningTime="2026-10-30 19:43:16.506794011 +0000 UTC m=+1644.077259610"
Oct 30 19:43:17.295642 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:43:17.295562    4892 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-7415809c-75wmh"]
Oct 30 19:43:17.299904 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:43:17.299878    4892 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7415809c-75wmh"
Oct 30 19:43:17.358018 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:43:17.357988    4892 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-7415809c-75wmh"]
Oct 30 19:43:17.404705 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:43:17.404667    4892 generic.go:356] "Generic (PLEG): container finished" podID="398bf4dc-e72f-46de-93fe-32dfeb6574d3" containerID="1e4bbc8e4a1be1435698519557f20d55c89fe7141636220fec506120abd765c9" exitCode=0
Oct 30 19:43:17.404705 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:43:17.404761    4892 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ff93af8f-7pjx4" event={"ID":"398bf4dc-e72f-46de-93fe-32dfeb6574d3","Type":"ContainerDied","Data":"1e4bbc8e4a1be1435698519557f20d55c89fe7141636220fec506120abd765c9"}
Oct 30 19:43:17.430612 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:43:17.430575    4892 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wzbsc\" (UniqueName: \"kubernetes.io/projected/a1a081b7-23cf-4337-9ad4-cb512b27fa7f-kube-api-access-wzbsc\") pod \"maas-api-revoke-sub-7415809c-75wmh\" (UID: \"a1a081b7-23cf-4337-9ad4-cb512b27fa7f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7415809c-75wmh"
Oct 30 19:43:17.430740 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:43:17.430637    4892 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a1a081b7-23cf-4337-9ad4-cb512b27fa7f-maas-api-xks-ca\") pod \"maas-api-revoke-sub-7415809c-75wmh\" (UID: \"a1a081b7-23cf-4337-9ad4-cb512b27fa7f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7415809c-75wmh"
Oct 30 19:43:17.430808 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:43:17.430787    4892 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a1a081b7-23cf-4337-9ad4-cb512b27fa7f-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-7415809c-75wmh\" (UID: \"a1a081b7-23cf-4337-9ad4-cb512b27fa7f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7415809c-75wmh"
Oct 30 19:43:17.532012 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:43:17.531785    4892 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a1a081b7-23cf-4337-9ad4-cb512b27fa7f-maas-api-xks-ca\") pod \"maas-api-revoke-sub-7415809c-75wmh\" (UID: \"a1a081b7-23cf-4337-9ad4-cb512b27fa7f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7415809c-75wmh"
Oct 30 19:43:17.532012 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:43:17.531987    4892 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a1a081b7-23cf-4337-9ad4-cb512b27fa7f-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-7415809c-75wmh\" (UID: \"a1a081b7-23cf-4337-9ad4-cb512b27fa7f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7415809c-75wmh"
Oct 30 19:43:17.532239 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:43:17.532170    4892 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wzbsc\" (UniqueName: \"kubernetes.io/projected/a1a081b7-23cf-4337-9ad4-cb512b27fa7f-kube-api-access-wzbsc\") pod \"maas-api-revoke-sub-7415809c-75wmh\" (UID: \"a1a081b7-23cf-4337-9ad4-cb512b27fa7f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7415809c-75wmh"
Oct 30 19:43:17.533495 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:43:17.533471    4892 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a1a081b7-23cf-4337-9ad4-cb512b27fa7f-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-7415809c-75wmh\" (UID: \"a1a081b7-23cf-4337-9ad4-cb512b27fa7f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7415809c-75wmh"
Oct 30 19:43:17.535465 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:43:17.535441    4892 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a1a081b7-23cf-4337-9ad4-cb512b27fa7f-maas-api-xks-ca\") pod \"maas-api-revoke-sub-7415809c-75wmh\" (UID: \"a1a081b7-23cf-4337-9ad4-cb512b27fa7f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7415809c-75wmh"
Oct 30 19:43:17.588709 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:43:17.588637    4892 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wzbsc\" (UniqueName: \"kubernetes.io/projected/a1a081b7-23cf-4337-9ad4-cb512b27fa7f-kube-api-access-wzbsc\") pod \"maas-api-revoke-sub-7415809c-75wmh\" (UID: \"a1a081b7-23cf-4337-9ad4-cb512b27fa7f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7415809c-75wmh"
Oct 30 19:43:17.613299 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:43:17.613257    4892 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7415809c-75wmh"
Oct 30 19:43:17.997301 ip-10-0-132-82 kubenswrapper[4892]: W1030 19:43:17.997268    4892 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda1a081b7_23cf_4337_9ad4_cb512b27fa7f.slice/crio-c74a789497c81feb4ec678b59627cda5c56d0640ead50f216afe0848c950a2ba WatchSource:0}: Error finding container c74a789497c81feb4ec678b59627cda5c56d0640ead50f216afe0848c950a2ba: Status 404 returned error can't find the container with id c74a789497c81feb4ec678b59627cda5c56d0640ead50f216afe0848c950a2ba
Oct 30 19:43:17.998373 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:43:17.998346    4892 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-7415809c-75wmh"]
Oct 30 19:43:18.423018 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:43:18.422948    4892 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7415809c-75wmh" event={"ID":"a1a081b7-23cf-4337-9ad4-cb512b27fa7f","Type":"ContainerStarted","Data":"c74a789497c81feb4ec678b59627cda5c56d0640ead50f216afe0848c950a2ba"}
Oct 30 19:43:18.647563 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:43:18.647538    4892 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ff93af8f-7pjx4"
Oct 30 19:43:18.747016 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:43:18.746955    4892 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/398bf4dc-e72f-46de-93fe-32dfeb6574d3-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/398bf4dc-e72f-46de-93fe-32dfeb6574d3-maas-api-xks-ca\") pod \"398bf4dc-e72f-46de-93fe-32dfeb6574d3\" (UID: \"398bf4dc-e72f-46de-93fe-32dfeb6574d3\") "
Oct 30 19:43:18.747168 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:43:18.747148    4892 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/398bf4dc-e72f-46de-93fe-32dfeb6574d3-kube-api-access-sxvpl\" (UniqueName: \"kubernetes.io/projected/398bf4dc-e72f-46de-93fe-32dfeb6574d3-kube-api-access-sxvpl\") pod \"398bf4dc-e72f-46de-93fe-32dfeb6574d3\" (UID: \"398bf4dc-e72f-46de-93fe-32dfeb6574d3\") "
Oct 30 19:43:18.747215 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:43:18.747197    4892 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/398bf4dc-e72f-46de-93fe-32dfeb6574d3-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/398bf4dc-e72f-46de-93fe-32dfeb6574d3-maas-api-openshift-ca\") pod \"398bf4dc-e72f-46de-93fe-32dfeb6574d3\" (UID: \"398bf4dc-e72f-46de-93fe-32dfeb6574d3\") "
Oct 30 19:43:18.747716 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:43:18.747653    4892 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/398bf4dc-e72f-46de-93fe-32dfeb6574d3-maas-api-openshift-ca" pod "398bf4dc-e72f-46de-93fe-32dfeb6574d3" (UID: "398bf4dc-e72f-46de-93fe-32dfeb6574d3"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 19:43:18.749169 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:43:18.749142    4892 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/398bf4dc-e72f-46de-93fe-32dfeb6574d3-maas-api-xks-ca" pod "398bf4dc-e72f-46de-93fe-32dfeb6574d3" (UID: "398bf4dc-e72f-46de-93fe-32dfeb6574d3"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 19:43:18.749456 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:43:18.749431    4892 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/398bf4dc-e72f-46de-93fe-32dfeb6574d3-kube-api-access-sxvpl" pod "398bf4dc-e72f-46de-93fe-32dfeb6574d3" (UID: "398bf4dc-e72f-46de-93fe-32dfeb6574d3"). InnerVolumeSpecName "kube-api-access-sxvpl". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:43:18.849100 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:43:18.849070    4892 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/398bf4dc-e72f-46de-93fe-32dfeb6574d3-maas-api-xks-ca\") on node \"ip-10-0-132-82.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 19:43:18.849100 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:43:18.849102    4892 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-sxvpl\" (UniqueName: \"kubernetes.io/projected/398bf4dc-e72f-46de-93fe-32dfeb6574d3-kube-api-access-sxvpl\") on node \"ip-10-0-132-82.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 19:43:18.849288 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:43:18.849119    4892 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/398bf4dc-e72f-46de-93fe-32dfeb6574d3-maas-api-openshift-ca\") on node \"ip-10-0-132-82.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 19:43:19.432253 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:43:19.432219    4892 generic.go:356] "Generic (PLEG): container finished" podID="a1a081b7-23cf-4337-9ad4-cb512b27fa7f" containerID="0cad2fcd9c5118cef7cb15d31b0b6314826d4bf1d98894a490c4464b77e8a715" exitCode=0
Oct 30 19:43:19.432647 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:43:19.432307    4892 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7415809c-75wmh" event={"ID":"a1a081b7-23cf-4337-9ad4-cb512b27fa7f","Type":"ContainerDied","Data":"0cad2fcd9c5118cef7cb15d31b0b6314826d4bf1d98894a490c4464b77e8a715"}
Oct 30 19:43:19.434032 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:43:19.434016    4892 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ff93af8f-7pjx4"
Oct 30 19:43:19.434129 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:43:19.434057    4892 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ff93af8f-7pjx4" event={"ID":"398bf4dc-e72f-46de-93fe-32dfeb6574d3","Type":"ContainerDied","Data":"39a7881238db437f8d956df406eb033331f4cb69072cf576f905d8d7a9302577"}
Oct 30 19:43:19.434129 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:43:19.434098    4892 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="39a7881238db437f8d956df406eb033331f4cb69072cf576f905d8d7a9302577"
Oct 30 19:43:20.603208 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:43:20.603182    4892 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7415809c-75wmh"
Oct 30 19:43:20.774334 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:43:20.774258    4892 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a1a081b7-23cf-4337-9ad4-cb512b27fa7f-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a1a081b7-23cf-4337-9ad4-cb512b27fa7f-maas-api-openshift-ca\") pod \"a1a081b7-23cf-4337-9ad4-cb512b27fa7f\" (UID: \"a1a081b7-23cf-4337-9ad4-cb512b27fa7f\") "
Oct 30 19:43:20.774496 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:43:20.774460    4892 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/a1a081b7-23cf-4337-9ad4-cb512b27fa7f-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a1a081b7-23cf-4337-9ad4-cb512b27fa7f-maas-api-xks-ca\") pod \"a1a081b7-23cf-4337-9ad4-cb512b27fa7f\" (UID: \"a1a081b7-23cf-4337-9ad4-cb512b27fa7f\") "
Oct 30 19:43:20.774567 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:43:20.774507    4892 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a1a081b7-23cf-4337-9ad4-cb512b27fa7f-kube-api-access-wzbsc\" (UniqueName: \"kubernetes.io/projected/a1a081b7-23cf-4337-9ad4-cb512b27fa7f-kube-api-access-wzbsc\") pod \"a1a081b7-23cf-4337-9ad4-cb512b27fa7f\" (UID: \"a1a081b7-23cf-4337-9ad4-cb512b27fa7f\") "
Oct 30 19:43:20.774624 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:43:20.774576    4892 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a1a081b7-23cf-4337-9ad4-cb512b27fa7f-maas-api-openshift-ca" pod "a1a081b7-23cf-4337-9ad4-cb512b27fa7f" (UID: "a1a081b7-23cf-4337-9ad4-cb512b27fa7f"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 19:43:20.774928 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:43:20.774910    4892 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a1a081b7-23cf-4337-9ad4-cb512b27fa7f-maas-api-openshift-ca\") on node \"ip-10-0-132-82.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 19:43:20.776629 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:43:20.776606    4892 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a1a081b7-23cf-4337-9ad4-cb512b27fa7f-kube-api-access-wzbsc" pod "a1a081b7-23cf-4337-9ad4-cb512b27fa7f" (UID: "a1a081b7-23cf-4337-9ad4-cb512b27fa7f"). InnerVolumeSpecName "kube-api-access-wzbsc". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:43:20.776707 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:43:20.776688    4892 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a1a081b7-23cf-4337-9ad4-cb512b27fa7f-maas-api-xks-ca" pod "a1a081b7-23cf-4337-9ad4-cb512b27fa7f" (UID: "a1a081b7-23cf-4337-9ad4-cb512b27fa7f"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 19:43:20.876454 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:43:20.876425    4892 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a1a081b7-23cf-4337-9ad4-cb512b27fa7f-maas-api-xks-ca\") on node \"ip-10-0-132-82.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 19:43:20.876454 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:43:20.876450    4892 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wzbsc\" (UniqueName: \"kubernetes.io/projected/a1a081b7-23cf-4337-9ad4-cb512b27fa7f-kube-api-access-wzbsc\") on node \"ip-10-0-132-82.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 19:43:21.451412 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:43:21.451382    4892 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7415809c-75wmh"
Oct 30 19:43:21.451572 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:43:21.451428    4892 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7415809c-75wmh" event={"ID":"a1a081b7-23cf-4337-9ad4-cb512b27fa7f","Type":"ContainerDied","Data":"c74a789497c81feb4ec678b59627cda5c56d0640ead50f216afe0848c950a2ba"}
Oct 30 19:43:21.451572 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:43:21.451481    4892 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="c74a789497c81feb4ec678b59627cda5c56d0640ead50f216afe0848c950a2ba"
Oct 30 19:43:24.846907 ip-10-0-132-82 kubenswrapper[4892]: E1030 19:43:24.846798    4892 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-jrm87" containerName="extractor"
Oct 30 19:43:26.097423 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:43:26.097372    4892 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-8964-629d1b"]
Oct 30 19:43:26.098199 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:43:26.098182    4892 cpu_manager.go:414] "containerMap: removing container" podUID="398bf4dc-e72f-46de-93fe-32dfeb6574d3" containerName="revoke-keys"
Oct 30 19:43:26.098248 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:43:26.098203    4892 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="398bf4dc-e72f-46de-93fe-32dfeb6574d3" containerName="revoke-keys"
Oct 30 19:43:26.098248 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:43:26.098228    4892 cpu_manager.go:414] "containerMap: removing container" podUID="a1a081b7-23cf-4337-9ad4-cb512b27fa7f" containerName="revoke-keys"
Oct 30 19:43:26.098248 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:43:26.098235    4892 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a1a081b7-23cf-4337-9ad4-cb512b27fa7f" containerName="revoke-keys"
Oct 30 19:43:26.098402 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:43:26.098391    4892 memory_manager.go:365] "RemoveStaleState removing state" podUID="a1a081b7-23cf-4337-9ad4-cb512b27fa7f" containerName="revoke-keys"
Oct 30 19:43:26.098444 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:43:26.098416    4892 memory_manager.go:365] "RemoveStaleState removing state" podUID="398bf4dc-e72f-46de-93fe-32dfeb6574d3" containerName="revoke-keys"
Oct 30 19:43:26.102677 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:43:26.102656    4892 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-8964-629d1b"
Oct 30 19:43:26.153012 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:43:26.152981    4892 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-8964-629d1b"]
Oct 30 19:43:26.196590 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:43:26.196570    4892 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"default-dockercfg-q2f25\""
Oct 30 19:43:26.242355 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:43:26.242329    4892 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nmtfs\" (UniqueName: \"kubernetes.io/projected/93d8a0c9-c95b-481c-9700-5609fe407cdc-kube-api-access-nmtfs\") pod \"test-curl-8964-629d1b\" (UID: \"93d8a0c9-c95b-481c-9700-5609fe407cdc\") " pod="opendatahub/test-curl-8964-629d1b"
Oct 30 19:43:26.343302 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:43:26.343268    4892 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nmtfs\" (UniqueName: \"kubernetes.io/projected/93d8a0c9-c95b-481c-9700-5609fe407cdc-kube-api-access-nmtfs\") pod \"test-curl-8964-629d1b\" (UID: \"93d8a0c9-c95b-481c-9700-5609fe407cdc\") " pod="opendatahub/test-curl-8964-629d1b"
Oct 30 19:43:26.400764 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:43:26.400740    4892 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nmtfs\" (UniqueName: \"kubernetes.io/projected/93d8a0c9-c95b-481c-9700-5609fe407cdc-kube-api-access-nmtfs\") pod \"test-curl-8964-629d1b\" (UID: \"93d8a0c9-c95b-481c-9700-5609fe407cdc\") " pod="opendatahub/test-curl-8964-629d1b"
Oct 30 19:43:26.415677 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:43:26.415657    4892 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-8964-629d1b"
Oct 30 19:43:26.796551 ip-10-0-132-82 kubenswrapper[4892]: W1030 19:43:26.796515    4892 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod93d8a0c9_c95b_481c_9700_5609fe407cdc.slice/crio-682698aebd8bb8d73cc4db7f56b243df190ddac51aff2f0e8d318c6d4bcf1773 WatchSource:0}: Error finding container 682698aebd8bb8d73cc4db7f56b243df190ddac51aff2f0e8d318c6d4bcf1773: Status 404 returned error can't find the container with id 682698aebd8bb8d73cc4db7f56b243df190ddac51aff2f0e8d318c6d4bcf1773
Oct 30 19:43:26.796551 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:43:26.796523    4892 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-8964-629d1b"]
Oct 30 19:43:26.798959 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:43:26.798942    4892 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Oct 30 19:43:27.505998 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:43:27.505942    4892 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-8964-629d1b" event={"ID":"93d8a0c9-c95b-481c-9700-5609fe407cdc","Type":"ContainerStarted","Data":"682698aebd8bb8d73cc4db7f56b243df190ddac51aff2f0e8d318c6d4bcf1773"}
Oct 30 19:43:27.989088 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:43:27.989062    4892 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b16664de-zhmcs"]
Oct 30 19:43:27.994476 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:43:27.994459    4892 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b16664de-zhmcs"
Oct 30 19:43:28.050098 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:43:28.050075    4892 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b16664de-zhmcs"]
Oct 30 19:43:28.068126 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:43:28.068098    4892 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 30 19:43:28.093409 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:43:28.093383    4892 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 30 19:43:28.093409 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:43:28.093391    4892 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 19:43:28.093570 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:43:28.093421    4892 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-nlxp4\""
Oct 30 19:43:28.164108 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:43:28.164078    4892 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/545e8fd9-b409-4437-9ed0-c227942ca7da-maas-api-xks-ca\") pod \"maas-api-revoke-sub-b16664de-zhmcs\" (UID: \"545e8fd9-b409-4437-9ed0-c227942ca7da\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b16664de-zhmcs"
Oct 30 19:43:28.164256 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:43:28.164118    4892 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/545e8fd9-b409-4437-9ed0-c227942ca7da-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-b16664de-zhmcs\" (UID: \"545e8fd9-b409-4437-9ed0-c227942ca7da\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b16664de-zhmcs"
Oct 30 19:43:28.164256 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:43:28.164151    4892 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-w8t5k\" (UniqueName: \"kubernetes.io/projected/545e8fd9-b409-4437-9ed0-c227942ca7da-kube-api-access-w8t5k\") pod \"maas-api-revoke-sub-b16664de-zhmcs\" (UID: \"545e8fd9-b409-4437-9ed0-c227942ca7da\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b16664de-zhmcs"
Oct 30 19:43:28.265461 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:43:28.265385    4892 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/545e8fd9-b409-4437-9ed0-c227942ca7da-maas-api-xks-ca\") pod \"maas-api-revoke-sub-b16664de-zhmcs\" (UID: \"545e8fd9-b409-4437-9ed0-c227942ca7da\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b16664de-zhmcs"
Oct 30 19:43:28.265461 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:43:28.265426    4892 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/545e8fd9-b409-4437-9ed0-c227942ca7da-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-b16664de-zhmcs\" (UID: \"545e8fd9-b409-4437-9ed0-c227942ca7da\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b16664de-zhmcs"
Oct 30 19:43:28.265461 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:43:28.265456    4892 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-w8t5k\" (UniqueName: \"kubernetes.io/projected/545e8fd9-b409-4437-9ed0-c227942ca7da-kube-api-access-w8t5k\") pod \"maas-api-revoke-sub-b16664de-zhmcs\" (UID: \"545e8fd9-b409-4437-9ed0-c227942ca7da\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b16664de-zhmcs"
Oct 30 19:43:28.266120 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:43:28.266097    4892 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/545e8fd9-b409-4437-9ed0-c227942ca7da-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-b16664de-zhmcs\" (UID: \"545e8fd9-b409-4437-9ed0-c227942ca7da\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b16664de-zhmcs"
Oct 30 19:43:28.267944 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:43:28.267922    4892 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/545e8fd9-b409-4437-9ed0-c227942ca7da-maas-api-xks-ca\") pod \"maas-api-revoke-sub-b16664de-zhmcs\" (UID: \"545e8fd9-b409-4437-9ed0-c227942ca7da\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b16664de-zhmcs"
Oct 30 19:43:28.323146 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:43:28.323122    4892 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-w8t5k\" (UniqueName: \"kubernetes.io/projected/545e8fd9-b409-4437-9ed0-c227942ca7da-kube-api-access-w8t5k\") pod \"maas-api-revoke-sub-b16664de-zhmcs\" (UID: \"545e8fd9-b409-4437-9ed0-c227942ca7da\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b16664de-zhmcs"
Oct 30 19:43:28.528100 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:43:28.528000    4892 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-8964-629d1b" event={"ID":"93d8a0c9-c95b-481c-9700-5609fe407cdc","Type":"ContainerStarted","Data":"f24b6e6de28027a2919befad6d44a2c23d1c6194139d51b0222d6cdceed2d903"}
Oct 30 19:43:28.621610 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:43:28.621582    4892 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b16664de-zhmcs"
Oct 30 19:43:28.646658 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:43:28.646255    4892 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/test-curl-8964-629d1b" podStartSLOduration=1.4810981 podStartE2EDuration="2.645224496s" totalImagesPullingTime="1.164126396s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:43:26 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:43:28.639077757 +0000 UTC m=+1656.209543372" watchObservedRunningTime="2026-10-30 19:43:28.645224496 +0000 UTC m=+1656.215690095"
Oct 30 19:43:29.005771 ip-10-0-132-82 kubenswrapper[4892]: W1030 19:43:29.005739    4892 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod545e8fd9_b409_4437_9ed0_c227942ca7da.slice/crio-4b51f7c5d9be7ae56728fd8a50a88f646269542793fe69583421fd506a4fac53 WatchSource:0}: Error finding container 4b51f7c5d9be7ae56728fd8a50a88f646269542793fe69583421fd506a4fac53: Status 404 returned error can't find the container with id 4b51f7c5d9be7ae56728fd8a50a88f646269542793fe69583421fd506a4fac53
Oct 30 19:43:29.006788 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:43:29.006764    4892 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b16664de-zhmcs"]
Oct 30 19:43:29.059290 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:43:29.059265    4892 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-6af60237-mnfct"]
Oct 30 19:43:29.063905 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:43:29.063881    4892 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-6af60237-mnfct"]
Oct 30 19:43:29.538462 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:43:29.538431    4892 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b16664de-zhmcs" event={"ID":"545e8fd9-b409-4437-9ed0-c227942ca7da","Type":"ContainerStarted","Data":"100400e8af473dfd0c5ee8c32ba2e4d297751b0570f398347d1851d22ccae74b"}
Oct 30 19:43:29.538859 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:43:29.538482    4892 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b16664de-zhmcs" event={"ID":"545e8fd9-b409-4437-9ed0-c227942ca7da","Type":"ContainerStarted","Data":"4b51f7c5d9be7ae56728fd8a50a88f646269542793fe69583421fd506a4fac53"}
Oct 30 19:43:29.703584 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:43:29.703538    4892 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b16664de-zhmcs" podStartSLOduration=2.7035209 podStartE2EDuration="2.7035209s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:43:27 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:43:29.651843601 +0000 UTC m=+1657.222309205" watchObservedRunningTime="2026-10-30 19:43:29.7035209 +0000 UTC m=+1657.273986500"
Oct 30 19:43:30.554753 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:43:30.554723    4892 generic.go:356] "Generic (PLEG): container finished" podID="545e8fd9-b409-4437-9ed0-c227942ca7da" containerID="100400e8af473dfd0c5ee8c32ba2e4d297751b0570f398347d1851d22ccae74b" exitCode=0
Oct 30 19:43:30.555219 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:43:30.554774    4892 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b16664de-zhmcs" event={"ID":"545e8fd9-b409-4437-9ed0-c227942ca7da","Type":"ContainerDied","Data":"100400e8af473dfd0c5ee8c32ba2e4d297751b0570f398347d1851d22ccae74b"}
Oct 30 19:43:30.571527 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:43:30.571496    4892 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f0f6128d-e2d6-4156-b261-c362b64ca062" path="/var/lib/kubelet/pods/f0f6128d-e2d6-4156-b261-c362b64ca062/volumes"
Oct 30 19:43:31.050623 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:43:31.050587    4892 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b19201f6-kv7xb"]
Oct 30 19:43:31.055332 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:43:31.055307    4892 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b19201f6-kv7xb"]
Oct 30 19:43:31.722785 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:43:31.722759    4892 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b16664de-zhmcs"
Oct 30 19:43:31.907955 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:43:31.907922    4892 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/545e8fd9-b409-4437-9ed0-c227942ca7da-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/545e8fd9-b409-4437-9ed0-c227942ca7da-maas-api-openshift-ca\") pod \"545e8fd9-b409-4437-9ed0-c227942ca7da\" (UID: \"545e8fd9-b409-4437-9ed0-c227942ca7da\") "
Oct 30 19:43:31.908143 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:43:31.908067    4892 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/545e8fd9-b409-4437-9ed0-c227942ca7da-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/545e8fd9-b409-4437-9ed0-c227942ca7da-maas-api-xks-ca\") pod \"545e8fd9-b409-4437-9ed0-c227942ca7da\" (UID: \"545e8fd9-b409-4437-9ed0-c227942ca7da\") "
Oct 30 19:43:31.908143 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:43:31.908111    4892 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/545e8fd9-b409-4437-9ed0-c227942ca7da-kube-api-access-w8t5k\" (UniqueName: \"kubernetes.io/projected/545e8fd9-b409-4437-9ed0-c227942ca7da-kube-api-access-w8t5k\") pod \"545e8fd9-b409-4437-9ed0-c227942ca7da\" (UID: \"545e8fd9-b409-4437-9ed0-c227942ca7da\") "
Oct 30 19:43:31.908274 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:43:31.908252    4892 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/545e8fd9-b409-4437-9ed0-c227942ca7da-maas-api-openshift-ca" pod "545e8fd9-b409-4437-9ed0-c227942ca7da" (UID: "545e8fd9-b409-4437-9ed0-c227942ca7da"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 19:43:31.908594 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:43:31.908571    4892 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/545e8fd9-b409-4437-9ed0-c227942ca7da-maas-api-openshift-ca\") on node \"ip-10-0-132-82.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 19:43:31.910383 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:43:31.910362    4892 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/545e8fd9-b409-4437-9ed0-c227942ca7da-maas-api-xks-ca" pod "545e8fd9-b409-4437-9ed0-c227942ca7da" (UID: "545e8fd9-b409-4437-9ed0-c227942ca7da"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 19:43:31.910822 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:43:31.910805    4892 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/545e8fd9-b409-4437-9ed0-c227942ca7da-kube-api-access-w8t5k" pod "545e8fd9-b409-4437-9ed0-c227942ca7da" (UID: "545e8fd9-b409-4437-9ed0-c227942ca7da"). InnerVolumeSpecName "kube-api-access-w8t5k". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:43:32.009731 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:43:32.009701    4892 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/545e8fd9-b409-4437-9ed0-c227942ca7da-maas-api-xks-ca\") on node \"ip-10-0-132-82.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 19:43:32.009731 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:43:32.009727    4892 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-w8t5k\" (UniqueName: \"kubernetes.io/projected/545e8fd9-b409-4437-9ed0-c227942ca7da-kube-api-access-w8t5k\") on node \"ip-10-0-132-82.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 19:43:32.300855 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:43:32.300780    4892 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-9837c7af-m8n4b"]
Oct 30 19:43:32.301371 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:43:32.301356    4892 cpu_manager.go:414] "containerMap: removing container" podUID="545e8fd9-b409-4437-9ed0-c227942ca7da" containerName="revoke-keys"
Oct 30 19:43:32.301428 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:43:32.301372    4892 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="545e8fd9-b409-4437-9ed0-c227942ca7da" containerName="revoke-keys"
Oct 30 19:43:32.301496 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:43:32.301487    4892 memory_manager.go:365] "RemoveStaleState removing state" podUID="545e8fd9-b409-4437-9ed0-c227942ca7da" containerName="revoke-keys"
Oct 30 19:43:32.305009 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:43:32.304987    4892 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9837c7af-m8n4b"
Oct 30 19:43:32.365006 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:43:32.364025    4892 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-9837c7af-m8n4b"]
Oct 30 19:43:32.415065 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:43:32.415032    4892 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ead14343-b5b5-4c17-94af-9790ed4c8c08-maas-api-xks-ca\") pod \"maas-api-revoke-sub-9837c7af-m8n4b\" (UID: \"ead14343-b5b5-4c17-94af-9790ed4c8c08\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9837c7af-m8n4b"
Oct 30 19:43:32.415203 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:43:32.415184    4892 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tqz9c\" (UniqueName: \"kubernetes.io/projected/ead14343-b5b5-4c17-94af-9790ed4c8c08-kube-api-access-tqz9c\") pod \"maas-api-revoke-sub-9837c7af-m8n4b\" (UID: \"ead14343-b5b5-4c17-94af-9790ed4c8c08\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9837c7af-m8n4b"
Oct 30 19:43:32.415330 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:43:32.415313    4892 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ead14343-b5b5-4c17-94af-9790ed4c8c08-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-9837c7af-m8n4b\" (UID: \"ead14343-b5b5-4c17-94af-9790ed4c8c08\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9837c7af-m8n4b"
Oct 30 19:43:32.516077 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:43:32.516042    4892 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ead14343-b5b5-4c17-94af-9790ed4c8c08-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-9837c7af-m8n4b\" (UID: \"ead14343-b5b5-4c17-94af-9790ed4c8c08\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9837c7af-m8n4b"
Oct 30 19:43:32.516263 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:43:32.516089    4892 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ead14343-b5b5-4c17-94af-9790ed4c8c08-maas-api-xks-ca\") pod \"maas-api-revoke-sub-9837c7af-m8n4b\" (UID: \"ead14343-b5b5-4c17-94af-9790ed4c8c08\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9837c7af-m8n4b"
Oct 30 19:43:32.516263 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:43:32.516201    4892 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tqz9c\" (UniqueName: \"kubernetes.io/projected/ead14343-b5b5-4c17-94af-9790ed4c8c08-kube-api-access-tqz9c\") pod \"maas-api-revoke-sub-9837c7af-m8n4b\" (UID: \"ead14343-b5b5-4c17-94af-9790ed4c8c08\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9837c7af-m8n4b"
Oct 30 19:43:32.516709 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:43:32.516670    4892 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ead14343-b5b5-4c17-94af-9790ed4c8c08-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-9837c7af-m8n4b\" (UID: \"ead14343-b5b5-4c17-94af-9790ed4c8c08\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9837c7af-m8n4b"
Oct 30 19:43:32.519184 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:43:32.519158    4892 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ead14343-b5b5-4c17-94af-9790ed4c8c08-maas-api-xks-ca\") pod \"maas-api-revoke-sub-9837c7af-m8n4b\" (UID: \"ead14343-b5b5-4c17-94af-9790ed4c8c08\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9837c7af-m8n4b"
Oct 30 19:43:32.572427 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:43:32.572348    4892 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="aa084871-b2e9-4cb2-bae0-f4add975912b" path="/var/lib/kubelet/pods/aa084871-b2e9-4cb2-bae0-f4add975912b/volumes"
Oct 30 19:43:32.574149 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:43:32.574119    4892 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tqz9c\" (UniqueName: \"kubernetes.io/projected/ead14343-b5b5-4c17-94af-9790ed4c8c08-kube-api-access-tqz9c\") pod \"maas-api-revoke-sub-9837c7af-m8n4b\" (UID: \"ead14343-b5b5-4c17-94af-9790ed4c8c08\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9837c7af-m8n4b"
Oct 30 19:43:32.577857 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:43:32.577829    4892 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b16664de-zhmcs" event={"ID":"545e8fd9-b409-4437-9ed0-c227942ca7da","Type":"ContainerDied","Data":"4b51f7c5d9be7ae56728fd8a50a88f646269542793fe69583421fd506a4fac53"}
Oct 30 19:43:32.577999 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:43:32.577865    4892 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b16664de-zhmcs"
Oct 30 19:43:32.577999 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:43:32.577881    4892 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="4b51f7c5d9be7ae56728fd8a50a88f646269542793fe69583421fd506a4fac53"
Oct 30 19:43:32.620309 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:43:32.620290    4892 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9837c7af-m8n4b"
Oct 30 19:43:33.005451 ip-10-0-132-82 kubenswrapper[4892]: W1030 19:43:33.005418    4892 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podead14343_b5b5_4c17_94af_9790ed4c8c08.slice/crio-2824de1eb7eedeb459bd8765b32a8edf58d47cd45f1cd277cc8afc055c04fa55 WatchSource:0}: Error finding container 2824de1eb7eedeb459bd8765b32a8edf58d47cd45f1cd277cc8afc055c04fa55: Status 404 returned error can't find the container with id 2824de1eb7eedeb459bd8765b32a8edf58d47cd45f1cd277cc8afc055c04fa55
Oct 30 19:43:33.006579 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:43:33.006555    4892 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-9837c7af-m8n4b"]
Oct 30 19:43:33.587388 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:43:33.587355    4892 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9837c7af-m8n4b" event={"ID":"ead14343-b5b5-4c17-94af-9790ed4c8c08","Type":"ContainerStarted","Data":"52bce220e3c1d35a73bed99b72853ee002edf1bcb14ee03dbc00b8f4f34a24b0"}
Oct 30 19:43:33.587583 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:43:33.587412    4892 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9837c7af-m8n4b" event={"ID":"ead14343-b5b5-4c17-94af-9790ed4c8c08","Type":"ContainerStarted","Data":"2824de1eb7eedeb459bd8765b32a8edf58d47cd45f1cd277cc8afc055c04fa55"}
Oct 30 19:43:33.704004 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:43:33.703939    4892 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9837c7af-m8n4b" podStartSLOduration=1.703924693 podStartE2EDuration="1.703924693s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:43:32 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:43:33.699293451 +0000 UTC m=+1661.269759050" watchObservedRunningTime="2026-10-30 19:43:33.703924693 +0000 UTC m=+1661.274390321"
Oct 30 19:43:34.596516 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:43:34.596482    4892 generic.go:356] "Generic (PLEG): container finished" podID="ead14343-b5b5-4c17-94af-9790ed4c8c08" containerID="52bce220e3c1d35a73bed99b72853ee002edf1bcb14ee03dbc00b8f4f34a24b0" exitCode=0
Oct 30 19:43:34.596894 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:43:34.596564    4892 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9837c7af-m8n4b" event={"ID":"ead14343-b5b5-4c17-94af-9790ed4c8c08","Type":"ContainerDied","Data":"52bce220e3c1d35a73bed99b72853ee002edf1bcb14ee03dbc00b8f4f34a24b0"}
Oct 30 19:43:35.766149 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:43:35.766120    4892 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9837c7af-m8n4b"
Oct 30 19:43:35.865245 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:43:35.865153    4892 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ead14343-b5b5-4c17-94af-9790ed4c8c08-kube-api-access-tqz9c\" (UniqueName: \"kubernetes.io/projected/ead14343-b5b5-4c17-94af-9790ed4c8c08-kube-api-access-tqz9c\") pod \"ead14343-b5b5-4c17-94af-9790ed4c8c08\" (UID: \"ead14343-b5b5-4c17-94af-9790ed4c8c08\") "
Oct 30 19:43:35.865404 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:43:35.865304    4892 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ead14343-b5b5-4c17-94af-9790ed4c8c08-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ead14343-b5b5-4c17-94af-9790ed4c8c08-maas-api-openshift-ca\") pod \"ead14343-b5b5-4c17-94af-9790ed4c8c08\" (UID: \"ead14343-b5b5-4c17-94af-9790ed4c8c08\") "
Oct 30 19:43:35.865508 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:43:35.865488    4892 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/ead14343-b5b5-4c17-94af-9790ed4c8c08-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ead14343-b5b5-4c17-94af-9790ed4c8c08-maas-api-xks-ca\") pod \"ead14343-b5b5-4c17-94af-9790ed4c8c08\" (UID: \"ead14343-b5b5-4c17-94af-9790ed4c8c08\") "
Oct 30 19:43:35.867957 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:43:35.867933    4892 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ead14343-b5b5-4c17-94af-9790ed4c8c08-maas-api-openshift-ca" pod "ead14343-b5b5-4c17-94af-9790ed4c8c08" (UID: "ead14343-b5b5-4c17-94af-9790ed4c8c08"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 19:43:35.868955 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:43:35.868720    4892 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ead14343-b5b5-4c17-94af-9790ed4c8c08-maas-api-xks-ca" pod "ead14343-b5b5-4c17-94af-9790ed4c8c08" (UID: "ead14343-b5b5-4c17-94af-9790ed4c8c08"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 19:43:35.876340 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:43:35.876293    4892 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ead14343-b5b5-4c17-94af-9790ed4c8c08-kube-api-access-tqz9c" pod "ead14343-b5b5-4c17-94af-9790ed4c8c08" (UID: "ead14343-b5b5-4c17-94af-9790ed4c8c08"). InnerVolumeSpecName "kube-api-access-tqz9c". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:43:35.967426 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:43:35.967383    4892 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ead14343-b5b5-4c17-94af-9790ed4c8c08-maas-api-openshift-ca\") on node \"ip-10-0-132-82.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 19:43:35.967426 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:43:35.967425    4892 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ead14343-b5b5-4c17-94af-9790ed4c8c08-maas-api-xks-ca\") on node \"ip-10-0-132-82.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 19:43:35.967625 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:43:35.967440    4892 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tqz9c\" (UniqueName: \"kubernetes.io/projected/ead14343-b5b5-4c17-94af-9790ed4c8c08-kube-api-access-tqz9c\") on node \"ip-10-0-132-82.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 19:43:36.614328 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:43:36.614296    4892 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9837c7af-m8n4b"
Oct 30 19:43:36.614477 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:43:36.614332    4892 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9837c7af-m8n4b" event={"ID":"ead14343-b5b5-4c17-94af-9790ed4c8c08","Type":"ContainerDied","Data":"2824de1eb7eedeb459bd8765b32a8edf58d47cd45f1cd277cc8afc055c04fa55"}
Oct 30 19:43:36.614477 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:43:36.614387    4892 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="2824de1eb7eedeb459bd8765b32a8edf58d47cd45f1cd277cc8afc055c04fa55"
Oct 30 19:43:37.857989 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:43:37.857621    4892 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-8964-629d1b"]
Oct 30 19:43:37.857989 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:43:37.857831    4892 kuberuntime_container.go:898] "Killing container with a grace period" pod="opendatahub/test-curl-8964-629d1b" podUID="93d8a0c9-c95b-481c-9700-5609fe407cdc" containerName="test-curl-8964-629d1b" containerID="cri-o://f24b6e6de28027a2919befad6d44a2c23d1c6194139d51b0222d6cdceed2d903" gracePeriod=2
Oct 30 19:43:37.878395 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:43:37.865613    4892 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-8964-629d1b"]
Oct 30 19:43:37.910819 ip-10-0-132-82 kubenswrapper[4892]: E1030 19:43:37.910783    4892 status_manager.go:1045] "Failed to get status for pod" err="pods \"test-curl-8964-629d1b\" is forbidden: User \"system:node:ip-10-0-132-82.us-west-1.compute.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-132-82.us-west-1.compute.internal' and this object" podUID="93d8a0c9-c95b-481c-9700-5609fe407cdc" pod="opendatahub/test-curl-8964-629d1b"
Oct 30 19:43:38.020763 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:43:38.020733    4892 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-8964-de9af3"]
Oct 30 19:43:38.021277 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:43:38.021264    4892 cpu_manager.go:414] "containerMap: removing container" podUID="ead14343-b5b5-4c17-94af-9790ed4c8c08" containerName="revoke-keys"
Oct 30 19:43:38.021337 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:43:38.021278    4892 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ead14343-b5b5-4c17-94af-9790ed4c8c08" containerName="revoke-keys"
Oct 30 19:43:38.021337 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:43:38.021307    4892 cpu_manager.go:414] "containerMap: removing container" podUID="93d8a0c9-c95b-481c-9700-5609fe407cdc" containerName="test-curl-8964-629d1b"
Oct 30 19:43:38.021337 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:43:38.021311    4892 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="93d8a0c9-c95b-481c-9700-5609fe407cdc" containerName="test-curl-8964-629d1b"
Oct 30 19:43:38.021433 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:43:38.021411    4892 memory_manager.go:365] "RemoveStaleState removing state" podUID="93d8a0c9-c95b-481c-9700-5609fe407cdc" containerName="test-curl-8964-629d1b"
Oct 30 19:43:38.021433 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:43:38.021423    4892 memory_manager.go:365] "RemoveStaleState removing state" podUID="ead14343-b5b5-4c17-94af-9790ed4c8c08" containerName="revoke-keys"
Oct 30 19:43:38.025123 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:43:38.025107    4892 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-8964-de9af3"
Oct 30 19:43:38.079661 ip-10-0-132-82 kubenswrapper[4892]: E1030 19:43:38.079624    4892 status_manager.go:1045] "Failed to get status for pod" err="pods \"test-curl-8964-629d1b\" is forbidden: User \"system:node:ip-10-0-132-82.us-west-1.compute.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-132-82.us-west-1.compute.internal' and this object" podUID="93d8a0c9-c95b-481c-9700-5609fe407cdc" pod="opendatahub/test-curl-8964-629d1b"
Oct 30 19:43:38.090740 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:43:38.090715    4892 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-8964-de9af3"]
Oct 30 19:43:38.093135 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:43:38.093108    4892 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-l5497\" (UniqueName: \"kubernetes.io/projected/1163665e-2ccb-4b87-8fab-abf927cbc704-kube-api-access-l5497\") pod \"test-curl-8964-de9af3\" (UID: \"1163665e-2ccb-4b87-8fab-abf927cbc704\") " pod="opendatahub/test-curl-8964-de9af3"
Oct 30 19:43:38.194756 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:43:38.194728    4892 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-l5497\" (UniqueName: \"kubernetes.io/projected/1163665e-2ccb-4b87-8fab-abf927cbc704-kube-api-access-l5497\") pod \"test-curl-8964-de9af3\" (UID: \"1163665e-2ccb-4b87-8fab-abf927cbc704\") " pod="opendatahub/test-curl-8964-de9af3"
Oct 30 19:43:38.252091 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:43:38.252058    4892 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-l5497\" (UniqueName: \"kubernetes.io/projected/1163665e-2ccb-4b87-8fab-abf927cbc704-kube-api-access-l5497\") pod \"test-curl-8964-de9af3\" (UID: \"1163665e-2ccb-4b87-8fab-abf927cbc704\") " pod="opendatahub/test-curl-8964-de9af3"
Oct 30 19:43:38.338142 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:43:38.338111    4892 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-8964-de9af3"
Oct 30 19:43:38.719409 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:43:38.719374    4892 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-8964-de9af3"]
Oct 30 19:43:39.642659 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:43:39.642615    4892 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-8964-de9af3" event={"ID":"1163665e-2ccb-4b87-8fab-abf927cbc704","Type":"ContainerStarted","Data":"3da9443879d1ac7cfb75cf4beb052464a02111fb085c7ad37774ea9e10f70716"}
Oct 30 19:43:39.643193 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:43:39.642671    4892 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-8964-de9af3" event={"ID":"1163665e-2ccb-4b87-8fab-abf927cbc704","Type":"ContainerStarted","Data":"721758bb9a0fece5cd5955e834d64fdd193c9ccf175720bae9b55d8f08b6c729"}
Oct 30 19:43:39.762010 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:43:39.761943    4892 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/test-curl-8964-de9af3" podStartSLOduration=2.761930795 podStartE2EDuration="2.761930795s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:43:37 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:43:39.756999984 +0000 UTC m=+1667.327465582" watchObservedRunningTime="2026-10-30 19:43:39.761930795 +0000 UTC m=+1667.332396394"
Oct 30 19:43:40.169806 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:43:40.169777    4892 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-8964-629d1b"
Oct 30 19:43:40.217557 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:43:40.217484    4892 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/93d8a0c9-c95b-481c-9700-5609fe407cdc-kube-api-access-nmtfs\" (UniqueName: \"kubernetes.io/projected/93d8a0c9-c95b-481c-9700-5609fe407cdc-kube-api-access-nmtfs\") pod \"93d8a0c9-c95b-481c-9700-5609fe407cdc\" (UID: \"93d8a0c9-c95b-481c-9700-5609fe407cdc\") "
Oct 30 19:43:40.219660 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:43:40.219636    4892 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/93d8a0c9-c95b-481c-9700-5609fe407cdc-kube-api-access-nmtfs" pod "93d8a0c9-c95b-481c-9700-5609fe407cdc" (UID: "93d8a0c9-c95b-481c-9700-5609fe407cdc"). InnerVolumeSpecName "kube-api-access-nmtfs". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:43:40.341993 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:43:40.339925    4892 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nmtfs\" (UniqueName: \"kubernetes.io/projected/93d8a0c9-c95b-481c-9700-5609fe407cdc-kube-api-access-nmtfs\") on node \"ip-10-0-132-82.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 19:43:40.613039 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:43:40.605761    4892 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="93d8a0c9-c95b-481c-9700-5609fe407cdc" path="/var/lib/kubelet/pods/93d8a0c9-c95b-481c-9700-5609fe407cdc/volumes"
Oct 30 19:43:40.719105 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:43:40.705716    4892 generic.go:356] "Generic (PLEG): container finished" podID="93d8a0c9-c95b-481c-9700-5609fe407cdc" containerID="f24b6e6de28027a2919befad6d44a2c23d1c6194139d51b0222d6cdceed2d903" exitCode=137
Oct 30 19:43:40.722999 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:43:40.719748    4892 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-8964-629d1b"
Oct 30 19:43:40.722999 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:43:40.720007    4892 scope.go:122] "RemoveContainer" containerID="f24b6e6de28027a2919befad6d44a2c23d1c6194139d51b0222d6cdceed2d903"
Oct 30 19:43:40.843054 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:43:40.842760    4892 scope.go:122] "RemoveContainer" containerID="f24b6e6de28027a2919befad6d44a2c23d1c6194139d51b0222d6cdceed2d903"
Oct 30 19:43:40.843370 ip-10-0-132-82 kubenswrapper[4892]: E1030 19:43:40.843277    4892 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f24b6e6de28027a2919befad6d44a2c23d1c6194139d51b0222d6cdceed2d903\": container with ID starting with f24b6e6de28027a2919befad6d44a2c23d1c6194139d51b0222d6cdceed2d903 not found: ID does not exist" containerID="f24b6e6de28027a2919befad6d44a2c23d1c6194139d51b0222d6cdceed2d903"
Oct 30 19:43:40.843370 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:43:40.843316    4892 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f24b6e6de28027a2919befad6d44a2c23d1c6194139d51b0222d6cdceed2d903"} err="failed to get container status \"f24b6e6de28027a2919befad6d44a2c23d1c6194139d51b0222d6cdceed2d903\": rpc error: code = NotFound desc = could not find container \"f24b6e6de28027a2919befad6d44a2c23d1c6194139d51b0222d6cdceed2d903\": container with ID starting with f24b6e6de28027a2919befad6d44a2c23d1c6194139d51b0222d6cdceed2d903 not found: ID does not exist"
Oct 30 19:43:41.079986 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:43:41.079379    4892 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-296bffe5-8sgf4"]
Oct 30 19:43:41.128128 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:43:41.122994    4892 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-296bffe5-8sgf4"]
Oct 30 19:43:41.835986 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:43:41.834102    4892 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-a3f9b7/simulator-w2-a3f9b7-kserve-79bbc4cb5f-8m488" podUID="50fb2283-bf3f-41da-835b-f407ee1a04f3" containerName="main" probeResult="failure" output="Get \"https://10.132.0.59:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:43:41.868990 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:43:41.865059    4892 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-d52df0/simulator-w1-d52df0-kserve-67f74bc577-q5bjz" podUID="81b7bb84-1003-4452-b7e0-b4aaba6fefa4" containerName="main" probeResult="failure" output="Get \"https://10.132.0.57:8000/ready\": context deadline exceeded"
Oct 30 19:43:41.868990 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:43:41.865932    4892 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-2d9861/premium-w0-2d9861-kserve-67945c6848-4n6kz" podUID="798988b2-c769-4f73-a7c8-ca15de0beef1" containerName="main" probeResult="failure" output="Get \"https://10.132.0.62:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:43:42.291990 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:43:42.291010    4892 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-ada8d7/simulator-w3-ada8d7-kserve-7d575cfcb5-5txgn" podUID="10a5c7c8-6146-47b0-9643-bb137cf10bf5" containerName="main" probeResult="failure" output="Get \"https://10.132.0.55:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:43:42.519990 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:43:42.519061    4892 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-5f55cd/simulator-w6-5f55cd-kserve-5ff86b7447-h9tm7" podUID="3cb47a99-8be1-4756-b31a-43b58cc6b8dc" containerName="main" probeResult="failure" output="Get \"https://10.132.0.58:8000/ready\": context deadline exceeded"
Oct 30 19:43:42.527000 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:43:42.524993    4892 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-d52df0/distinct-2-w1-d52df0-kserve-7846558464-xrkdc" podUID="9c614ab6-30e2-4fe1-8609-d5e2dadc198a" containerName="main" probeResult="failure" output="Get \"https://10.132.0.83:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:43:42.627040 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:43:42.626087    4892 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="aee774c4-0eaf-4fa5-bec7-5b371c039f4a" path="/var/lib/kubelet/pods/aee774c4-0eaf-4fa5-bec7-5b371c039f4a/volumes"
Oct 30 19:43:42.837996 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:43:42.837046    4892 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-a3f9b7/simulator-w2-a3f9b7-kserve-79bbc4cb5f-8m488" podUID="50fb2283-bf3f-41da-835b-f407ee1a04f3" containerName="main" probeResult="failure" output="Get \"https://10.132.0.59:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:43:42.837996 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:43:42.837455    4892 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-a3f9b7/simulator-w2-a3f9b7-kserve-79bbc4cb5f-8m488"
Oct 30 19:43:42.869747 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:43:42.868388    4892 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-2d9861/premium-w0-2d9861-kserve-67945c6848-4n6kz" podUID="798988b2-c769-4f73-a7c8-ca15de0beef1" containerName="main" probeResult="failure" output="Get \"https://10.132.0.62:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:43:42.869747 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:43:42.868901    4892 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-2d9861/premium-w0-2d9861-kserve-67945c6848-4n6kz"
Oct 30 19:43:42.885046 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:43:42.870903    4892 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-d52df0/simulator-w1-d52df0-kserve-67f74bc577-q5bjz" podUID="81b7bb84-1003-4452-b7e0-b4aaba6fefa4" containerName="main" probeResult="failure" output="Get \"https://10.132.0.57:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:43:42.885046 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:43:42.871504    4892 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-d52df0/simulator-w1-d52df0-kserve-67f74bc577-q5bjz"
Oct 30 19:43:43.219068 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:43:43.094009    4892 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-5f55cd/premium-w6-5f55cd-kserve-7f785c544b-pcvhb" podUID="191593a2-e3a9-4ce4-a1c6-c089723b0237" containerName="main" probeResult="failure" output="Get \"https://10.132.0.60:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:43:43.226993 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:43:43.220240    4892 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-686689/distinct-2-w5-686689-kserve-5ff578b4cd-rkc6s" podUID="a8829732-ac6f-4238-b8ce-fc781084668b" containerName="main" probeResult="failure" output="Get \"https://10.132.0.87:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:43:43.303989 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:43:43.302724    4892 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-b5178f/distinct-2-w4-b5178f-kserve-69ddf8b665-9pdth" podUID="fb71d961-8500-4367-b48c-88d385962bd5" containerName="main" probeResult="failure" output="Get \"https://10.132.0.90:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:43:43.329990 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:43:43.304280    4892 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-ada8d7/simulator-w3-ada8d7-kserve-7d575cfcb5-5txgn" podUID="10a5c7c8-6146-47b0-9643-bb137cf10bf5" containerName="main" probeResult="failure" output="Get \"https://10.132.0.55:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:43:43.346997 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:43:43.339141    4892 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-ada8d7/simulator-w3-ada8d7-kserve-7d575cfcb5-5txgn"
Oct 30 19:43:43.544158 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:43:43.533238    4892 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-d52df0/distinct-2-w1-d52df0-kserve-7846558464-xrkdc" podUID="9c614ab6-30e2-4fe1-8609-d5e2dadc198a" containerName="main" probeResult="failure" output="Get \"https://10.132.0.83:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:43:43.544158 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:43:43.533652    4892 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-d52df0/distinct-2-w1-d52df0-kserve-7846558464-xrkdc"
Oct 30 19:43:43.883043 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:43:43.878461    4892 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-d52df0/simulator-w1-d52df0-kserve-67f74bc577-q5bjz" podUID="81b7bb84-1003-4452-b7e0-b4aaba6fefa4" containerName="main" probeResult="failure" output="Get \"https://10.132.0.57:8000/ready\": context deadline exceeded"
Oct 30 19:43:43.883043 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:43:43.878155    4892 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-2d9861/premium-w0-2d9861-kserve-67945c6848-4n6kz" podUID="798988b2-c769-4f73-a7c8-ca15de0beef1" containerName="main" probeResult="failure" output="Get \"https://10.132.0.62:8000/ready\": context deadline exceeded"
Oct 30 19:43:44.106088 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:43:43.897715    4892 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-a3f9b7/simulator-w2-a3f9b7-kserve-79bbc4cb5f-8m488" podUID="50fb2283-bf3f-41da-835b-f407ee1a04f3" containerName="main" probeResult="failure" output="Get \"https://10.132.0.59:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:43:44.106088 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:43:44.067772    4892 patch_prober.go:29] interesting pod/e2e-worker-w5-686689-openshift-default-5fbf5787bf-jrt7h container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.80:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 30 19:43:44.106088 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:43:44.067863    4892 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w5-686689-openshift-default-5fbf5787bf-jrt7h" podUID="fa668020-72be-46a7-aa53-e8249130c4f1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.80:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:43:44.106088 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:43:44.068005    4892 patch_prober.go:29] interesting pod/e2e-worker-w1-d52df0-openshift-default-6d5f887b45-985gh container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.64:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 30 19:43:44.106088 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:43:44.068032    4892 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w1-d52df0-openshift-default-6d5f887b45-985gh" podUID="e453311b-d7b1-4819-a57a-d1de4886de26" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.64:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:43:44.113185 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:43:44.110818    4892 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-5f55cd/premium-w6-5f55cd-kserve-7f785c544b-pcvhb" podUID="191593a2-e3a9-4ce4-a1c6-c089723b0237" containerName="main" probeResult="failure" output="Get \"https://10.132.0.60:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:43:44.113185 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:43:44.112146    4892 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w6-5f55cd/premium-w6-5f55cd-kserve-7f785c544b-pcvhb"
Oct 30 19:43:44.238951 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:43:44.229053    4892 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-686689/distinct-2-w5-686689-kserve-5ff578b4cd-rkc6s" podUID="a8829732-ac6f-4238-b8ce-fc781084668b" containerName="main" probeResult="failure" output="Get \"https://10.132.0.87:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:43:44.238951 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:43:44.230198    4892 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w5-686689/distinct-2-w5-686689-kserve-5ff578b4cd-rkc6s"
Oct 30 19:43:44.312991 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:43:44.312011    4892 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-b5178f/distinct-2-w4-b5178f-kserve-69ddf8b665-9pdth" podUID="fb71d961-8500-4367-b48c-88d385962bd5" containerName="main" probeResult="failure" output="Get \"https://10.132.0.90:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:43:44.316990 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:43:44.313270    4892 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-b5178f/distinct-2-w4-b5178f-kserve-69ddf8b665-9pdth"
Oct 30 19:43:44.333473 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:43:44.333429    4892 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-ada8d7/simulator-w3-ada8d7-kserve-7d575cfcb5-5txgn" podUID="10a5c7c8-6146-47b0-9643-bb137cf10bf5" containerName="main" probeResult="failure" output="Get \"https://10.132.0.55:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:43:44.405246 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:43:44.405216    4892 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-ada8d7/simulator-w3-ada8d7-kserve-7d575cfcb5-5txgn"
Oct 30 19:43:44.450354 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:43:44.436126    4892 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a429595a-fnjb5"]
Oct 30 19:43:44.483506 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:43:44.483473    4892 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a429595a-fnjb5"
Oct 30 19:43:44.535291 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:43:44.535189    4892 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-d52df0/distinct-2-w1-d52df0-kserve-7846558464-xrkdc" podUID="9c614ab6-30e2-4fe1-8609-d5e2dadc198a" containerName="main" probeResult="failure" output="Get \"https://10.132.0.83:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:43:44.554902 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:43:44.554873    4892 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-nlxp4\""
Oct 30 19:43:44.555228 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:43:44.555208    4892 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 19:43:44.566814 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:43:44.566786    4892 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 30 19:43:44.661120 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:43:44.661079    4892 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/65aca2dc-d914-46c8-9b84-20a864873a1d-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-a429595a-fnjb5\" (UID: \"65aca2dc-d914-46c8-9b84-20a864873a1d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a429595a-fnjb5"
Oct 30 19:43:44.678998 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:43:44.672086    4892 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/65aca2dc-d914-46c8-9b84-20a864873a1d-maas-api-xks-ca\") pod \"maas-api-revoke-sub-a429595a-fnjb5\" (UID: \"65aca2dc-d914-46c8-9b84-20a864873a1d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a429595a-fnjb5"
Oct 30 19:43:44.687014 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:43:44.675852    4892 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 30 19:43:44.723570 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:43:44.723432    4892 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6t4sm\" (UniqueName: \"kubernetes.io/projected/65aca2dc-d914-46c8-9b84-20a864873a1d-kube-api-access-6t4sm\") pod \"maas-api-revoke-sub-a429595a-fnjb5\" (UID: \"65aca2dc-d914-46c8-9b84-20a864873a1d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a429595a-fnjb5"
Oct 30 19:43:44.815239 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:43:44.815156    4892 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w5-686689/distinct-2-w5-686689-kserve-5ff578b4cd-rkc6s"
Oct 30 19:43:44.910807 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:43:44.907250    4892 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-d52df0/simulator-w1-d52df0-kserve-67f74bc577-q5bjz" podUID="81b7bb84-1003-4452-b7e0-b4aaba6fefa4" containerName="main" probeResult="failure" output="Get \"https://10.132.0.57:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:43:44.975998 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:43:44.913919    4892 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/65aca2dc-d914-46c8-9b84-20a864873a1d-maas-api-xks-ca\") pod \"maas-api-revoke-sub-a429595a-fnjb5\" (UID: \"65aca2dc-d914-46c8-9b84-20a864873a1d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a429595a-fnjb5"
Oct 30 19:43:44.975998 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:43:44.919446    4892 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-a3f9b7/simulator-w2-a3f9b7-kserve-79bbc4cb5f-8m488" podUID="50fb2283-bf3f-41da-835b-f407ee1a04f3" containerName="main" probeResult="failure" output="Get \"https://10.132.0.59:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:43:44.975998 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:43:44.935738    4892 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-2d9861/premium-w0-2d9861-kserve-67945c6848-4n6kz" podUID="798988b2-c769-4f73-a7c8-ca15de0beef1" containerName="main" probeResult="failure" output="Get \"https://10.132.0.62:8000/ready\": context deadline exceeded"
Oct 30 19:43:45.007000 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:43:44.980080    4892 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6t4sm\" (UniqueName: \"kubernetes.io/projected/65aca2dc-d914-46c8-9b84-20a864873a1d-kube-api-access-6t4sm\") pod \"maas-api-revoke-sub-a429595a-fnjb5\" (UID: \"65aca2dc-d914-46c8-9b84-20a864873a1d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a429595a-fnjb5"
Oct 30 19:43:45.007000 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:43:44.980174    4892 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/65aca2dc-d914-46c8-9b84-20a864873a1d-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-a429595a-fnjb5\" (UID: \"65aca2dc-d914-46c8-9b84-20a864873a1d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a429595a-fnjb5"
Oct 30 19:43:45.014991 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:43:45.007832    4892 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/65aca2dc-d914-46c8-9b84-20a864873a1d-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-a429595a-fnjb5\" (UID: \"65aca2dc-d914-46c8-9b84-20a864873a1d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a429595a-fnjb5"
Oct 30 19:43:45.019026 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:43:45.015624    4892 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/65aca2dc-d914-46c8-9b84-20a864873a1d-maas-api-xks-ca\") pod \"maas-api-revoke-sub-a429595a-fnjb5\" (UID: \"65aca2dc-d914-46c8-9b84-20a864873a1d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a429595a-fnjb5"
Oct 30 19:43:45.100045 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:43:45.096736    4892 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6t4sm\" (UniqueName: \"kubernetes.io/projected/65aca2dc-d914-46c8-9b84-20a864873a1d-kube-api-access-6t4sm\") pod \"maas-api-revoke-sub-a429595a-fnjb5\" (UID: \"65aca2dc-d914-46c8-9b84-20a864873a1d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a429595a-fnjb5"
Oct 30 19:43:45.139021 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:43:45.136335    4892 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a429595a-fnjb5"]
Oct 30 19:43:45.139021 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:43:45.138855    4892 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-5f55cd/premium-w6-5f55cd-kserve-7f785c544b-pcvhb" podUID="191593a2-e3a9-4ce4-a1c6-c089723b0237" containerName="main" probeResult="failure" output="Get \"https://10.132.0.60:8000/ready\": context deadline exceeded"
Oct 30 19:43:45.215398 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:43:45.215368    4892 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a429595a-fnjb5"
Oct 30 19:43:45.380035 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:43:45.371912    4892 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-b5178f/distinct-2-w4-b5178f-kserve-69ddf8b665-9pdth" podUID="fb71d961-8500-4367-b48c-88d385962bd5" containerName="main" probeResult="failure" output="Get \"https://10.132.0.90:8000/ready\": context deadline exceeded"
Oct 30 19:43:45.380304 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:43:45.380279    4892 patch_prober.go:29] interesting pod/e2e-worker-w1-d52df0-openshift-default-6d5f887b45-2f4xc container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.88:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 30 19:43:45.380458 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:43:45.380433    4892 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w1-d52df0-openshift-default-6d5f887b45-2f4xc" podUID="6befc94b-40e5-42a4-adf5-392b5ecde123" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.88:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:43:45.535991 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:43:45.518073    4892 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-5f55cd/simulator-w6-5f55cd-kserve-5ff86b7447-h9tm7" podUID="3cb47a99-8be1-4756-b31a-43b58cc6b8dc" containerName="main" probeResult="failure" output="Get \"https://10.132.0.58:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:43:45.599988 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:43:45.564759    4892 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-d52df0/distinct-2-w1-d52df0-kserve-7846558464-xrkdc" podUID="9c614ab6-30e2-4fe1-8609-d5e2dadc198a" containerName="main" probeResult="failure" output="Get \"https://10.132.0.83:8000/ready\": context deadline exceeded"
Oct 30 19:43:45.817935 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:43:45.817889    4892 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-686689/distinct-2-w5-686689-kserve-5ff578b4cd-rkc6s" podUID="a8829732-ac6f-4238-b8ce-fc781084668b" containerName="main" probeResult="failure" output="Get \"https://10.132.0.87:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:43:45.935235 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:43:45.934264    4892 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-a3f9b7/simulator-w2-a3f9b7-kserve-79bbc4cb5f-8m488" podUID="50fb2283-bf3f-41da-835b-f407ee1a04f3" containerName="main" probeResult="failure" output="Get \"https://10.132.0.59:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:43:45.935235 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:43:45.934643    4892 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-d52df0/simulator-w1-d52df0-kserve-67f74bc577-q5bjz" podUID="81b7bb84-1003-4452-b7e0-b4aaba6fefa4" containerName="main" probeResult="failure" output="Get \"https://10.132.0.57:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:43:45.940771 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:43:45.940738    4892 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-2d9861/premium-w0-2d9861-kserve-67945c6848-4n6kz" podUID="798988b2-c769-4f73-a7c8-ca15de0beef1" containerName="main" probeResult="failure" output="Get \"https://10.132.0.62:8000/ready\": context deadline exceeded"
Oct 30 19:43:46.060689 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:43:46.060623    4892 patch_prober.go:29] interesting pod/e2e-worker-w5-686689-openshift-default-5fbf5787bf-7b547 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.114:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 30 19:43:46.061071 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:43:46.061009    4892 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w5-686689-openshift-default-5fbf5787bf-7b547" podUID="098876de-0b44-493b-a9da-45280afe2cdf" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.114:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:43:46.141240 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:43:46.141196    4892 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-5f55cd/premium-w6-5f55cd-kserve-7f785c544b-pcvhb" podUID="191593a2-e3a9-4ce4-a1c6-c089723b0237" containerName="main" probeResult="failure" output="Get \"https://10.132.0.60:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:43:46.320992 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:43:46.320870    4892 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e2375738-tbbvw"]
Oct 30 19:43:46.368992 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:43:46.365467    4892 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-ada8d7/simulator-w3-ada8d7-kserve-7d575cfcb5-5txgn" podUID="10a5c7c8-6146-47b0-9643-bb137cf10bf5" containerName="main" probeResult="failure" output="Get \"https://10.132.0.55:8000/ready\": context deadline exceeded"
Oct 30 19:43:46.405089 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:43:46.404667    4892 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-b5178f/distinct-2-w4-b5178f-kserve-69ddf8b665-9pdth" podUID="fb71d961-8500-4367-b48c-88d385962bd5" containerName="main" probeResult="failure" output="Get \"https://10.132.0.90:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:43:46.466992 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:43:46.441509    4892 patch_prober.go:29] interesting pod/e2e-worker-w1-d52df0-openshift-default-6d5f887b45-lxbtl container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.102:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 30 19:43:46.466992 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:43:46.459666    4892 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w1-d52df0-openshift-default-6d5f887b45-lxbtl" podUID="dc4774a0-3da8-4c65-a0b3-887a476ab465" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.102:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:43:46.469076 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:43:46.469042    4892 patch_prober.go:29] interesting pod/e2e-worker-w5-686689-openshift-default-5fbf5787bf-nsql4 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.120:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 30 19:43:46.469294 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:43:46.469269    4892 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w5-686689-openshift-default-5fbf5787bf-nsql4" podUID="dffc80d1-bd47-4493-8cd9-87530034d485" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.120:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:43:46.534488 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:43:46.534440    4892 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-5f55cd/simulator-w6-5f55cd-kserve-5ff86b7447-h9tm7" podUID="3cb47a99-8be1-4756-b31a-43b58cc6b8dc" containerName="main" probeResult="failure" output="Get \"https://10.132.0.58:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:43:46.535705 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:43:46.535681    4892 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w6-5f55cd/simulator-w6-5f55cd-kserve-5ff86b7447-h9tm7"
Oct 30 19:43:46.724075 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:43:46.706493    4892 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-d52df0/distinct-2-w1-d52df0-kserve-7846558464-xrkdc" podUID="9c614ab6-30e2-4fe1-8609-d5e2dadc198a" containerName="main" probeResult="failure" output="Get \"https://10.132.0.83:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:43:46.828356 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:43:46.828312    4892 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-686689/distinct-2-w5-686689-kserve-5ff578b4cd-rkc6s" podUID="a8829732-ac6f-4238-b8ce-fc781084668b" containerName="main" probeResult="failure" output="Get \"https://10.132.0.87:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:43:46.985037 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:43:46.939391    4892 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-d52df0/simulator-w1-d52df0-kserve-67f74bc577-q5bjz" podUID="81b7bb84-1003-4452-b7e0-b4aaba6fefa4" containerName="main" probeResult="failure" output="Get \"https://10.132.0.57:8000/ready\": context deadline exceeded"
Oct 30 19:43:46.985037 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:43:46.939695    4892 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-a3f9b7/simulator-w2-a3f9b7-kserve-79bbc4cb5f-8m488" podUID="50fb2283-bf3f-41da-835b-f407ee1a04f3" containerName="main" probeResult="failure" output="Get \"https://10.132.0.59:8000/ready\": context deadline exceeded"
Oct 30 19:43:46.985037 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:43:46.964485    4892 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-2d9861/premium-w0-2d9861-kserve-67945c6848-4n6kz" podUID="798988b2-c769-4f73-a7c8-ca15de0beef1" containerName="main" probeResult="failure" output="Get \"https://10.132.0.62:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:43:47.138877 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:43:47.138836    4892 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e2375738-tbbvw"]
Oct 30 19:43:47.139292 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:43:47.139267    4892 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w5-686689/distinct-2-w5-686689-kserve-5ff578b4cd-rkc6s"
Oct 30 19:43:47.207205 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:43:47.196080    4892 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-5f55cd/premium-w6-5f55cd-kserve-7f785c544b-pcvhb" podUID="191593a2-e3a9-4ce4-a1c6-c089723b0237" containerName="main" probeResult="failure" output="Get \"https://10.132.0.60:8000/ready\": context deadline exceeded"
Oct 30 19:43:47.489989 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:43:47.483118    4892 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-ada8d7/simulator-w3-ada8d7-kserve-7d575cfcb5-5txgn" podUID="10a5c7c8-6146-47b0-9643-bb137cf10bf5" containerName="main" probeResult="failure" output="Get \"https://10.132.0.55:8000/ready\": context deadline exceeded"
Oct 30 19:43:47.522217 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:43:47.522097    4892 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-b5178f/distinct-2-w4-b5178f-kserve-69ddf8b665-9pdth" podUID="fb71d961-8500-4367-b48c-88d385962bd5" containerName="main" probeResult="failure" output="Get \"https://10.132.0.90:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:43:47.525492 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:43:47.523138    4892 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-ada8d7/simulator-w3-ada8d7-kserve-7d575cfcb5-5txgn"
Oct 30 19:43:47.546413 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:43:47.546363    4892 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-5f55cd/simulator-w6-5f55cd-kserve-5ff86b7447-h9tm7" podUID="3cb47a99-8be1-4756-b31a-43b58cc6b8dc" containerName="main" probeResult="failure" output="Get \"https://10.132.0.58:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:43:47.708685 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:43:47.708645    4892 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-d52df0/distinct-2-w1-d52df0-kserve-7846558464-xrkdc" podUID="9c614ab6-30e2-4fe1-8609-d5e2dadc198a" containerName="main" probeResult="failure" output="Get \"https://10.132.0.83:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:43:47.932528 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:43:47.932471    4892 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-686689/distinct-2-w5-686689-kserve-5ff578b4cd-rkc6s" podUID="a8829732-ac6f-4238-b8ce-fc781084668b" containerName="main" probeResult="failure" output="Get \"https://10.132.0.87:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:43:47.940821 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:43:47.940761    4892 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-a3f9b7/simulator-w2-a3f9b7-kserve-79bbc4cb5f-8m488" podUID="50fb2283-bf3f-41da-835b-f407ee1a04f3" containerName="main" probeResult="failure" output="Get \"https://10.132.0.59:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:43:47.941450 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:43:47.941405    4892 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-d52df0/simulator-w1-d52df0-kserve-67f74bc577-q5bjz" podUID="81b7bb84-1003-4452-b7e0-b4aaba6fefa4" containerName="main" probeResult="failure" output="Get \"https://10.132.0.57:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:43:47.950214 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:43:47.950134    4892 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-2d9861/premium-w0-2d9861-kserve-67945c6848-4n6kz"
Oct 30 19:43:48.080045 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:43:48.079724    4892 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-8964-de9af3"]
Oct 30 19:43:48.100845 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:43:48.079948    4892 kuberuntime_container.go:898] "Killing container with a grace period" pod="opendatahub/test-curl-8964-de9af3" podUID="1163665e-2ccb-4b87-8fab-abf927cbc704" containerName="test-curl-8964-de9af3" containerID="cri-o://3da9443879d1ac7cfb75cf4beb052464a02111fb085c7ad37774ea9e10f70716" gracePeriod=2
Oct 30 19:43:48.156006 ip-10-0-132-82 kubenswrapper[4892]: E1030 19:43:48.154954    4892 status_manager.go:1045] "Failed to get status for pod" err="pods \"test-curl-8964-de9af3\" is forbidden: User \"system:node:ip-10-0-132-82.us-west-1.compute.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-132-82.us-west-1.compute.internal' and this object" podUID="1163665e-2ccb-4b87-8fab-abf927cbc704" pod="opendatahub/test-curl-8964-de9af3"
Oct 30 19:43:48.229044 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:43:48.220899    4892 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w6-5f55cd/simulator-w6-5f55cd-kserve-5ff86b7447-h9tm7"
Oct 30 19:43:48.229044 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:43:48.221737    4892 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-5f55cd/premium-w6-5f55cd-kserve-7f785c544b-pcvhb" podUID="191593a2-e3a9-4ce4-a1c6-c089723b0237" containerName="main" probeResult="failure" output="Get \"https://10.132.0.60:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:43:48.272723 ip-10-0-132-82 kubenswrapper[4892]: E1030 19:43:48.272663    4892 status_manager.go:1045] "Failed to get status for pod" err="pods \"test-curl-8964-de9af3\" is forbidden: User \"system:node:ip-10-0-132-82.us-west-1.compute.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-132-82.us-west-1.compute.internal' and this object" podUID="1163665e-2ccb-4b87-8fab-abf927cbc704" pod="opendatahub/test-curl-8964-de9af3"
Oct 30 19:43:48.345752 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:43:48.345679    4892 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-8964-de9af3"]
Oct 30 19:43:48.391028 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:43:48.383329    4892 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a429595a-fnjb5" event={"ID":"65aca2dc-d914-46c8-9b84-20a864873a1d","Type":"ContainerStarted","Data":"19b6c23831a67cd2578c62f8e7d9ce8b4a65df7a11de7a823d030d4ef350218a"}
Oct 30 19:43:48.465156 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:43:48.465120    4892 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a429595a-fnjb5"]
Oct 30 19:43:48.491315 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:43:48.491227    4892 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-ada8d7/simulator-w3-ada8d7-kserve-7d575cfcb5-5txgn" podUID="10a5c7c8-6146-47b0-9643-bb137cf10bf5" containerName="main" probeResult="failure" output="Get \"https://10.132.0.55:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:43:48.545319 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:43:48.531493    4892 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-b5178f/distinct-2-w4-b5178f-kserve-69ddf8b665-9pdth" podUID="fb71d961-8500-4367-b48c-88d385962bd5" containerName="main" probeResult="failure" output="Get \"https://10.132.0.90:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:43:48.581315 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:43:48.552069    4892 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w2-a3f9b7/simulator-w2-a3f9b7-kserve-79bbc4cb5f-8m488" podUID="50fb2283-bf3f-41da-835b-f407ee1a04f3" containerName="main" probeResult="failure" output="Get \"https://10.132.0.59:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:43:48.652359 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:43:48.633341    4892 patch_prober.go:29] interesting pod/e2e-worker-w4-b5178f-openshift-default-c8fbdff8c-bjzz7 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.75:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 30 19:43:48.652359 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:43:48.633409    4892 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w4-b5178f-openshift-default-c8fbdff8c-bjzz7" podUID="98aae325-07a2-49fc-a4ba-3e6709dc7c31" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.75:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:43:48.735108 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:43:48.735068    4892 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-d52df0/distinct-2-w1-d52df0-kserve-7846558464-xrkdc" podUID="9c614ab6-30e2-4fe1-8609-d5e2dadc198a" containerName="main" probeResult="failure" output="Get \"https://10.132.0.83:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:43:48.767046 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:43:48.759893    4892 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="06d9238c-da95-4e55-b07d-d95dae4f2b29" path="/var/lib/kubelet/pods/06d9238c-da95-4e55-b07d-d95dae4f2b29/volumes"
Oct 30 19:43:48.802637 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:43:48.783038    4892 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w5-686689/distinct-2-w5-686689-kserve-5ff578b4cd-rkc6s"
Oct 30 19:43:48.981428 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:43:48.951165    4892 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-d52df0/simulator-w1-d52df0-kserve-67f74bc577-q5bjz" podUID="81b7bb84-1003-4452-b7e0-b4aaba6fefa4" containerName="main" probeResult="failure" output="Get \"https://10.132.0.57:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:43:48.981428 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:43:48.951513    4892 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-a3f9b7/simulator-w2-a3f9b7-kserve-79bbc4cb5f-8m488" podUID="50fb2283-bf3f-41da-835b-f407ee1a04f3" containerName="main" probeResult="failure" output="Get \"https://10.132.0.59:8000/ready\": context deadline exceeded"
Oct 30 19:43:48.994000 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:43:48.992182    4892 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w0-2d9861/distinct-2-w0-2d9861-kserve-679c6b56dd-nmhbm"]
Oct 30 19:43:48.994000 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:43:48.993024    4892 cpu_manager.go:414] "containerMap: removing container" podUID="1163665e-2ccb-4b87-8fab-abf927cbc704" containerName="test-curl-8964-de9af3"
Oct 30 19:43:48.994000 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:43:48.993041    4892 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1163665e-2ccb-4b87-8fab-abf927cbc704" containerName="test-curl-8964-de9af3"
Oct 30 19:43:48.994000 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:43:48.993198    4892 memory_manager.go:365] "RemoveStaleState removing state" podUID="1163665e-2ccb-4b87-8fab-abf927cbc704" containerName="test-curl-8964-de9af3"
Oct 30 19:43:49.001742 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:43:49.001630    4892 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w6-5f55cd/premium-w6-5f55cd-kserve-7f785c544b-pcvhb" podUID="191593a2-e3a9-4ce4-a1c6-c089723b0237" containerName="main" probeResult="failure" output="Get \"https://10.132.0.60:8000/health\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:43:49.197384 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:43:49.196628    4892 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-d52df0/simulator-w1-d52df0-kserve-67f74bc577-q5bjz"
Oct 30 19:43:49.197384 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:43:49.196685    4892 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-b5178f/distinct-2-w4-b5178f-kserve-69ddf8b665-9pdth"
Oct 30 19:43:49.197384 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:43:49.196721    4892 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-a3f9b7/simulator-w2-a3f9b7-kserve-79bbc4cb5f-8m488"
Oct 30 19:43:49.197384 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:43:49.196755    4892 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w6-5f55cd/premium-w6-5f55cd-kserve-7f785c544b-pcvhb"
Oct 30 19:43:49.206327 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:43:49.201119    4892 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-2d9861/distinct-2-w0-2d9861-kserve-679c6b56dd-nmhbm"
Oct 30 19:43:49.275988 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:43:49.264170    4892 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w0-2d9861\"/\"distinct-2-w0-2d9861-kserve-self-signed-certs\""
Oct 30 19:43:49.275988 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:43:49.265989    4892 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-ada8d7/simulator-w3-ada8d7-kserve-7d575cfcb5-5txgn"
Oct 30 19:43:49.288690 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:43:49.276706    4892 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w0-2d9861/distinct-2-w0-2d9861-kserve-679c6b56dd-nmhbm"]
Oct 30 19:43:49.288690 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:43:49.279015    4892 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-9ea0f852-lt2xc"]
Oct 30 19:43:49.290928 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:43:49.290909    4892 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-d52df0/distinct-2-w1-d52df0-kserve-7846558464-xrkdc"
Oct 30 19:43:49.291256 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:43:49.291240    4892 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9ea0f852-lt2xc"
Oct 30 19:43:49.305182 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:43:49.302976    4892 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/65b0586a-6593-48cc-adf8-8ff48417ece4-model-cache\") pod \"distinct-2-w0-2d9861-kserve-679c6b56dd-nmhbm\" (UID: \"65b0586a-6593-48cc-adf8-8ff48417ece4\") " pod="e2e-models-e2e-worker-w0-2d9861/distinct-2-w0-2d9861-kserve-679c6b56dd-nmhbm"
Oct 30 19:43:49.305182 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:43:49.303029    4892 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9grrd\" (UniqueName: \"kubernetes.io/projected/65b0586a-6593-48cc-adf8-8ff48417ece4-kube-api-access-9grrd\") pod \"distinct-2-w0-2d9861-kserve-679c6b56dd-nmhbm\" (UID: \"65b0586a-6593-48cc-adf8-8ff48417ece4\") " pod="e2e-models-e2e-worker-w0-2d9861/distinct-2-w0-2d9861-kserve-679c6b56dd-nmhbm"
Oct 30 19:43:49.305182 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:43:49.303092    4892 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/65b0586a-6593-48cc-adf8-8ff48417ece4-home\") pod \"distinct-2-w0-2d9861-kserve-679c6b56dd-nmhbm\" (UID: \"65b0586a-6593-48cc-adf8-8ff48417ece4\") " pod="e2e-models-e2e-worker-w0-2d9861/distinct-2-w0-2d9861-kserve-679c6b56dd-nmhbm"
Oct 30 19:43:49.305182 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:43:49.303182    4892 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/65b0586a-6593-48cc-adf8-8ff48417ece4-dshm\") pod \"distinct-2-w0-2d9861-kserve-679c6b56dd-nmhbm\" (UID: \"65b0586a-6593-48cc-adf8-8ff48417ece4\") " pod="e2e-models-e2e-worker-w0-2d9861/distinct-2-w0-2d9861-kserve-679c6b56dd-nmhbm"
Oct 30 19:43:49.305182 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:43:49.304011    4892 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/65b0586a-6593-48cc-adf8-8ff48417ece4-tmp-dir\") pod \"distinct-2-w0-2d9861-kserve-679c6b56dd-nmhbm\" (UID: \"65b0586a-6593-48cc-adf8-8ff48417ece4\") " pod="e2e-models-e2e-worker-w0-2d9861/distinct-2-w0-2d9861-kserve-679c6b56dd-nmhbm"
Oct 30 19:43:49.305182 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:43:49.304225    4892 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/65b0586a-6593-48cc-adf8-8ff48417ece4-tls-certs\") pod \"distinct-2-w0-2d9861-kserve-679c6b56dd-nmhbm\" (UID: \"65b0586a-6593-48cc-adf8-8ff48417ece4\") " pod="e2e-models-e2e-worker-w0-2d9861/distinct-2-w0-2d9861-kserve-679c6b56dd-nmhbm"
Oct 30 19:43:49.306232 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:43:49.306211    4892 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-9ea0f852-lt2xc"]
Oct 30 19:43:49.494633 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:43:49.434010    4892 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/65b0586a-6593-48cc-adf8-8ff48417ece4-tls-certs\") pod \"distinct-2-w0-2d9861-kserve-679c6b56dd-nmhbm\" (UID: \"65b0586a-6593-48cc-adf8-8ff48417ece4\") " pod="e2e-models-e2e-worker-w0-2d9861/distinct-2-w0-2d9861-kserve-679c6b56dd-nmhbm"
Oct 30 19:43:49.494633 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:43:49.434119    4892 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qg8fj\" (UniqueName: \"kubernetes.io/projected/f2c3a6a5-3357-4220-a7c2-75c96d152aa0-kube-api-access-qg8fj\") pod \"maas-api-revoke-sub-9ea0f852-lt2xc\" (UID: \"f2c3a6a5-3357-4220-a7c2-75c96d152aa0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9ea0f852-lt2xc"
Oct 30 19:43:49.494633 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:43:49.434212    4892 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f2c3a6a5-3357-4220-a7c2-75c96d152aa0-maas-api-xks-ca\") pod \"maas-api-revoke-sub-9ea0f852-lt2xc\" (UID: \"f2c3a6a5-3357-4220-a7c2-75c96d152aa0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9ea0f852-lt2xc"
Oct 30 19:43:49.494633 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:43:49.434260    4892 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/65b0586a-6593-48cc-adf8-8ff48417ece4-model-cache\") pod \"distinct-2-w0-2d9861-kserve-679c6b56dd-nmhbm\" (UID: \"65b0586a-6593-48cc-adf8-8ff48417ece4\") " pod="e2e-models-e2e-worker-w0-2d9861/distinct-2-w0-2d9861-kserve-679c6b56dd-nmhbm"
Oct 30 19:43:49.494633 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:43:49.434302    4892 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9grrd\" (UniqueName: \"kubernetes.io/projected/65b0586a-6593-48cc-adf8-8ff48417ece4-kube-api-access-9grrd\") pod \"distinct-2-w0-2d9861-kserve-679c6b56dd-nmhbm\" (UID: \"65b0586a-6593-48cc-adf8-8ff48417ece4\") " pod="e2e-models-e2e-worker-w0-2d9861/distinct-2-w0-2d9861-kserve-679c6b56dd-nmhbm"
Oct 30 19:43:49.494633 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:43:49.434344    4892 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/65b0586a-6593-48cc-adf8-8ff48417ece4-home\") pod \"distinct-2-w0-2d9861-kserve-679c6b56dd-nmhbm\" (UID: \"65b0586a-6593-48cc-adf8-8ff48417ece4\") " pod="e2e-models-e2e-worker-w0-2d9861/distinct-2-w0-2d9861-kserve-679c6b56dd-nmhbm"
Oct 30 19:43:49.494633 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:43:49.434369    4892 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f2c3a6a5-3357-4220-a7c2-75c96d152aa0-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-9ea0f852-lt2xc\" (UID: \"f2c3a6a5-3357-4220-a7c2-75c96d152aa0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9ea0f852-lt2xc"
Oct 30 19:43:49.494633 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:43:49.434479    4892 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/65b0586a-6593-48cc-adf8-8ff48417ece4-dshm\") pod \"distinct-2-w0-2d9861-kserve-679c6b56dd-nmhbm\" (UID: \"65b0586a-6593-48cc-adf8-8ff48417ece4\") " pod="e2e-models-e2e-worker-w0-2d9861/distinct-2-w0-2d9861-kserve-679c6b56dd-nmhbm"
Oct 30 19:43:49.494633 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:43:49.434563    4892 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/65b0586a-6593-48cc-adf8-8ff48417ece4-tmp-dir\") pod \"distinct-2-w0-2d9861-kserve-679c6b56dd-nmhbm\" (UID: \"65b0586a-6593-48cc-adf8-8ff48417ece4\") " pod="e2e-models-e2e-worker-w0-2d9861/distinct-2-w0-2d9861-kserve-679c6b56dd-nmhbm"
Oct 30 19:43:49.494633 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:43:49.474605    4892 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/65b0586a-6593-48cc-adf8-8ff48417ece4-model-cache\") pod \"distinct-2-w0-2d9861-kserve-679c6b56dd-nmhbm\" (UID: \"65b0586a-6593-48cc-adf8-8ff48417ece4\") " pod="e2e-models-e2e-worker-w0-2d9861/distinct-2-w0-2d9861-kserve-679c6b56dd-nmhbm"
Oct 30 19:43:49.494633 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:43:49.474938    4892 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/65b0586a-6593-48cc-adf8-8ff48417ece4-home\") pod \"distinct-2-w0-2d9861-kserve-679c6b56dd-nmhbm\" (UID: \"65b0586a-6593-48cc-adf8-8ff48417ece4\") " pod="e2e-models-e2e-worker-w0-2d9861/distinct-2-w0-2d9861-kserve-679c6b56dd-nmhbm"
Oct 30 19:43:49.494633 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:43:49.478021    4892 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/65b0586a-6593-48cc-adf8-8ff48417ece4-tmp-dir\") pod \"distinct-2-w0-2d9861-kserve-679c6b56dd-nmhbm\" (UID: \"65b0586a-6593-48cc-adf8-8ff48417ece4\") " pod="e2e-models-e2e-worker-w0-2d9861/distinct-2-w0-2d9861-kserve-679c6b56dd-nmhbm"
Oct 30 19:43:49.551158 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:43:49.547215    4892 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f2c3a6a5-3357-4220-a7c2-75c96d152aa0-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-9ea0f852-lt2xc\" (UID: \"f2c3a6a5-3357-4220-a7c2-75c96d152aa0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9ea0f852-lt2xc"
Oct 30 19:43:49.551158 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:43:49.547567    4892 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qg8fj\" (UniqueName: \"kubernetes.io/projected/f2c3a6a5-3357-4220-a7c2-75c96d152aa0-kube-api-access-qg8fj\") pod \"maas-api-revoke-sub-9ea0f852-lt2xc\" (UID: \"f2c3a6a5-3357-4220-a7c2-75c96d152aa0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9ea0f852-lt2xc"
Oct 30 19:43:49.551158 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:43:49.547644    4892 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f2c3a6a5-3357-4220-a7c2-75c96d152aa0-maas-api-xks-ca\") pod \"maas-api-revoke-sub-9ea0f852-lt2xc\" (UID: \"f2c3a6a5-3357-4220-a7c2-75c96d152aa0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9ea0f852-lt2xc"
Oct 30 19:43:49.571063 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:43:49.570934    4892 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/65b0586a-6593-48cc-adf8-8ff48417ece4-dshm\") pod \"distinct-2-w0-2d9861-kserve-679c6b56dd-nmhbm\" (UID: \"65b0586a-6593-48cc-adf8-8ff48417ece4\") " pod="e2e-models-e2e-worker-w0-2d9861/distinct-2-w0-2d9861-kserve-679c6b56dd-nmhbm"
Oct 30 19:43:49.623128 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:43:49.581232    4892 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f2c3a6a5-3357-4220-a7c2-75c96d152aa0-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-9ea0f852-lt2xc\" (UID: \"f2c3a6a5-3357-4220-a7c2-75c96d152aa0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9ea0f852-lt2xc"
Oct 30 19:43:49.623455 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:43:49.590696    4892 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f2c3a6a5-3357-4220-a7c2-75c96d152aa0-maas-api-xks-ca\") pod \"maas-api-revoke-sub-9ea0f852-lt2xc\" (UID: \"f2c3a6a5-3357-4220-a7c2-75c96d152aa0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9ea0f852-lt2xc"
Oct 30 19:43:49.623593 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:43:49.621535    4892 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/65b0586a-6593-48cc-adf8-8ff48417ece4-tls-certs\") pod \"distinct-2-w0-2d9861-kserve-679c6b56dd-nmhbm\" (UID: \"65b0586a-6593-48cc-adf8-8ff48417ece4\") " pod="e2e-models-e2e-worker-w0-2d9861/distinct-2-w0-2d9861-kserve-679c6b56dd-nmhbm"
Oct 30 19:43:49.623703 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:43:49.623055    4892 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9grrd\" (UniqueName: \"kubernetes.io/projected/65b0586a-6593-48cc-adf8-8ff48417ece4-kube-api-access-9grrd\") pod \"distinct-2-w0-2d9861-kserve-679c6b56dd-nmhbm\" (UID: \"65b0586a-6593-48cc-adf8-8ff48417ece4\") " pod="e2e-models-e2e-worker-w0-2d9861/distinct-2-w0-2d9861-kserve-679c6b56dd-nmhbm"
Oct 30 19:43:49.691390 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:43:49.691351    4892 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qg8fj\" (UniqueName: \"kubernetes.io/projected/f2c3a6a5-3357-4220-a7c2-75c96d152aa0-kube-api-access-qg8fj\") pod \"maas-api-revoke-sub-9ea0f852-lt2xc\" (UID: \"f2c3a6a5-3357-4220-a7c2-75c96d152aa0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9ea0f852-lt2xc"
Oct 30 19:43:49.694365 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:43:49.694325    4892 generic.go:356] "Generic (PLEG): container finished" podID="65aca2dc-d914-46c8-9b84-20a864873a1d" containerID="c8ec16c869191b30e60c22a2527e52f0af8397c65126fd4b67a07aebd051e74c" exitCode=0
Oct 30 19:43:49.694504 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:43:49.694459    4892 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a429595a-fnjb5" event={"ID":"65aca2dc-d914-46c8-9b84-20a864873a1d","Type":"ContainerDied","Data":"c8ec16c869191b30e60c22a2527e52f0af8397c65126fd4b67a07aebd051e74c"}
Oct 30 19:43:49.891249 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:43:49.891217    4892 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-2d9861/distinct-2-w0-2d9861-kserve-679c6b56dd-nmhbm"
Oct 30 19:43:49.913930 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:43:49.913904    4892 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9ea0f852-lt2xc"
Oct 30 19:43:50.332735 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:43:50.332684    4892 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w0-2d9861/distinct-2-w0-2d9861-kserve-679c6b56dd-nmhbm"]
Oct 30 19:43:50.341020 ip-10-0-132-82 kubenswrapper[4892]: W1030 19:43:50.335135    4892 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod65b0586a_6593_48cc_adf8_8ff48417ece4.slice/crio-c5b217d2de2cd2b38101716994243231ac57195716d24dc1bf6e839eb2efe482 WatchSource:0}: Error finding container c5b217d2de2cd2b38101716994243231ac57195716d24dc1bf6e839eb2efe482: Status 404 returned error can't find the container with id c5b217d2de2cd2b38101716994243231ac57195716d24dc1bf6e839eb2efe482
Oct 30 19:43:50.459454 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:43:50.459406    4892 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-8964-de9af3"
Oct 30 19:43:50.501280 ip-10-0-132-82 kubenswrapper[4892]: W1030 19:43:50.501248    4892 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf2c3a6a5_3357_4220_a7c2_75c96d152aa0.slice/crio-d35824b15013b2306c2d719a36ab916e38d2a40cf3707f75d70d924b11d10260 WatchSource:0}: Error finding container d35824b15013b2306c2d719a36ab916e38d2a40cf3707f75d70d924b11d10260: Status 404 returned error can't find the container with id d35824b15013b2306c2d719a36ab916e38d2a40cf3707f75d70d924b11d10260
Oct 30 19:43:50.502817 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:43:50.502794    4892 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-9ea0f852-lt2xc"]
Oct 30 19:43:50.580610 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:43:50.580583    4892 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1163665e-2ccb-4b87-8fab-abf927cbc704-kube-api-access-l5497\" (UniqueName: \"kubernetes.io/projected/1163665e-2ccb-4b87-8fab-abf927cbc704-kube-api-access-l5497\") pod \"1163665e-2ccb-4b87-8fab-abf927cbc704\" (UID: \"1163665e-2ccb-4b87-8fab-abf927cbc704\") "
Oct 30 19:43:50.582845 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:43:50.582818    4892 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1163665e-2ccb-4b87-8fab-abf927cbc704-kube-api-access-l5497" pod "1163665e-2ccb-4b87-8fab-abf927cbc704" (UID: "1163665e-2ccb-4b87-8fab-abf927cbc704"). InnerVolumeSpecName "kube-api-access-l5497". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:43:50.682431 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:43:50.682358    4892 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-l5497\" (UniqueName: \"kubernetes.io/projected/1163665e-2ccb-4b87-8fab-abf927cbc704-kube-api-access-l5497\") on node \"ip-10-0-132-82.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 19:43:50.705641 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:43:50.705595    4892 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-2d9861/distinct-2-w0-2d9861-kserve-679c6b56dd-nmhbm" event={"ID":"65b0586a-6593-48cc-adf8-8ff48417ece4","Type":"ContainerStarted","Data":"d666bdabcb260b813cdb2b7190a2f57774d6bbcd0232768e893ed49ef2d99b29"}
Oct 30 19:43:50.705826 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:43:50.705656    4892 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-2d9861/distinct-2-w0-2d9861-kserve-679c6b56dd-nmhbm" event={"ID":"65b0586a-6593-48cc-adf8-8ff48417ece4","Type":"ContainerStarted","Data":"c5b217d2de2cd2b38101716994243231ac57195716d24dc1bf6e839eb2efe482"}
Oct 30 19:43:50.707135 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:43:50.707104    4892 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9ea0f852-lt2xc" event={"ID":"f2c3a6a5-3357-4220-a7c2-75c96d152aa0","Type":"ContainerStarted","Data":"d35824b15013b2306c2d719a36ab916e38d2a40cf3707f75d70d924b11d10260"}
Oct 30 19:43:50.708584 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:43:50.708559    4892 generic.go:356] "Generic (PLEG): container finished" podID="1163665e-2ccb-4b87-8fab-abf927cbc704" containerID="3da9443879d1ac7cfb75cf4beb052464a02111fb085c7ad37774ea9e10f70716" exitCode=137
Oct 30 19:43:50.708654 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:43:50.708603    4892 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-8964-de9af3"
Oct 30 19:43:50.708654 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:43:50.708647    4892 scope.go:122] "RemoveContainer" containerID="3da9443879d1ac7cfb75cf4beb052464a02111fb085c7ad37774ea9e10f70716"
Oct 30 19:43:50.722508 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:43:50.722477    4892 scope.go:122] "RemoveContainer" containerID="3da9443879d1ac7cfb75cf4beb052464a02111fb085c7ad37774ea9e10f70716"
Oct 30 19:43:50.722777 ip-10-0-132-82 kubenswrapper[4892]: E1030 19:43:50.722760    4892 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3da9443879d1ac7cfb75cf4beb052464a02111fb085c7ad37774ea9e10f70716\": container with ID starting with 3da9443879d1ac7cfb75cf4beb052464a02111fb085c7ad37774ea9e10f70716 not found: ID does not exist" containerID="3da9443879d1ac7cfb75cf4beb052464a02111fb085c7ad37774ea9e10f70716"
Oct 30 19:43:50.722828 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:43:50.722786    4892 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3da9443879d1ac7cfb75cf4beb052464a02111fb085c7ad37774ea9e10f70716"} err="failed to get container status \"3da9443879d1ac7cfb75cf4beb052464a02111fb085c7ad37774ea9e10f70716\": rpc error: code = NotFound desc = could not find container \"3da9443879d1ac7cfb75cf4beb052464a02111fb085c7ad37774ea9e10f70716\": container with ID starting with 3da9443879d1ac7cfb75cf4beb052464a02111fb085c7ad37774ea9e10f70716 not found: ID does not exist"
Oct 30 19:43:50.854908 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:43:50.854847    4892 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w0-2d9861/distinct-2-w0-2d9861-kserve-679c6b56dd-nmhbm" podStartSLOduration=2.854829053 podStartE2EDuration="2.854829053s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:43:48 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:43:50.850769803 +0000 UTC m=+1678.421235415" watchObservedRunningTime="2026-10-30 19:43:50.854829053 +0000 UTC m=+1678.425294655"
Oct 30 19:43:50.880331 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:43:50.880306    4892 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a429595a-fnjb5"
Oct 30 19:43:50.891572 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:43:50.891550    4892 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-2d9861/distinct-2-w0-2d9861-kserve-679c6b56dd-nmhbm"
Oct 30 19:43:50.985943 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:43:50.985863    4892 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/65aca2dc-d914-46c8-9b84-20a864873a1d-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/65aca2dc-d914-46c8-9b84-20a864873a1d-maas-api-xks-ca\") pod \"65aca2dc-d914-46c8-9b84-20a864873a1d\" (UID: \"65aca2dc-d914-46c8-9b84-20a864873a1d\") "
Oct 30 19:43:50.985943 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:43:50.985908    4892 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/65aca2dc-d914-46c8-9b84-20a864873a1d-kube-api-access-6t4sm\" (UniqueName: \"kubernetes.io/projected/65aca2dc-d914-46c8-9b84-20a864873a1d-kube-api-access-6t4sm\") pod \"65aca2dc-d914-46c8-9b84-20a864873a1d\" (UID: \"65aca2dc-d914-46c8-9b84-20a864873a1d\") "
Oct 30 19:43:50.986221 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:43:50.985990    4892 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/65aca2dc-d914-46c8-9b84-20a864873a1d-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/65aca2dc-d914-46c8-9b84-20a864873a1d-maas-api-openshift-ca\") pod \"65aca2dc-d914-46c8-9b84-20a864873a1d\" (UID: \"65aca2dc-d914-46c8-9b84-20a864873a1d\") "
Oct 30 19:43:50.987022 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:43:50.986945    4892 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/65aca2dc-d914-46c8-9b84-20a864873a1d-maas-api-openshift-ca" pod "65aca2dc-d914-46c8-9b84-20a864873a1d" (UID: "65aca2dc-d914-46c8-9b84-20a864873a1d"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 19:43:50.992702 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:43:50.992672    4892 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/65aca2dc-d914-46c8-9b84-20a864873a1d-maas-api-xks-ca" pod "65aca2dc-d914-46c8-9b84-20a864873a1d" (UID: "65aca2dc-d914-46c8-9b84-20a864873a1d"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 19:43:50.992850 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:43:50.992793    4892 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/65aca2dc-d914-46c8-9b84-20a864873a1d-kube-api-access-6t4sm" pod "65aca2dc-d914-46c8-9b84-20a864873a1d" (UID: "65aca2dc-d914-46c8-9b84-20a864873a1d"). InnerVolumeSpecName "kube-api-access-6t4sm". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:43:51.095036 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:43:51.091219    4892 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/65aca2dc-d914-46c8-9b84-20a864873a1d-maas-api-xks-ca\") on node \"ip-10-0-132-82.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 19:43:51.095036 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:43:51.091256    4892 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6t4sm\" (UniqueName: \"kubernetes.io/projected/65aca2dc-d914-46c8-9b84-20a864873a1d-kube-api-access-6t4sm\") on node \"ip-10-0-132-82.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 19:43:51.095036 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:43:51.091270    4892 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/65aca2dc-d914-46c8-9b84-20a864873a1d-maas-api-openshift-ca\") on node \"ip-10-0-132-82.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 19:43:51.718503 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:43:51.718472    4892 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a429595a-fnjb5"
Oct 30 19:43:51.718894 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:43:51.718505    4892 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a429595a-fnjb5" event={"ID":"65aca2dc-d914-46c8-9b84-20a864873a1d","Type":"ContainerDied","Data":"19b6c23831a67cd2578c62f8e7d9ce8b4a65df7a11de7a823d030d4ef350218a"}
Oct 30 19:43:51.718894 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:43:51.718552    4892 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="19b6c23831a67cd2578c62f8e7d9ce8b4a65df7a11de7a823d030d4ef350218a"
Oct 30 19:43:51.720765 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:43:51.720735    4892 generic.go:356] "Generic (PLEG): container finished" podID="f2c3a6a5-3357-4220-a7c2-75c96d152aa0" containerID="533908949b3e37aae1637ec976559e6519109a433712a90acd9c5909c5f8dfc7" exitCode=0
Oct 30 19:43:51.720927 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:43:51.720811    4892 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9ea0f852-lt2xc" event={"ID":"f2c3a6a5-3357-4220-a7c2-75c96d152aa0","Type":"ContainerDied","Data":"533908949b3e37aae1637ec976559e6519109a433712a90acd9c5909c5f8dfc7"}
Oct 30 19:43:52.572975 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:43:52.572928    4892 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="1163665e-2ccb-4b87-8fab-abf927cbc704" path="/var/lib/kubelet/pods/1163665e-2ccb-4b87-8fab-abf927cbc704/volumes"
Oct 30 19:43:52.894416 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:43:52.894393    4892 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9ea0f852-lt2xc"
Oct 30 19:43:53.043082 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:43:53.043050    4892 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f2c3a6a5-3357-4220-a7c2-75c96d152aa0-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f2c3a6a5-3357-4220-a7c2-75c96d152aa0-maas-api-openshift-ca\") pod \"f2c3a6a5-3357-4220-a7c2-75c96d152aa0\" (UID: \"f2c3a6a5-3357-4220-a7c2-75c96d152aa0\") "
Oct 30 19:43:53.043280 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:43:53.043260    4892 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f2c3a6a5-3357-4220-a7c2-75c96d152aa0-kube-api-access-qg8fj\" (UniqueName: \"kubernetes.io/projected/f2c3a6a5-3357-4220-a7c2-75c96d152aa0-kube-api-access-qg8fj\") pod \"f2c3a6a5-3357-4220-a7c2-75c96d152aa0\" (UID: \"f2c3a6a5-3357-4220-a7c2-75c96d152aa0\") "
Oct 30 19:43:53.043355 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:43:53.043331    4892 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/f2c3a6a5-3357-4220-a7c2-75c96d152aa0-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f2c3a6a5-3357-4220-a7c2-75c96d152aa0-maas-api-xks-ca\") pod \"f2c3a6a5-3357-4220-a7c2-75c96d152aa0\" (UID: \"f2c3a6a5-3357-4220-a7c2-75c96d152aa0\") "
Oct 30 19:43:53.043475 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:43:53.043447    4892 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f2c3a6a5-3357-4220-a7c2-75c96d152aa0-maas-api-openshift-ca" pod "f2c3a6a5-3357-4220-a7c2-75c96d152aa0" (UID: "f2c3a6a5-3357-4220-a7c2-75c96d152aa0"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 19:43:53.043935 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:43:53.043909    4892 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f2c3a6a5-3357-4220-a7c2-75c96d152aa0-maas-api-openshift-ca\") on node \"ip-10-0-132-82.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 19:43:53.046035 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:43:53.045942    4892 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f2c3a6a5-3357-4220-a7c2-75c96d152aa0-kube-api-access-qg8fj" pod "f2c3a6a5-3357-4220-a7c2-75c96d152aa0" (UID: "f2c3a6a5-3357-4220-a7c2-75c96d152aa0"). InnerVolumeSpecName "kube-api-access-qg8fj". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:43:53.046035 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:43:53.046003    4892 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f2c3a6a5-3357-4220-a7c2-75c96d152aa0-maas-api-xks-ca" pod "f2c3a6a5-3357-4220-a7c2-75c96d152aa0" (UID: "f2c3a6a5-3357-4220-a7c2-75c96d152aa0"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 19:43:53.145091 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:43:53.145054    4892 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qg8fj\" (UniqueName: \"kubernetes.io/projected/f2c3a6a5-3357-4220-a7c2-75c96d152aa0-kube-api-access-qg8fj\") on node \"ip-10-0-132-82.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 19:43:53.145091 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:43:53.145086    4892 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f2c3a6a5-3357-4220-a7c2-75c96d152aa0-maas-api-xks-ca\") on node \"ip-10-0-132-82.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 19:43:53.277466 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:43:53.277431    4892 scope.go:122] "RemoveContainer" containerID="9fee3f79a37da6e1108cb6ff7056fee62b1a22fd6cbb811d782289d007f5ce9c"
Oct 30 19:43:53.288803 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:43:53.288772    4892 scope.go:122] "RemoveContainer" containerID="3933dfedcf561e55b4d9ec4f30a9ccaf8ed137755b4d03b662f3067b9a2d692b"
Oct 30 19:43:53.741782 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:43:53.741748    4892 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9ea0f852-lt2xc"
Oct 30 19:43:53.741782 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:43:53.741773    4892 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9ea0f852-lt2xc" event={"ID":"f2c3a6a5-3357-4220-a7c2-75c96d152aa0","Type":"ContainerDied","Data":"d35824b15013b2306c2d719a36ab916e38d2a40cf3707f75d70d924b11d10260"}
Oct 30 19:43:53.742017 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:43:53.741820    4892 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="d35824b15013b2306c2d719a36ab916e38d2a40cf3707f75d70d924b11d10260"
Oct 30 19:43:54.685069 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:43:54.685033    4892 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-94581ab2-688dd7dd6-rxx2h"]
Oct 30 19:43:54.685608 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:43:54.685594    4892 cpu_manager.go:414] "containerMap: removing container" podUID="65aca2dc-d914-46c8-9b84-20a864873a1d" containerName="revoke-keys"
Oct 30 19:43:54.685651 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:43:54.685609    4892 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="65aca2dc-d914-46c8-9b84-20a864873a1d" containerName="revoke-keys"
Oct 30 19:43:54.685651 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:43:54.685634    4892 cpu_manager.go:414] "containerMap: removing container" podUID="f2c3a6a5-3357-4220-a7c2-75c96d152aa0" containerName="revoke-keys"
Oct 30 19:43:54.685651 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:43:54.685639    4892 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f2c3a6a5-3357-4220-a7c2-75c96d152aa0" containerName="revoke-keys"
Oct 30 19:43:54.685778 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:43:54.685768    4892 memory_manager.go:365] "RemoveStaleState removing state" podUID="65aca2dc-d914-46c8-9b84-20a864873a1d" containerName="revoke-keys"
Oct 30 19:43:54.685809 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:43:54.685784    4892 memory_manager.go:365] "RemoveStaleState removing state" podUID="f2c3a6a5-3357-4220-a7c2-75c96d152aa0" containerName="revoke-keys"
Oct 30 19:43:54.691033 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:43:54.691014    4892 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-94581ab2-688dd7dd6-rxx2h"
Oct 30 19:43:54.776985 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:43:54.776482    4892 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ait-94581ab2\""
Oct 30 19:43:54.801988 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:43:54.798436    4892 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e4da7424-8179-4b09-82cf-d2f3ed9f4894-plugins-config-volume\") pod \"payload-processing-e2e-ait-94581ab2-688dd7dd6-rxx2h\" (UID: \"e4da7424-8179-4b09-82cf-d2f3ed9f4894\") " pod="openshift-ingress/payload-processing-e2e-ait-94581ab2-688dd7dd6-rxx2h"
Oct 30 19:43:54.801988 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:43:54.798570    4892 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9b7t8\" (UniqueName: \"kubernetes.io/projected/e4da7424-8179-4b09-82cf-d2f3ed9f4894-kube-api-access-9b7t8\") pod \"payload-processing-e2e-ait-94581ab2-688dd7dd6-rxx2h\" (UID: \"e4da7424-8179-4b09-82cf-d2f3ed9f4894\") " pod="openshift-ingress/payload-processing-e2e-ait-94581ab2-688dd7dd6-rxx2h"
Oct 30 19:43:54.842132 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:43:54.832159    4892 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ait-94581ab2-dockercfg-prfsc\""
Oct 30 19:43:54.926749 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:43:54.903937    4892 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9b7t8\" (UniqueName: \"kubernetes.io/projected/e4da7424-8179-4b09-82cf-d2f3ed9f4894-kube-api-access-9b7t8\") pod \"payload-processing-e2e-ait-94581ab2-688dd7dd6-rxx2h\" (UID: \"e4da7424-8179-4b09-82cf-d2f3ed9f4894\") " pod="openshift-ingress/payload-processing-e2e-ait-94581ab2-688dd7dd6-rxx2h"
Oct 30 19:43:54.926749 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:43:54.904175    4892 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e4da7424-8179-4b09-82cf-d2f3ed9f4894-plugins-config-volume\") pod \"payload-processing-e2e-ait-94581ab2-688dd7dd6-rxx2h\" (UID: \"e4da7424-8179-4b09-82cf-d2f3ed9f4894\") " pod="openshift-ingress/payload-processing-e2e-ait-94581ab2-688dd7dd6-rxx2h"
Oct 30 19:43:54.926749 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:43:54.923874    4892 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e4da7424-8179-4b09-82cf-d2f3ed9f4894-plugins-config-volume\") pod \"payload-processing-e2e-ait-94581ab2-688dd7dd6-rxx2h\" (UID: \"e4da7424-8179-4b09-82cf-d2f3ed9f4894\") " pod="openshift-ingress/payload-processing-e2e-ait-94581ab2-688dd7dd6-rxx2h"
Oct 30 19:43:54.950367 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:43:54.948148    4892 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-94581ab2-688dd7dd6-rxx2h"]
Oct 30 19:43:54.958239 ip-10-0-132-82 kubenswrapper[4892]: E1030 19:43:54.958190    4892 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-jrm87" containerName="extractor"
Oct 30 19:43:54.989768 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:43:54.988428    4892 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9b7t8\" (UniqueName: \"kubernetes.io/projected/e4da7424-8179-4b09-82cf-d2f3ed9f4894-kube-api-access-9b7t8\") pod \"payload-processing-e2e-ait-94581ab2-688dd7dd6-rxx2h\" (UID: \"e4da7424-8179-4b09-82cf-d2f3ed9f4894\") " pod="openshift-ingress/payload-processing-e2e-ait-94581ab2-688dd7dd6-rxx2h"
Oct 30 19:43:55.005239 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:43:55.005206    4892 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-94581ab2-688dd7dd6-rxx2h"
Oct 30 19:43:55.392633 ip-10-0-132-82 kubenswrapper[4892]: W1030 19:43:55.392607    4892 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode4da7424_8179_4b09_82cf_d2f3ed9f4894.slice/crio-e1aaf116034b42bb3532cd47f3a5be2dc4204750e120ca5997f7f5c7390f2c7b WatchSource:0}: Error finding container e1aaf116034b42bb3532cd47f3a5be2dc4204750e120ca5997f7f5c7390f2c7b: Status 404 returned error can't find the container with id e1aaf116034b42bb3532cd47f3a5be2dc4204750e120ca5997f7f5c7390f2c7b
Oct 30 19:43:55.395344 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:43:55.395312    4892 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-94581ab2-688dd7dd6-rxx2h"]
Oct 30 19:43:55.837104 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:43:55.837017    4892 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-94581ab2-688dd7dd6-rxx2h" event={"ID":"e4da7424-8179-4b09-82cf-d2f3ed9f4894","Type":"ContainerStarted","Data":"c69391ca1e932e304e80741189ad5f815e5d3d059e1057e9ef008a49916e23e3"}
Oct 30 19:43:55.837104 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:43:55.837070    4892 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-94581ab2-688dd7dd6-rxx2h" event={"ID":"e4da7424-8179-4b09-82cf-d2f3ed9f4894","Type":"ContainerStarted","Data":"e1aaf116034b42bb3532cd47f3a5be2dc4204750e120ca5997f7f5c7390f2c7b"}
Oct 30 19:43:55.837531 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:43:55.837110    4892 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-ait-94581ab2-688dd7dd6-rxx2h"
Oct 30 19:43:55.995169 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:43:55.995113    4892 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-ait-94581ab2-688dd7dd6-rxx2h" podStartSLOduration=1.995099927 podStartE2EDuration="1.995099927s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:43:54 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:43:55.993315335 +0000 UTC m=+1683.563780935" watchObservedRunningTime="2026-10-30 19:43:55.995099927 +0000 UTC m=+1683.565565525"
Oct 30 19:43:59.891492 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:43:59.891464    4892 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w0-2d9861/distinct-2-w0-2d9861-kserve-679c6b56dd-nmhbm"
Oct 30 19:43:59.904200 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:43:59.904178    4892 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w0-2d9861/distinct-2-w0-2d9861-kserve-679c6b56dd-nmhbm"
Oct 30 19:44:00.888423 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:44:00.888391    4892 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-2d9861/distinct-2-w0-2d9861-kserve-679c6b56dd-nmhbm"
Oct 30 19:44:02.734201 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:44:02.734133    4892 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-migrate-d74c1bd9-5f777c6ff-qbwsw"]
Oct 30 19:44:02.741412 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:44:02.741386    4892 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-migrate-d74c1bd9-5f777c6ff-qbwsw"
Oct 30 19:44:02.788505 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:44:02.788472    4892 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-migrate-d74c1bd9-5f777c6ff-qbwsw"]
Oct 30 19:44:02.828506 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:44:02.828469    4892 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-migrate-d74c1bd9\""
Oct 30 19:44:02.828906 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:44:02.828890    4892 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-migrate-d74c1bd9-dockercfg-sf7hp\""
Oct 30 19:44:02.847760 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:44:02.847656    4892 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jncj8\" (UniqueName: \"kubernetes.io/projected/28480114-29bf-4642-8da2-a7e714ba3460-kube-api-access-jncj8\") pod \"payload-pre-processing-e2e-migrate-d74c1bd9-5f777c6ff-qbwsw\" (UID: \"28480114-29bf-4642-8da2-a7e714ba3460\") " pod="openshift-ingress/payload-pre-processing-e2e-migrate-d74c1bd9-5f777c6ff-qbwsw"
Oct 30 19:44:02.848244 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:44:02.848197    4892 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/28480114-29bf-4642-8da2-a7e714ba3460-plugins-config-volume\") pod \"payload-pre-processing-e2e-migrate-d74c1bd9-5f777c6ff-qbwsw\" (UID: \"28480114-29bf-4642-8da2-a7e714ba3460\") " pod="openshift-ingress/payload-pre-processing-e2e-migrate-d74c1bd9-5f777c6ff-qbwsw"
Oct 30 19:44:02.965471 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:44:02.954175    4892 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/28480114-29bf-4642-8da2-a7e714ba3460-plugins-config-volume\") pod \"payload-pre-processing-e2e-migrate-d74c1bd9-5f777c6ff-qbwsw\" (UID: \"28480114-29bf-4642-8da2-a7e714ba3460\") " pod="openshift-ingress/payload-pre-processing-e2e-migrate-d74c1bd9-5f777c6ff-qbwsw"
Oct 30 19:44:02.965471 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:44:02.954373    4892 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jncj8\" (UniqueName: \"kubernetes.io/projected/28480114-29bf-4642-8da2-a7e714ba3460-kube-api-access-jncj8\") pod \"payload-pre-processing-e2e-migrate-d74c1bd9-5f777c6ff-qbwsw\" (UID: \"28480114-29bf-4642-8da2-a7e714ba3460\") " pod="openshift-ingress/payload-pre-processing-e2e-migrate-d74c1bd9-5f777c6ff-qbwsw"
Oct 30 19:44:02.965471 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:44:02.955652    4892 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/28480114-29bf-4642-8da2-a7e714ba3460-plugins-config-volume\") pod \"payload-pre-processing-e2e-migrate-d74c1bd9-5f777c6ff-qbwsw\" (UID: \"28480114-29bf-4642-8da2-a7e714ba3460\") " pod="openshift-ingress/payload-pre-processing-e2e-migrate-d74c1bd9-5f777c6ff-qbwsw"
Oct 30 19:44:03.003989 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:44:03.003901    4892 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-migrate-d74c1bd9-7c87cbbf88-2j9tg"]
Oct 30 19:44:03.022251 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:44:03.022223    4892 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-migrate-d74c1bd9-7c87cbbf88-2j9tg"
Oct 30 19:44:03.035309 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:44:03.035277    4892 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-migrate-d74c1bd9-7c87cbbf88-2j9tg"]
Oct 30 19:44:03.041008 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:44:03.040809    4892 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jncj8\" (UniqueName: \"kubernetes.io/projected/28480114-29bf-4642-8da2-a7e714ba3460-kube-api-access-jncj8\") pod \"payload-pre-processing-e2e-migrate-d74c1bd9-5f777c6ff-qbwsw\" (UID: \"28480114-29bf-4642-8da2-a7e714ba3460\") " pod="openshift-ingress/payload-pre-processing-e2e-migrate-d74c1bd9-5f777c6ff-qbwsw"
Oct 30 19:44:03.063744 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:44:03.063694    4892 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-migrate-d74c1bd9-5f777c6ff-qbwsw"
Oct 30 19:44:03.115021 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:44:03.068254    4892 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/79ae5280-77c7-451b-bd82-ade64806cdd7-plugins-config-volume\") pod \"payload-processing-e2e-migrate-d74c1bd9-7c87cbbf88-2j9tg\" (UID: \"79ae5280-77c7-451b-bd82-ade64806cdd7\") " pod="openshift-ingress/payload-processing-e2e-migrate-d74c1bd9-7c87cbbf88-2j9tg"
Oct 30 19:44:03.115021 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:44:03.074127    4892 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vj45m\" (UniqueName: \"kubernetes.io/projected/79ae5280-77c7-451b-bd82-ade64806cdd7-kube-api-access-vj45m\") pod \"payload-processing-e2e-migrate-d74c1bd9-7c87cbbf88-2j9tg\" (UID: \"79ae5280-77c7-451b-bd82-ade64806cdd7\") " pod="openshift-ingress/payload-processing-e2e-migrate-d74c1bd9-7c87cbbf88-2j9tg"
Oct 30 19:44:03.178879 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:44:03.178839    4892 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/79ae5280-77c7-451b-bd82-ade64806cdd7-plugins-config-volume\") pod \"payload-processing-e2e-migrate-d74c1bd9-7c87cbbf88-2j9tg\" (UID: \"79ae5280-77c7-451b-bd82-ade64806cdd7\") " pod="openshift-ingress/payload-processing-e2e-migrate-d74c1bd9-7c87cbbf88-2j9tg"
Oct 30 19:44:03.179010 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:44:03.178919    4892 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vj45m\" (UniqueName: \"kubernetes.io/projected/79ae5280-77c7-451b-bd82-ade64806cdd7-kube-api-access-vj45m\") pod \"payload-processing-e2e-migrate-d74c1bd9-7c87cbbf88-2j9tg\" (UID: \"79ae5280-77c7-451b-bd82-ade64806cdd7\") " pod="openshift-ingress/payload-processing-e2e-migrate-d74c1bd9-7c87cbbf88-2j9tg"
Oct 30 19:44:03.180256 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:44:03.180231    4892 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/79ae5280-77c7-451b-bd82-ade64806cdd7-plugins-config-volume\") pod \"payload-processing-e2e-migrate-d74c1bd9-7c87cbbf88-2j9tg\" (UID: \"79ae5280-77c7-451b-bd82-ade64806cdd7\") " pod="openshift-ingress/payload-processing-e2e-migrate-d74c1bd9-7c87cbbf88-2j9tg"
Oct 30 19:44:03.239000 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:44:03.238455    4892 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vj45m\" (UniqueName: \"kubernetes.io/projected/79ae5280-77c7-451b-bd82-ade64806cdd7-kube-api-access-vj45m\") pod \"payload-processing-e2e-migrate-d74c1bd9-7c87cbbf88-2j9tg\" (UID: \"79ae5280-77c7-451b-bd82-ade64806cdd7\") " pod="openshift-ingress/payload-processing-e2e-migrate-d74c1bd9-7c87cbbf88-2j9tg"
Oct 30 19:44:03.436613 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:44:03.436584    4892 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-migrate-d74c1bd9-7c87cbbf88-2j9tg"
Oct 30 19:44:03.544138 ip-10-0-132-82 kubenswrapper[4892]: W1030 19:44:03.544103    4892 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod28480114_29bf_4642_8da2_a7e714ba3460.slice/crio-a3acfc7ccdc5bac38140ceeda6c8549cf3efb74ef09bede38d8a76e191e5fec7 WatchSource:0}: Error finding container a3acfc7ccdc5bac38140ceeda6c8549cf3efb74ef09bede38d8a76e191e5fec7: Status 404 returned error can't find the container with id a3acfc7ccdc5bac38140ceeda6c8549cf3efb74ef09bede38d8a76e191e5fec7
Oct 30 19:44:03.545557 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:44:03.545528    4892 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-migrate-d74c1bd9-5f777c6ff-qbwsw"]
Oct 30 19:44:03.823066 ip-10-0-132-82 kubenswrapper[4892]: W1030 19:44:03.823032    4892 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod79ae5280_77c7_451b_bd82_ade64806cdd7.slice/crio-308ff9019c9f212fb1f96eb28ac25a246d143c053660d531cc08bbc739fb0c32 WatchSource:0}: Error finding container 308ff9019c9f212fb1f96eb28ac25a246d143c053660d531cc08bbc739fb0c32: Status 404 returned error can't find the container with id 308ff9019c9f212fb1f96eb28ac25a246d143c053660d531cc08bbc739fb0c32
Oct 30 19:44:03.823904 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:44:03.823877    4892 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-migrate-d74c1bd9-7c87cbbf88-2j9tg"]
Oct 30 19:44:04.003677 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:44:04.003645    4892 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-migrate-d74c1bd9-5f777c6ff-qbwsw" event={"ID":"28480114-29bf-4642-8da2-a7e714ba3460","Type":"ContainerStarted","Data":"c6a83984b85dacf4ffe0b4e353826977e934b3bd8a85c4cc66d48f2b58d42aeb"}
Oct 30 19:44:04.003942 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:44:04.003701    4892 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-migrate-d74c1bd9-5f777c6ff-qbwsw" event={"ID":"28480114-29bf-4642-8da2-a7e714ba3460","Type":"ContainerStarted","Data":"a3acfc7ccdc5bac38140ceeda6c8549cf3efb74ef09bede38d8a76e191e5fec7"}
Oct 30 19:44:04.003942 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:44:04.003731    4892 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-migrate-d74c1bd9-5f777c6ff-qbwsw"
Oct 30 19:44:04.005325 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:44:04.005300    4892 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-migrate-d74c1bd9-7c87cbbf88-2j9tg" event={"ID":"79ae5280-77c7-451b-bd82-ade64806cdd7","Type":"ContainerStarted","Data":"af8b326ce902f6443927be0567200322cbe282bd21efd5d4aab337d071585235"}
Oct 30 19:44:04.005466 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:44:04.005343    4892 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-migrate-d74c1bd9-7c87cbbf88-2j9tg" event={"ID":"79ae5280-77c7-451b-bd82-ade64806cdd7","Type":"ContainerStarted","Data":"308ff9019c9f212fb1f96eb28ac25a246d143c053660d531cc08bbc739fb0c32"}
Oct 30 19:44:04.005466 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:44:04.005370    4892 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-migrate-d74c1bd9-7c87cbbf88-2j9tg"
Oct 30 19:44:04.122277 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:44:04.122171    4892 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-migrate-d74c1bd9-5f777c6ff-qbwsw" podStartSLOduration=2.122153004 podStartE2EDuration="2.122153004s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:44:02 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:44:04.11736437 +0000 UTC m=+1691.687829968" watchObservedRunningTime="2026-10-30 19:44:04.122153004 +0000 UTC m=+1691.692618604"
Oct 30 19:44:04.260377 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:44:04.258066    4892 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-migrate-d74c1bd9-7c87cbbf88-2j9tg" podStartSLOduration=2.257940662 podStartE2EDuration="2.257940662s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:44:02 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:44:04.255696104 +0000 UTC m=+1691.826161720" watchObservedRunningTime="2026-10-30 19:44:04.257940662 +0000 UTC m=+1691.828406263"
Oct 30 19:44:06.847797 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:44:06.847762    4892 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-ait-94581ab2-688dd7dd6-rxx2h"
Oct 30 19:44:09.057389 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:44:09.057355    4892 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c4294e09-5b4hh"]
Oct 30 19:44:09.106078 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:44:09.106050    4892 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c4294e09-5b4hh"]
Oct 30 19:44:10.570126 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:44:10.570093    4892 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="59b8e731-59dc-4dd9-a71c-04425e64db5b" path="/var/lib/kubelet/pods/59b8e731-59dc-4dd9-a71c-04425e64db5b/volumes"
Oct 30 19:44:10.709330 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:44:10.709304    4892 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3884c8de-l6x8q"]
Oct 30 19:44:10.714489 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:44:10.714470    4892 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3884c8de-l6x8q"
Oct 30 19:44:10.775531 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:44:10.775500    4892 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3884c8de-l6x8q"]
Oct 30 19:44:10.777688 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:44:10.777654    4892 pod_container_manager_linux.go:218] "Failed to delete cgroup paths" cgroupName=["kubepods","besteffort","pod93d8a0c9-c95b-481c-9700-5609fe407cdc"] err="unable to destroy cgroup paths for cgroup [kubepods besteffort pod93d8a0c9-c95b-481c-9700-5609fe407cdc] : Timed out while waiting for systemd to remove kubepods-besteffort-pod93d8a0c9_c95b_481c_9700_5609fe407cdc.slice"
Oct 30 19:44:10.777688 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:44:10.777684    4892 kubelet_pods.go:2797] "Failed to destroy orphaned pod cgroup" podUID="93d8a0c9-c95b-481c-9700-5609fe407cdc" err="failed to delete cgroup paths for [kubepods besteffort pod93d8a0c9-c95b-481c-9700-5609fe407cdc] : unable to destroy cgroup paths for cgroup [kubepods besteffort pod93d8a0c9-c95b-481c-9700-5609fe407cdc] : Timed out while waiting for systemd to remove kubepods-besteffort-pod93d8a0c9_c95b_481c_9700_5609fe407cdc.slice"
Oct 30 19:44:10.782623 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:44:10.782594    4892 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-grsnr\" (UniqueName: \"kubernetes.io/projected/ef9ff75d-23ce-4a8c-9271-0dd12f292e00-kube-api-access-grsnr\") pod \"maas-api-revoke-sub-3884c8de-l6x8q\" (UID: \"ef9ff75d-23ce-4a8c-9271-0dd12f292e00\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3884c8de-l6x8q"
Oct 30 19:44:10.782769 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:44:10.782725    4892 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ef9ff75d-23ce-4a8c-9271-0dd12f292e00-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3884c8de-l6x8q\" (UID: \"ef9ff75d-23ce-4a8c-9271-0dd12f292e00\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3884c8de-l6x8q"
Oct 30 19:44:10.782842 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:44:10.782787    4892 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ef9ff75d-23ce-4a8c-9271-0dd12f292e00-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3884c8de-l6x8q\" (UID: \"ef9ff75d-23ce-4a8c-9271-0dd12f292e00\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3884c8de-l6x8q"
Oct 30 19:44:10.818037 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:44:10.818001    4892 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 19:44:10.818037 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:44:10.818015    4892 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 30 19:44:10.818037 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:44:10.818026    4892 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-nlxp4\""
Oct 30 19:44:10.818230 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:44:10.818072    4892 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 30 19:44:10.883942 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:44:10.883911    4892 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-grsnr\" (UniqueName: \"kubernetes.io/projected/ef9ff75d-23ce-4a8c-9271-0dd12f292e00-kube-api-access-grsnr\") pod \"maas-api-revoke-sub-3884c8de-l6x8q\" (UID: \"ef9ff75d-23ce-4a8c-9271-0dd12f292e00\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3884c8de-l6x8q"
Oct 30 19:44:10.884109 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:44:10.883996    4892 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ef9ff75d-23ce-4a8c-9271-0dd12f292e00-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3884c8de-l6x8q\" (UID: \"ef9ff75d-23ce-4a8c-9271-0dd12f292e00\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3884c8de-l6x8q"
Oct 30 19:44:10.884109 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:44:10.884032    4892 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ef9ff75d-23ce-4a8c-9271-0dd12f292e00-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3884c8de-l6x8q\" (UID: \"ef9ff75d-23ce-4a8c-9271-0dd12f292e00\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3884c8de-l6x8q"
Oct 30 19:44:10.884751 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:44:10.884731    4892 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ef9ff75d-23ce-4a8c-9271-0dd12f292e00-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3884c8de-l6x8q\" (UID: \"ef9ff75d-23ce-4a8c-9271-0dd12f292e00\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3884c8de-l6x8q"
Oct 30 19:44:10.886684 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:44:10.886660    4892 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ef9ff75d-23ce-4a8c-9271-0dd12f292e00-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3884c8de-l6x8q\" (UID: \"ef9ff75d-23ce-4a8c-9271-0dd12f292e00\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3884c8de-l6x8q"
Oct 30 19:44:10.944117 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:44:10.944087    4892 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-grsnr\" (UniqueName: \"kubernetes.io/projected/ef9ff75d-23ce-4a8c-9271-0dd12f292e00-kube-api-access-grsnr\") pod \"maas-api-revoke-sub-3884c8de-l6x8q\" (UID: \"ef9ff75d-23ce-4a8c-9271-0dd12f292e00\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3884c8de-l6x8q"
Oct 30 19:44:11.028811 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:44:11.028779    4892 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3884c8de-l6x8q"
Oct 30 19:44:11.419267 ip-10-0-132-82 kubenswrapper[4892]: W1030 19:44:11.419234    4892 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podef9ff75d_23ce_4a8c_9271_0dd12f292e00.slice/crio-2d0dd243d9625545af7d16bcb4a02fd3fd3ee040c786ab349a0e13bdc9e3abf5 WatchSource:0}: Error finding container 2d0dd243d9625545af7d16bcb4a02fd3fd3ee040c786ab349a0e13bdc9e3abf5: Status 404 returned error can't find the container with id 2d0dd243d9625545af7d16bcb4a02fd3fd3ee040c786ab349a0e13bdc9e3abf5
Oct 30 19:44:11.426064 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:44:11.424315    4892 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3884c8de-l6x8q"]
Oct 30 19:44:12.082179 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:44:12.082093    4892 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3884c8de-l6x8q" event={"ID":"ef9ff75d-23ce-4a8c-9271-0dd12f292e00","Type":"ContainerStarted","Data":"bea3091d950188a93bb395ce0766482893c449df4c5fa38957304cb79b74db1f"}
Oct 30 19:44:12.082179 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:44:12.082154    4892 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3884c8de-l6x8q" event={"ID":"ef9ff75d-23ce-4a8c-9271-0dd12f292e00","Type":"ContainerStarted","Data":"2d0dd243d9625545af7d16bcb4a02fd3fd3ee040c786ab349a0e13bdc9e3abf5"}
Oct 30 19:44:12.250816 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:44:12.250768    4892 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3884c8de-l6x8q" podStartSLOduration=2.250754109 podStartE2EDuration="2.250754109s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:44:10 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:44:12.198756888 +0000 UTC m=+1699.769222482" watchObservedRunningTime="2026-10-30 19:44:12.250754109 +0000 UTC m=+1699.821219736"
Oct 30 19:44:12.591756 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:44:12.591721    4892 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-fsfvk"]
Oct 30 19:44:13.092446 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:44:13.092416    4892 generic.go:356] "Generic (PLEG): container finished" podID="ef9ff75d-23ce-4a8c-9271-0dd12f292e00" containerID="bea3091d950188a93bb395ce0766482893c449df4c5fa38957304cb79b74db1f" exitCode=0
Oct 30 19:44:13.092836 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:44:13.092503    4892 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3884c8de-l6x8q" event={"ID":"ef9ff75d-23ce-4a8c-9271-0dd12f292e00","Type":"ContainerDied","Data":"bea3091d950188a93bb395ce0766482893c449df4c5fa38957304cb79b74db1f"}
Oct 30 19:44:14.957041 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:44:14.943218    4892 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-686689/distinct-2-w5-686689-kserve-5ff578b4cd-rkc6s" podUID="a8829732-ac6f-4238-b8ce-fc781084668b" containerName="main" probeResult="failure" output="Get \"https://10.132.0.87:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:44:15.065351 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:44:15.056598    4892 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-2d9861/distinct-2-w0-2d9861-kserve-679c6b56dd-nmhbm" podUID="65b0586a-6593-48cc-adf8-8ff48417ece4" containerName="main" probeResult="failure" output="Get \"https://10.132.0.133:8000/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:44:15.072987 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:44:15.070197    4892 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-migrate-d74c1bd9-5f777c6ff-qbwsw"
Oct 30 19:44:15.096816 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:44:15.074333    4892 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-migrate-d74c1bd9-7c87cbbf88-2j9tg"
Oct 30 19:44:15.118897 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:44:14.933499    4892 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-a3f9b7/simulator-w2-a3f9b7-kserve-79bbc4cb5f-8m488" podUID="50fb2283-bf3f-41da-835b-f407ee1a04f3" containerName="main" probeResult="failure" output="Get \"https://10.132.0.59:8000/ready\": context deadline exceeded"
Oct 30 19:44:15.281348 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:44:15.278238    4892 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-ada8d7/simulator-w3-ada8d7-kserve-7d575cfcb5-5txgn" podUID="10a5c7c8-6146-47b0-9643-bb137cf10bf5" containerName="main" probeResult="failure" output="Get \"https://10.132.0.55:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:44:15.319991 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:44:15.299044    4892 patch_prober.go:29] interesting pod/e2e-worker-w1-d52df0-openshift-default-6d5f887b45-2f4xc container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.88:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 30 19:44:15.320311 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:44:15.320223    4892 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w1-d52df0-openshift-default-6d5f887b45-2f4xc" podUID="6befc94b-40e5-42a4-adf5-392b5ecde123" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.88:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:44:15.387231 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:44:15.387183    4892 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-b5178f/distinct-2-w4-b5178f-kserve-69ddf8b665-9pdth" podUID="fb71d961-8500-4367-b48c-88d385962bd5" containerName="main" probeResult="failure" output="Get \"https://10.132.0.90:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:44:15.492995 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:44:15.449307    4892 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-32855537-vqhss"]
Oct 30 19:44:15.589104 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:44:15.510596    4892 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3884c8de-l6x8q"
Oct 30 19:44:15.589104 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:44:15.586715    4892 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-5f55cd/simulator-w6-5f55cd-kserve-5ff86b7447-h9tm7" podUID="3cb47a99-8be1-4756-b31a-43b58cc6b8dc" containerName="main" probeResult="failure" output="Get \"https://10.132.0.58:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:44:15.625122 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:44:15.616673    4892 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-32855537-vqhss"]
Oct 30 19:44:15.649940 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:44:15.649628    4892 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3884c8de-l6x8q" event={"ID":"ef9ff75d-23ce-4a8c-9271-0dd12f292e00","Type":"ContainerDied","Data":"2d0dd243d9625545af7d16bcb4a02fd3fd3ee040c786ab349a0e13bdc9e3abf5"}
Oct 30 19:44:15.649940 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:44:15.649695    4892 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="2d0dd243d9625545af7d16bcb4a02fd3fd3ee040c786ab349a0e13bdc9e3abf5"
Oct 30 19:44:15.649940 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:44:15.649768    4892 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3884c8de-l6x8q"
Oct 30 19:44:15.718823 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:44:15.717788    4892 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ef9ff75d-23ce-4a8c-9271-0dd12f292e00-kube-api-access-grsnr\" (UniqueName: \"kubernetes.io/projected/ef9ff75d-23ce-4a8c-9271-0dd12f292e00-kube-api-access-grsnr\") pod \"ef9ff75d-23ce-4a8c-9271-0dd12f292e00\" (UID: \"ef9ff75d-23ce-4a8c-9271-0dd12f292e00\") "
Oct 30 19:44:15.718823 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:44:15.717907    4892 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ef9ff75d-23ce-4a8c-9271-0dd12f292e00-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ef9ff75d-23ce-4a8c-9271-0dd12f292e00-maas-api-openshift-ca\") pod \"ef9ff75d-23ce-4a8c-9271-0dd12f292e00\" (UID: \"ef9ff75d-23ce-4a8c-9271-0dd12f292e00\") "
Oct 30 19:44:15.718823 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:44:15.718019    4892 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/ef9ff75d-23ce-4a8c-9271-0dd12f292e00-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ef9ff75d-23ce-4a8c-9271-0dd12f292e00-maas-api-xks-ca\") pod \"ef9ff75d-23ce-4a8c-9271-0dd12f292e00\" (UID: \"ef9ff75d-23ce-4a8c-9271-0dd12f292e00\") "
Oct 30 19:44:15.730862 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:44:15.730828    4892 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ef9ff75d-23ce-4a8c-9271-0dd12f292e00-maas-api-openshift-ca" pod "ef9ff75d-23ce-4a8c-9271-0dd12f292e00" (UID: "ef9ff75d-23ce-4a8c-9271-0dd12f292e00"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 19:44:15.857125 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:44:15.820815    4892 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ef9ff75d-23ce-4a8c-9271-0dd12f292e00-maas-api-openshift-ca\") on node \"ip-10-0-132-82.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 19:44:16.003524 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:44:15.989815    4892 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-2d9861/premium-w0-2d9861-kserve-67945c6848-4n6kz" podUID="798988b2-c769-4f73-a7c8-ca15de0beef1" containerName="main" probeResult="failure" output="Get \"https://10.132.0.62:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:44:16.003524 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:44:15.990027    4892 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-d52df0/simulator-w1-d52df0-kserve-67f74bc577-q5bjz" podUID="81b7bb84-1003-4452-b7e0-b4aaba6fefa4" containerName="main" probeResult="failure" output="Get \"https://10.132.0.57:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:44:16.003524 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:44:15.993294    4892 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ef9ff75d-23ce-4a8c-9271-0dd12f292e00-kube-api-access-grsnr" pod "ef9ff75d-23ce-4a8c-9271-0dd12f292e00" (UID: "ef9ff75d-23ce-4a8c-9271-0dd12f292e00"). InnerVolumeSpecName "kube-api-access-grsnr". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:44:16.003524 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:44:15.996773    4892 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-686689/distinct-2-w5-686689-kserve-5ff578b4cd-rkc6s" podUID="a8829732-ac6f-4238-b8ce-fc781084668b" containerName="main" probeResult="failure" output="Get \"https://10.132.0.87:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:44:16.019016 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:44:16.018555    4892 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w5-686689/distinct-2-w5-686689-kserve-5ff578b4cd-rkc6s"
Oct 30 19:44:16.023986 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:44:16.020956    4892 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-grsnr\" (UniqueName: \"kubernetes.io/projected/ef9ff75d-23ce-4a8c-9271-0dd12f292e00-kube-api-access-grsnr\") on node \"ip-10-0-132-82.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 19:44:16.059320 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:44:16.059276    4892 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-2d9861/distinct-2-w0-2d9861-kserve-679c6b56dd-nmhbm" podUID="65b0586a-6593-48cc-adf8-8ff48417ece4" containerName="main" probeResult="failure" output="Get \"https://10.132.0.133:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:44:16.062586 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:44:16.062564    4892 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-2d9861/distinct-2-w0-2d9861-kserve-679c6b56dd-nmhbm"
Oct 30 19:44:16.111282 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:44:16.111206    4892 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ef9ff75d-23ce-4a8c-9271-0dd12f292e00-maas-api-xks-ca" pod "ef9ff75d-23ce-4a8c-9271-0dd12f292e00" (UID: "ef9ff75d-23ce-4a8c-9271-0dd12f292e00"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 19:44:16.232992 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:44:16.206161    4892 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-5f55cd/premium-w6-5f55cd-kserve-7f785c544b-pcvhb" podUID="191593a2-e3a9-4ce4-a1c6-c089723b0237" containerName="main" probeResult="failure" output="Get \"https://10.132.0.60:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:44:16.290130 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:44:16.279061    4892 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-ada8d7/simulator-w3-ada8d7-kserve-7d575cfcb5-5txgn" podUID="10a5c7c8-6146-47b0-9643-bb137cf10bf5" containerName="main" probeResult="failure" output="Get \"https://10.132.0.55:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:44:16.290130 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:44:16.279638    4892 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-ada8d7/simulator-w3-ada8d7-kserve-7d575cfcb5-5txgn"
Oct 30 19:44:16.352100 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:44:16.336264    4892 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ef9ff75d-23ce-4a8c-9271-0dd12f292e00-maas-api-xks-ca\") on node \"ip-10-0-132-82.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 19:44:16.409998 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:44:16.393927    4892 patch_prober.go:29] interesting pod/e2e-worker-w1-d52df0-openshift-default-6d5f887b45-lxbtl container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.102:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 30 19:44:16.409998 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:44:16.394000    4892 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w1-d52df0-openshift-default-6d5f887b45-lxbtl" podUID="dc4774a0-3da8-4c65-a0b3-887a476ab465" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.102:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:44:16.433001 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:44:16.420365    4892 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-2d9861/distinct-2-w0-2d9861-kserve-679c6b56dd-nmhbm"
Oct 30 19:44:16.503419 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:44:16.503392    4892 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w5-686689/distinct-2-w5-686689-kserve-5ff578b4cd-rkc6s"
Oct 30 19:44:16.711844 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:44:16.711792    4892 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-b5178f/distinct-2-w4-b5178f-kserve-69ddf8b665-9pdth" podUID="fb71d961-8500-4367-b48c-88d385962bd5" containerName="main" probeResult="failure" output="Get \"https://10.132.0.90:8000/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:44:16.959995 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:44:16.959942    4892 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="5bbe30fd-2c9a-475d-976f-77a7e8c41401" path="/var/lib/kubelet/pods/5bbe30fd-2c9a-475d-976f-77a7e8c41401/volumes"
Oct 30 19:44:16.961209 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:44:16.961190    4892 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-b5178f/distinct-2-w4-b5178f-kserve-69ddf8b665-9pdth"
Oct 30 19:44:16.961710 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:44:16.961693    4892 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-ada8d7/simulator-w3-ada8d7-kserve-7d575cfcb5-5txgn"
Oct 30 19:44:16.962887 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:44:16.962837    4892 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-b5178f/distinct-2-w4-b5178f-kserve-69ddf8b665-9pdth"
Oct 30 19:44:22.677190 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:44:22.677162    4892 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w3-ada8d7-openshift-default-5f8b49bdfc-pkd8j"]
Oct 30 19:44:22.677740 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:44:22.677726    4892 cpu_manager.go:414] "containerMap: removing container" podUID="ef9ff75d-23ce-4a8c-9271-0dd12f292e00" containerName="revoke-keys"
Oct 30 19:44:22.677740 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:44:22.677741    4892 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ef9ff75d-23ce-4a8c-9271-0dd12f292e00" containerName="revoke-keys"
Oct 30 19:44:22.677853 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:44:22.677843    4892 memory_manager.go:365] "RemoveStaleState removing state" podUID="ef9ff75d-23ce-4a8c-9271-0dd12f292e00" containerName="revoke-keys"
Oct 30 19:44:22.682897 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:44:22.682876    4892 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-ada8d7-openshift-default-5f8b49bdfc-pkd8j"
Oct 30 19:44:22.778660 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:44:22.778630    4892 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-x9k2k\" (UniqueName: \"kubernetes.io/projected/fcc8f06d-659a-42e9-9ee8-4390a2591f5b-kube-api-access-x9k2k\") pod \"e2e-worker-w3-ada8d7-openshift-default-5f8b49bdfc-pkd8j\" (UID: \"fcc8f06d-659a-42e9-9ee8-4390a2591f5b\") " pod="openshift-ingress/e2e-worker-w3-ada8d7-openshift-default-5f8b49bdfc-pkd8j"
Oct 30 19:44:22.778847 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:44:22.778784    4892 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/fcc8f06d-659a-42e9-9ee8-4390a2591f5b-istio-podinfo\") pod \"e2e-worker-w3-ada8d7-openshift-default-5f8b49bdfc-pkd8j\" (UID: \"fcc8f06d-659a-42e9-9ee8-4390a2591f5b\") " pod="openshift-ingress/e2e-worker-w3-ada8d7-openshift-default-5f8b49bdfc-pkd8j"
Oct 30 19:44:22.778847 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:44:22.778842    4892 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/fcc8f06d-659a-42e9-9ee8-4390a2591f5b-istio-envoy\") pod \"e2e-worker-w3-ada8d7-openshift-default-5f8b49bdfc-pkd8j\" (UID: \"fcc8f06d-659a-42e9-9ee8-4390a2591f5b\") " pod="openshift-ingress/e2e-worker-w3-ada8d7-openshift-default-5f8b49bdfc-pkd8j"
Oct 30 19:44:22.779000 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:44:22.778920    4892 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/fcc8f06d-659a-42e9-9ee8-4390a2591f5b-istio-token\") pod \"e2e-worker-w3-ada8d7-openshift-default-5f8b49bdfc-pkd8j\" (UID: \"fcc8f06d-659a-42e9-9ee8-4390a2591f5b\") " pod="openshift-ingress/e2e-worker-w3-ada8d7-openshift-default-5f8b49bdfc-pkd8j"
Oct 30 19:44:22.779095 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:44:22.778954    4892 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/fcc8f06d-659a-42e9-9ee8-4390a2591f5b-credential-socket\") pod \"e2e-worker-w3-ada8d7-openshift-default-5f8b49bdfc-pkd8j\" (UID: \"fcc8f06d-659a-42e9-9ee8-4390a2591f5b\") " pod="openshift-ingress/e2e-worker-w3-ada8d7-openshift-default-5f8b49bdfc-pkd8j"
Oct 30 19:44:22.779095 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:44:22.779079    4892 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/fcc8f06d-659a-42e9-9ee8-4390a2591f5b-workload-socket\") pod \"e2e-worker-w3-ada8d7-openshift-default-5f8b49bdfc-pkd8j\" (UID: \"fcc8f06d-659a-42e9-9ee8-4390a2591f5b\") " pod="openshift-ingress/e2e-worker-w3-ada8d7-openshift-default-5f8b49bdfc-pkd8j"
Oct 30 19:44:22.779199 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:44:22.779160    4892 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/fcc8f06d-659a-42e9-9ee8-4390a2591f5b-istiod-ca-cert\") pod \"e2e-worker-w3-ada8d7-openshift-default-5f8b49bdfc-pkd8j\" (UID: \"fcc8f06d-659a-42e9-9ee8-4390a2591f5b\") " pod="openshift-ingress/e2e-worker-w3-ada8d7-openshift-default-5f8b49bdfc-pkd8j"
Oct 30 19:44:22.779270 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:44:22.779252    4892 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/fcc8f06d-659a-42e9-9ee8-4390a2591f5b-istio-data\") pod \"e2e-worker-w3-ada8d7-openshift-default-5f8b49bdfc-pkd8j\" (UID: \"fcc8f06d-659a-42e9-9ee8-4390a2591f5b\") " pod="openshift-ingress/e2e-worker-w3-ada8d7-openshift-default-5f8b49bdfc-pkd8j"
Oct 30 19:44:22.779323 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:44:22.779284    4892 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/fcc8f06d-659a-42e9-9ee8-4390a2591f5b-workload-certs\") pod \"e2e-worker-w3-ada8d7-openshift-default-5f8b49bdfc-pkd8j\" (UID: \"fcc8f06d-659a-42e9-9ee8-4390a2591f5b\") " pod="openshift-ingress/e2e-worker-w3-ada8d7-openshift-default-5f8b49bdfc-pkd8j"
Oct 30 19:44:22.790381 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:44:22.790354    4892 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-ada8d7-openshift-default-5f8b49bdfc-pkd8j"]
Oct 30 19:44:22.880209 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:44:22.880181    4892 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/fcc8f06d-659a-42e9-9ee8-4390a2591f5b-workload-socket\") pod \"e2e-worker-w3-ada8d7-openshift-default-5f8b49bdfc-pkd8j\" (UID: \"fcc8f06d-659a-42e9-9ee8-4390a2591f5b\") " pod="openshift-ingress/e2e-worker-w3-ada8d7-openshift-default-5f8b49bdfc-pkd8j"
Oct 30 19:44:22.880209 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:44:22.880213    4892 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/fcc8f06d-659a-42e9-9ee8-4390a2591f5b-istiod-ca-cert\") pod \"e2e-worker-w3-ada8d7-openshift-default-5f8b49bdfc-pkd8j\" (UID: \"fcc8f06d-659a-42e9-9ee8-4390a2591f5b\") " pod="openshift-ingress/e2e-worker-w3-ada8d7-openshift-default-5f8b49bdfc-pkd8j"
Oct 30 19:44:22.880470 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:44:22.880251    4892 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/fcc8f06d-659a-42e9-9ee8-4390a2591f5b-istio-data\") pod \"e2e-worker-w3-ada8d7-openshift-default-5f8b49bdfc-pkd8j\" (UID: \"fcc8f06d-659a-42e9-9ee8-4390a2591f5b\") " pod="openshift-ingress/e2e-worker-w3-ada8d7-openshift-default-5f8b49bdfc-pkd8j"
Oct 30 19:44:22.880470 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:44:22.880271    4892 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/fcc8f06d-659a-42e9-9ee8-4390a2591f5b-workload-certs\") pod \"e2e-worker-w3-ada8d7-openshift-default-5f8b49bdfc-pkd8j\" (UID: \"fcc8f06d-659a-42e9-9ee8-4390a2591f5b\") " pod="openshift-ingress/e2e-worker-w3-ada8d7-openshift-default-5f8b49bdfc-pkd8j"
Oct 30 19:44:22.880470 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:44:22.880361    4892 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-x9k2k\" (UniqueName: \"kubernetes.io/projected/fcc8f06d-659a-42e9-9ee8-4390a2591f5b-kube-api-access-x9k2k\") pod \"e2e-worker-w3-ada8d7-openshift-default-5f8b49bdfc-pkd8j\" (UID: \"fcc8f06d-659a-42e9-9ee8-4390a2591f5b\") " pod="openshift-ingress/e2e-worker-w3-ada8d7-openshift-default-5f8b49bdfc-pkd8j"
Oct 30 19:44:22.880470 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:44:22.880443    4892 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/fcc8f06d-659a-42e9-9ee8-4390a2591f5b-istio-podinfo\") pod \"e2e-worker-w3-ada8d7-openshift-default-5f8b49bdfc-pkd8j\" (UID: \"fcc8f06d-659a-42e9-9ee8-4390a2591f5b\") " pod="openshift-ingress/e2e-worker-w3-ada8d7-openshift-default-5f8b49bdfc-pkd8j"
Oct 30 19:44:22.880696 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:44:22.880494    4892 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/fcc8f06d-659a-42e9-9ee8-4390a2591f5b-istio-envoy\") pod \"e2e-worker-w3-ada8d7-openshift-default-5f8b49bdfc-pkd8j\" (UID: \"fcc8f06d-659a-42e9-9ee8-4390a2591f5b\") " pod="openshift-ingress/e2e-worker-w3-ada8d7-openshift-default-5f8b49bdfc-pkd8j"
Oct 30 19:44:22.880696 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:44:22.880542    4892 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/fcc8f06d-659a-42e9-9ee8-4390a2591f5b-istio-token\") pod \"e2e-worker-w3-ada8d7-openshift-default-5f8b49bdfc-pkd8j\" (UID: \"fcc8f06d-659a-42e9-9ee8-4390a2591f5b\") " pod="openshift-ingress/e2e-worker-w3-ada8d7-openshift-default-5f8b49bdfc-pkd8j"
Oct 30 19:44:22.880696 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:44:22.880575    4892 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/fcc8f06d-659a-42e9-9ee8-4390a2591f5b-credential-socket\") pod \"e2e-worker-w3-ada8d7-openshift-default-5f8b49bdfc-pkd8j\" (UID: \"fcc8f06d-659a-42e9-9ee8-4390a2591f5b\") " pod="openshift-ingress/e2e-worker-w3-ada8d7-openshift-default-5f8b49bdfc-pkd8j"
Oct 30 19:44:22.880696 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:44:22.880671    4892 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/fcc8f06d-659a-42e9-9ee8-4390a2591f5b-workload-socket\") pod \"e2e-worker-w3-ada8d7-openshift-default-5f8b49bdfc-pkd8j\" (UID: \"fcc8f06d-659a-42e9-9ee8-4390a2591f5b\") " pod="openshift-ingress/e2e-worker-w3-ada8d7-openshift-default-5f8b49bdfc-pkd8j"
Oct 30 19:44:22.880914 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:44:22.880810    4892 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/fcc8f06d-659a-42e9-9ee8-4390a2591f5b-istio-data\") pod \"e2e-worker-w3-ada8d7-openshift-default-5f8b49bdfc-pkd8j\" (UID: \"fcc8f06d-659a-42e9-9ee8-4390a2591f5b\") " pod="openshift-ingress/e2e-worker-w3-ada8d7-openshift-default-5f8b49bdfc-pkd8j"
Oct 30 19:44:22.880914 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:44:22.880819    4892 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/fcc8f06d-659a-42e9-9ee8-4390a2591f5b-workload-certs\") pod \"e2e-worker-w3-ada8d7-openshift-default-5f8b49bdfc-pkd8j\" (UID: \"fcc8f06d-659a-42e9-9ee8-4390a2591f5b\") " pod="openshift-ingress/e2e-worker-w3-ada8d7-openshift-default-5f8b49bdfc-pkd8j"
Oct 30 19:44:22.881070 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:44:22.881014    4892 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/fcc8f06d-659a-42e9-9ee8-4390a2591f5b-credential-socket\") pod \"e2e-worker-w3-ada8d7-openshift-default-5f8b49bdfc-pkd8j\" (UID: \"fcc8f06d-659a-42e9-9ee8-4390a2591f5b\") " pod="openshift-ingress/e2e-worker-w3-ada8d7-openshift-default-5f8b49bdfc-pkd8j"
Oct 30 19:44:22.881070 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:44:22.881041    4892 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/fcc8f06d-659a-42e9-9ee8-4390a2591f5b-istiod-ca-cert\") pod \"e2e-worker-w3-ada8d7-openshift-default-5f8b49bdfc-pkd8j\" (UID: \"fcc8f06d-659a-42e9-9ee8-4390a2591f5b\") " pod="openshift-ingress/e2e-worker-w3-ada8d7-openshift-default-5f8b49bdfc-pkd8j"
Oct 30 19:44:22.883001 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:44:22.882949    4892 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/fcc8f06d-659a-42e9-9ee8-4390a2591f5b-istio-envoy\") pod \"e2e-worker-w3-ada8d7-openshift-default-5f8b49bdfc-pkd8j\" (UID: \"fcc8f06d-659a-42e9-9ee8-4390a2591f5b\") " pod="openshift-ingress/e2e-worker-w3-ada8d7-openshift-default-5f8b49bdfc-pkd8j"
Oct 30 19:44:22.883402 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:44:22.883366    4892 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/fcc8f06d-659a-42e9-9ee8-4390a2591f5b-istio-podinfo\") pod \"e2e-worker-w3-ada8d7-openshift-default-5f8b49bdfc-pkd8j\" (UID: \"fcc8f06d-659a-42e9-9ee8-4390a2591f5b\") " pod="openshift-ingress/e2e-worker-w3-ada8d7-openshift-default-5f8b49bdfc-pkd8j"
Oct 30 19:44:22.946520 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:44:22.946457    4892 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-x9k2k\" (UniqueName: \"kubernetes.io/projected/fcc8f06d-659a-42e9-9ee8-4390a2591f5b-kube-api-access-x9k2k\") pod \"e2e-worker-w3-ada8d7-openshift-default-5f8b49bdfc-pkd8j\" (UID: \"fcc8f06d-659a-42e9-9ee8-4390a2591f5b\") " pod="openshift-ingress/e2e-worker-w3-ada8d7-openshift-default-5f8b49bdfc-pkd8j"
Oct 30 19:44:22.947505 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:44:22.947483    4892 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/fcc8f06d-659a-42e9-9ee8-4390a2591f5b-istio-token\") pod \"e2e-worker-w3-ada8d7-openshift-default-5f8b49bdfc-pkd8j\" (UID: \"fcc8f06d-659a-42e9-9ee8-4390a2591f5b\") " pod="openshift-ingress/e2e-worker-w3-ada8d7-openshift-default-5f8b49bdfc-pkd8j"
Oct 30 19:44:22.995861 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:44:22.995836    4892 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-ada8d7-openshift-default-5f8b49bdfc-pkd8j"
Oct 30 19:44:23.435431 ip-10-0-132-82 kubenswrapper[4892]: W1030 19:44:23.435405    4892 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfcc8f06d_659a_42e9_9ee8_4390a2591f5b.slice/crio-a122e028e90ced031b5a7154a9b4dffe9025cb8ed9fea71a815712d597041556 WatchSource:0}: Error finding container a122e028e90ced031b5a7154a9b4dffe9025cb8ed9fea71a815712d597041556: Status 404 returned error can't find the container with id a122e028e90ced031b5a7154a9b4dffe9025cb8ed9fea71a815712d597041556
Oct 30 19:44:23.437284 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:44:23.437259    4892 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-ada8d7-openshift-default-5f8b49bdfc-pkd8j"]
Oct 30 19:44:23.438481 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:44:23.438447    4892 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892000Ki","pods":"250"}
Oct 30 19:44:23.438552 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:44:23.438520    4892 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892000Ki","pods":"250"}
Oct 30 19:44:23.438591 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:44:23.438550    4892 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892000Ki","pods":"250"}
Oct 30 19:44:23.976385 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:44:23.976336    4892 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-ada8d7-openshift-default-5f8b49bdfc-pkd8j" event={"ID":"fcc8f06d-659a-42e9-9ee8-4390a2591f5b","Type":"ContainerStarted","Data":"be881d28f681b44ed2009fa0469ff21b153a84e2544d23cd03188c1351c5c990"}
Oct 30 19:44:23.976907 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:44:23.976440    4892 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-ada8d7-openshift-default-5f8b49bdfc-pkd8j" event={"ID":"fcc8f06d-659a-42e9-9ee8-4390a2591f5b","Type":"ContainerStarted","Data":"a122e028e90ced031b5a7154a9b4dffe9025cb8ed9fea71a815712d597041556"}
Oct 30 19:44:23.996877 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:44:23.996843    4892 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w3-ada8d7-openshift-default-5f8b49bdfc-pkd8j"
Oct 30 19:44:24.098260 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:44:24.098206    4892 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w3-ada8d7-openshift-default-5f8b49bdfc-pkd8j" podStartSLOduration=2.098190501 podStartE2EDuration="2.098190501s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:44:22 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:44:24.093334456 +0000 UTC m=+1711.663800055" watchObservedRunningTime="2026-10-30 19:44:24.098190501 +0000 UTC m=+1711.668656100"
Oct 30 19:44:24.874620 ip-10-0-132-82 kubenswrapper[4892]: E1030 19:44:24.874585    4892 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-jrm87" containerName="extractor"
Oct 30 19:44:24.998708 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:44:24.998673    4892 patch_prober.go:29] interesting pod/e2e-worker-w3-ada8d7-openshift-default-5f8b49bdfc-pkd8j container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.139:15021/healthz/ready\": dial tcp 10.132.0.139:15021: connect: connection refused" start-of-body=
Oct 30 19:44:24.999100 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:44:24.998725    4892 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-ada8d7-openshift-default-5f8b49bdfc-pkd8j" podUID="fcc8f06d-659a-42e9-9ee8-4390a2591f5b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.139:15021/healthz/ready\": dial tcp 10.132.0.139:15021: connect: connection refused"
Oct 30 19:44:25.996799 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:44:25.996764    4892 patch_prober.go:29] interesting pod/e2e-worker-w3-ada8d7-openshift-default-5f8b49bdfc-pkd8j container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.139:15021/healthz/ready\": dial tcp 10.132.0.139:15021: connect: connection refused" start-of-body=
Oct 30 19:44:25.997012 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:44:25.996827    4892 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-ada8d7-openshift-default-5f8b49bdfc-pkd8j" podUID="fcc8f06d-659a-42e9-9ee8-4390a2591f5b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.139:15021/healthz/ready\": dial tcp 10.132.0.139:15021: connect: connection refused"
Oct 30 19:44:26.079655 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:44:26.079614    4892 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-6517618f-nph8j"]
Oct 30 19:44:26.111996 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:44:26.111951    4892 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-6517618f-nph8j"]
Oct 30 19:44:26.572693 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:44:26.572662    4892 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ad5c24d2-e4cc-4737-aab6-bf77769262e5" path="/var/lib/kubelet/pods/ad5c24d2-e4cc-4737-aab6-bf77769262e5/volumes"
Oct 30 19:44:26.996674 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:44:26.996635    4892 patch_prober.go:29] interesting pod/e2e-worker-w3-ada8d7-openshift-default-5f8b49bdfc-pkd8j container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.139:15021/healthz/ready\": dial tcp 10.132.0.139:15021: connect: connection refused" start-of-body=
Oct 30 19:44:26.996893 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:44:26.996694    4892 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-ada8d7-openshift-default-5f8b49bdfc-pkd8j" podUID="fcc8f06d-659a-42e9-9ee8-4390a2591f5b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.139:15021/healthz/ready\": dial tcp 10.132.0.139:15021: connect: connection refused"
Oct 30 19:44:27.996957 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:44:27.996920    4892 patch_prober.go:29] interesting pod/e2e-worker-w3-ada8d7-openshift-default-5f8b49bdfc-pkd8j container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.139:15021/healthz/ready\": dial tcp 10.132.0.139:15021: connect: connection refused" start-of-body=
Oct 30 19:44:27.997369 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:44:27.996995    4892 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-ada8d7-openshift-default-5f8b49bdfc-pkd8j" podUID="fcc8f06d-659a-42e9-9ee8-4390a2591f5b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.139:15021/healthz/ready\": dial tcp 10.132.0.139:15021: connect: connection refused"
Oct 30 19:44:28.996366 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:44:28.996332    4892 patch_prober.go:29] interesting pod/e2e-worker-w3-ada8d7-openshift-default-5f8b49bdfc-pkd8j container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.139:15021/healthz/ready\": dial tcp 10.132.0.139:15021: connect: connection refused" start-of-body=
Oct 30 19:44:28.996528 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:44:28.996393    4892 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-ada8d7-openshift-default-5f8b49bdfc-pkd8j" podUID="fcc8f06d-659a-42e9-9ee8-4390a2591f5b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.139:15021/healthz/ready\": dial tcp 10.132.0.139:15021: connect: connection refused"
Oct 30 19:44:29.996954 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:44:29.996919    4892 patch_prober.go:29] interesting pod/e2e-worker-w3-ada8d7-openshift-default-5f8b49bdfc-pkd8j container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.139:15021/healthz/ready\": dial tcp 10.132.0.139:15021: connect: connection refused" start-of-body=
Oct 30 19:44:29.997354 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:44:29.996985    4892 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-ada8d7-openshift-default-5f8b49bdfc-pkd8j" podUID="fcc8f06d-659a-42e9-9ee8-4390a2591f5b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.139:15021/healthz/ready\": dial tcp 10.132.0.139:15021: connect: connection refused"
Oct 30 19:44:31.000822 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:44:31.000783    4892 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w3-ada8d7-openshift-default-5f8b49bdfc-pkd8j"
Oct 30 19:44:31.001355 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:44:31.001102    4892 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w3-ada8d7-openshift-default-5f8b49bdfc-pkd8j"
Oct 30 19:44:31.001876 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:44:31.001858    4892 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w3-ada8d7-openshift-default-5f8b49bdfc-pkd8j"
Oct 30 19:44:32.634410 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:44:32.634376    4892 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-lgwz5"]
Oct 30 19:44:32.651721 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:44:32.651687    4892 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-lgwz5"
Oct 30 19:44:32.750840 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:44:32.750800    4892 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c63df543-fcb1-4bee-9d52-c50286ac6f91-workload-socket\") pod \"maas-default-gateway-openshift-default-6db586dd6c-lgwz5\" (UID: \"c63df543-fcb1-4bee-9d52-c50286ac6f91\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-lgwz5"
Oct 30 19:44:32.751021 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:44:32.750850    4892 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c63df543-fcb1-4bee-9d52-c50286ac6f91-istio-envoy\") pod \"maas-default-gateway-openshift-default-6db586dd6c-lgwz5\" (UID: \"c63df543-fcb1-4bee-9d52-c50286ac6f91\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-lgwz5"
Oct 30 19:44:32.751169 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:44:32.751153    4892 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c63df543-fcb1-4bee-9d52-c50286ac6f91-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-6db586dd6c-lgwz5\" (UID: \"c63df543-fcb1-4bee-9d52-c50286ac6f91\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-lgwz5"
Oct 30 19:44:32.751289 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:44:32.751264    4892 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c63df543-fcb1-4bee-9d52-c50286ac6f91-istio-data\") pod \"maas-default-gateway-openshift-default-6db586dd6c-lgwz5\" (UID: \"c63df543-fcb1-4bee-9d52-c50286ac6f91\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-lgwz5"
Oct 30 19:44:32.751332 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:44:32.751304    4892 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c63df543-fcb1-4bee-9d52-c50286ac6f91-istio-podinfo\") pod \"maas-default-gateway-openshift-default-6db586dd6c-lgwz5\" (UID: \"c63df543-fcb1-4bee-9d52-c50286ac6f91\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-lgwz5"
Oct 30 19:44:32.751489 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:44:32.751467    4892 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pqlj4\" (UniqueName: \"kubernetes.io/projected/c63df543-fcb1-4bee-9d52-c50286ac6f91-kube-api-access-pqlj4\") pod \"maas-default-gateway-openshift-default-6db586dd6c-lgwz5\" (UID: \"c63df543-fcb1-4bee-9d52-c50286ac6f91\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-lgwz5"
Oct 30 19:44:32.751563 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:44:32.751532    4892 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c63df543-fcb1-4bee-9d52-c50286ac6f91-istio-token\") pod \"maas-default-gateway-openshift-default-6db586dd6c-lgwz5\" (UID: \"c63df543-fcb1-4bee-9d52-c50286ac6f91\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-lgwz5"
Oct 30 19:44:32.751803 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:44:32.751597    4892 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c63df543-fcb1-4bee-9d52-c50286ac6f91-credential-socket\") pod \"maas-default-gateway-openshift-default-6db586dd6c-lgwz5\" (UID: \"c63df543-fcb1-4bee-9d52-c50286ac6f91\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-lgwz5"
Oct 30 19:44:32.751803 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:44:32.751678    4892 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c63df543-fcb1-4bee-9d52-c50286ac6f91-workload-certs\") pod \"maas-default-gateway-openshift-default-6db586dd6c-lgwz5\" (UID: \"c63df543-fcb1-4bee-9d52-c50286ac6f91\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-lgwz5"
Oct 30 19:44:32.769638 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:44:32.769617    4892 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"maas-default-gateway-openshift-default-dockercfg-f2npg\""
Oct 30 19:44:32.782639 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:44:32.782604    4892 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-lgwz5"]
Oct 30 19:44:32.830602 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:44:32.830571    4892 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w0-2d9861-openshift-default-58ddf574b9-w2r9m"]
Oct 30 19:44:32.835781 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:44:32.835758    4892 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w0-2d9861-openshift-default-58ddf574b9-lrwlb"]
Oct 30 19:44:32.835933 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:44:32.835883    4892 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-2d9861-openshift-default-58ddf574b9-w2r9m"
Oct 30 19:44:32.841611 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:44:32.841590    4892 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-2d9861-openshift-default-58ddf574b9-lrwlb"
Oct 30 19:44:32.858693 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:44:32.858665    4892 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c63df543-fcb1-4bee-9d52-c50286ac6f91-workload-certs\") pod \"maas-default-gateway-openshift-default-6db586dd6c-lgwz5\" (UID: \"c63df543-fcb1-4bee-9d52-c50286ac6f91\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-lgwz5"
Oct 30 19:44:32.858822 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:44:32.858728    4892 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0f4fa4be-451c-43b5-bc35-cdf507a2dfb3-workload-socket\") pod \"e2e-worker-w0-2d9861-openshift-default-58ddf574b9-w2r9m\" (UID: \"0f4fa4be-451c-43b5-bc35-cdf507a2dfb3\") " pod="openshift-ingress/e2e-worker-w0-2d9861-openshift-default-58ddf574b9-w2r9m"
Oct 30 19:44:32.859440 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:44:32.859396    4892 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0f4fa4be-451c-43b5-bc35-cdf507a2dfb3-workload-certs\") pod \"e2e-worker-w0-2d9861-openshift-default-58ddf574b9-w2r9m\" (UID: \"0f4fa4be-451c-43b5-bc35-cdf507a2dfb3\") " pod="openshift-ingress/e2e-worker-w0-2d9861-openshift-default-58ddf574b9-w2r9m"
Oct 30 19:44:32.859545 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:44:32.859463    4892 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0f4fa4be-451c-43b5-bc35-cdf507a2dfb3-istio-data\") pod \"e2e-worker-w0-2d9861-openshift-default-58ddf574b9-w2r9m\" (UID: \"0f4fa4be-451c-43b5-bc35-cdf507a2dfb3\") " pod="openshift-ingress/e2e-worker-w0-2d9861-openshift-default-58ddf574b9-w2r9m"
Oct 30 19:44:32.859545 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:44:32.859505    4892 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0f4fa4be-451c-43b5-bc35-cdf507a2dfb3-istio-token\") pod \"e2e-worker-w0-2d9861-openshift-default-58ddf574b9-w2r9m\" (UID: \"0f4fa4be-451c-43b5-bc35-cdf507a2dfb3\") " pod="openshift-ingress/e2e-worker-w0-2d9861-openshift-default-58ddf574b9-w2r9m"
Oct 30 19:44:32.859671 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:44:32.859584    4892 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c63df543-fcb1-4bee-9d52-c50286ac6f91-workload-socket\") pod \"maas-default-gateway-openshift-default-6db586dd6c-lgwz5\" (UID: \"c63df543-fcb1-4bee-9d52-c50286ac6f91\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-lgwz5"
Oct 30 19:44:32.859671 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:44:32.859614    4892 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c63df543-fcb1-4bee-9d52-c50286ac6f91-istio-envoy\") pod \"maas-default-gateway-openshift-default-6db586dd6c-lgwz5\" (UID: \"c63df543-fcb1-4bee-9d52-c50286ac6f91\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-lgwz5"
Oct 30 19:44:32.859671 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:44:32.859660    4892 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0f4fa4be-451c-43b5-bc35-cdf507a2dfb3-istio-podinfo\") pod \"e2e-worker-w0-2d9861-openshift-default-58ddf574b9-w2r9m\" (UID: \"0f4fa4be-451c-43b5-bc35-cdf507a2dfb3\") " pod="openshift-ingress/e2e-worker-w0-2d9861-openshift-default-58ddf574b9-w2r9m"
Oct 30 19:44:32.859844 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:44:32.859740    4892 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c63df543-fcb1-4bee-9d52-c50286ac6f91-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-6db586dd6c-lgwz5\" (UID: \"c63df543-fcb1-4bee-9d52-c50286ac6f91\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-lgwz5"
Oct 30 19:44:32.859844 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:44:32.859812    4892 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c63df543-fcb1-4bee-9d52-c50286ac6f91-istio-data\") pod \"maas-default-gateway-openshift-default-6db586dd6c-lgwz5\" (UID: \"c63df543-fcb1-4bee-9d52-c50286ac6f91\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-lgwz5"
Oct 30 19:44:32.859844 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:44:32.859838    4892 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c63df543-fcb1-4bee-9d52-c50286ac6f91-istio-podinfo\") pod \"maas-default-gateway-openshift-default-6db586dd6c-lgwz5\" (UID: \"c63df543-fcb1-4bee-9d52-c50286ac6f91\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-lgwz5"
Oct 30 19:44:32.860077 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:44:32.859904    4892 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tbvsg\" (UniqueName: \"kubernetes.io/projected/0f4fa4be-451c-43b5-bc35-cdf507a2dfb3-kube-api-access-tbvsg\") pod \"e2e-worker-w0-2d9861-openshift-default-58ddf574b9-w2r9m\" (UID: \"0f4fa4be-451c-43b5-bc35-cdf507a2dfb3\") " pod="openshift-ingress/e2e-worker-w0-2d9861-openshift-default-58ddf574b9-w2r9m"
Oct 30 19:44:32.860077 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:44:32.859998    4892 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0f4fa4be-451c-43b5-bc35-cdf507a2dfb3-istiod-ca-cert\") pod \"e2e-worker-w0-2d9861-openshift-default-58ddf574b9-w2r9m\" (UID: \"0f4fa4be-451c-43b5-bc35-cdf507a2dfb3\") " pod="openshift-ingress/e2e-worker-w0-2d9861-openshift-default-58ddf574b9-w2r9m"
Oct 30 19:44:32.860198 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:44:32.860080    4892 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0f4fa4be-451c-43b5-bc35-cdf507a2dfb3-istio-envoy\") pod \"e2e-worker-w0-2d9861-openshift-default-58ddf574b9-w2r9m\" (UID: \"0f4fa4be-451c-43b5-bc35-cdf507a2dfb3\") " pod="openshift-ingress/e2e-worker-w0-2d9861-openshift-default-58ddf574b9-w2r9m"
Oct 30 19:44:32.860198 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:44:32.860123    4892 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pqlj4\" (UniqueName: \"kubernetes.io/projected/c63df543-fcb1-4bee-9d52-c50286ac6f91-kube-api-access-pqlj4\") pod \"maas-default-gateway-openshift-default-6db586dd6c-lgwz5\" (UID: \"c63df543-fcb1-4bee-9d52-c50286ac6f91\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-lgwz5"
Oct 30 19:44:32.860198 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:44:32.860173    4892 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0f4fa4be-451c-43b5-bc35-cdf507a2dfb3-credential-socket\") pod \"e2e-worker-w0-2d9861-openshift-default-58ddf574b9-w2r9m\" (UID: \"0f4fa4be-451c-43b5-bc35-cdf507a2dfb3\") " pod="openshift-ingress/e2e-worker-w0-2d9861-openshift-default-58ddf574b9-w2r9m"
Oct 30 19:44:32.860352 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:44:32.860259    4892 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c63df543-fcb1-4bee-9d52-c50286ac6f91-istio-token\") pod \"maas-default-gateway-openshift-default-6db586dd6c-lgwz5\" (UID: \"c63df543-fcb1-4bee-9d52-c50286ac6f91\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-lgwz5"
Oct 30 19:44:32.860352 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:44:32.860312    4892 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c63df543-fcb1-4bee-9d52-c50286ac6f91-credential-socket\") pod \"maas-default-gateway-openshift-default-6db586dd6c-lgwz5\" (UID: \"c63df543-fcb1-4bee-9d52-c50286ac6f91\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-lgwz5"
Oct 30 19:44:32.861295 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:44:32.861014    4892 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c63df543-fcb1-4bee-9d52-c50286ac6f91-credential-socket\") pod \"maas-default-gateway-openshift-default-6db586dd6c-lgwz5\" (UID: \"c63df543-fcb1-4bee-9d52-c50286ac6f91\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-lgwz5"
Oct 30 19:44:32.861777 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:44:32.861736    4892 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c63df543-fcb1-4bee-9d52-c50286ac6f91-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-6db586dd6c-lgwz5\" (UID: \"c63df543-fcb1-4bee-9d52-c50286ac6f91\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-lgwz5"
Oct 30 19:44:32.862482 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:44:32.862460    4892 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c63df543-fcb1-4bee-9d52-c50286ac6f91-istio-data\") pod \"maas-default-gateway-openshift-default-6db586dd6c-lgwz5\" (UID: \"c63df543-fcb1-4bee-9d52-c50286ac6f91\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-lgwz5"
Oct 30 19:44:32.863149 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:44:32.863122    4892 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c63df543-fcb1-4bee-9d52-c50286ac6f91-workload-certs\") pod \"maas-default-gateway-openshift-default-6db586dd6c-lgwz5\" (UID: \"c63df543-fcb1-4bee-9d52-c50286ac6f91\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-lgwz5"
Oct 30 19:44:32.863446 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:44:32.863419    4892 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c63df543-fcb1-4bee-9d52-c50286ac6f91-workload-socket\") pod \"maas-default-gateway-openshift-default-6db586dd6c-lgwz5\" (UID: \"c63df543-fcb1-4bee-9d52-c50286ac6f91\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-lgwz5"
Oct 30 19:44:32.869714 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:44:32.869688    4892 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c63df543-fcb1-4bee-9d52-c50286ac6f91-istio-envoy\") pod \"maas-default-gateway-openshift-default-6db586dd6c-lgwz5\" (UID: \"c63df543-fcb1-4bee-9d52-c50286ac6f91\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-lgwz5"
Oct 30 19:44:32.871933 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:44:32.871908    4892 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c63df543-fcb1-4bee-9d52-c50286ac6f91-istio-podinfo\") pod \"maas-default-gateway-openshift-default-6db586dd6c-lgwz5\" (UID: \"c63df543-fcb1-4bee-9d52-c50286ac6f91\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-lgwz5"
Oct 30 19:44:32.902265 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:44:32.902195    4892 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-2d9861-openshift-default-58ddf574b9-w2r9m"]
Oct 30 19:44:32.909312 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:44:32.909264    4892 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-2d9861-openshift-default-58ddf574b9-lrwlb"]
Oct 30 19:44:32.925568 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:44:32.925520    4892 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-pqlj4\" (UniqueName: \"kubernetes.io/projected/c63df543-fcb1-4bee-9d52-c50286ac6f91-kube-api-access-pqlj4\") pod \"maas-default-gateway-openshift-default-6db586dd6c-lgwz5\" (UID: \"c63df543-fcb1-4bee-9d52-c50286ac6f91\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-lgwz5"
Oct 30 19:44:32.926212 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:44:32.926174    4892 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c63df543-fcb1-4bee-9d52-c50286ac6f91-istio-token\") pod \"maas-default-gateway-openshift-default-6db586dd6c-lgwz5\" (UID: \"c63df543-fcb1-4bee-9d52-c50286ac6f91\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-lgwz5"
Oct 30 19:44:32.961250 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:44:32.961224    4892 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/fd2378f0-12b6-4a3a-a526-beb05ec13e02-istio-envoy\") pod \"e2e-worker-w0-2d9861-openshift-default-58ddf574b9-lrwlb\" (UID: \"fd2378f0-12b6-4a3a-a526-beb05ec13e02\") " pod="openshift-ingress/e2e-worker-w0-2d9861-openshift-default-58ddf574b9-lrwlb"
Oct 30 19:44:32.961383 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:44:32.961287    4892 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/fd2378f0-12b6-4a3a-a526-beb05ec13e02-workload-certs\") pod \"e2e-worker-w0-2d9861-openshift-default-58ddf574b9-lrwlb\" (UID: \"fd2378f0-12b6-4a3a-a526-beb05ec13e02\") " pod="openshift-ingress/e2e-worker-w0-2d9861-openshift-default-58ddf574b9-lrwlb"
Oct 30 19:44:32.961383 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:44:32.961350    4892 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0f4fa4be-451c-43b5-bc35-cdf507a2dfb3-workload-socket\") pod \"e2e-worker-w0-2d9861-openshift-default-58ddf574b9-w2r9m\" (UID: \"0f4fa4be-451c-43b5-bc35-cdf507a2dfb3\") " pod="openshift-ingress/e2e-worker-w0-2d9861-openshift-default-58ddf574b9-w2r9m"
Oct 30 19:44:32.961506 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:44:32.961401    4892 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0f4fa4be-451c-43b5-bc35-cdf507a2dfb3-workload-certs\") pod \"e2e-worker-w0-2d9861-openshift-default-58ddf574b9-w2r9m\" (UID: \"0f4fa4be-451c-43b5-bc35-cdf507a2dfb3\") " pod="openshift-ingress/e2e-worker-w0-2d9861-openshift-default-58ddf574b9-w2r9m"
Oct 30 19:44:32.961506 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:44:32.961421    4892 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0f4fa4be-451c-43b5-bc35-cdf507a2dfb3-istio-data\") pod \"e2e-worker-w0-2d9861-openshift-default-58ddf574b9-w2r9m\" (UID: \"0f4fa4be-451c-43b5-bc35-cdf507a2dfb3\") " pod="openshift-ingress/e2e-worker-w0-2d9861-openshift-default-58ddf574b9-w2r9m"
Oct 30 19:44:32.961506 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:44:32.961437    4892 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0f4fa4be-451c-43b5-bc35-cdf507a2dfb3-istio-token\") pod \"e2e-worker-w0-2d9861-openshift-default-58ddf574b9-w2r9m\" (UID: \"0f4fa4be-451c-43b5-bc35-cdf507a2dfb3\") " pod="openshift-ingress/e2e-worker-w0-2d9861-openshift-default-58ddf574b9-w2r9m"
Oct 30 19:44:32.961506 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:44:32.961466    4892 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/fd2378f0-12b6-4a3a-a526-beb05ec13e02-istio-token\") pod \"e2e-worker-w0-2d9861-openshift-default-58ddf574b9-lrwlb\" (UID: \"fd2378f0-12b6-4a3a-a526-beb05ec13e02\") " pod="openshift-ingress/e2e-worker-w0-2d9861-openshift-default-58ddf574b9-lrwlb"
Oct 30 19:44:32.961686 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:44:32.961509    4892 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0f4fa4be-451c-43b5-bc35-cdf507a2dfb3-istio-podinfo\") pod \"e2e-worker-w0-2d9861-openshift-default-58ddf574b9-w2r9m\" (UID: \"0f4fa4be-451c-43b5-bc35-cdf507a2dfb3\") " pod="openshift-ingress/e2e-worker-w0-2d9861-openshift-default-58ddf574b9-w2r9m"
Oct 30 19:44:32.961686 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:44:32.961539    4892 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/fd2378f0-12b6-4a3a-a526-beb05ec13e02-istio-data\") pod \"e2e-worker-w0-2d9861-openshift-default-58ddf574b9-lrwlb\" (UID: \"fd2378f0-12b6-4a3a-a526-beb05ec13e02\") " pod="openshift-ingress/e2e-worker-w0-2d9861-openshift-default-58ddf574b9-lrwlb"
Oct 30 19:44:32.961686 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:44:32.961557    4892 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/fd2378f0-12b6-4a3a-a526-beb05ec13e02-istio-podinfo\") pod \"e2e-worker-w0-2d9861-openshift-default-58ddf574b9-lrwlb\" (UID: \"fd2378f0-12b6-4a3a-a526-beb05ec13e02\") " pod="openshift-ingress/e2e-worker-w0-2d9861-openshift-default-58ddf574b9-lrwlb"
Oct 30 19:44:32.961686 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:44:32.961600    4892 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/fd2378f0-12b6-4a3a-a526-beb05ec13e02-credential-socket\") pod \"e2e-worker-w0-2d9861-openshift-default-58ddf574b9-lrwlb\" (UID: \"fd2378f0-12b6-4a3a-a526-beb05ec13e02\") " pod="openshift-ingress/e2e-worker-w0-2d9861-openshift-default-58ddf574b9-lrwlb"
Oct 30 19:44:32.961686 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:44:32.961649    4892 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/fd2378f0-12b6-4a3a-a526-beb05ec13e02-istiod-ca-cert\") pod \"e2e-worker-w0-2d9861-openshift-default-58ddf574b9-lrwlb\" (UID: \"fd2378f0-12b6-4a3a-a526-beb05ec13e02\") " pod="openshift-ingress/e2e-worker-w0-2d9861-openshift-default-58ddf574b9-lrwlb"
Oct 30 19:44:32.961686 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:44:32.961669    4892 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tbvsg\" (UniqueName: \"kubernetes.io/projected/0f4fa4be-451c-43b5-bc35-cdf507a2dfb3-kube-api-access-tbvsg\") pod \"e2e-worker-w0-2d9861-openshift-default-58ddf574b9-w2r9m\" (UID: \"0f4fa4be-451c-43b5-bc35-cdf507a2dfb3\") " pod="openshift-ingress/e2e-worker-w0-2d9861-openshift-default-58ddf574b9-w2r9m"
Oct 30 19:44:32.961924 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:44:32.961699    4892 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-54k6d\" (UniqueName: \"kubernetes.io/projected/fd2378f0-12b6-4a3a-a526-beb05ec13e02-kube-api-access-54k6d\") pod \"e2e-worker-w0-2d9861-openshift-default-58ddf574b9-lrwlb\" (UID: \"fd2378f0-12b6-4a3a-a526-beb05ec13e02\") " pod="openshift-ingress/e2e-worker-w0-2d9861-openshift-default-58ddf574b9-lrwlb"
Oct 30 19:44:32.961924 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:44:32.961727    4892 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0f4fa4be-451c-43b5-bc35-cdf507a2dfb3-istiod-ca-cert\") pod \"e2e-worker-w0-2d9861-openshift-default-58ddf574b9-w2r9m\" (UID: \"0f4fa4be-451c-43b5-bc35-cdf507a2dfb3\") " pod="openshift-ingress/e2e-worker-w0-2d9861-openshift-default-58ddf574b9-w2r9m"
Oct 30 19:44:32.961924 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:44:32.961770    4892 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0f4fa4be-451c-43b5-bc35-cdf507a2dfb3-istio-envoy\") pod \"e2e-worker-w0-2d9861-openshift-default-58ddf574b9-w2r9m\" (UID: \"0f4fa4be-451c-43b5-bc35-cdf507a2dfb3\") " pod="openshift-ingress/e2e-worker-w0-2d9861-openshift-default-58ddf574b9-w2r9m"
Oct 30 19:44:32.961924 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:44:32.961796    4892 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0f4fa4be-451c-43b5-bc35-cdf507a2dfb3-credential-socket\") pod \"e2e-worker-w0-2d9861-openshift-default-58ddf574b9-w2r9m\" (UID: \"0f4fa4be-451c-43b5-bc35-cdf507a2dfb3\") " pod="openshift-ingress/e2e-worker-w0-2d9861-openshift-default-58ddf574b9-w2r9m"
Oct 30 19:44:32.961924 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:44:32.961820    4892 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0f4fa4be-451c-43b5-bc35-cdf507a2dfb3-workload-certs\") pod \"e2e-worker-w0-2d9861-openshift-default-58ddf574b9-w2r9m\" (UID: \"0f4fa4be-451c-43b5-bc35-cdf507a2dfb3\") " pod="openshift-ingress/e2e-worker-w0-2d9861-openshift-default-58ddf574b9-w2r9m"
Oct 30 19:44:32.961924 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:44:32.961835    4892 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/fd2378f0-12b6-4a3a-a526-beb05ec13e02-workload-socket\") pod \"e2e-worker-w0-2d9861-openshift-default-58ddf574b9-lrwlb\" (UID: \"fd2378f0-12b6-4a3a-a526-beb05ec13e02\") " pod="openshift-ingress/e2e-worker-w0-2d9861-openshift-default-58ddf574b9-lrwlb"
Oct 30 19:44:32.962262 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:44:32.962138    4892 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0f4fa4be-451c-43b5-bc35-cdf507a2dfb3-istio-data\") pod \"e2e-worker-w0-2d9861-openshift-default-58ddf574b9-w2r9m\" (UID: \"0f4fa4be-451c-43b5-bc35-cdf507a2dfb3\") " pod="openshift-ingress/e2e-worker-w0-2d9861-openshift-default-58ddf574b9-w2r9m"
Oct 30 19:44:32.962262 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:44:32.962203    4892 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0f4fa4be-451c-43b5-bc35-cdf507a2dfb3-workload-socket\") pod \"e2e-worker-w0-2d9861-openshift-default-58ddf574b9-w2r9m\" (UID: \"0f4fa4be-451c-43b5-bc35-cdf507a2dfb3\") " pod="openshift-ingress/e2e-worker-w0-2d9861-openshift-default-58ddf574b9-w2r9m"
Oct 30 19:44:32.962919 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:44:32.962890    4892 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0f4fa4be-451c-43b5-bc35-cdf507a2dfb3-credential-socket\") pod \"e2e-worker-w0-2d9861-openshift-default-58ddf574b9-w2r9m\" (UID: \"0f4fa4be-451c-43b5-bc35-cdf507a2dfb3\") " pod="openshift-ingress/e2e-worker-w0-2d9861-openshift-default-58ddf574b9-w2r9m"
Oct 30 19:44:32.963040 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:44:32.962958    4892 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0f4fa4be-451c-43b5-bc35-cdf507a2dfb3-istiod-ca-cert\") pod \"e2e-worker-w0-2d9861-openshift-default-58ddf574b9-w2r9m\" (UID: \"0f4fa4be-451c-43b5-bc35-cdf507a2dfb3\") " pod="openshift-ingress/e2e-worker-w0-2d9861-openshift-default-58ddf574b9-w2r9m"
Oct 30 19:44:32.965071 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:44:32.965006    4892 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0f4fa4be-451c-43b5-bc35-cdf507a2dfb3-istio-envoy\") pod \"e2e-worker-w0-2d9861-openshift-default-58ddf574b9-w2r9m\" (UID: \"0f4fa4be-451c-43b5-bc35-cdf507a2dfb3\") " pod="openshift-ingress/e2e-worker-w0-2d9861-openshift-default-58ddf574b9-w2r9m"
Oct 30 19:44:32.966203 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:44:32.966180    4892 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0f4fa4be-451c-43b5-bc35-cdf507a2dfb3-istio-podinfo\") pod \"e2e-worker-w0-2d9861-openshift-default-58ddf574b9-w2r9m\" (UID: \"0f4fa4be-451c-43b5-bc35-cdf507a2dfb3\") " pod="openshift-ingress/e2e-worker-w0-2d9861-openshift-default-58ddf574b9-w2r9m"
Oct 30 19:44:32.970246 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:44:32.970227    4892 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-lgwz5"
Oct 30 19:44:33.020571 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:44:33.020534    4892 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tbvsg\" (UniqueName: \"kubernetes.io/projected/0f4fa4be-451c-43b5-bc35-cdf507a2dfb3-kube-api-access-tbvsg\") pod \"e2e-worker-w0-2d9861-openshift-default-58ddf574b9-w2r9m\" (UID: \"0f4fa4be-451c-43b5-bc35-cdf507a2dfb3\") " pod="openshift-ingress/e2e-worker-w0-2d9861-openshift-default-58ddf574b9-w2r9m"
Oct 30 19:44:33.020721 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:44:33.020704    4892 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0f4fa4be-451c-43b5-bc35-cdf507a2dfb3-istio-token\") pod \"e2e-worker-w0-2d9861-openshift-default-58ddf574b9-w2r9m\" (UID: \"0f4fa4be-451c-43b5-bc35-cdf507a2dfb3\") " pod="openshift-ingress/e2e-worker-w0-2d9861-openshift-default-58ddf574b9-w2r9m"
Oct 30 19:44:33.065919 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:44:33.064055    4892 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/fd2378f0-12b6-4a3a-a526-beb05ec13e02-workload-socket\") pod \"e2e-worker-w0-2d9861-openshift-default-58ddf574b9-lrwlb\" (UID: \"fd2378f0-12b6-4a3a-a526-beb05ec13e02\") " pod="openshift-ingress/e2e-worker-w0-2d9861-openshift-default-58ddf574b9-lrwlb"
Oct 30 19:44:33.065919 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:44:33.064478    4892 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/fd2378f0-12b6-4a3a-a526-beb05ec13e02-workload-socket\") pod \"e2e-worker-w0-2d9861-openshift-default-58ddf574b9-lrwlb\" (UID: \"fd2378f0-12b6-4a3a-a526-beb05ec13e02\") " pod="openshift-ingress/e2e-worker-w0-2d9861-openshift-default-58ddf574b9-lrwlb"
Oct 30 19:44:33.065919 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:44:33.064583    4892 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/fd2378f0-12b6-4a3a-a526-beb05ec13e02-istio-envoy\") pod \"e2e-worker-w0-2d9861-openshift-default-58ddf574b9-lrwlb\" (UID: \"fd2378f0-12b6-4a3a-a526-beb05ec13e02\") " pod="openshift-ingress/e2e-worker-w0-2d9861-openshift-default-58ddf574b9-lrwlb"
Oct 30 19:44:33.065919 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:44:33.064653    4892 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/fd2378f0-12b6-4a3a-a526-beb05ec13e02-workload-certs\") pod \"e2e-worker-w0-2d9861-openshift-default-58ddf574b9-lrwlb\" (UID: \"fd2378f0-12b6-4a3a-a526-beb05ec13e02\") " pod="openshift-ingress/e2e-worker-w0-2d9861-openshift-default-58ddf574b9-lrwlb"
Oct 30 19:44:33.065919 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:44:33.064743    4892 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/fd2378f0-12b6-4a3a-a526-beb05ec13e02-istio-token\") pod \"e2e-worker-w0-2d9861-openshift-default-58ddf574b9-lrwlb\" (UID: \"fd2378f0-12b6-4a3a-a526-beb05ec13e02\") " pod="openshift-ingress/e2e-worker-w0-2d9861-openshift-default-58ddf574b9-lrwlb"
Oct 30 19:44:33.065919 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:44:33.064818    4892 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/fd2378f0-12b6-4a3a-a526-beb05ec13e02-istio-data\") pod \"e2e-worker-w0-2d9861-openshift-default-58ddf574b9-lrwlb\" (UID: \"fd2378f0-12b6-4a3a-a526-beb05ec13e02\") " pod="openshift-ingress/e2e-worker-w0-2d9861-openshift-default-58ddf574b9-lrwlb"
Oct 30 19:44:33.065919 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:44:33.064845    4892 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/fd2378f0-12b6-4a3a-a526-beb05ec13e02-istio-podinfo\") pod \"e2e-worker-w0-2d9861-openshift-default-58ddf574b9-lrwlb\" (UID: \"fd2378f0-12b6-4a3a-a526-beb05ec13e02\") " pod="openshift-ingress/e2e-worker-w0-2d9861-openshift-default-58ddf574b9-lrwlb"
Oct 30 19:44:33.065919 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:44:33.064895    4892 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/fd2378f0-12b6-4a3a-a526-beb05ec13e02-credential-socket\") pod \"e2e-worker-w0-2d9861-openshift-default-58ddf574b9-lrwlb\" (UID: \"fd2378f0-12b6-4a3a-a526-beb05ec13e02\") " pod="openshift-ingress/e2e-worker-w0-2d9861-openshift-default-58ddf574b9-lrwlb"
Oct 30 19:44:33.065919 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:44:33.064954    4892 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/fd2378f0-12b6-4a3a-a526-beb05ec13e02-istiod-ca-cert\") pod \"e2e-worker-w0-2d9861-openshift-default-58ddf574b9-lrwlb\" (UID: \"fd2378f0-12b6-4a3a-a526-beb05ec13e02\") " pod="openshift-ingress/e2e-worker-w0-2d9861-openshift-default-58ddf574b9-lrwlb"
Oct 30 19:44:33.065919 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:44:33.065015    4892 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-54k6d\" (UniqueName: \"kubernetes.io/projected/fd2378f0-12b6-4a3a-a526-beb05ec13e02-kube-api-access-54k6d\") pod \"e2e-worker-w0-2d9861-openshift-default-58ddf574b9-lrwlb\" (UID: \"fd2378f0-12b6-4a3a-a526-beb05ec13e02\") " pod="openshift-ingress/e2e-worker-w0-2d9861-openshift-default-58ddf574b9-lrwlb"
Oct 30 19:44:33.066556 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:44:33.066271    4892 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/fd2378f0-12b6-4a3a-a526-beb05ec13e02-istio-data\") pod \"e2e-worker-w0-2d9861-openshift-default-58ddf574b9-lrwlb\" (UID: \"fd2378f0-12b6-4a3a-a526-beb05ec13e02\") " pod="openshift-ingress/e2e-worker-w0-2d9861-openshift-default-58ddf574b9-lrwlb"
Oct 30 19:44:33.066556 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:44:33.066328    4892 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/fd2378f0-12b6-4a3a-a526-beb05ec13e02-credential-socket\") pod \"e2e-worker-w0-2d9861-openshift-default-58ddf574b9-lrwlb\" (UID: \"fd2378f0-12b6-4a3a-a526-beb05ec13e02\") " pod="openshift-ingress/e2e-worker-w0-2d9861-openshift-default-58ddf574b9-lrwlb"
Oct 30 19:44:33.066556 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:44:33.066519    4892 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/fd2378f0-12b6-4a3a-a526-beb05ec13e02-workload-certs\") pod \"e2e-worker-w0-2d9861-openshift-default-58ddf574b9-lrwlb\" (UID: \"fd2378f0-12b6-4a3a-a526-beb05ec13e02\") " pod="openshift-ingress/e2e-worker-w0-2d9861-openshift-default-58ddf574b9-lrwlb"
Oct 30 19:44:33.067921 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:44:33.066895    4892 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/fd2378f0-12b6-4a3a-a526-beb05ec13e02-istiod-ca-cert\") pod \"e2e-worker-w0-2d9861-openshift-default-58ddf574b9-lrwlb\" (UID: \"fd2378f0-12b6-4a3a-a526-beb05ec13e02\") " pod="openshift-ingress/e2e-worker-w0-2d9861-openshift-default-58ddf574b9-lrwlb"
Oct 30 19:44:33.068833 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:44:33.068140    4892 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/fd2378f0-12b6-4a3a-a526-beb05ec13e02-istio-envoy\") pod \"e2e-worker-w0-2d9861-openshift-default-58ddf574b9-lrwlb\" (UID: \"fd2378f0-12b6-4a3a-a526-beb05ec13e02\") " pod="openshift-ingress/e2e-worker-w0-2d9861-openshift-default-58ddf574b9-lrwlb"
Oct 30 19:44:33.070896 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:44:33.070876    4892 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/fd2378f0-12b6-4a3a-a526-beb05ec13e02-istio-podinfo\") pod \"e2e-worker-w0-2d9861-openshift-default-58ddf574b9-lrwlb\" (UID: \"fd2378f0-12b6-4a3a-a526-beb05ec13e02\") " pod="openshift-ingress/e2e-worker-w0-2d9861-openshift-default-58ddf574b9-lrwlb"
Oct 30 19:44:33.139727 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:44:33.139701    4892 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-54k6d\" (UniqueName: \"kubernetes.io/projected/fd2378f0-12b6-4a3a-a526-beb05ec13e02-kube-api-access-54k6d\") pod \"e2e-worker-w0-2d9861-openshift-default-58ddf574b9-lrwlb\" (UID: \"fd2378f0-12b6-4a3a-a526-beb05ec13e02\") " pod="openshift-ingress/e2e-worker-w0-2d9861-openshift-default-58ddf574b9-lrwlb"
Oct 30 19:44:33.145530 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:44:33.145497    4892 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/fd2378f0-12b6-4a3a-a526-beb05ec13e02-istio-token\") pod \"e2e-worker-w0-2d9861-openshift-default-58ddf574b9-lrwlb\" (UID: \"fd2378f0-12b6-4a3a-a526-beb05ec13e02\") " pod="openshift-ingress/e2e-worker-w0-2d9861-openshift-default-58ddf574b9-lrwlb"
Oct 30 19:44:33.162009 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:44:33.161913    4892 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-2d9861-openshift-default-58ddf574b9-w2r9m"
Oct 30 19:44:33.171582 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:44:33.171532    4892 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-2d9861-openshift-default-58ddf574b9-lrwlb"
Oct 30 19:44:33.466753 ip-10-0-132-82 kubenswrapper[4892]: W1030 19:44:33.465908    4892 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc63df543_fcb1_4bee_9d52_c50286ac6f91.slice/crio-c19f80eaa33b3667ecb85e2df383a166608f3edf890e9032668b820777400f78 WatchSource:0}: Error finding container c19f80eaa33b3667ecb85e2df383a166608f3edf890e9032668b820777400f78: Status 404 returned error can't find the container with id c19f80eaa33b3667ecb85e2df383a166608f3edf890e9032668b820777400f78
Oct 30 19:44:33.469449 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:44:33.469406    4892 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892000Ki","pods":"250"}
Oct 30 19:44:33.469585 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:44:33.469549    4892 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892000Ki","pods":"250"}
Oct 30 19:44:33.469644 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:44:33.469609    4892 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892000Ki","pods":"250"}
Oct 30 19:44:33.497040 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:44:33.491302    4892 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-lgwz5"]
Oct 30 19:44:33.591243 ip-10-0-132-82 kubenswrapper[4892]: W1030 19:44:33.591219    4892 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0f4fa4be_451c_43b5_bc35_cdf507a2dfb3.slice/crio-19c39f69dacc7b2dbe4c49b40289e50a3c265c3da952b59487f4042ad43ae610 WatchSource:0}: Error finding container 19c39f69dacc7b2dbe4c49b40289e50a3c265c3da952b59487f4042ad43ae610: Status 404 returned error can't find the container with id 19c39f69dacc7b2dbe4c49b40289e50a3c265c3da952b59487f4042ad43ae610
Oct 30 19:44:33.591534 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:44:33.591514    4892 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-2d9861-openshift-default-58ddf574b9-w2r9m"]
Oct 30 19:44:33.593832 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:44:33.593794    4892 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892000Ki","pods":"250"}
Oct 30 19:44:33.593941 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:44:33.593866    4892 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892000Ki","pods":"250"}
Oct 30 19:44:33.593941 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:44:33.593900    4892 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892000Ki","pods":"250"}
Oct 30 19:44:33.644004 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:44:33.643955    4892 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-2d9861-openshift-default-58ddf574b9-lrwlb"]
Oct 30 19:44:33.650293 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:44:33.648436    4892 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892000Ki","pods":"250"}
Oct 30 19:44:33.650293 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:44:33.648514    4892 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892000Ki","pods":"250"}
Oct 30 19:44:33.650293 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:44:33.648557    4892 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892000Ki","pods":"250"}
Oct 30 19:44:34.090595 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:44:34.090505    4892 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-2d9861-openshift-default-58ddf574b9-lrwlb" event={"ID":"fd2378f0-12b6-4a3a-a526-beb05ec13e02","Type":"ContainerStarted","Data":"760dd2a07edede572e89e2ad80b741b750963f274a497b5857f369d9e1d4377a"}
Oct 30 19:44:34.090595 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:44:34.090570    4892 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-2d9861-openshift-default-58ddf574b9-lrwlb" event={"ID":"fd2378f0-12b6-4a3a-a526-beb05ec13e02","Type":"ContainerStarted","Data":"823ece61e9500ad7b075eb2e2f78eb18567cd129bb68a3041207eedbf339f464"}
Oct 30 19:44:34.092937 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:44:34.092903    4892 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-2d9861-openshift-default-58ddf574b9-w2r9m" event={"ID":"0f4fa4be-451c-43b5-bc35-cdf507a2dfb3","Type":"ContainerStarted","Data":"772e8c1633aca86a1b725cee1d4db72ab0e0927004a64e78f0ea6d7cb13f9bf6"}
Oct 30 19:44:34.093122 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:44:34.093018    4892 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-2d9861-openshift-default-58ddf574b9-w2r9m" event={"ID":"0f4fa4be-451c-43b5-bc35-cdf507a2dfb3","Type":"ContainerStarted","Data":"19c39f69dacc7b2dbe4c49b40289e50a3c265c3da952b59487f4042ad43ae610"}
Oct 30 19:44:34.095284 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:44:34.095242    4892 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-lgwz5" event={"ID":"c63df543-fcb1-4bee-9d52-c50286ac6f91","Type":"ContainerStarted","Data":"deff73fe4ec3bff9c4fc74f415b493a91a87921ea03ab30b4b7ccc64fab8a84c"}
Oct 30 19:44:34.095430 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:44:34.095311    4892 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-lgwz5" event={"ID":"c63df543-fcb1-4bee-9d52-c50286ac6f91","Type":"ContainerStarted","Data":"c19f80eaa33b3667ecb85e2df383a166608f3edf890e9032668b820777400f78"}
Oct 30 19:44:34.166987 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:44:34.163205    4892 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w0-2d9861-openshift-default-58ddf574b9-w2r9m"
Oct 30 19:44:34.166987 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:44:34.165263    4892 patch_prober.go:29] interesting pod/e2e-worker-w0-2d9861-openshift-default-58ddf574b9-w2r9m container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.141:15021/healthz/ready\": dial tcp 10.132.0.141:15021: connect: connection refused" start-of-body=
Oct 30 19:44:34.166987 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:44:34.165320    4892 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-2d9861-openshift-default-58ddf574b9-w2r9m" podUID="0f4fa4be-451c-43b5-bc35-cdf507a2dfb3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.141:15021/healthz/ready\": dial tcp 10.132.0.141:15021: connect: connection refused"
Oct 30 19:44:34.175004 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:44:34.173775    4892 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w0-2d9861-openshift-default-58ddf574b9-lrwlb"
Oct 30 19:44:34.175004 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:44:34.173878    4892 patch_prober.go:29] interesting pod/e2e-worker-w0-2d9861-openshift-default-58ddf574b9-lrwlb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.142:15021/healthz/ready\": dial tcp 10.132.0.142:15021: connect: connection refused" start-of-body=
Oct 30 19:44:34.175004 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:44:34.173920    4892 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-2d9861-openshift-default-58ddf574b9-lrwlb" podUID="fd2378f0-12b6-4a3a-a526-beb05ec13e02" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.142:15021/healthz/ready\": dial tcp 10.132.0.142:15021: connect: connection refused"
Oct 30 19:44:34.220796 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:44:34.220740    4892 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w0-2d9861-openshift-default-58ddf574b9-lrwlb" podStartSLOduration=2.220726031 podStartE2EDuration="2.220726031s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:44:32 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:44:34.215427926 +0000 UTC m=+1721.785893526" watchObservedRunningTime="2026-10-30 19:44:34.220726031 +0000 UTC m=+1721.791191629"
Oct 30 19:44:34.350115 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:44:34.349997    4892 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w0-2d9861-openshift-default-58ddf574b9-w2r9m" podStartSLOduration=2.3499466399999998 podStartE2EDuration="2.34994664s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:44:32 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:44:34.347257758 +0000 UTC m=+1721.917723356" watchObservedRunningTime="2026-10-30 19:44:34.34994664 +0000 UTC m=+1721.920412281"
Oct 30 19:44:34.491951 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:44:34.491888    4892 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-lgwz5" podStartSLOduration=2.491872982 podStartE2EDuration="2.491872982s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:44:32 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:44:34.490758675 +0000 UTC m=+1722.061224274" watchObservedRunningTime="2026-10-30 19:44:34.491872982 +0000 UTC m=+1722.062338644"
Oct 30 19:44:34.975763 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:44:34.975729    4892 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-lgwz5"
Oct 30 19:44:34.976527 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:44:34.976502    4892 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6db586dd6c-lgwz5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.140:15021/healthz/ready\": dial tcp 10.132.0.140:15021: connect: connection refused" start-of-body=
Oct 30 19:44:34.976691 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:44:34.976667    4892 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-lgwz5" podUID="c63df543-fcb1-4bee-9d52-c50286ac6f91" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.140:15021/healthz/ready\": dial tcp 10.132.0.140:15021: connect: connection refused"
Oct 30 19:44:35.164995 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:44:35.164395    4892 patch_prober.go:29] interesting pod/e2e-worker-w0-2d9861-openshift-default-58ddf574b9-w2r9m container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.141:15021/healthz/ready\": dial tcp 10.132.0.141:15021: connect: connection refused" start-of-body=
Oct 30 19:44:35.164995 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:44:35.164472    4892 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-2d9861-openshift-default-58ddf574b9-w2r9m" podUID="0f4fa4be-451c-43b5-bc35-cdf507a2dfb3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.141:15021/healthz/ready\": dial tcp 10.132.0.141:15021: connect: connection refused"
Oct 30 19:44:35.174990 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:44:35.173348    4892 patch_prober.go:29] interesting pod/e2e-worker-w0-2d9861-openshift-default-58ddf574b9-lrwlb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.142:15021/healthz/ready\": dial tcp 10.132.0.142:15021: connect: connection refused" start-of-body=
Oct 30 19:44:35.174990 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:44:35.173411    4892 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-2d9861-openshift-default-58ddf574b9-lrwlb" podUID="fd2378f0-12b6-4a3a-a526-beb05ec13e02" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.142:15021/healthz/ready\": dial tcp 10.132.0.142:15021: connect: connection refused"
Oct 30 19:44:35.971990 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:44:35.971374    4892 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6db586dd6c-lgwz5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.140:15021/healthz/ready\": dial tcp 10.132.0.140:15021: connect: connection refused" start-of-body=
Oct 30 19:44:35.971990 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:44:35.971452    4892 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-lgwz5" podUID="c63df543-fcb1-4bee-9d52-c50286ac6f91" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.140:15021/healthz/ready\": dial tcp 10.132.0.140:15021: connect: connection refused"
Oct 30 19:44:36.164989 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:44:36.164356    4892 patch_prober.go:29] interesting pod/e2e-worker-w0-2d9861-openshift-default-58ddf574b9-w2r9m container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.141:15021/healthz/ready\": dial tcp 10.132.0.141:15021: connect: connection refused" start-of-body=
Oct 30 19:44:36.164989 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:44:36.164413    4892 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-2d9861-openshift-default-58ddf574b9-w2r9m" podUID="0f4fa4be-451c-43b5-bc35-cdf507a2dfb3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.141:15021/healthz/ready\": dial tcp 10.132.0.141:15021: connect: connection refused"
Oct 30 19:44:36.177578 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:44:36.177546    4892 patch_prober.go:29] interesting pod/e2e-worker-w0-2d9861-openshift-default-58ddf574b9-lrwlb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.142:15021/healthz/ready\": dial tcp 10.132.0.142:15021: connect: connection refused" start-of-body=
Oct 30 19:44:36.177847 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:44:36.177814    4892 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-2d9861-openshift-default-58ddf574b9-lrwlb" podUID="fd2378f0-12b6-4a3a-a526-beb05ec13e02" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.142:15021/healthz/ready\": dial tcp 10.132.0.142:15021: connect: connection refused"
Oct 30 19:44:36.982394 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:44:36.982358    4892 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6db586dd6c-lgwz5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.140:15021/healthz/ready\": dial tcp 10.132.0.140:15021: connect: connection refused" start-of-body=
Oct 30 19:44:36.982692 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:44:36.982665    4892 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-lgwz5" podUID="c63df543-fcb1-4bee-9d52-c50286ac6f91" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.140:15021/healthz/ready\": dial tcp 10.132.0.140:15021: connect: connection refused"
Oct 30 19:44:37.166992 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:44:37.166766    4892 patch_prober.go:29] interesting pod/e2e-worker-w0-2d9861-openshift-default-58ddf574b9-w2r9m container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.141:15021/healthz/ready\": dial tcp 10.132.0.141:15021: connect: connection refused" start-of-body=
Oct 30 19:44:37.170986 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:44:37.168017    4892 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-2d9861-openshift-default-58ddf574b9-w2r9m" podUID="0f4fa4be-451c-43b5-bc35-cdf507a2dfb3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.141:15021/healthz/ready\": dial tcp 10.132.0.141:15021: connect: connection refused"
Oct 30 19:44:37.178491 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:44:37.178351    4892 patch_prober.go:29] interesting pod/e2e-worker-w0-2d9861-openshift-default-58ddf574b9-lrwlb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.142:15021/healthz/ready\": dial tcp 10.132.0.142:15021: connect: connection refused" start-of-body=
Oct 30 19:44:37.178491 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:44:37.178419    4892 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-2d9861-openshift-default-58ddf574b9-lrwlb" podUID="fd2378f0-12b6-4a3a-a526-beb05ec13e02" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.142:15021/healthz/ready\": dial tcp 10.132.0.142:15021: connect: connection refused"
Oct 30 19:44:38.000992 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:44:37.975351    4892 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6db586dd6c-lgwz5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.140:15021/healthz/ready\": dial tcp 10.132.0.140:15021: connect: connection refused" start-of-body=
Oct 30 19:44:38.000992 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:44:37.975414    4892 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-lgwz5" podUID="c63df543-fcb1-4bee-9d52-c50286ac6f91" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.140:15021/healthz/ready\": dial tcp 10.132.0.140:15021: connect: connection refused"
Oct 30 19:44:38.320286 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:44:38.320188    4892 patch_prober.go:29] interesting pod/e2e-worker-w0-2d9861-openshift-default-58ddf574b9-lrwlb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.142:15021/healthz/ready\": dial tcp 10.132.0.142:15021: connect: connection refused" start-of-body=
Oct 30 19:44:38.320761 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:44:38.320291    4892 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-2d9861-openshift-default-58ddf574b9-lrwlb" podUID="fd2378f0-12b6-4a3a-a526-beb05ec13e02" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.142:15021/healthz/ready\": dial tcp 10.132.0.142:15021: connect: connection refused"
Oct 30 19:44:38.326920 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:44:38.326885    4892 patch_prober.go:29] interesting pod/e2e-worker-w0-2d9861-openshift-default-58ddf574b9-w2r9m container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.141:15021/healthz/ready\": dial tcp 10.132.0.141:15021: connect: connection refused" start-of-body=
Oct 30 19:44:38.340108 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:44:38.327015    4892 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-2d9861-openshift-default-58ddf574b9-w2r9m" podUID="0f4fa4be-451c-43b5-bc35-cdf507a2dfb3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.141:15021/healthz/ready\": dial tcp 10.132.0.141:15021: connect: connection refused"
Oct 30 19:44:39.002127 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:44:39.002089    4892 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6db586dd6c-lgwz5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.140:15021/healthz/ready\": dial tcp 10.132.0.140:15021: connect: connection refused" start-of-body=
Oct 30 19:44:39.002406 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:44:39.002377    4892 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-lgwz5" podUID="c63df543-fcb1-4bee-9d52-c50286ac6f91" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.140:15021/healthz/ready\": dial tcp 10.132.0.140:15021: connect: connection refused"
Oct 30 19:44:39.044375 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:44:39.044330    4892 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-d52df0/simulator-w1-d52df0-kserve-67f74bc577-q5bjz" podUID="81b7bb84-1003-4452-b7e0-b4aaba6fefa4" containerName="main" probeResult="failure" output="Get \"https://10.132.0.57:8000/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:44:39.181122 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:44:39.181087    4892 patch_prober.go:29] interesting pod/e2e-worker-w0-2d9861-openshift-default-58ddf574b9-lrwlb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.142:15021/healthz/ready\": dial tcp 10.132.0.142:15021: connect: connection refused" start-of-body=
Oct 30 19:44:39.181414 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:44:39.181382    4892 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-2d9861-openshift-default-58ddf574b9-lrwlb" podUID="fd2378f0-12b6-4a3a-a526-beb05ec13e02" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.142:15021/healthz/ready\": dial tcp 10.132.0.142:15021: connect: connection refused"
Oct 30 19:44:39.214229 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:44:39.214187    4892 patch_prober.go:29] interesting pod/e2e-worker-w0-2d9861-openshift-default-58ddf574b9-w2r9m container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.141:15021/healthz/ready\": dial tcp 10.132.0.141:15021: connect: connection refused" start-of-body=
Oct 30 19:44:39.226524 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:44:39.226485    4892 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-2d9861-openshift-default-58ddf574b9-w2r9m" podUID="0f4fa4be-451c-43b5-bc35-cdf507a2dfb3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.141:15021/healthz/ready\": dial tcp 10.132.0.141:15021: connect: connection refused"
Oct 30 19:44:39.386936 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:44:39.386134    4892 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-b5178f/distinct-2-w4-b5178f-kserve-69ddf8b665-9pdth" podUID="fb71d961-8500-4367-b48c-88d385962bd5" containerName="main" probeResult="failure" output="Get \"https://10.132.0.90:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:44:39.496126 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:44:39.496083    4892 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-2d9861/distinct-2-w0-2d9861-kserve-679c6b56dd-nmhbm" podUID="65b0586a-6593-48cc-adf8-8ff48417ece4" containerName="main" probeResult="failure" output="Get \"https://10.132.0.133:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:44:39.986151 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:44:39.982692    4892 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6db586dd6c-lgwz5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.140:15021/healthz/ready\": dial tcp 10.132.0.140:15021: connect: connection refused" start-of-body=
Oct 30 19:44:39.986151 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:44:39.982749    4892 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-lgwz5" podUID="c63df543-fcb1-4bee-9d52-c50286ac6f91" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.140:15021/healthz/ready\": dial tcp 10.132.0.140:15021: connect: connection refused"
Oct 30 19:44:40.163508 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:44:40.163473    4892 patch_prober.go:29] interesting pod/e2e-worker-w0-2d9861-openshift-default-58ddf574b9-w2r9m container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.141:15021/healthz/ready\": dial tcp 10.132.0.141:15021: connect: connection refused" start-of-body=
Oct 30 19:44:40.163820 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:44:40.163770    4892 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-2d9861-openshift-default-58ddf574b9-w2r9m" podUID="0f4fa4be-451c-43b5-bc35-cdf507a2dfb3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.141:15021/healthz/ready\": dial tcp 10.132.0.141:15021: connect: connection refused"
Oct 30 19:44:40.180201 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:44:40.180162    4892 patch_prober.go:29] interesting pod/e2e-worker-w0-2d9861-openshift-default-58ddf574b9-lrwlb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.142:15021/healthz/ready\": dial tcp 10.132.0.142:15021: connect: connection refused" start-of-body=
Oct 30 19:44:40.180557 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:44:40.180491    4892 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-2d9861-openshift-default-58ddf574b9-lrwlb" podUID="fd2378f0-12b6-4a3a-a526-beb05ec13e02" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.142:15021/healthz/ready\": dial tcp 10.132.0.142:15021: connect: connection refused"
Oct 30 19:44:40.976158 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:44:40.976118    4892 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6db586dd6c-lgwz5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.140:15021/healthz/ready\": dial tcp 10.132.0.140:15021: connect: connection refused" start-of-body=
Oct 30 19:44:40.976771 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:44:40.976738    4892 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-lgwz5" podUID="c63df543-fcb1-4bee-9d52-c50286ac6f91" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.140:15021/healthz/ready\": dial tcp 10.132.0.140:15021: connect: connection refused"
Oct 30 19:44:41.162801 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:44:41.162762    4892 patch_prober.go:29] interesting pod/e2e-worker-w0-2d9861-openshift-default-58ddf574b9-w2r9m container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.141:15021/healthz/ready\": dial tcp 10.132.0.141:15021: connect: connection refused" start-of-body=
Oct 30 19:44:41.163030 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:44:41.162823    4892 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-2d9861-openshift-default-58ddf574b9-w2r9m" podUID="0f4fa4be-451c-43b5-bc35-cdf507a2dfb3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.141:15021/healthz/ready\": dial tcp 10.132.0.141:15021: connect: connection refused"
Oct 30 19:44:41.172019 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:44:41.171955    4892 patch_prober.go:29] interesting pod/e2e-worker-w0-2d9861-openshift-default-58ddf574b9-lrwlb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.142:15021/healthz/ready\": dial tcp 10.132.0.142:15021: connect: connection refused" start-of-body=
Oct 30 19:44:41.172234 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:44:41.172038    4892 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-2d9861-openshift-default-58ddf574b9-lrwlb" podUID="fd2378f0-12b6-4a3a-a526-beb05ec13e02" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.142:15021/healthz/ready\": dial tcp 10.132.0.142:15021: connect: connection refused"
Oct 30 19:44:41.971654 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:44:41.971614    4892 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6db586dd6c-lgwz5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.140:15021/healthz/ready\": dial tcp 10.132.0.140:15021: connect: connection refused" start-of-body=
Oct 30 19:44:41.971882 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:44:41.971689    4892 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-lgwz5" podUID="c63df543-fcb1-4bee-9d52-c50286ac6f91" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.140:15021/healthz/ready\": dial tcp 10.132.0.140:15021: connect: connection refused"
Oct 30 19:44:42.917071 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:44:42.917031    4892 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w0-2d9861-openshift-default-58ddf574b9-lrwlb"
Oct 30 19:44:42.918651 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:44:42.918627    4892 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w0-2d9861-openshift-default-58ddf574b9-lrwlb"
Oct 30 19:44:42.919633 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:44:42.919608    4892 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w0-2d9861-openshift-default-58ddf574b9-lrwlb"
Oct 30 19:44:42.948587 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:44:42.947136    4892 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w0-2d9861-openshift-default-58ddf574b9-w2r9m"
Oct 30 19:44:42.948587 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:44:42.948502    4892 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w0-2d9861-openshift-default-58ddf574b9-w2r9m"
Oct 30 19:44:42.951207 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:44:42.951188    4892 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w0-2d9861-openshift-default-58ddf574b9-w2r9m"
Oct 30 19:44:42.971296 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:44:42.971266    4892 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6db586dd6c-lgwz5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.140:15021/healthz/ready\": dial tcp 10.132.0.140:15021: connect: connection refused" start-of-body=
Oct 30 19:44:42.971423 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:44:42.971318    4892 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-lgwz5" podUID="c63df543-fcb1-4bee-9d52-c50286ac6f91" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.140:15021/healthz/ready\": dial tcp 10.132.0.140:15021: connect: connection refused"
Oct 30 19:44:44.195210 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:44:44.195176    4892 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-lgwz5"
Oct 30 19:44:44.195677 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:44:44.195532    4892 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-lgwz5"
Oct 30 19:44:44.196398 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:44:44.196380    4892 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-lgwz5"
Oct 30 19:44:46.042412 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:44:46.042371    4892 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ae4eb22f-mxz6s"]
Oct 30 19:44:46.048279 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:44:46.048253    4892 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ae4eb22f-mxz6s"
Oct 30 19:44:46.103893 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:44:46.103866    4892 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-nlxp4\""
Oct 30 19:44:46.154094 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:44:46.154039    4892 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 30 19:44:46.158896 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:44:46.158077    4892 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 30 19:44:46.165792 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:44:46.165761    4892 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ae4eb22f-mxz6s"]
Oct 30 19:44:46.182258 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:44:46.182228    4892 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d299acaf-6665-4ab0-8333-cca9953d8f47-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-ae4eb22f-mxz6s\" (UID: \"d299acaf-6665-4ab0-8333-cca9953d8f47\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ae4eb22f-mxz6s"
Oct 30 19:44:46.182406 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:44:46.182344    4892 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6tksl\" (UniqueName: \"kubernetes.io/projected/d299acaf-6665-4ab0-8333-cca9953d8f47-kube-api-access-6tksl\") pod \"maas-api-revoke-sub-ae4eb22f-mxz6s\" (UID: \"d299acaf-6665-4ab0-8333-cca9953d8f47\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ae4eb22f-mxz6s"
Oct 30 19:44:46.182476 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:44:46.182414    4892 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d299acaf-6665-4ab0-8333-cca9953d8f47-maas-api-xks-ca\") pod \"maas-api-revoke-sub-ae4eb22f-mxz6s\" (UID: \"d299acaf-6665-4ab0-8333-cca9953d8f47\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ae4eb22f-mxz6s"
Oct 30 19:44:46.202666 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:44:46.202646    4892 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 19:44:46.284454 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:44:46.284398    4892 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d299acaf-6665-4ab0-8333-cca9953d8f47-maas-api-xks-ca\") pod \"maas-api-revoke-sub-ae4eb22f-mxz6s\" (UID: \"d299acaf-6665-4ab0-8333-cca9953d8f47\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ae4eb22f-mxz6s"
Oct 30 19:44:46.284652 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:44:46.284637    4892 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d299acaf-6665-4ab0-8333-cca9953d8f47-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-ae4eb22f-mxz6s\" (UID: \"d299acaf-6665-4ab0-8333-cca9953d8f47\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ae4eb22f-mxz6s"
Oct 30 19:44:46.284780 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:44:46.284763    4892 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6tksl\" (UniqueName: \"kubernetes.io/projected/d299acaf-6665-4ab0-8333-cca9953d8f47-kube-api-access-6tksl\") pod \"maas-api-revoke-sub-ae4eb22f-mxz6s\" (UID: \"d299acaf-6665-4ab0-8333-cca9953d8f47\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ae4eb22f-mxz6s"
Oct 30 19:44:46.285556 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:44:46.285474    4892 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d299acaf-6665-4ab0-8333-cca9953d8f47-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-ae4eb22f-mxz6s\" (UID: \"d299acaf-6665-4ab0-8333-cca9953d8f47\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ae4eb22f-mxz6s"
Oct 30 19:44:46.287495 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:44:46.287426    4892 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d299acaf-6665-4ab0-8333-cca9953d8f47-maas-api-xks-ca\") pod \"maas-api-revoke-sub-ae4eb22f-mxz6s\" (UID: \"d299acaf-6665-4ab0-8333-cca9953d8f47\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ae4eb22f-mxz6s"
Oct 30 19:44:46.366295 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:44:46.363052    4892 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6tksl\" (UniqueName: \"kubernetes.io/projected/d299acaf-6665-4ab0-8333-cca9953d8f47-kube-api-access-6tksl\") pod \"maas-api-revoke-sub-ae4eb22f-mxz6s\" (UID: \"d299acaf-6665-4ab0-8333-cca9953d8f47\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ae4eb22f-mxz6s"
Oct 30 19:44:46.370659 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:44:46.370628    4892 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ae4eb22f-mxz6s"
Oct 30 19:44:46.778152 ip-10-0-132-82 kubenswrapper[4892]: W1030 19:44:46.778123    4892 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd299acaf_6665_4ab0_8333_cca9953d8f47.slice/crio-ae1fad81af5091f24c8155fd6e1e55c0de85edfcf9824dda359937a61a44f162 WatchSource:0}: Error finding container ae1fad81af5091f24c8155fd6e1e55c0de85edfcf9824dda359937a61a44f162: Status 404 returned error can't find the container with id ae1fad81af5091f24c8155fd6e1e55c0de85edfcf9824dda359937a61a44f162
Oct 30 19:44:46.778932 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:44:46.778908    4892 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ae4eb22f-mxz6s"]
Oct 30 19:44:47.547686 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:44:47.547646    4892 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ae4eb22f-mxz6s" event={"ID":"d299acaf-6665-4ab0-8333-cca9953d8f47","Type":"ContainerStarted","Data":"41779301ffee7b2cd47f02b4782841173bc03bd11947d1459aa6d7c6189ae9a2"}
Oct 30 19:44:47.550848 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:44:47.550821    4892 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ae4eb22f-mxz6s" event={"ID":"d299acaf-6665-4ab0-8333-cca9953d8f47","Type":"ContainerStarted","Data":"ae1fad81af5091f24c8155fd6e1e55c0de85edfcf9824dda359937a61a44f162"}
Oct 30 19:44:47.687590 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:44:47.687529    4892 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ae4eb22f-mxz6s" podStartSLOduration=2.687514212 podStartE2EDuration="2.687514212s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:44:45 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:44:47.6797111 +0000 UTC m=+1735.250176700" watchObservedRunningTime="2026-10-30 19:44:47.687514212 +0000 UTC m=+1735.257979810"
Oct 30 19:44:47.739998 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:44:47.739129    4892 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-a-w3-cf9b6f-openshift-default-56db6769f7-xl4xv"]
Oct 30 19:44:47.743990 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:44:47.743941    4892 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w3-cf9b6f-openshift-default-56db6769f7-xl4xv"
Oct 30 19:44:47.784479 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:44:47.784448    4892 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-lqrjf"]
Oct 30 19:44:47.789206 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:44:47.789181    4892 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-lqrjf"
Oct 30 19:44:47.800276 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:44:47.800251    4892 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-shared-a-w3-cf9b6f-openshift-default-dockercfg-qcqbf\""
Oct 30 19:44:47.813011 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:44:47.811583    4892 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w3-cf9b6f-openshift-default-56db6769f7-xl4xv"]
Oct 30 19:44:47.836840 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:44:47.836792    4892 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/fd212b75-a818-45a3-946a-864a230b7707-istiod-ca-cert\") pod \"e2e-shared-a-w3-cf9b6f-openshift-default-56db6769f7-xl4xv\" (UID: \"fd212b75-a818-45a3-946a-864a230b7707\") " pod="openshift-ingress/e2e-shared-a-w3-cf9b6f-openshift-default-56db6769f7-xl4xv"
Oct 30 19:44:47.837046 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:44:47.836847    4892 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/fd212b75-a818-45a3-946a-864a230b7707-istio-data\") pod \"e2e-shared-a-w3-cf9b6f-openshift-default-56db6769f7-xl4xv\" (UID: \"fd212b75-a818-45a3-946a-864a230b7707\") " pod="openshift-ingress/e2e-shared-a-w3-cf9b6f-openshift-default-56db6769f7-xl4xv"
Oct 30 19:44:47.837046 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:44:47.836981    4892 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/fd212b75-a818-45a3-946a-864a230b7707-workload-certs\") pod \"e2e-shared-a-w3-cf9b6f-openshift-default-56db6769f7-xl4xv\" (UID: \"fd212b75-a818-45a3-946a-864a230b7707\") " pod="openshift-ingress/e2e-shared-a-w3-cf9b6f-openshift-default-56db6769f7-xl4xv"
Oct 30 19:44:47.837046 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:44:47.837033    4892 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/fd212b75-a818-45a3-946a-864a230b7707-istio-envoy\") pod \"e2e-shared-a-w3-cf9b6f-openshift-default-56db6769f7-xl4xv\" (UID: \"fd212b75-a818-45a3-946a-864a230b7707\") " pod="openshift-ingress/e2e-shared-a-w3-cf9b6f-openshift-default-56db6769f7-xl4xv"
Oct 30 19:44:47.837252 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:44:47.837070    4892 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/fd212b75-a818-45a3-946a-864a230b7707-credential-socket\") pod \"e2e-shared-a-w3-cf9b6f-openshift-default-56db6769f7-xl4xv\" (UID: \"fd212b75-a818-45a3-946a-864a230b7707\") " pod="openshift-ingress/e2e-shared-a-w3-cf9b6f-openshift-default-56db6769f7-xl4xv"
Oct 30 19:44:47.837252 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:44:47.837101    4892 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ntlts\" (UniqueName: \"kubernetes.io/projected/fd212b75-a818-45a3-946a-864a230b7707-kube-api-access-ntlts\") pod \"e2e-shared-a-w3-cf9b6f-openshift-default-56db6769f7-xl4xv\" (UID: \"fd212b75-a818-45a3-946a-864a230b7707\") " pod="openshift-ingress/e2e-shared-a-w3-cf9b6f-openshift-default-56db6769f7-xl4xv"
Oct 30 19:44:47.837252 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:44:47.837213    4892 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/fd212b75-a818-45a3-946a-864a230b7707-istio-token\") pod \"e2e-shared-a-w3-cf9b6f-openshift-default-56db6769f7-xl4xv\" (UID: \"fd212b75-a818-45a3-946a-864a230b7707\") " pod="openshift-ingress/e2e-shared-a-w3-cf9b6f-openshift-default-56db6769f7-xl4xv"
Oct 30 19:44:47.837479 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:44:47.837265    4892 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/fd212b75-a818-45a3-946a-864a230b7707-workload-socket\") pod \"e2e-shared-a-w3-cf9b6f-openshift-default-56db6769f7-xl4xv\" (UID: \"fd212b75-a818-45a3-946a-864a230b7707\") " pod="openshift-ingress/e2e-shared-a-w3-cf9b6f-openshift-default-56db6769f7-xl4xv"
Oct 30 19:44:47.837479 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:44:47.837328    4892 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/fd212b75-a818-45a3-946a-864a230b7707-istio-podinfo\") pod \"e2e-shared-a-w3-cf9b6f-openshift-default-56db6769f7-xl4xv\" (UID: \"fd212b75-a818-45a3-946a-864a230b7707\") " pod="openshift-ingress/e2e-shared-a-w3-cf9b6f-openshift-default-56db6769f7-xl4xv"
Oct 30 19:44:47.948006 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:44:47.942213    4892 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/fd212b75-a818-45a3-946a-864a230b7707-istio-data\") pod \"e2e-shared-a-w3-cf9b6f-openshift-default-56db6769f7-xl4xv\" (UID: \"fd212b75-a818-45a3-946a-864a230b7707\") " pod="openshift-ingress/e2e-shared-a-w3-cf9b6f-openshift-default-56db6769f7-xl4xv"
Oct 30 19:44:47.948006 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:44:47.942283    4892 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c5510c1c-0da3-4b39-95ee-a9195c32e990-istio-data\") pod \"maas-default-gateway-openshift-default-6db586dd6c-lqrjf\" (UID: \"c5510c1c-0da3-4b39-95ee-a9195c32e990\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-lqrjf"
Oct 30 19:44:47.948006 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:44:47.942324    4892 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c5510c1c-0da3-4b39-95ee-a9195c32e990-workload-socket\") pod \"maas-default-gateway-openshift-default-6db586dd6c-lqrjf\" (UID: \"c5510c1c-0da3-4b39-95ee-a9195c32e990\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-lqrjf"
Oct 30 19:44:47.948006 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:44:47.942419    4892 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/fd212b75-a818-45a3-946a-864a230b7707-workload-certs\") pod \"e2e-shared-a-w3-cf9b6f-openshift-default-56db6769f7-xl4xv\" (UID: \"fd212b75-a818-45a3-946a-864a230b7707\") " pod="openshift-ingress/e2e-shared-a-w3-cf9b6f-openshift-default-56db6769f7-xl4xv"
Oct 30 19:44:47.948006 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:44:47.942459    4892 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/fd212b75-a818-45a3-946a-864a230b7707-istio-envoy\") pod \"e2e-shared-a-w3-cf9b6f-openshift-default-56db6769f7-xl4xv\" (UID: \"fd212b75-a818-45a3-946a-864a230b7707\") " pod="openshift-ingress/e2e-shared-a-w3-cf9b6f-openshift-default-56db6769f7-xl4xv"
Oct 30 19:44:47.948006 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:44:47.942492    4892 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/fd212b75-a818-45a3-946a-864a230b7707-credential-socket\") pod \"e2e-shared-a-w3-cf9b6f-openshift-default-56db6769f7-xl4xv\" (UID: \"fd212b75-a818-45a3-946a-864a230b7707\") " pod="openshift-ingress/e2e-shared-a-w3-cf9b6f-openshift-default-56db6769f7-xl4xv"
Oct 30 19:44:47.948006 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:44:47.942524    4892 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c5510c1c-0da3-4b39-95ee-a9195c32e990-credential-socket\") pod \"maas-default-gateway-openshift-default-6db586dd6c-lqrjf\" (UID: \"c5510c1c-0da3-4b39-95ee-a9195c32e990\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-lqrjf"
Oct 30 19:44:47.948006 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:44:47.942570    4892 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c5510c1c-0da3-4b39-95ee-a9195c32e990-istio-envoy\") pod \"maas-default-gateway-openshift-default-6db586dd6c-lqrjf\" (UID: \"c5510c1c-0da3-4b39-95ee-a9195c32e990\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-lqrjf"
Oct 30 19:44:47.948006 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:44:47.942611    4892 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/fd212b75-a818-45a3-946a-864a230b7707-istio-token\") pod \"e2e-shared-a-w3-cf9b6f-openshift-default-56db6769f7-xl4xv\" (UID: \"fd212b75-a818-45a3-946a-864a230b7707\") " pod="openshift-ingress/e2e-shared-a-w3-cf9b6f-openshift-default-56db6769f7-xl4xv"
Oct 30 19:44:47.948006 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:44:47.942662    4892 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/fd212b75-a818-45a3-946a-864a230b7707-workload-socket\") pod \"e2e-shared-a-w3-cf9b6f-openshift-default-56db6769f7-xl4xv\" (UID: \"fd212b75-a818-45a3-946a-864a230b7707\") " pod="openshift-ingress/e2e-shared-a-w3-cf9b6f-openshift-default-56db6769f7-xl4xv"
Oct 30 19:44:47.948006 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:44:47.942688    4892 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6zkft\" (UniqueName: \"kubernetes.io/projected/c5510c1c-0da3-4b39-95ee-a9195c32e990-kube-api-access-6zkft\") pod \"maas-default-gateway-openshift-default-6db586dd6c-lqrjf\" (UID: \"c5510c1c-0da3-4b39-95ee-a9195c32e990\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-lqrjf"
Oct 30 19:44:47.948006 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:44:47.942727    4892 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/fd212b75-a818-45a3-946a-864a230b7707-istio-podinfo\") pod \"e2e-shared-a-w3-cf9b6f-openshift-default-56db6769f7-xl4xv\" (UID: \"fd212b75-a818-45a3-946a-864a230b7707\") " pod="openshift-ingress/e2e-shared-a-w3-cf9b6f-openshift-default-56db6769f7-xl4xv"
Oct 30 19:44:47.948006 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:44:47.942783    4892 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/fd212b75-a818-45a3-946a-864a230b7707-istiod-ca-cert\") pod \"e2e-shared-a-w3-cf9b6f-openshift-default-56db6769f7-xl4xv\" (UID: \"fd212b75-a818-45a3-946a-864a230b7707\") " pod="openshift-ingress/e2e-shared-a-w3-cf9b6f-openshift-default-56db6769f7-xl4xv"
Oct 30 19:44:47.948006 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:44:47.942839    4892 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c5510c1c-0da3-4b39-95ee-a9195c32e990-istio-podinfo\") pod \"maas-default-gateway-openshift-default-6db586dd6c-lqrjf\" (UID: \"c5510c1c-0da3-4b39-95ee-a9195c32e990\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-lqrjf"
Oct 30 19:44:47.948006 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:44:47.942881    4892 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c5510c1c-0da3-4b39-95ee-a9195c32e990-workload-certs\") pod \"maas-default-gateway-openshift-default-6db586dd6c-lqrjf\" (UID: \"c5510c1c-0da3-4b39-95ee-a9195c32e990\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-lqrjf"
Oct 30 19:44:47.948006 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:44:47.942909    4892 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c5510c1c-0da3-4b39-95ee-a9195c32e990-istio-token\") pod \"maas-default-gateway-openshift-default-6db586dd6c-lqrjf\" (UID: \"c5510c1c-0da3-4b39-95ee-a9195c32e990\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-lqrjf"
Oct 30 19:44:47.948006 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:44:47.942949    4892 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c5510c1c-0da3-4b39-95ee-a9195c32e990-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-6db586dd6c-lqrjf\" (UID: \"c5510c1c-0da3-4b39-95ee-a9195c32e990\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-lqrjf"
Oct 30 19:44:47.948006 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:44:47.943015    4892 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ntlts\" (UniqueName: \"kubernetes.io/projected/fd212b75-a818-45a3-946a-864a230b7707-kube-api-access-ntlts\") pod \"e2e-shared-a-w3-cf9b6f-openshift-default-56db6769f7-xl4xv\" (UID: \"fd212b75-a818-45a3-946a-864a230b7707\") " pod="openshift-ingress/e2e-shared-a-w3-cf9b6f-openshift-default-56db6769f7-xl4xv"
Oct 30 19:44:47.948006 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:44:47.943745    4892 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/fd212b75-a818-45a3-946a-864a230b7707-istio-data\") pod \"e2e-shared-a-w3-cf9b6f-openshift-default-56db6769f7-xl4xv\" (UID: \"fd212b75-a818-45a3-946a-864a230b7707\") " pod="openshift-ingress/e2e-shared-a-w3-cf9b6f-openshift-default-56db6769f7-xl4xv"
Oct 30 19:44:48.005029 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:44:48.003174    4892 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/fd212b75-a818-45a3-946a-864a230b7707-credential-socket\") pod \"e2e-shared-a-w3-cf9b6f-openshift-default-56db6769f7-xl4xv\" (UID: \"fd212b75-a818-45a3-946a-864a230b7707\") " pod="openshift-ingress/e2e-shared-a-w3-cf9b6f-openshift-default-56db6769f7-xl4xv"
Oct 30 19:44:48.005029 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:44:48.003818    4892 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/fd212b75-a818-45a3-946a-864a230b7707-workload-socket\") pod \"e2e-shared-a-w3-cf9b6f-openshift-default-56db6769f7-xl4xv\" (UID: \"fd212b75-a818-45a3-946a-864a230b7707\") " pod="openshift-ingress/e2e-shared-a-w3-cf9b6f-openshift-default-56db6769f7-xl4xv"
Oct 30 19:44:48.137037 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:44:48.104376    4892 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/fd212b75-a818-45a3-946a-864a230b7707-workload-certs\") pod \"e2e-shared-a-w3-cf9b6f-openshift-default-56db6769f7-xl4xv\" (UID: \"fd212b75-a818-45a3-946a-864a230b7707\") " pod="openshift-ingress/e2e-shared-a-w3-cf9b6f-openshift-default-56db6769f7-xl4xv"
Oct 30 19:44:48.137412 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:44:48.129769    4892 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c5510c1c-0da3-4b39-95ee-a9195c32e990-credential-socket\") pod \"maas-default-gateway-openshift-default-6db586dd6c-lqrjf\" (UID: \"c5510c1c-0da3-4b39-95ee-a9195c32e990\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-lqrjf"
Oct 30 19:44:48.137621 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:44:48.137604    4892 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c5510c1c-0da3-4b39-95ee-a9195c32e990-istio-envoy\") pod \"maas-default-gateway-openshift-default-6db586dd6c-lqrjf\" (UID: \"c5510c1c-0da3-4b39-95ee-a9195c32e990\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-lqrjf"
Oct 30 19:44:48.137729 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:44:48.130674    4892 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-lqrjf"]
Oct 30 19:44:48.137791 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:44:48.134547    4892 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c5510c1c-0da3-4b39-95ee-a9195c32e990-credential-socket\") pod \"maas-default-gateway-openshift-default-6db586dd6c-lqrjf\" (UID: \"c5510c1c-0da3-4b39-95ee-a9195c32e990\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-lqrjf"
Oct 30 19:44:48.137928 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:44:48.137911    4892 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6zkft\" (UniqueName: \"kubernetes.io/projected/c5510c1c-0da3-4b39-95ee-a9195c32e990-kube-api-access-6zkft\") pod \"maas-default-gateway-openshift-default-6db586dd6c-lqrjf\" (UID: \"c5510c1c-0da3-4b39-95ee-a9195c32e990\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-lqrjf"
Oct 30 19:44:48.138164 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:44:48.138144    4892 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c5510c1c-0da3-4b39-95ee-a9195c32e990-istio-podinfo\") pod \"maas-default-gateway-openshift-default-6db586dd6c-lqrjf\" (UID: \"c5510c1c-0da3-4b39-95ee-a9195c32e990\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-lqrjf"
Oct 30 19:44:48.138327 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:44:48.132122    4892 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/fd212b75-a818-45a3-946a-864a230b7707-istiod-ca-cert\") pod \"e2e-shared-a-w3-cf9b6f-openshift-default-56db6769f7-xl4xv\" (UID: \"fd212b75-a818-45a3-946a-864a230b7707\") " pod="openshift-ingress/e2e-shared-a-w3-cf9b6f-openshift-default-56db6769f7-xl4xv"
Oct 30 19:44:48.138430 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:44:48.138412    4892 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c5510c1c-0da3-4b39-95ee-a9195c32e990-workload-certs\") pod \"maas-default-gateway-openshift-default-6db586dd6c-lqrjf\" (UID: \"c5510c1c-0da3-4b39-95ee-a9195c32e990\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-lqrjf"
Oct 30 19:44:48.138545 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:44:48.138529    4892 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c5510c1c-0da3-4b39-95ee-a9195c32e990-istio-token\") pod \"maas-default-gateway-openshift-default-6db586dd6c-lqrjf\" (UID: \"c5510c1c-0da3-4b39-95ee-a9195c32e990\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-lqrjf"
Oct 30 19:44:48.138665 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:44:48.138651    4892 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c5510c1c-0da3-4b39-95ee-a9195c32e990-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-6db586dd6c-lqrjf\" (UID: \"c5510c1c-0da3-4b39-95ee-a9195c32e990\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-lqrjf"
Oct 30 19:44:48.139874 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:44:48.138891    4892 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c5510c1c-0da3-4b39-95ee-a9195c32e990-workload-certs\") pod \"maas-default-gateway-openshift-default-6db586dd6c-lqrjf\" (UID: \"c5510c1c-0da3-4b39-95ee-a9195c32e990\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-lqrjf"
Oct 30 19:44:48.139874 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:44:48.138914    4892 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c5510c1c-0da3-4b39-95ee-a9195c32e990-istio-data\") pod \"maas-default-gateway-openshift-default-6db586dd6c-lqrjf\" (UID: \"c5510c1c-0da3-4b39-95ee-a9195c32e990\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-lqrjf"
Oct 30 19:44:48.139874 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:44:48.138985    4892 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c5510c1c-0da3-4b39-95ee-a9195c32e990-workload-socket\") pod \"maas-default-gateway-openshift-default-6db586dd6c-lqrjf\" (UID: \"c5510c1c-0da3-4b39-95ee-a9195c32e990\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-lqrjf"
Oct 30 19:44:48.139874 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:44:48.139390    4892 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c5510c1c-0da3-4b39-95ee-a9195c32e990-workload-socket\") pod \"maas-default-gateway-openshift-default-6db586dd6c-lqrjf\" (UID: \"c5510c1c-0da3-4b39-95ee-a9195c32e990\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-lqrjf"
Oct 30 19:44:48.139874 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:44:48.139811    4892 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c5510c1c-0da3-4b39-95ee-a9195c32e990-istio-data\") pod \"maas-default-gateway-openshift-default-6db586dd6c-lqrjf\" (UID: \"c5510c1c-0da3-4b39-95ee-a9195c32e990\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-lqrjf"
Oct 30 19:44:48.140290 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:44:48.140095    4892 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c5510c1c-0da3-4b39-95ee-a9195c32e990-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-6db586dd6c-lqrjf\" (UID: \"c5510c1c-0da3-4b39-95ee-a9195c32e990\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-lqrjf"
Oct 30 19:44:48.143721 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:44:48.143656    4892 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/fd212b75-a818-45a3-946a-864a230b7707-istio-token\") pod \"e2e-shared-a-w3-cf9b6f-openshift-default-56db6769f7-xl4xv\" (UID: \"fd212b75-a818-45a3-946a-864a230b7707\") " pod="openshift-ingress/e2e-shared-a-w3-cf9b6f-openshift-default-56db6769f7-xl4xv"
Oct 30 19:44:48.301794 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:44:48.236374    4892 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/fd212b75-a818-45a3-946a-864a230b7707-istio-podinfo\") pod \"e2e-shared-a-w3-cf9b6f-openshift-default-56db6769f7-xl4xv\" (UID: \"fd212b75-a818-45a3-946a-864a230b7707\") " pod="openshift-ingress/e2e-shared-a-w3-cf9b6f-openshift-default-56db6769f7-xl4xv"
Oct 30 19:44:48.301794 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:44:48.241826    4892 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c5510c1c-0da3-4b39-95ee-a9195c32e990-istio-token\") pod \"maas-default-gateway-openshift-default-6db586dd6c-lqrjf\" (UID: \"c5510c1c-0da3-4b39-95ee-a9195c32e990\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-lqrjf"
Oct 30 19:44:48.333677 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:44:48.333603    4892 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c5510c1c-0da3-4b39-95ee-a9195c32e990-istio-podinfo\") pod \"maas-default-gateway-openshift-default-6db586dd6c-lqrjf\" (UID: \"c5510c1c-0da3-4b39-95ee-a9195c32e990\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-lqrjf"
Oct 30 19:44:48.364770 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:44:48.364689    4892 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ntlts\" (UniqueName: \"kubernetes.io/projected/fd212b75-a818-45a3-946a-864a230b7707-kube-api-access-ntlts\") pod \"e2e-shared-a-w3-cf9b6f-openshift-default-56db6769f7-xl4xv\" (UID: \"fd212b75-a818-45a3-946a-864a230b7707\") " pod="openshift-ingress/e2e-shared-a-w3-cf9b6f-openshift-default-56db6769f7-xl4xv"
Oct 30 19:44:48.365201 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:44:48.365177    4892 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/fd212b75-a818-45a3-946a-864a230b7707-istio-envoy\") pod \"e2e-shared-a-w3-cf9b6f-openshift-default-56db6769f7-xl4xv\" (UID: \"fd212b75-a818-45a3-946a-864a230b7707\") " pod="openshift-ingress/e2e-shared-a-w3-cf9b6f-openshift-default-56db6769f7-xl4xv"
Oct 30 19:44:48.365576 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:44:48.365522    4892 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c5510c1c-0da3-4b39-95ee-a9195c32e990-istio-envoy\") pod \"maas-default-gateway-openshift-default-6db586dd6c-lqrjf\" (UID: \"c5510c1c-0da3-4b39-95ee-a9195c32e990\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-lqrjf"
Oct 30 19:44:48.379160 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:44:48.379126    4892 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6zkft\" (UniqueName: \"kubernetes.io/projected/c5510c1c-0da3-4b39-95ee-a9195c32e990-kube-api-access-6zkft\") pod \"maas-default-gateway-openshift-default-6db586dd6c-lqrjf\" (UID: \"c5510c1c-0da3-4b39-95ee-a9195c32e990\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-lqrjf"
Oct 30 19:44:48.386504 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:44:48.386228    4892 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w3-cf9b6f-openshift-default-56db6769f7-xl4xv"
Oct 30 19:44:48.602797 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:44:48.601728    4892 generic.go:356] "Generic (PLEG): container finished" podID="d299acaf-6665-4ab0-8333-cca9953d8f47" containerID="41779301ffee7b2cd47f02b4782841173bc03bd11947d1459aa6d7c6189ae9a2" exitCode=0
Oct 30 19:44:48.602797 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:44:48.601861    4892 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ae4eb22f-mxz6s" event={"ID":"d299acaf-6665-4ab0-8333-cca9953d8f47","Type":"ContainerDied","Data":"41779301ffee7b2cd47f02b4782841173bc03bd11947d1459aa6d7c6189ae9a2"}
Oct 30 19:44:48.641861 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:44:48.641828    4892 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-lqrjf"
Oct 30 19:44:49.116871 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:44:49.116778    4892 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w3-cf9b6f-openshift-default-56db6769f7-xl4xv"]
Oct 30 19:44:49.146647 ip-10-0-132-82 kubenswrapper[4892]: W1030 19:44:49.146620    4892 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfd212b75_a818_45a3_946a_864a230b7707.slice/crio-f0dbedfeca2a18f12365cac61a397772359acb31916d626a2010eb1eca587504 WatchSource:0}: Error finding container f0dbedfeca2a18f12365cac61a397772359acb31916d626a2010eb1eca587504: Status 404 returned error can't find the container with id f0dbedfeca2a18f12365cac61a397772359acb31916d626a2010eb1eca587504
Oct 30 19:44:49.149810 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:44:49.149766    4892 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892000Ki","pods":"250"}
Oct 30 19:44:49.149902 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:44:49.149863    4892 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892000Ki","pods":"250"}
Oct 30 19:44:49.149939 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:44:49.149908    4892 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892000Ki","pods":"250"}
Oct 30 19:44:49.166844 ip-10-0-132-82 kubenswrapper[4892]: W1030 19:44:49.166817    4892 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc5510c1c_0da3_4b39_95ee_a9195c32e990.slice/crio-9b6ccaf73a864a336a802613dde37395e73b532d240198ea85b05cc63718ac53 WatchSource:0}: Error finding container 9b6ccaf73a864a336a802613dde37395e73b532d240198ea85b05cc63718ac53: Status 404 returned error can't find the container with id 9b6ccaf73a864a336a802613dde37395e73b532d240198ea85b05cc63718ac53
Oct 30 19:44:49.172639 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:44:49.172593    4892 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-lqrjf"]
Oct 30 19:44:49.174985 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:44:49.174469    4892 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892000Ki","pods":"250"}
Oct 30 19:44:49.174985 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:44:49.174550    4892 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892000Ki","pods":"250"}
Oct 30 19:44:49.174985 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:44:49.174593    4892 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892000Ki","pods":"250"}
Oct 30 19:44:49.616233 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:44:49.616192    4892 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-lqrjf" event={"ID":"c5510c1c-0da3-4b39-95ee-a9195c32e990","Type":"ContainerStarted","Data":"cef76da3f34b3aaeb3e66a0565011103bbfd83112c3a11cc808885abcec85f90"}
Oct 30 19:44:49.616744 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:44:49.616265    4892 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-lqrjf" event={"ID":"c5510c1c-0da3-4b39-95ee-a9195c32e990","Type":"ContainerStarted","Data":"9b6ccaf73a864a336a802613dde37395e73b532d240198ea85b05cc63718ac53"}
Oct 30 19:44:49.619338 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:44:49.619241    4892 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w3-cf9b6f-openshift-default-56db6769f7-xl4xv" event={"ID":"fd212b75-a818-45a3-946a-864a230b7707","Type":"ContainerStarted","Data":"658c14d7f26cca7441ed312ad29eb8f102c041b9138faff109a3707e3f723c75"}
Oct 30 19:44:49.619338 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:44:49.619295    4892 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w3-cf9b6f-openshift-default-56db6769f7-xl4xv" event={"ID":"fd212b75-a818-45a3-946a-864a230b7707","Type":"ContainerStarted","Data":"f0dbedfeca2a18f12365cac61a397772359acb31916d626a2010eb1eca587504"}
Oct 30 19:44:49.642410 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:44:49.642372    4892 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-lqrjf"
Oct 30 19:44:49.759266 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:44:49.759202    4892 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-lqrjf" podStartSLOduration=2.759182686 podStartE2EDuration="2.759182686s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:44:47 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:44:49.757547511 +0000 UTC m=+1737.328013112" watchObservedRunningTime="2026-10-30 19:44:49.759182686 +0000 UTC m=+1737.329648286"
Oct 30 19:44:49.825222 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:44:49.825194    4892 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ae4eb22f-mxz6s"
Oct 30 19:44:49.828408 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:44:49.828384    4892 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d299acaf-6665-4ab0-8333-cca9953d8f47-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d299acaf-6665-4ab0-8333-cca9953d8f47-maas-api-openshift-ca\") pod \"d299acaf-6665-4ab0-8333-cca9953d8f47\" (UID: \"d299acaf-6665-4ab0-8333-cca9953d8f47\") "
Oct 30 19:44:49.828581 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:44:49.828466    4892 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/d299acaf-6665-4ab0-8333-cca9953d8f47-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d299acaf-6665-4ab0-8333-cca9953d8f47-maas-api-xks-ca\") pod \"d299acaf-6665-4ab0-8333-cca9953d8f47\" (UID: \"d299acaf-6665-4ab0-8333-cca9953d8f47\") "
Oct 30 19:44:49.829030 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:44:49.828999    4892 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d299acaf-6665-4ab0-8333-cca9953d8f47-maas-api-openshift-ca" pod "d299acaf-6665-4ab0-8333-cca9953d8f47" (UID: "d299acaf-6665-4ab0-8333-cca9953d8f47"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 19:44:49.829759 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:44:49.829250    4892 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d299acaf-6665-4ab0-8333-cca9953d8f47-maas-api-openshift-ca\") on node \"ip-10-0-132-82.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 19:44:49.831826 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:44:49.831800    4892 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d299acaf-6665-4ab0-8333-cca9953d8f47-maas-api-xks-ca" pod "d299acaf-6665-4ab0-8333-cca9953d8f47" (UID: "d299acaf-6665-4ab0-8333-cca9953d8f47"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 19:44:49.905600 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:44:49.905539    4892 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-a-w3-cf9b6f-openshift-default-56db6769f7-xl4xv" podStartSLOduration=2.905520763 podStartE2EDuration="2.905520763s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:44:47 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:44:49.897460152 +0000 UTC m=+1737.467925751" watchObservedRunningTime="2026-10-30 19:44:49.905520763 +0000 UTC m=+1737.475986363"
Oct 30 19:44:49.929844 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:44:49.929704    4892 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d299acaf-6665-4ab0-8333-cca9953d8f47-kube-api-access-6tksl\" (UniqueName: \"kubernetes.io/projected/d299acaf-6665-4ab0-8333-cca9953d8f47-kube-api-access-6tksl\") pod \"d299acaf-6665-4ab0-8333-cca9953d8f47\" (UID: \"d299acaf-6665-4ab0-8333-cca9953d8f47\") "
Oct 30 19:44:49.930700 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:44:49.930673    4892 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d299acaf-6665-4ab0-8333-cca9953d8f47-maas-api-xks-ca\") on node \"ip-10-0-132-82.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 19:44:49.932221 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:44:49.932196    4892 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d299acaf-6665-4ab0-8333-cca9953d8f47-kube-api-access-6tksl" pod "d299acaf-6665-4ab0-8333-cca9953d8f47" (UID: "d299acaf-6665-4ab0-8333-cca9953d8f47"). InnerVolumeSpecName "kube-api-access-6tksl". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:44:50.044083 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:44:50.043626    4892 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6tksl\" (UniqueName: \"kubernetes.io/projected/d299acaf-6665-4ab0-8333-cca9953d8f47-kube-api-access-6tksl\") on node \"ip-10-0-132-82.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 19:44:50.387053 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:44:50.387024    4892 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-a-w3-cf9b6f-openshift-default-56db6769f7-xl4xv"
Oct 30 19:44:50.395001 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:44:50.394932    4892 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-a-w3-cf9b6f-openshift-default-56db6769f7-xl4xv"
Oct 30 19:44:50.649082 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:44:50.644486    4892 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6db586dd6c-lqrjf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.145:15021/healthz/ready\": dial tcp 10.132.0.145:15021: connect: connection refused" start-of-body=
Oct 30 19:44:50.649082 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:44:50.644553    4892 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-lqrjf" podUID="c5510c1c-0da3-4b39-95ee-a9195c32e990" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.145:15021/healthz/ready\": dial tcp 10.132.0.145:15021: connect: connection refused"
Oct 30 19:44:50.674841 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:44:50.674812    4892 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ae4eb22f-mxz6s"
Oct 30 19:44:50.676011 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:44:50.674835    4892 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ae4eb22f-mxz6s" event={"ID":"d299acaf-6665-4ab0-8333-cca9953d8f47","Type":"ContainerDied","Data":"ae1fad81af5091f24c8155fd6e1e55c0de85edfcf9824dda359937a61a44f162"}
Oct 30 19:44:50.676011 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:44:50.674890    4892 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="ae1fad81af5091f24c8155fd6e1e55c0de85edfcf9824dda359937a61a44f162"
Oct 30 19:44:50.676011 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:44:50.675388    4892 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-a-w3-cf9b6f-openshift-default-56db6769f7-xl4xv"
Oct 30 19:44:50.676528 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:44:50.676509    4892 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-a-w3-cf9b6f-openshift-default-56db6769f7-xl4xv"
Oct 30 19:44:51.646374 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:44:51.646334    4892 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6db586dd6c-lqrjf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.145:15021/healthz/ready\": dial tcp 10.132.0.145:15021: connect: connection refused" start-of-body=
Oct 30 19:44:51.646667 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:44:51.646627    4892 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-lqrjf" podUID="c5510c1c-0da3-4b39-95ee-a9195c32e990" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.145:15021/healthz/ready\": dial tcp 10.132.0.145:15021: connect: connection refused"
Oct 30 19:44:52.653704 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:44:52.653663    4892 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6db586dd6c-lqrjf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.145:15021/healthz/ready\": dial tcp 10.132.0.145:15021: connect: connection refused" start-of-body=
Oct 30 19:44:52.654264 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:44:52.653730    4892 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-lqrjf" podUID="c5510c1c-0da3-4b39-95ee-a9195c32e990" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.145:15021/healthz/ready\": dial tcp 10.132.0.145:15021: connect: connection refused"
Oct 30 19:44:53.069523 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:44:53.069442    4892 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-203da51c-5zcqh"]
Oct 30 19:44:53.117812 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:44:53.117785    4892 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-203da51c-5zcqh"]
Oct 30 19:44:53.357047 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:44:53.356884    4892 scope.go:122] "RemoveContainer" containerID="8c2fda42837807c71d2e1d6969eaf6df339c1781989f8c4376e2430afc419be6"
Oct 30 19:44:53.372854 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:44:53.372833    4892 scope.go:122] "RemoveContainer" containerID="43bf6d0c5d9060c1c318ee33f6aee895a3b122ee22e8ba36475fc912923bc1b5"
Oct 30 19:44:53.384653 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:44:53.384626    4892 scope.go:122] "RemoveContainer" containerID="d4e63a468abe6bde7fe79679cfdcae37580628eb0787861866792f48124d2d43"
Oct 30 19:44:53.395246 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:44:53.395223    4892 scope.go:122] "RemoveContainer" containerID="8e04dd468eafc106b973c87398222e366ead23eab487a8de1beed4e51918f279"
Oct 30 19:44:53.663847 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:44:53.662348    4892 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6db586dd6c-lqrjf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.145:15021/healthz/ready\": dial tcp 10.132.0.145:15021: connect: connection refused" start-of-body=
Oct 30 19:44:53.663847 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:44:53.662395    4892 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-lqrjf" podUID="c5510c1c-0da3-4b39-95ee-a9195c32e990" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.145:15021/healthz/ready\": dial tcp 10.132.0.145:15021: connect: connection refused"
Oct 30 19:44:54.571357 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:44:54.571323    4892 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="7cc551c1-9fcd-4df4-afa9-db440c768fc5" path="/var/lib/kubelet/pods/7cc551c1-9fcd-4df4-afa9-db440c768fc5/volumes"
Oct 30 19:44:54.651991 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:44:54.645297    4892 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6db586dd6c-lqrjf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.145:15021/healthz/ready\": dial tcp 10.132.0.145:15021: connect: connection refused" start-of-body=
Oct 30 19:44:54.651991 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:44:54.645365    4892 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-lqrjf" podUID="c5510c1c-0da3-4b39-95ee-a9195c32e990" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.145:15021/healthz/ready\": dial tcp 10.132.0.145:15021: connect: connection refused"
Oct 30 19:44:54.874161 ip-10-0-132-82 kubenswrapper[4892]: E1030 19:44:54.874080    4892 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-jrm87" containerName="extractor"
Oct 30 19:44:55.642829 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:44:55.642787    4892 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6db586dd6c-lqrjf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.145:15021/healthz/ready\": dial tcp 10.132.0.145:15021: connect: connection refused" start-of-body=
Oct 30 19:44:55.643060 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:44:55.642855    4892 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-lqrjf" podUID="c5510c1c-0da3-4b39-95ee-a9195c32e990" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.145:15021/healthz/ready\": dial tcp 10.132.0.145:15021: connect: connection refused"
Oct 30 19:44:56.650987 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:44:56.643240    4892 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6db586dd6c-lqrjf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.145:15021/healthz/ready\": dial tcp 10.132.0.145:15021: connect: connection refused" start-of-body=
Oct 30 19:44:56.650987 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:44:56.643320    4892 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-lqrjf" podUID="c5510c1c-0da3-4b39-95ee-a9195c32e990" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.145:15021/healthz/ready\": dial tcp 10.132.0.145:15021: connect: connection refused"
Oct 30 19:44:56.763632 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:44:56.763594    4892 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w3-cf9b6f-59d848ddcf-jxlzh"]
Oct 30 19:44:56.764804 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:44:56.764777    4892 cpu_manager.go:414] "containerMap: removing container" podUID="d299acaf-6665-4ab0-8333-cca9953d8f47" containerName="revoke-keys"
Oct 30 19:44:56.764994 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:44:56.764956    4892 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d299acaf-6665-4ab0-8333-cca9953d8f47" containerName="revoke-keys"
Oct 30 19:44:56.765502 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:44:56.765481    4892 memory_manager.go:365] "RemoveStaleState removing state" podUID="d299acaf-6665-4ab0-8333-cca9953d8f47" containerName="revoke-keys"
Oct 30 19:44:56.813291 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:44:56.813260    4892 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-cf9b6f-59d848ddcf-jxlzh"
Oct 30 19:44:56.830992 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:44:56.825532    4892 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w3-cf9b6f-59d848ddcf-jxlzh"]
Oct 30 19:44:56.857022 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:44:56.856554    4892 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qwskv\" (UniqueName: \"kubernetes.io/projected/4bd85267-2235-4bc6-80cd-473a48b0098c-kube-api-access-qwskv\") pod \"payload-pre-processing-e2e-shared-a-w3-cf9b6f-59d848ddcf-jxlzh\" (UID: \"4bd85267-2235-4bc6-80cd-473a48b0098c\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-cf9b6f-59d848ddcf-jxlzh"
Oct 30 19:44:56.857022 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:44:56.856734    4892 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4bd85267-2235-4bc6-80cd-473a48b0098c-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-a-w3-cf9b6f-59d848ddcf-jxlzh\" (UID: \"4bd85267-2235-4bc6-80cd-473a48b0098c\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-cf9b6f-59d848ddcf-jxlzh"
Oct 30 19:44:56.872987 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:44:56.870884    4892 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-shared-a-w3-cf9b6f\""
Oct 30 19:44:56.872987 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:44:56.871075    4892 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-shared-a-w3-cf9b6f-dockercfg-h7twk\""
Oct 30 19:44:57.001997 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:44:57.001888    4892 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4bd85267-2235-4bc6-80cd-473a48b0098c-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-a-w3-cf9b6f-59d848ddcf-jxlzh\" (UID: \"4bd85267-2235-4bc6-80cd-473a48b0098c\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-cf9b6f-59d848ddcf-jxlzh"
Oct 30 19:44:57.019104 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:44:57.019063    4892 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qwskv\" (UniqueName: \"kubernetes.io/projected/4bd85267-2235-4bc6-80cd-473a48b0098c-kube-api-access-qwskv\") pod \"payload-pre-processing-e2e-shared-a-w3-cf9b6f-59d848ddcf-jxlzh\" (UID: \"4bd85267-2235-4bc6-80cd-473a48b0098c\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-cf9b6f-59d848ddcf-jxlzh"
Oct 30 19:44:57.046994 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:44:57.020662    4892 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4bd85267-2235-4bc6-80cd-473a48b0098c-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-a-w3-cf9b6f-59d848ddcf-jxlzh\" (UID: \"4bd85267-2235-4bc6-80cd-473a48b0098c\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-cf9b6f-59d848ddcf-jxlzh"
Oct 30 19:44:57.054095 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:44:57.037367    4892 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w3-cf9b6f-7bb7c4ccf4-zc2s9"]
Oct 30 19:44:57.180321 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:44:57.180252    4892 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qwskv\" (UniqueName: \"kubernetes.io/projected/4bd85267-2235-4bc6-80cd-473a48b0098c-kube-api-access-qwskv\") pod \"payload-pre-processing-e2e-shared-a-w3-cf9b6f-59d848ddcf-jxlzh\" (UID: \"4bd85267-2235-4bc6-80cd-473a48b0098c\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-cf9b6f-59d848ddcf-jxlzh"
Oct 30 19:44:57.200207 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:44:57.200179    4892 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w3-cf9b6f-7bb7c4ccf4-zc2s9"]
Oct 30 19:44:57.200565 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:44:57.200493    4892 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w3-cf9b6f-7bb7c4ccf4-zc2s9"
Oct 30 19:44:57.238217 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:44:57.238184    4892 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mpw2q\" (UniqueName: \"kubernetes.io/projected/44646b7f-870a-49c1-bd3d-8e3e7817d6a3-kube-api-access-mpw2q\") pod \"payload-processing-e2e-shared-a-w3-cf9b6f-7bb7c4ccf4-zc2s9\" (UID: \"44646b7f-870a-49c1-bd3d-8e3e7817d6a3\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w3-cf9b6f-7bb7c4ccf4-zc2s9"
Oct 30 19:44:57.238496 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:44:57.238470    4892 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/44646b7f-870a-49c1-bd3d-8e3e7817d6a3-plugins-config-volume\") pod \"payload-processing-e2e-shared-a-w3-cf9b6f-7bb7c4ccf4-zc2s9\" (UID: \"44646b7f-870a-49c1-bd3d-8e3e7817d6a3\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w3-cf9b6f-7bb7c4ccf4-zc2s9"
Oct 30 19:44:57.278049 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:44:57.277952    4892 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-cf9b6f-59d848ddcf-jxlzh"
Oct 30 19:44:57.341740 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:44:57.340897    4892 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mpw2q\" (UniqueName: \"kubernetes.io/projected/44646b7f-870a-49c1-bd3d-8e3e7817d6a3-kube-api-access-mpw2q\") pod \"payload-processing-e2e-shared-a-w3-cf9b6f-7bb7c4ccf4-zc2s9\" (UID: \"44646b7f-870a-49c1-bd3d-8e3e7817d6a3\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w3-cf9b6f-7bb7c4ccf4-zc2s9"
Oct 30 19:44:57.341740 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:44:57.341326    4892 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/44646b7f-870a-49c1-bd3d-8e3e7817d6a3-plugins-config-volume\") pod \"payload-processing-e2e-shared-a-w3-cf9b6f-7bb7c4ccf4-zc2s9\" (UID: \"44646b7f-870a-49c1-bd3d-8e3e7817d6a3\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w3-cf9b6f-7bb7c4ccf4-zc2s9"
Oct 30 19:44:57.343031 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:44:57.342412    4892 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/44646b7f-870a-49c1-bd3d-8e3e7817d6a3-plugins-config-volume\") pod \"payload-processing-e2e-shared-a-w3-cf9b6f-7bb7c4ccf4-zc2s9\" (UID: \"44646b7f-870a-49c1-bd3d-8e3e7817d6a3\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w3-cf9b6f-7bb7c4ccf4-zc2s9"
Oct 30 19:44:57.411409 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:44:57.411012    4892 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mpw2q\" (UniqueName: \"kubernetes.io/projected/44646b7f-870a-49c1-bd3d-8e3e7817d6a3-kube-api-access-mpw2q\") pod \"payload-processing-e2e-shared-a-w3-cf9b6f-7bb7c4ccf4-zc2s9\" (UID: \"44646b7f-870a-49c1-bd3d-8e3e7817d6a3\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w3-cf9b6f-7bb7c4ccf4-zc2s9"
Oct 30 19:44:57.538452 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:44:57.538362    4892 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w3-cf9b6f-7bb7c4ccf4-zc2s9"
Oct 30 19:44:57.643208 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:44:57.643177    4892 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6db586dd6c-lqrjf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.145:15021/healthz/ready\": dial tcp 10.132.0.145:15021: connect: connection refused" start-of-body=
Oct 30 19:44:57.643369 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:44:57.643233    4892 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-lqrjf" podUID="c5510c1c-0da3-4b39-95ee-a9195c32e990" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.145:15021/healthz/ready\": dial tcp 10.132.0.145:15021: connect: connection refused"
Oct 30 19:44:57.673332 ip-10-0-132-82 kubenswrapper[4892]: W1030 19:44:57.673300    4892 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4bd85267_2235_4bc6_80cd_473a48b0098c.slice/crio-d9968232891cfdb768d249fec1a11fc94db1f847274e87aa519cfcbaf1f891b2 WatchSource:0}: Error finding container d9968232891cfdb768d249fec1a11fc94db1f847274e87aa519cfcbaf1f891b2: Status 404 returned error can't find the container with id d9968232891cfdb768d249fec1a11fc94db1f847274e87aa519cfcbaf1f891b2
Oct 30 19:44:57.677473 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:44:57.676620    4892 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w3-cf9b6f-59d848ddcf-jxlzh"]
Oct 30 19:44:57.771934 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:44:57.771891    4892 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-cf9b6f-59d848ddcf-jxlzh" event={"ID":"4bd85267-2235-4bc6-80cd-473a48b0098c","Type":"ContainerStarted","Data":"d9968232891cfdb768d249fec1a11fc94db1f847274e87aa519cfcbaf1f891b2"}
Oct 30 19:44:57.772068 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:44:57.772041    4892 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-cf9b6f-59d848ddcf-jxlzh"
Oct 30 19:44:57.895729 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:44:57.895658    4892 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-cf9b6f-59d848ddcf-jxlzh" podStartSLOduration=1.895644618 podStartE2EDuration="1.895644618s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:44:56 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:44:57.888158877 +0000 UTC m=+1745.458624480" watchObservedRunningTime="2026-10-30 19:44:57.895644618 +0000 UTC m=+1745.466110229"
Oct 30 19:44:57.935119 ip-10-0-132-82 kubenswrapper[4892]: W1030 19:44:57.935086    4892 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod44646b7f_870a_49c1_bd3d_8e3e7817d6a3.slice/crio-7d97dd2f5c482b3d6d85d7e9a11c2b16df4ac9483c52b0dc9fbca1000e1fa67f WatchSource:0}: Error finding container 7d97dd2f5c482b3d6d85d7e9a11c2b16df4ac9483c52b0dc9fbca1000e1fa67f: Status 404 returned error can't find the container with id 7d97dd2f5c482b3d6d85d7e9a11c2b16df4ac9483c52b0dc9fbca1000e1fa67f
Oct 30 19:44:57.936719 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:44:57.936654    4892 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w3-cf9b6f-7bb7c4ccf4-zc2s9"]
Oct 30 19:44:58.785889 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:44:58.785848    4892 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-cf9b6f-59d848ddcf-jxlzh" event={"ID":"4bd85267-2235-4bc6-80cd-473a48b0098c","Type":"ContainerStarted","Data":"b14502fe4af0f3ad1bdf46b8cc4c5b7c04040ee3b14e2eb804d39f95a2741824"}
Oct 30 19:44:58.788123 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:44:58.788092    4892 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w3-cf9b6f-7bb7c4ccf4-zc2s9" event={"ID":"44646b7f-870a-49c1-bd3d-8e3e7817d6a3","Type":"ContainerStarted","Data":"4d233e5567a3b6abc12cf1ec1841e234250403e061929b81c33958ee9a4b79fc"}
Oct 30 19:44:58.788285 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:44:58.788144    4892 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w3-cf9b6f-7bb7c4ccf4-zc2s9" event={"ID":"44646b7f-870a-49c1-bd3d-8e3e7817d6a3","Type":"ContainerStarted","Data":"7d97dd2f5c482b3d6d85d7e9a11c2b16df4ac9483c52b0dc9fbca1000e1fa67f"}
Oct 30 19:44:58.788414 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:44:58.788352    4892 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-shared-a-w3-cf9b6f-7bb7c4ccf4-zc2s9"
Oct 30 19:44:58.885847 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:44:58.885818    4892 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-lqrjf"
Oct 30 19:44:58.886584 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:44:58.886552    4892 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-lqrjf"
Oct 30 19:44:58.887530 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:44:58.887507    4892 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-lqrjf"
Oct 30 19:44:58.907541 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:44:58.907490    4892 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-shared-a-w3-cf9b6f-7bb7c4ccf4-zc2s9" podStartSLOduration=2.907475701 podStartE2EDuration="2.907475701s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:44:56 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:44:58.905514951 +0000 UTC m=+1746.475980556" watchObservedRunningTime="2026-10-30 19:44:58.907475701 +0000 UTC m=+1746.477941307"
Oct 30 19:45:00.241262 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:45:00.241231    4892 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-2d9861-29889825-ngvs4"]
Oct 30 19:45:00.245701 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:45:00.245682    4892 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-2d9861-29889825-ngvs4"
Oct 30 19:45:00.246459 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:45:00.246441    4892 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-ada8d7-29889825-mzfrg"]
Oct 30 19:45:00.250756 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:45:00.250738    4892 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-ada8d7-29889825-mzfrg"
Oct 30 19:45:00.251385 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:45:00.251365    4892 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-5f55cd-29889825-fb6g6"]
Oct 30 19:45:00.259623 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:45:00.259593    4892 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-5f55cd-29889825-fb6g6"
Oct 30 19:45:00.290101 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:45:00.290075    4892 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-d52df0-29889825-kxd49"]
Oct 30 19:45:00.293895 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:45:00.293870    4892 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2r4fg\" (UniqueName: \"kubernetes.io/projected/0bbf2f20-664b-4c26-82c8-e779d16a54e3-kube-api-access-2r4fg\") pod \"maas-api-key-cleanup-e2e-worker-w0-2d9861-29889825-ngvs4\" (UID: \"0bbf2f20-664b-4c26-82c8-e779d16a54e3\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-2d9861-29889825-ngvs4"
Oct 30 19:45:00.294033 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:45:00.294017    4892 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/0bbf2f20-664b-4c26-82c8-e779d16a54e3-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w0-2d9861-29889825-ngvs4\" (UID: \"0bbf2f20-664b-4c26-82c8-e779d16a54e3\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-2d9861-29889825-ngvs4"
Oct 30 19:45:00.294089 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:45:00.294044    4892 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/01eed39c-1a18-4a99-b1a6-4ffba9703c24-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w6-5f55cd-29889825-fb6g6\" (UID: \"01eed39c-1a18-4a99-b1a6-4ffba9703c24\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-5f55cd-29889825-fb6g6"
Oct 30 19:45:00.294155 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:45:00.294098    4892 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/bac2c08f-5b25-4ac6-9fb3-bf8e539c6ed8-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w3-ada8d7-29889825-mzfrg\" (UID: \"bac2c08f-5b25-4ac6-9fb3-bf8e539c6ed8\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-ada8d7-29889825-mzfrg"
Oct 30 19:45:00.294209 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:45:00.294170    4892 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mn42q\" (UniqueName: \"kubernetes.io/projected/bac2c08f-5b25-4ac6-9fb3-bf8e539c6ed8-kube-api-access-mn42q\") pod \"maas-api-key-cleanup-e2e-worker-w3-ada8d7-29889825-mzfrg\" (UID: \"bac2c08f-5b25-4ac6-9fb3-bf8e539c6ed8\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-ada8d7-29889825-mzfrg"
Oct 30 19:45:00.294331 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:45:00.294282    4892 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-96kbh\" (UniqueName: \"kubernetes.io/projected/01eed39c-1a18-4a99-b1a6-4ffba9703c24-kube-api-access-96kbh\") pod \"maas-api-key-cleanup-e2e-worker-w6-5f55cd-29889825-fb6g6\" (UID: \"01eed39c-1a18-4a99-b1a6-4ffba9703c24\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-5f55cd-29889825-fb6g6"
Oct 30 19:45:00.294490 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:45:00.294472    4892 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-d52df0-29889825-kxd49"
Oct 30 19:45:00.295171 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:45:00.295145    4892 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-b5178f-29889825-zncxf"]
Oct 30 19:45:00.299691 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:45:00.299671    4892 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-b5178f-29889825-zncxf"
Oct 30 19:45:00.300462 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:45:00.300444    4892 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-a3f9b7-29889825-wdl5n"]
Oct 30 19:45:00.305245 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:45:00.305229    4892 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-a3f9b7-29889825-wdl5n"
Oct 30 19:45:00.314654 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:45:00.314616    4892 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 30 19:45:00.315097 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:45:00.315079    4892 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 30 19:45:00.334076 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:45:00.334050    4892 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-nlxp4\""
Oct 30 19:45:00.339923 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:45:00.339896    4892 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-migrate-d74c1bd9-29889825-wwbj8"]
Oct 30 19:45:00.344915 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:45:00.344893    4892 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-migrate-d74c1bd9-29889825-wwbj8"
Oct 30 19:45:00.349673 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:45:00.348381    4892 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-2d9861-29889825-ngvs4"]
Oct 30 19:45:00.356431 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:45:00.356406    4892 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-ada8d7-29889825-mzfrg"]
Oct 30 19:45:00.366715 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:45:00.366694    4892 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-5f55cd-29889825-fb6g6"]
Oct 30 19:45:00.387476 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:45:00.386864    4892 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-b5178f-29889825-zncxf"]
Oct 30 19:45:00.392842 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:45:00.392809    4892 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-a3f9b7-29889825-wdl5n"]
Oct 30 19:45:00.395332 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:45:00.395309    4892 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dv64b\" (UniqueName: \"kubernetes.io/projected/d9801904-680f-4253-9104-04bbf36d9b35-kube-api-access-dv64b\") pod \"maas-api-key-cleanup-e2e-worker-w1-d52df0-29889825-kxd49\" (UID: \"d9801904-680f-4253-9104-04bbf36d9b35\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-d52df0-29889825-kxd49"
Oct 30 19:45:00.395441 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:45:00.395342    4892 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mn42q\" (UniqueName: \"kubernetes.io/projected/bac2c08f-5b25-4ac6-9fb3-bf8e539c6ed8-kube-api-access-mn42q\") pod \"maas-api-key-cleanup-e2e-worker-w3-ada8d7-29889825-mzfrg\" (UID: \"bac2c08f-5b25-4ac6-9fb3-bf8e539c6ed8\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-ada8d7-29889825-mzfrg"
Oct 30 19:45:00.395441 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:45:00.395371    4892 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/8277bac6-a556-4e2d-bc2c-17121d44e4b5-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w2-a3f9b7-29889825-wdl5n\" (UID: \"8277bac6-a556-4e2d-bc2c-17121d44e4b5\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-a3f9b7-29889825-wdl5n"
Oct 30 19:45:00.395567 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:45:00.395482    4892 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fz6b7\" (UniqueName: \"kubernetes.io/projected/8277bac6-a556-4e2d-bc2c-17121d44e4b5-kube-api-access-fz6b7\") pod \"maas-api-key-cleanup-e2e-worker-w2-a3f9b7-29889825-wdl5n\" (UID: \"8277bac6-a556-4e2d-bc2c-17121d44e4b5\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-a3f9b7-29889825-wdl5n"
Oct 30 19:45:00.395567 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:45:00.395520    4892 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-m8hqb\" (UniqueName: \"kubernetes.io/projected/c18a2644-72a4-4b63-bf7d-c55bcb715f7c-kube-api-access-m8hqb\") pod \"maas-api-key-cleanup-e2e-worker-w4-b5178f-29889825-zncxf\" (UID: \"c18a2644-72a4-4b63-bf7d-c55bcb715f7c\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-b5178f-29889825-zncxf"
Oct 30 19:45:00.395651 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:45:00.395624    4892 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/d9801904-680f-4253-9104-04bbf36d9b35-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w1-d52df0-29889825-kxd49\" (UID: \"d9801904-680f-4253-9104-04bbf36d9b35\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-d52df0-29889825-kxd49"
Oct 30 19:45:00.395696 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:45:00.395662    4892 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-96kbh\" (UniqueName: \"kubernetes.io/projected/01eed39c-1a18-4a99-b1a6-4ffba9703c24-kube-api-access-96kbh\") pod \"maas-api-key-cleanup-e2e-worker-w6-5f55cd-29889825-fb6g6\" (UID: \"01eed39c-1a18-4a99-b1a6-4ffba9703c24\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-5f55cd-29889825-fb6g6"
Oct 30 19:45:00.395751 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:45:00.395697    4892 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/b8655bf2-a203-4072-8894-994c2a780657-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-migrate-d74c1bd9-29889825-wwbj8\" (UID: \"b8655bf2-a203-4072-8894-994c2a780657\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-migrate-d74c1bd9-29889825-wwbj8"
Oct 30 19:45:00.395751 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:45:00.395742    4892 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2r4fg\" (UniqueName: \"kubernetes.io/projected/0bbf2f20-664b-4c26-82c8-e779d16a54e3-kube-api-access-2r4fg\") pod \"maas-api-key-cleanup-e2e-worker-w0-2d9861-29889825-ngvs4\" (UID: \"0bbf2f20-664b-4c26-82c8-e779d16a54e3\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-2d9861-29889825-ngvs4"
Oct 30 19:45:00.395884 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:45:00.395861    4892 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/0bbf2f20-664b-4c26-82c8-e779d16a54e3-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w0-2d9861-29889825-ngvs4\" (UID: \"0bbf2f20-664b-4c26-82c8-e779d16a54e3\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-2d9861-29889825-ngvs4"
Oct 30 19:45:00.395953 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:45:00.395902    4892 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/01eed39c-1a18-4a99-b1a6-4ffba9703c24-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w6-5f55cd-29889825-fb6g6\" (UID: \"01eed39c-1a18-4a99-b1a6-4ffba9703c24\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-5f55cd-29889825-fb6g6"
Oct 30 19:45:00.395953 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:45:00.395935    4892 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/c18a2644-72a4-4b63-bf7d-c55bcb715f7c-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w4-b5178f-29889825-zncxf\" (UID: \"c18a2644-72a4-4b63-bf7d-c55bcb715f7c\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-b5178f-29889825-zncxf"
Oct 30 19:45:00.396068 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:45:00.396020    4892 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/bac2c08f-5b25-4ac6-9fb3-bf8e539c6ed8-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w3-ada8d7-29889825-mzfrg\" (UID: \"bac2c08f-5b25-4ac6-9fb3-bf8e539c6ed8\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-ada8d7-29889825-mzfrg"
Oct 30 19:45:00.396068 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:45:00.396048    4892 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-l7rq8\" (UniqueName: \"kubernetes.io/projected/b8655bf2-a203-4072-8894-994c2a780657-kube-api-access-l7rq8\") pod \"maas-api-key-cleanup-e2e-migrate-d74c1bd9-29889825-wwbj8\" (UID: \"b8655bf2-a203-4072-8894-994c2a780657\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-migrate-d74c1bd9-29889825-wwbj8"
Oct 30 19:45:00.396568 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:45:00.396551    4892 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/bac2c08f-5b25-4ac6-9fb3-bf8e539c6ed8-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w3-ada8d7-29889825-mzfrg\" (UID: \"bac2c08f-5b25-4ac6-9fb3-bf8e539c6ed8\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-ada8d7-29889825-mzfrg"
Oct 30 19:45:00.396685 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:45:00.396659    4892 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/0bbf2f20-664b-4c26-82c8-e779d16a54e3-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w0-2d9861-29889825-ngvs4\" (UID: \"0bbf2f20-664b-4c26-82c8-e779d16a54e3\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-2d9861-29889825-ngvs4"
Oct 30 19:45:00.396758 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:45:00.396679    4892 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/01eed39c-1a18-4a99-b1a6-4ffba9703c24-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w6-5f55cd-29889825-fb6g6\" (UID: \"01eed39c-1a18-4a99-b1a6-4ffba9703c24\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-5f55cd-29889825-fb6g6"
Oct 30 19:45:00.438035 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:45:00.438011    4892 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-d52df0-29889825-kxd49"]
Oct 30 19:45:00.463126 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:45:00.463096    4892 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-migrate-d74c1bd9-29889825-wwbj8"]
Oct 30 19:45:00.471431 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:45:00.471407    4892 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-96kbh\" (UniqueName: \"kubernetes.io/projected/01eed39c-1a18-4a99-b1a6-4ffba9703c24-kube-api-access-96kbh\") pod \"maas-api-key-cleanup-e2e-worker-w6-5f55cd-29889825-fb6g6\" (UID: \"01eed39c-1a18-4a99-b1a6-4ffba9703c24\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-5f55cd-29889825-fb6g6"
Oct 30 19:45:00.471551 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:45:00.471473    4892 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2r4fg\" (UniqueName: \"kubernetes.io/projected/0bbf2f20-664b-4c26-82c8-e779d16a54e3-kube-api-access-2r4fg\") pod \"maas-api-key-cleanup-e2e-worker-w0-2d9861-29889825-ngvs4\" (UID: \"0bbf2f20-664b-4c26-82c8-e779d16a54e3\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-2d9861-29889825-ngvs4"
Oct 30 19:45:00.472300 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:45:00.472283    4892 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mn42q\" (UniqueName: \"kubernetes.io/projected/bac2c08f-5b25-4ac6-9fb3-bf8e539c6ed8-kube-api-access-mn42q\") pod \"maas-api-key-cleanup-e2e-worker-w3-ada8d7-29889825-mzfrg\" (UID: \"bac2c08f-5b25-4ac6-9fb3-bf8e539c6ed8\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-ada8d7-29889825-mzfrg"
Oct 30 19:45:00.497987 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:45:00.497906    4892 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/c18a2644-72a4-4b63-bf7d-c55bcb715f7c-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w4-b5178f-29889825-zncxf\" (UID: \"c18a2644-72a4-4b63-bf7d-c55bcb715f7c\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-b5178f-29889825-zncxf"
Oct 30 19:45:00.498098 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:45:00.497983    4892 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-l7rq8\" (UniqueName: \"kubernetes.io/projected/b8655bf2-a203-4072-8894-994c2a780657-kube-api-access-l7rq8\") pod \"maas-api-key-cleanup-e2e-migrate-d74c1bd9-29889825-wwbj8\" (UID: \"b8655bf2-a203-4072-8894-994c2a780657\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-migrate-d74c1bd9-29889825-wwbj8"
Oct 30 19:45:00.498098 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:45:00.498093    4892 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dv64b\" (UniqueName: \"kubernetes.io/projected/d9801904-680f-4253-9104-04bbf36d9b35-kube-api-access-dv64b\") pod \"maas-api-key-cleanup-e2e-worker-w1-d52df0-29889825-kxd49\" (UID: \"d9801904-680f-4253-9104-04bbf36d9b35\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-d52df0-29889825-kxd49"
Oct 30 19:45:00.498241 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:45:00.498137    4892 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/8277bac6-a556-4e2d-bc2c-17121d44e4b5-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w2-a3f9b7-29889825-wdl5n\" (UID: \"8277bac6-a556-4e2d-bc2c-17121d44e4b5\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-a3f9b7-29889825-wdl5n"
Oct 30 19:45:00.498241 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:45:00.498183    4892 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fz6b7\" (UniqueName: \"kubernetes.io/projected/8277bac6-a556-4e2d-bc2c-17121d44e4b5-kube-api-access-fz6b7\") pod \"maas-api-key-cleanup-e2e-worker-w2-a3f9b7-29889825-wdl5n\" (UID: \"8277bac6-a556-4e2d-bc2c-17121d44e4b5\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-a3f9b7-29889825-wdl5n"
Oct 30 19:45:00.498241 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:45:00.498208    4892 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-m8hqb\" (UniqueName: \"kubernetes.io/projected/c18a2644-72a4-4b63-bf7d-c55bcb715f7c-kube-api-access-m8hqb\") pod \"maas-api-key-cleanup-e2e-worker-w4-b5178f-29889825-zncxf\" (UID: \"c18a2644-72a4-4b63-bf7d-c55bcb715f7c\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-b5178f-29889825-zncxf"
Oct 30 19:45:00.498476 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:45:00.498286    4892 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/d9801904-680f-4253-9104-04bbf36d9b35-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w1-d52df0-29889825-kxd49\" (UID: \"d9801904-680f-4253-9104-04bbf36d9b35\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-d52df0-29889825-kxd49"
Oct 30 19:45:00.498476 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:45:00.498381    4892 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/b8655bf2-a203-4072-8894-994c2a780657-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-migrate-d74c1bd9-29889825-wwbj8\" (UID: \"b8655bf2-a203-4072-8894-994c2a780657\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-migrate-d74c1bd9-29889825-wwbj8"
Oct 30 19:45:00.498692 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:45:00.498671    4892 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/c18a2644-72a4-4b63-bf7d-c55bcb715f7c-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w4-b5178f-29889825-zncxf\" (UID: \"c18a2644-72a4-4b63-bf7d-c55bcb715f7c\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-b5178f-29889825-zncxf"
Oct 30 19:45:00.499094 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:45:00.499071    4892 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/8277bac6-a556-4e2d-bc2c-17121d44e4b5-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w2-a3f9b7-29889825-wdl5n\" (UID: \"8277bac6-a556-4e2d-bc2c-17121d44e4b5\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-a3f9b7-29889825-wdl5n"
Oct 30 19:45:00.499250 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:45:00.499229    4892 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/b8655bf2-a203-4072-8894-994c2a780657-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-migrate-d74c1bd9-29889825-wwbj8\" (UID: \"b8655bf2-a203-4072-8894-994c2a780657\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-migrate-d74c1bd9-29889825-wwbj8"
Oct 30 19:45:00.499343 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:45:00.499321    4892 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/d9801904-680f-4253-9104-04bbf36d9b35-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w1-d52df0-29889825-kxd49\" (UID: \"d9801904-680f-4253-9104-04bbf36d9b35\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-d52df0-29889825-kxd49"
Oct 30 19:45:00.557488 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:45:00.557460    4892 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fz6b7\" (UniqueName: \"kubernetes.io/projected/8277bac6-a556-4e2d-bc2c-17121d44e4b5-kube-api-access-fz6b7\") pod \"maas-api-key-cleanup-e2e-worker-w2-a3f9b7-29889825-wdl5n\" (UID: \"8277bac6-a556-4e2d-bc2c-17121d44e4b5\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-a3f9b7-29889825-wdl5n"
Oct 30 19:45:00.557842 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:45:00.557820    4892 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-m8hqb\" (UniqueName: \"kubernetes.io/projected/c18a2644-72a4-4b63-bf7d-c55bcb715f7c-kube-api-access-m8hqb\") pod \"maas-api-key-cleanup-e2e-worker-w4-b5178f-29889825-zncxf\" (UID: \"c18a2644-72a4-4b63-bf7d-c55bcb715f7c\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-b5178f-29889825-zncxf"
Oct 30 19:45:00.557901 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:45:00.557851    4892 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dv64b\" (UniqueName: \"kubernetes.io/projected/d9801904-680f-4253-9104-04bbf36d9b35-kube-api-access-dv64b\") pod \"maas-api-key-cleanup-e2e-worker-w1-d52df0-29889825-kxd49\" (UID: \"d9801904-680f-4253-9104-04bbf36d9b35\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-d52df0-29889825-kxd49"
Oct 30 19:45:00.558003 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:45:00.557984    4892 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-l7rq8\" (UniqueName: \"kubernetes.io/projected/b8655bf2-a203-4072-8894-994c2a780657-kube-api-access-l7rq8\") pod \"maas-api-key-cleanup-e2e-migrate-d74c1bd9-29889825-wwbj8\" (UID: \"b8655bf2-a203-4072-8894-994c2a780657\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-migrate-d74c1bd9-29889825-wwbj8"
Oct 30 19:45:00.568685 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:45:00.568666    4892 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-2d9861-29889825-ngvs4"
Oct 30 19:45:00.576643 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:45:00.576622    4892 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-ada8d7-29889825-mzfrg"
Oct 30 19:45:00.583627 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:45:00.583607    4892 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-5f55cd-29889825-fb6g6"
Oct 30 19:45:00.610522 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:45:00.610487    4892 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-d52df0-29889825-kxd49"
Oct 30 19:45:00.618250 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:45:00.618080    4892 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-b5178f-29889825-zncxf"
Oct 30 19:45:00.638750 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:45:00.638188    4892 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-a3f9b7-29889825-wdl5n"
Oct 30 19:45:00.663765 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:45:00.663654    4892 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-migrate-d74c1bd9-29889825-wwbj8"
Oct 30 19:45:01.218918 ip-10-0-132-82 kubenswrapper[4892]: W1030 19:45:01.218891    4892 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0bbf2f20_664b_4c26_82c8_e779d16a54e3.slice/crio-cb6b4ba662c32f4a0a7fcccf53e8517c1610388f47c1fafa5263a732e798abb8 WatchSource:0}: Error finding container cb6b4ba662c32f4a0a7fcccf53e8517c1610388f47c1fafa5263a732e798abb8: Status 404 returned error can't find the container with id cb6b4ba662c32f4a0a7fcccf53e8517c1610388f47c1fafa5263a732e798abb8
Oct 30 19:45:01.222867 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:45:01.222847    4892 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-2d9861-29889825-ngvs4"]
Oct 30 19:45:01.253821 ip-10-0-132-82 kubenswrapper[4892]: W1030 19:45:01.253794    4892 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod01eed39c_1a18_4a99_b1a6_4ffba9703c24.slice/crio-47c1ecc95a66fe77f9b591197f067c689aaf573e5324c24a0546fa952cc1685f WatchSource:0}: Error finding container 47c1ecc95a66fe77f9b591197f067c689aaf573e5324c24a0546fa952cc1685f: Status 404 returned error can't find the container with id 47c1ecc95a66fe77f9b591197f067c689aaf573e5324c24a0546fa952cc1685f
Oct 30 19:45:01.255863 ip-10-0-132-82 kubenswrapper[4892]: W1030 19:45:01.255839    4892 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbac2c08f_5b25_4ac6_9fb3_bf8e539c6ed8.slice/crio-1ce8788f196033f21b40487a70dbfde5420e7acd49deda8b62b1a291c96b765d WatchSource:0}: Error finding container 1ce8788f196033f21b40487a70dbfde5420e7acd49deda8b62b1a291c96b765d: Status 404 returned error can't find the container with id 1ce8788f196033f21b40487a70dbfde5420e7acd49deda8b62b1a291c96b765d
Oct 30 19:45:01.256640 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:45:01.256618    4892 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-5f55cd-29889825-fb6g6"]
Oct 30 19:45:01.266222 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:45:01.266034    4892 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-ada8d7-29889825-mzfrg"]
Oct 30 19:45:01.325719 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:45:01.325690    4892 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-b5178f-29889825-zncxf"]
Oct 30 19:45:01.366374 ip-10-0-132-82 kubenswrapper[4892]: W1030 19:45:01.364178    4892 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc18a2644_72a4_4b63_bf7d_c55bcb715f7c.slice/crio-261a20b2bb1d731953a711a9c505a7a2567ec6dbc9549defac988a6905607bb3 WatchSource:0}: Error finding container 261a20b2bb1d731953a711a9c505a7a2567ec6dbc9549defac988a6905607bb3: Status 404 returned error can't find the container with id 261a20b2bb1d731953a711a9c505a7a2567ec6dbc9549defac988a6905607bb3
Oct 30 19:45:01.418002 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:45:01.401012    4892 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-d52df0-29889825-kxd49"]
Oct 30 19:45:01.435065 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:45:01.434819    4892 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-a3f9b7-29889825-wdl5n"]
Oct 30 19:45:01.442132 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:45:01.441778    4892 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-migrate-d74c1bd9-29889825-wwbj8"]
Oct 30 19:45:01.824893 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:45:01.824813    4892 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-migrate-d74c1bd9-29889825-wwbj8" event={"ID":"b8655bf2-a203-4072-8894-994c2a780657","Type":"ContainerStarted","Data":"03c04f553c5f2b3f5e00b290566947fb2639b0d3673ae0e3d34983d693c9471e"}
Oct 30 19:45:01.826086 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:45:01.826060    4892 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-d52df0-29889825-kxd49" event={"ID":"d9801904-680f-4253-9104-04bbf36d9b35","Type":"ContainerStarted","Data":"8e36bf401639745d224a23b9494493fb29adc69284f513dca0150533d261873b"}
Oct 30 19:45:01.827503 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:45:01.827483    4892 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-b5178f-29889825-zncxf" event={"ID":"c18a2644-72a4-4b63-bf7d-c55bcb715f7c","Type":"ContainerStarted","Data":"261a20b2bb1d731953a711a9c505a7a2567ec6dbc9549defac988a6905607bb3"}
Oct 30 19:45:01.830919 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:45:01.830896    4892 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-5f55cd-29889825-fb6g6" event={"ID":"01eed39c-1a18-4a99-b1a6-4ffba9703c24","Type":"ContainerStarted","Data":"eb699c013cb2fcdf869565df6600dad56abae5fb1cd23d5cfb3144f61ba36ad9"}
Oct 30 19:45:01.831683 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:45:01.830940    4892 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-5f55cd-29889825-fb6g6" event={"ID":"01eed39c-1a18-4a99-b1a6-4ffba9703c24","Type":"ContainerStarted","Data":"47c1ecc95a66fe77f9b591197f067c689aaf573e5324c24a0546fa952cc1685f"}
Oct 30 19:45:01.833710 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:45:01.833690    4892 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-ada8d7-29889825-mzfrg" event={"ID":"bac2c08f-5b25-4ac6-9fb3-bf8e539c6ed8","Type":"ContainerStarted","Data":"1ce8788f196033f21b40487a70dbfde5420e7acd49deda8b62b1a291c96b765d"}
Oct 30 19:45:01.835283 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:45:01.835262    4892 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-a3f9b7-29889825-wdl5n" event={"ID":"8277bac6-a556-4e2d-bc2c-17121d44e4b5","Type":"ContainerStarted","Data":"a1067345d8ba62d1d11940446e94e89eb332edd964fdfa4f36d83568267d44f8"}
Oct 30 19:45:01.836764 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:45:01.836746    4892 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-2d9861-29889825-ngvs4" event={"ID":"0bbf2f20-664b-4c26-82c8-e779d16a54e3","Type":"ContainerStarted","Data":"478a245f20ac443b5b5f6593295f10bfc388765cf89b14097f626db4efde4105"}
Oct 30 19:45:01.836849 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:45:01.836779    4892 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-2d9861-29889825-ngvs4" event={"ID":"0bbf2f20-664b-4c26-82c8-e779d16a54e3","Type":"ContainerStarted","Data":"cb6b4ba662c32f4a0a7fcccf53e8517c1610388f47c1fafa5263a732e798abb8"}
Oct 30 19:45:01.971728 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:45:01.971683    4892 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-5f55cd-29889825-fb6g6" podStartSLOduration=1.971668539 podStartE2EDuration="1.971668539s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:45:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:45:01.969665608 +0000 UTC m=+1749.540131200" watchObservedRunningTime="2026-10-30 19:45:01.971668539 +0000 UTC m=+1749.542134116"
Oct 30 19:45:02.104946 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:45:02.104834    4892 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-2d9861-29889825-ngvs4" podStartSLOduration=2.104819696 podStartE2EDuration="2.104819696s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:45:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:45:02.103508429 +0000 UTC m=+1749.673974029" watchObservedRunningTime="2026-10-30 19:45:02.104819696 +0000 UTC m=+1749.675285326"
Oct 30 19:45:02.847603 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:45:02.847573    4892 generic.go:356] "Generic (PLEG): container finished" podID="b8655bf2-a203-4072-8894-994c2a780657" containerID="b705238c41edb050cff62c03168bb336920a2caf2b1ca77250ea1b31d829a770" exitCode=60
Oct 30 19:45:02.848139 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:45:02.847651    4892 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-migrate-d74c1bd9-29889825-wwbj8" event={"ID":"b8655bf2-a203-4072-8894-994c2a780657","Type":"ContainerDied","Data":"b705238c41edb050cff62c03168bb336920a2caf2b1ca77250ea1b31d829a770"}
Oct 30 19:45:02.848139 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:45:02.847994    4892 scope.go:122] "RemoveContainer" containerID="b705238c41edb050cff62c03168bb336920a2caf2b1ca77250ea1b31d829a770"
Oct 30 19:45:02.849647 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:45:02.849514    4892 generic.go:356] "Generic (PLEG): container finished" podID="d9801904-680f-4253-9104-04bbf36d9b35" containerID="ea0a1823430969dfc2386883c3658086c78c79e4127c128b2a2db0a401befae8" exitCode=60
Oct 30 19:45:02.849647 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:45:02.849586    4892 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-d52df0-29889825-kxd49" event={"ID":"d9801904-680f-4253-9104-04bbf36d9b35","Type":"ContainerDied","Data":"ea0a1823430969dfc2386883c3658086c78c79e4127c128b2a2db0a401befae8"}
Oct 30 19:45:02.849903 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:45:02.849885    4892 scope.go:122] "RemoveContainer" containerID="ea0a1823430969dfc2386883c3658086c78c79e4127c128b2a2db0a401befae8"
Oct 30 19:45:02.851804 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:45:02.851776    4892 generic.go:356] "Generic (PLEG): container finished" podID="c18a2644-72a4-4b63-bf7d-c55bcb715f7c" containerID="69c1b73a3ab4c2b50541de2e374466fa95f23a5ab77709d9cbdd12433ffe62c3" exitCode=60
Oct 30 19:45:02.851906 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:45:02.851860    4892 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-b5178f-29889825-zncxf" event={"ID":"c18a2644-72a4-4b63-bf7d-c55bcb715f7c","Type":"ContainerDied","Data":"69c1b73a3ab4c2b50541de2e374466fa95f23a5ab77709d9cbdd12433ffe62c3"}
Oct 30 19:45:02.852207 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:45:02.852178    4892 scope.go:122] "RemoveContainer" containerID="69c1b73a3ab4c2b50541de2e374466fa95f23a5ab77709d9cbdd12433ffe62c3"
Oct 30 19:45:02.853935 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:45:02.853877    4892 generic.go:356] "Generic (PLEG): container finished" podID="01eed39c-1a18-4a99-b1a6-4ffba9703c24" containerID="eb699c013cb2fcdf869565df6600dad56abae5fb1cd23d5cfb3144f61ba36ad9" exitCode=60
Oct 30 19:45:02.854020 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:45:02.853952    4892 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-5f55cd-29889825-fb6g6" event={"ID":"01eed39c-1a18-4a99-b1a6-4ffba9703c24","Type":"ContainerDied","Data":"eb699c013cb2fcdf869565df6600dad56abae5fb1cd23d5cfb3144f61ba36ad9"}
Oct 30 19:45:02.854196 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:45:02.854179    4892 scope.go:122] "RemoveContainer" containerID="eb699c013cb2fcdf869565df6600dad56abae5fb1cd23d5cfb3144f61ba36ad9"
Oct 30 19:45:02.856047 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:45:02.856027    4892 generic.go:356] "Generic (PLEG): container finished" podID="bac2c08f-5b25-4ac6-9fb3-bf8e539c6ed8" containerID="e98a1b3cf5faf8c27bd74ce309c2a81c75bd63a9b8ebe50284ebef444c1fb339" exitCode=60
Oct 30 19:45:02.856129 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:45:02.856114    4892 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-ada8d7-29889825-mzfrg" event={"ID":"bac2c08f-5b25-4ac6-9fb3-bf8e539c6ed8","Type":"ContainerDied","Data":"e98a1b3cf5faf8c27bd74ce309c2a81c75bd63a9b8ebe50284ebef444c1fb339"}
Oct 30 19:45:02.856295 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:45:02.856278    4892 scope.go:122] "RemoveContainer" containerID="e98a1b3cf5faf8c27bd74ce309c2a81c75bd63a9b8ebe50284ebef444c1fb339"
Oct 30 19:45:02.858373 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:45:02.858345    4892 generic.go:356] "Generic (PLEG): container finished" podID="8277bac6-a556-4e2d-bc2c-17121d44e4b5" containerID="6c3bc7ed0a279a56f2875224cbacaa89fbddf8d8c27a0da8de0101393daf1acb" exitCode=60
Oct 30 19:45:02.858477 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:45:02.858459    4892 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-a3f9b7-29889825-wdl5n" event={"ID":"8277bac6-a556-4e2d-bc2c-17121d44e4b5","Type":"ContainerDied","Data":"6c3bc7ed0a279a56f2875224cbacaa89fbddf8d8c27a0da8de0101393daf1acb"}
Oct 30 19:45:02.858644 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:45:02.858624    4892 scope.go:122] "RemoveContainer" containerID="6c3bc7ed0a279a56f2875224cbacaa89fbddf8d8c27a0da8de0101393daf1acb"
Oct 30 19:45:02.860579 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:45:02.860559    4892 generic.go:356] "Generic (PLEG): container finished" podID="0bbf2f20-664b-4c26-82c8-e779d16a54e3" containerID="478a245f20ac443b5b5f6593295f10bfc388765cf89b14097f626db4efde4105" exitCode=60
Oct 30 19:45:02.860668 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:45:02.860640    4892 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-2d9861-29889825-ngvs4" event={"ID":"0bbf2f20-664b-4c26-82c8-e779d16a54e3","Type":"ContainerDied","Data":"478a245f20ac443b5b5f6593295f10bfc388765cf89b14097f626db4efde4105"}
Oct 30 19:45:02.861063 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:45:02.861047    4892 scope.go:122] "RemoveContainer" containerID="478a245f20ac443b5b5f6593295f10bfc388765cf89b14097f626db4efde4105"
Oct 30 19:45:03.872471 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:45:03.872378    4892 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-migrate-d74c1bd9-29889825-wwbj8" event={"ID":"b8655bf2-a203-4072-8894-994c2a780657","Type":"ContainerStarted","Data":"2f4b23be1720bfcf9d8b07787c38ddefd5cdbfc1ff2390936ff7061f3cd6384a"}
Oct 30 19:45:03.874944 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:45:03.874892    4892 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-d52df0-29889825-kxd49" event={"ID":"d9801904-680f-4253-9104-04bbf36d9b35","Type":"ContainerStarted","Data":"ce19add82cdb7592ef183f063d82701e09f4eec9e7709979ecdf3b23982adba7"}
Oct 30 19:45:03.877563 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:45:03.877536    4892 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-b5178f-29889825-zncxf" event={"ID":"c18a2644-72a4-4b63-bf7d-c55bcb715f7c","Type":"ContainerStarted","Data":"50632fd13d588708610b7d20ea54a35a8f72d6d50d25828b76302aa5adc061d7"}
Oct 30 19:45:03.879691 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:45:03.879667    4892 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-5f55cd-29889825-fb6g6" event={"ID":"01eed39c-1a18-4a99-b1a6-4ffba9703c24","Type":"ContainerStarted","Data":"c9680a63d8ecfec6d8e2a85f5da6afdd96d13d499105cc77892f0c459b04dcc1"}
Oct 30 19:45:03.881671 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:45:03.881644    4892 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-ada8d7-29889825-mzfrg" event={"ID":"bac2c08f-5b25-4ac6-9fb3-bf8e539c6ed8","Type":"ContainerStarted","Data":"4b4ffdf90f11bee9621a51249916185c41bd56246a7d020f9446f596581acd1f"}
Oct 30 19:45:03.883816 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:45:03.883781    4892 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-a3f9b7-29889825-wdl5n" event={"ID":"8277bac6-a556-4e2d-bc2c-17121d44e4b5","Type":"ContainerStarted","Data":"1cb3ed8747f4dbfe7467a2194a9a2c3b19e2b6c3b34498ea564a7628f0ffd361"}
Oct 30 19:45:03.885874 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:45:03.885855    4892 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-2d9861-29889825-ngvs4" event={"ID":"0bbf2f20-664b-4c26-82c8-e779d16a54e3","Type":"ContainerStarted","Data":"d9002af700a71cd5d16d95d9e82dc1d38062162c81f2b72d90a75ecc18df4c5f"}
Oct 30 19:45:03.993617 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:45:03.993564    4892 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-migrate-d74c1bd9-29889825-wwbj8" podStartSLOduration=3.9935516680000003 podStartE2EDuration="3.993551668s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:45:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:45:03.988105108 +0000 UTC m=+1751.558570707" watchObservedRunningTime="2026-10-30 19:45:03.993551668 +0000 UTC m=+1751.564017267"
Oct 30 19:45:04.234225 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:45:04.234162    4892 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-d52df0-29889825-kxd49" podStartSLOduration=4.234144868 podStartE2EDuration="4.234144868s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:45:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:45:04.228762263 +0000 UTC m=+1751.799227862" watchObservedRunningTime="2026-10-30 19:45:04.234144868 +0000 UTC m=+1751.804610468"
Oct 30 19:45:04.344179 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:45:04.344128    4892 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-b5178f-29889825-zncxf" podStartSLOduration=4.3441151940000005 podStartE2EDuration="4.344115194s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:45:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:45:04.34210862 +0000 UTC m=+1751.912574242" watchObservedRunningTime="2026-10-30 19:45:04.344115194 +0000 UTC m=+1751.914580792"
Oct 30 19:45:04.578034 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:45:04.577947    4892 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-a3f9b7-29889825-wdl5n" podStartSLOduration=4.577934052 podStartE2EDuration="4.577934052s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:45:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:45:04.573221537 +0000 UTC m=+1752.143687138" watchObservedRunningTime="2026-10-30 19:45:04.577934052 +0000 UTC m=+1752.148399650"
Oct 30 19:45:04.692292 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:45:04.692240    4892 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-ada8d7-29889825-mzfrg" podStartSLOduration=4.692225936 podStartE2EDuration="4.692225936s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:45:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:45:04.687938718 +0000 UTC m=+1752.258404317" watchObservedRunningTime="2026-10-30 19:45:04.692225936 +0000 UTC m=+1752.262691564"
Oct 30 19:45:04.896682 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:45:04.896651    4892 generic.go:356] "Generic (PLEG): container finished" podID="c18a2644-72a4-4b63-bf7d-c55bcb715f7c" containerID="50632fd13d588708610b7d20ea54a35a8f72d6d50d25828b76302aa5adc061d7" exitCode=60
Oct 30 19:45:04.897177 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:45:04.896736    4892 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-b5178f-29889825-zncxf" event={"ID":"c18a2644-72a4-4b63-bf7d-c55bcb715f7c","Type":"ContainerDied","Data":"50632fd13d588708610b7d20ea54a35a8f72d6d50d25828b76302aa5adc061d7"}
Oct 30 19:45:04.897177 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:45:04.896805    4892 scope.go:122] "RemoveContainer" containerID="69c1b73a3ab4c2b50541de2e374466fa95f23a5ab77709d9cbdd12433ffe62c3"
Oct 30 19:45:04.897177 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:45:04.897024    4892 scope.go:122] "RemoveContainer" containerID="50632fd13d588708610b7d20ea54a35a8f72d6d50d25828b76302aa5adc061d7"
Oct 30 19:45:04.897396 ip-10-0-132-82 kubenswrapper[4892]: E1030 19:45:04.897370    4892 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w4-b5178f-29889825-zncxf_odh-ai-gateway-infra(c18a2644-72a4-4b63-bf7d-c55bcb715f7c)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-b5178f-29889825-zncxf" podUID="c18a2644-72a4-4b63-bf7d-c55bcb715f7c"
Oct 30 19:45:04.898849 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:45:04.898828    4892 generic.go:356] "Generic (PLEG): container finished" podID="01eed39c-1a18-4a99-b1a6-4ffba9703c24" containerID="c9680a63d8ecfec6d8e2a85f5da6afdd96d13d499105cc77892f0c459b04dcc1" exitCode=60
Oct 30 19:45:04.898935 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:45:04.898908    4892 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-5f55cd-29889825-fb6g6" event={"ID":"01eed39c-1a18-4a99-b1a6-4ffba9703c24","Type":"ContainerDied","Data":"c9680a63d8ecfec6d8e2a85f5da6afdd96d13d499105cc77892f0c459b04dcc1"}
Oct 30 19:45:04.899158 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:45:04.899143    4892 scope.go:122] "RemoveContainer" containerID="c9680a63d8ecfec6d8e2a85f5da6afdd96d13d499105cc77892f0c459b04dcc1"
Oct 30 19:45:04.899430 ip-10-0-132-82 kubenswrapper[4892]: E1030 19:45:04.899406    4892 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w6-5f55cd-29889825-fb6g6_odh-ai-gateway-infra(01eed39c-1a18-4a99-b1a6-4ffba9703c24)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-5f55cd-29889825-fb6g6" podUID="01eed39c-1a18-4a99-b1a6-4ffba9703c24"
Oct 30 19:45:04.900788 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:45:04.900768    4892 generic.go:356] "Generic (PLEG): container finished" podID="bac2c08f-5b25-4ac6-9fb3-bf8e539c6ed8" containerID="4b4ffdf90f11bee9621a51249916185c41bd56246a7d020f9446f596581acd1f" exitCode=60
Oct 30 19:45:04.900885 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:45:04.900843    4892 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-ada8d7-29889825-mzfrg" event={"ID":"bac2c08f-5b25-4ac6-9fb3-bf8e539c6ed8","Type":"ContainerDied","Data":"4b4ffdf90f11bee9621a51249916185c41bd56246a7d020f9446f596581acd1f"}
Oct 30 19:45:04.901099 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:45:04.901082    4892 scope.go:122] "RemoveContainer" containerID="4b4ffdf90f11bee9621a51249916185c41bd56246a7d020f9446f596581acd1f"
Oct 30 19:45:04.901361 ip-10-0-132-82 kubenswrapper[4892]: E1030 19:45:04.901341    4892 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w3-ada8d7-29889825-mzfrg_odh-ai-gateway-infra(bac2c08f-5b25-4ac6-9fb3-bf8e539c6ed8)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-ada8d7-29889825-mzfrg" podUID="bac2c08f-5b25-4ac6-9fb3-bf8e539c6ed8"
Oct 30 19:45:04.902994 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:45:04.902941    4892 generic.go:356] "Generic (PLEG): container finished" podID="8277bac6-a556-4e2d-bc2c-17121d44e4b5" containerID="1cb3ed8747f4dbfe7467a2194a9a2c3b19e2b6c3b34498ea564a7628f0ffd361" exitCode=60
Oct 30 19:45:04.903074 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:45:04.902990    4892 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-a3f9b7-29889825-wdl5n" event={"ID":"8277bac6-a556-4e2d-bc2c-17121d44e4b5","Type":"ContainerDied","Data":"1cb3ed8747f4dbfe7467a2194a9a2c3b19e2b6c3b34498ea564a7628f0ffd361"}
Oct 30 19:45:04.903252 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:45:04.903234    4892 scope.go:122] "RemoveContainer" containerID="1cb3ed8747f4dbfe7467a2194a9a2c3b19e2b6c3b34498ea564a7628f0ffd361"
Oct 30 19:45:04.903515 ip-10-0-132-82 kubenswrapper[4892]: E1030 19:45:04.903495    4892 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w2-a3f9b7-29889825-wdl5n_odh-ai-gateway-infra(8277bac6-a556-4e2d-bc2c-17121d44e4b5)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-a3f9b7-29889825-wdl5n" podUID="8277bac6-a556-4e2d-bc2c-17121d44e4b5"
Oct 30 19:45:04.905327 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:45:04.905304    4892 generic.go:356] "Generic (PLEG): container finished" podID="0bbf2f20-664b-4c26-82c8-e779d16a54e3" containerID="d9002af700a71cd5d16d95d9e82dc1d38062162c81f2b72d90a75ecc18df4c5f" exitCode=60
Oct 30 19:45:04.905419 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:45:04.905387    4892 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-2d9861-29889825-ngvs4" event={"ID":"0bbf2f20-664b-4c26-82c8-e779d16a54e3","Type":"ContainerDied","Data":"d9002af700a71cd5d16d95d9e82dc1d38062162c81f2b72d90a75ecc18df4c5f"}
Oct 30 19:45:04.905581 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:45:04.905565    4892 scope.go:122] "RemoveContainer" containerID="d9002af700a71cd5d16d95d9e82dc1d38062162c81f2b72d90a75ecc18df4c5f"
Oct 30 19:45:04.905832 ip-10-0-132-82 kubenswrapper[4892]: E1030 19:45:04.905814    4892 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w0-2d9861-29889825-ngvs4_odh-ai-gateway-infra(0bbf2f20-664b-4c26-82c8-e779d16a54e3)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-2d9861-29889825-ngvs4" podUID="0bbf2f20-664b-4c26-82c8-e779d16a54e3"
Oct 30 19:45:04.908054 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:45:04.908032    4892 generic.go:356] "Generic (PLEG): container finished" podID="b8655bf2-a203-4072-8894-994c2a780657" containerID="2f4b23be1720bfcf9d8b07787c38ddefd5cdbfc1ff2390936ff7061f3cd6384a" exitCode=60
Oct 30 19:45:04.908141 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:45:04.908099    4892 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-migrate-d74c1bd9-29889825-wwbj8" event={"ID":"b8655bf2-a203-4072-8894-994c2a780657","Type":"ContainerDied","Data":"2f4b23be1720bfcf9d8b07787c38ddefd5cdbfc1ff2390936ff7061f3cd6384a"}
Oct 30 19:45:04.908297 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:45:04.908282    4892 scope.go:122] "RemoveContainer" containerID="2f4b23be1720bfcf9d8b07787c38ddefd5cdbfc1ff2390936ff7061f3cd6384a"
Oct 30 19:45:04.908544 ip-10-0-132-82 kubenswrapper[4892]: E1030 19:45:04.908520    4892 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-migrate-d74c1bd9-29889825-wwbj8_odh-ai-gateway-infra(b8655bf2-a203-4072-8894-994c2a780657)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-migrate-d74c1bd9-29889825-wwbj8" podUID="b8655bf2-a203-4072-8894-994c2a780657"
Oct 30 19:45:04.910204 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:45:04.910184    4892 generic.go:356] "Generic (PLEG): container finished" podID="d9801904-680f-4253-9104-04bbf36d9b35" containerID="ce19add82cdb7592ef183f063d82701e09f4eec9e7709979ecdf3b23982adba7" exitCode=60
Oct 30 19:45:04.910293 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:45:04.910228    4892 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-d52df0-29889825-kxd49" event={"ID":"d9801904-680f-4253-9104-04bbf36d9b35","Type":"ContainerDied","Data":"ce19add82cdb7592ef183f063d82701e09f4eec9e7709979ecdf3b23982adba7"}
Oct 30 19:45:04.910462 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:45:04.910446    4892 scope.go:122] "RemoveContainer" containerID="ce19add82cdb7592ef183f063d82701e09f4eec9e7709979ecdf3b23982adba7"
Oct 30 19:45:04.910716 ip-10-0-132-82 kubenswrapper[4892]: E1030 19:45:04.910697    4892 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w1-d52df0-29889825-kxd49_odh-ai-gateway-infra(d9801904-680f-4253-9104-04bbf36d9b35)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-d52df0-29889825-kxd49" podUID="d9801904-680f-4253-9104-04bbf36d9b35"
Oct 30 19:45:04.912529 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:45:04.912515    4892 scope.go:122] "RemoveContainer" containerID="eb699c013cb2fcdf869565df6600dad56abae5fb1cd23d5cfb3144f61ba36ad9"
Oct 30 19:45:04.926333 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:45:04.926312    4892 scope.go:122] "RemoveContainer" containerID="e98a1b3cf5faf8c27bd74ce309c2a81c75bd63a9b8ebe50284ebef444c1fb339"
Oct 30 19:45:04.945958 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:45:04.945936    4892 scope.go:122] "RemoveContainer" containerID="6c3bc7ed0a279a56f2875224cbacaa89fbddf8d8c27a0da8de0101393daf1acb"
Oct 30 19:45:04.962708 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:45:04.962687    4892 scope.go:122] "RemoveContainer" containerID="478a245f20ac443b5b5f6593295f10bfc388765cf89b14097f626db4efde4105"
Oct 30 19:45:04.976088 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:45:04.976061    4892 scope.go:122] "RemoveContainer" containerID="b705238c41edb050cff62c03168bb336920a2caf2b1ca77250ea1b31d829a770"
Oct 30 19:45:04.989254 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:45:04.989236    4892 scope.go:122] "RemoveContainer" containerID="ea0a1823430969dfc2386883c3658086c78c79e4127c128b2a2db0a401befae8"
Oct 30 19:45:05.923067 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:45:05.923036    4892 scope.go:122] "RemoveContainer" containerID="2f4b23be1720bfcf9d8b07787c38ddefd5cdbfc1ff2390936ff7061f3cd6384a"
Oct 30 19:45:05.923556 ip-10-0-132-82 kubenswrapper[4892]: E1030 19:45:05.923295    4892 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-migrate-d74c1bd9-29889825-wwbj8_odh-ai-gateway-infra(b8655bf2-a203-4072-8894-994c2a780657)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-migrate-d74c1bd9-29889825-wwbj8" podUID="b8655bf2-a203-4072-8894-994c2a780657"
Oct 30 19:45:05.924565 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:45:05.924541    4892 scope.go:122] "RemoveContainer" containerID="ce19add82cdb7592ef183f063d82701e09f4eec9e7709979ecdf3b23982adba7"
Oct 30 19:45:05.924805 ip-10-0-132-82 kubenswrapper[4892]: E1030 19:45:05.924784    4892 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w1-d52df0-29889825-kxd49_odh-ai-gateway-infra(d9801904-680f-4253-9104-04bbf36d9b35)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-d52df0-29889825-kxd49" podUID="d9801904-680f-4253-9104-04bbf36d9b35"
Oct 30 19:45:05.926151 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:45:05.926131    4892 scope.go:122] "RemoveContainer" containerID="50632fd13d588708610b7d20ea54a35a8f72d6d50d25828b76302aa5adc061d7"
Oct 30 19:45:05.926365 ip-10-0-132-82 kubenswrapper[4892]: E1030 19:45:05.926339    4892 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w4-b5178f-29889825-zncxf_odh-ai-gateway-infra(c18a2644-72a4-4b63-bf7d-c55bcb715f7c)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-b5178f-29889825-zncxf" podUID="c18a2644-72a4-4b63-bf7d-c55bcb715f7c"
Oct 30 19:45:05.927854 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:45:05.927833    4892 scope.go:122] "RemoveContainer" containerID="c9680a63d8ecfec6d8e2a85f5da6afdd96d13d499105cc77892f0c459b04dcc1"
Oct 30 19:45:05.928114 ip-10-0-132-82 kubenswrapper[4892]: E1030 19:45:05.928096    4892 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w6-5f55cd-29889825-fb6g6_odh-ai-gateway-infra(01eed39c-1a18-4a99-b1a6-4ffba9703c24)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-5f55cd-29889825-fb6g6" podUID="01eed39c-1a18-4a99-b1a6-4ffba9703c24"
Oct 30 19:45:05.929659 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:45:05.929642    4892 scope.go:122] "RemoveContainer" containerID="4b4ffdf90f11bee9621a51249916185c41bd56246a7d020f9446f596581acd1f"
Oct 30 19:45:05.929911 ip-10-0-132-82 kubenswrapper[4892]: E1030 19:45:05.929884    4892 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w3-ada8d7-29889825-mzfrg_odh-ai-gateway-infra(bac2c08f-5b25-4ac6-9fb3-bf8e539c6ed8)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-ada8d7-29889825-mzfrg" podUID="bac2c08f-5b25-4ac6-9fb3-bf8e539c6ed8"
Oct 30 19:45:05.931442 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:45:05.931425    4892 scope.go:122] "RemoveContainer" containerID="1cb3ed8747f4dbfe7467a2194a9a2c3b19e2b6c3b34498ea564a7628f0ffd361"
Oct 30 19:45:05.931661 ip-10-0-132-82 kubenswrapper[4892]: E1030 19:45:05.931638    4892 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w2-a3f9b7-29889825-wdl5n_odh-ai-gateway-infra(8277bac6-a556-4e2d-bc2c-17121d44e4b5)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-a3f9b7-29889825-wdl5n" podUID="8277bac6-a556-4e2d-bc2c-17121d44e4b5"
Oct 30 19:45:05.933142 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:45:05.933125    4892 scope.go:122] "RemoveContainer" containerID="d9002af700a71cd5d16d95d9e82dc1d38062162c81f2b72d90a75ecc18df4c5f"
Oct 30 19:45:05.933327 ip-10-0-132-82 kubenswrapper[4892]: E1030 19:45:05.933308    4892 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w0-2d9861-29889825-ngvs4_odh-ai-gateway-infra(0bbf2f20-664b-4c26-82c8-e779d16a54e3)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-2d9861-29889825-ngvs4" podUID="0bbf2f20-664b-4c26-82c8-e779d16a54e3"
Oct 30 19:45:09.800571 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:45:09.800542    4892 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-cf9b6f-59d848ddcf-jxlzh"
Oct 30 19:45:09.800991 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:45:09.800628    4892 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-shared-a-w3-cf9b6f-7bb7c4ccf4-zc2s9"
Oct 30 19:45:11.444818 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:45:11.444781    4892 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-fsfvk"]
Oct 30 19:45:14.563047 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:45:14.563012    4892 scope.go:122] "RemoveContainer" containerID="4b4ffdf90f11bee9621a51249916185c41bd56246a7d020f9446f596581acd1f"
Oct 30 19:45:15.094371 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:45:15.094326    4892 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-ada8d7-29889825-mzfrg" event={"ID":"bac2c08f-5b25-4ac6-9fb3-bf8e539c6ed8","Type":"ContainerStarted","Data":"b355b13b771b0c32ef507933d373ad00752d168b0e946d43839bcabc29affec6"}
Oct 30 19:45:15.562694 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:45:15.562663    4892 scope.go:122] "RemoveContainer" containerID="2f4b23be1720bfcf9d8b07787c38ddefd5cdbfc1ff2390936ff7061f3cd6384a"
Oct 30 19:45:16.198761 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:45:16.198660    4892 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-migrate-d74c1bd9-29889825-wwbj8" event={"ID":"b8655bf2-a203-4072-8894-994c2a780657","Type":"ContainerStarted","Data":"d45f022382b2a8d1c3bfe9f2eb11b72200a414fbdb088e77d7534cfb1784a4f0"}
Oct 30 19:45:16.235281 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:45:16.235240    4892 generic.go:356] "Generic (PLEG): container finished" podID="bac2c08f-5b25-4ac6-9fb3-bf8e539c6ed8" containerID="b355b13b771b0c32ef507933d373ad00752d168b0e946d43839bcabc29affec6" exitCode=60
Oct 30 19:45:16.235669 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:45:16.235519    4892 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-ada8d7-29889825-mzfrg" event={"ID":"bac2c08f-5b25-4ac6-9fb3-bf8e539c6ed8","Type":"ContainerDied","Data":"b355b13b771b0c32ef507933d373ad00752d168b0e946d43839bcabc29affec6"}
Oct 30 19:45:16.235950 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:45:16.235904    4892 scope.go:122] "RemoveContainer" containerID="4b4ffdf90f11bee9621a51249916185c41bd56246a7d020f9446f596581acd1f"
Oct 30 19:45:16.238657 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:45:16.238639    4892 scope.go:122] "RemoveContainer" containerID="b355b13b771b0c32ef507933d373ad00752d168b0e946d43839bcabc29affec6"
Oct 30 19:45:16.241087 ip-10-0-132-82 kubenswrapper[4892]: E1030 19:45:16.241047    4892 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w3-ada8d7-29889825-mzfrg_odh-ai-gateway-infra(bac2c08f-5b25-4ac6-9fb3-bf8e539c6ed8)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-ada8d7-29889825-mzfrg" podUID="bac2c08f-5b25-4ac6-9fb3-bf8e539c6ed8"
Oct 30 19:45:16.252643 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:45:16.252619    4892 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-ada8d7-29889825-mzfrg"]
Oct 30 19:45:16.563476 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:45:16.563377    4892 scope.go:122] "RemoveContainer" containerID="1cb3ed8747f4dbfe7467a2194a9a2c3b19e2b6c3b34498ea564a7628f0ffd361"
Oct 30 19:45:16.563866 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:45:16.563829    4892 scope.go:122] "RemoveContainer" containerID="d9002af700a71cd5d16d95d9e82dc1d38062162c81f2b72d90a75ecc18df4c5f"
Oct 30 19:45:16.564044 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:45:16.564020    4892 scope.go:122] "RemoveContainer" containerID="c9680a63d8ecfec6d8e2a85f5da6afdd96d13d499105cc77892f0c459b04dcc1"
Oct 30 19:45:17.060649 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:45:17.060620    4892 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a348f93c-jgspj"]
Oct 30 19:45:17.127225 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:45:17.069870    4892 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a348f93c-jgspj"]
Oct 30 19:45:17.260170 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:45:17.260021    4892 generic.go:356] "Generic (PLEG): container finished" podID="b8655bf2-a203-4072-8894-994c2a780657" containerID="d45f022382b2a8d1c3bfe9f2eb11b72200a414fbdb088e77d7534cfb1784a4f0" exitCode=60
Oct 30 19:45:17.260170 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:45:17.260058    4892 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-migrate-d74c1bd9-29889825-wwbj8" event={"ID":"b8655bf2-a203-4072-8894-994c2a780657","Type":"ContainerDied","Data":"d45f022382b2a8d1c3bfe9f2eb11b72200a414fbdb088e77d7534cfb1784a4f0"}
Oct 30 19:45:17.260170 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:45:17.260130    4892 scope.go:122] "RemoveContainer" containerID="2f4b23be1720bfcf9d8b07787c38ddefd5cdbfc1ff2390936ff7061f3cd6384a"
Oct 30 19:45:17.260718 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:45:17.260500    4892 scope.go:122] "RemoveContainer" containerID="d45f022382b2a8d1c3bfe9f2eb11b72200a414fbdb088e77d7534cfb1784a4f0"
Oct 30 19:45:17.261025 ip-10-0-132-82 kubenswrapper[4892]: E1030 19:45:17.260833    4892 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-migrate-d74c1bd9-29889825-wwbj8_odh-ai-gateway-infra(b8655bf2-a203-4072-8894-994c2a780657)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-migrate-d74c1bd9-29889825-wwbj8" podUID="b8655bf2-a203-4072-8894-994c2a780657"
Oct 30 19:45:17.264614 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:45:17.264588    4892 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-5f55cd-29889825-fb6g6" event={"ID":"01eed39c-1a18-4a99-b1a6-4ffba9703c24","Type":"ContainerStarted","Data":"3df4db6b8eb03d61c0985ce02cb0cd17125b07fc10e725355ccf54397d990a92"}
Oct 30 19:45:17.269330 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:45:17.269308    4892 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-a3f9b7-29889825-wdl5n" event={"ID":"8277bac6-a556-4e2d-bc2c-17121d44e4b5","Type":"ContainerStarted","Data":"73d7b8e4edf0806f220ce5da180473e34485ccca35dd3218df6e93afb5c0fd7f"}
Oct 30 19:45:17.562833 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:45:17.562803    4892 scope.go:122] "RemoveContainer" containerID="ce19add82cdb7592ef183f063d82701e09f4eec9e7709979ecdf3b23982adba7"
Oct 30 19:45:17.617147 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:45:17.617123    4892 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-ada8d7-29889825-mzfrg"
Oct 30 19:45:17.672247 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:45:17.672217    4892 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/bac2c08f-5b25-4ac6-9fb3-bf8e539c6ed8-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/bac2c08f-5b25-4ac6-9fb3-bf8e539c6ed8-maas-api-service-ca\") pod \"bac2c08f-5b25-4ac6-9fb3-bf8e539c6ed8\" (UID: \"bac2c08f-5b25-4ac6-9fb3-bf8e539c6ed8\") "
Oct 30 19:45:17.672368 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:45:17.672355    4892 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/bac2c08f-5b25-4ac6-9fb3-bf8e539c6ed8-kube-api-access-mn42q\" (UniqueName: \"kubernetes.io/projected/bac2c08f-5b25-4ac6-9fb3-bf8e539c6ed8-kube-api-access-mn42q\") pod \"bac2c08f-5b25-4ac6-9fb3-bf8e539c6ed8\" (UID: \"bac2c08f-5b25-4ac6-9fb3-bf8e539c6ed8\") "
Oct 30 19:45:17.672625 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:45:17.672586    4892 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/bac2c08f-5b25-4ac6-9fb3-bf8e539c6ed8-maas-api-service-ca" pod "bac2c08f-5b25-4ac6-9fb3-bf8e539c6ed8" (UID: "bac2c08f-5b25-4ac6-9fb3-bf8e539c6ed8"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 19:45:17.673088 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:45:17.673063    4892 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/bac2c08f-5b25-4ac6-9fb3-bf8e539c6ed8-maas-api-service-ca\") on node \"ip-10-0-132-82.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 19:45:17.674554 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:45:17.674530    4892 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bac2c08f-5b25-4ac6-9fb3-bf8e539c6ed8-kube-api-access-mn42q" pod "bac2c08f-5b25-4ac6-9fb3-bf8e539c6ed8" (UID: "bac2c08f-5b25-4ac6-9fb3-bf8e539c6ed8"). InnerVolumeSpecName "kube-api-access-mn42q". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:45:17.774095 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:45:17.774026    4892 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mn42q\" (UniqueName: \"kubernetes.io/projected/bac2c08f-5b25-4ac6-9fb3-bf8e539c6ed8-kube-api-access-mn42q\") on node \"ip-10-0-132-82.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 19:45:18.060802 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:45:18.060773    4892 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-83bd7b88-nzq6n"]
Oct 30 19:45:18.073095 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:45:18.072918    4892 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-83bd7b88-nzq6n"]
Oct 30 19:45:18.282631 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:45:18.282597    4892 scope.go:122] "RemoveContainer" containerID="d45f022382b2a8d1c3bfe9f2eb11b72200a414fbdb088e77d7534cfb1784a4f0"
Oct 30 19:45:18.283182 ip-10-0-132-82 kubenswrapper[4892]: E1030 19:45:18.282936    4892 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-migrate-d74c1bd9-29889825-wwbj8_odh-ai-gateway-infra(b8655bf2-a203-4072-8894-994c2a780657)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-migrate-d74c1bd9-29889825-wwbj8" podUID="b8655bf2-a203-4072-8894-994c2a780657"
Oct 30 19:45:18.284590 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:45:18.284531    4892 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-d52df0-29889825-kxd49" event={"ID":"d9801904-680f-4253-9104-04bbf36d9b35","Type":"ContainerStarted","Data":"4018e0aefbb79c880ae8a85612b0445261080cacbdbe2c1a91f14e3fdca5b7f4"}
Oct 30 19:45:18.286382 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:45:18.286359    4892 generic.go:356] "Generic (PLEG): container finished" podID="01eed39c-1a18-4a99-b1a6-4ffba9703c24" containerID="3df4db6b8eb03d61c0985ce02cb0cd17125b07fc10e725355ccf54397d990a92" exitCode=60
Oct 30 19:45:18.286481 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:45:18.286426    4892 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-5f55cd-29889825-fb6g6" event={"ID":"01eed39c-1a18-4a99-b1a6-4ffba9703c24","Type":"ContainerDied","Data":"3df4db6b8eb03d61c0985ce02cb0cd17125b07fc10e725355ccf54397d990a92"}
Oct 30 19:45:18.286481 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:45:18.286474    4892 scope.go:122] "RemoveContainer" containerID="c9680a63d8ecfec6d8e2a85f5da6afdd96d13d499105cc77892f0c459b04dcc1"
Oct 30 19:45:18.286735 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:45:18.286716    4892 scope.go:122] "RemoveContainer" containerID="3df4db6b8eb03d61c0985ce02cb0cd17125b07fc10e725355ccf54397d990a92"
Oct 30 19:45:18.286984 ip-10-0-132-82 kubenswrapper[4892]: E1030 19:45:18.286945    4892 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w6-5f55cd-29889825-fb6g6_odh-ai-gateway-infra(01eed39c-1a18-4a99-b1a6-4ffba9703c24)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-5f55cd-29889825-fb6g6" podUID="01eed39c-1a18-4a99-b1a6-4ffba9703c24"
Oct 30 19:45:18.287880 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:45:18.287861    4892 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-ada8d7-29889825-mzfrg"
Oct 30 19:45:18.287998 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:45:18.287914    4892 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-ada8d7-29889825-mzfrg" event={"ID":"bac2c08f-5b25-4ac6-9fb3-bf8e539c6ed8","Type":"ContainerDied","Data":"1ce8788f196033f21b40487a70dbfde5420e7acd49deda8b62b1a291c96b765d"}
Oct 30 19:45:18.289887 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:45:18.289867    4892 generic.go:356] "Generic (PLEG): container finished" podID="8277bac6-a556-4e2d-bc2c-17121d44e4b5" containerID="73d7b8e4edf0806f220ce5da180473e34485ccca35dd3218df6e93afb5c0fd7f" exitCode=60
Oct 30 19:45:18.289986 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:45:18.289890    4892 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-a3f9b7-29889825-wdl5n" event={"ID":"8277bac6-a556-4e2d-bc2c-17121d44e4b5","Type":"ContainerDied","Data":"73d7b8e4edf0806f220ce5da180473e34485ccca35dd3218df6e93afb5c0fd7f"}
Oct 30 19:45:18.290202 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:45:18.290186    4892 scope.go:122] "RemoveContainer" containerID="73d7b8e4edf0806f220ce5da180473e34485ccca35dd3218df6e93afb5c0fd7f"
Oct 30 19:45:18.290462 ip-10-0-132-82 kubenswrapper[4892]: E1030 19:45:18.290437    4892 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w2-a3f9b7-29889825-wdl5n_odh-ai-gateway-infra(8277bac6-a556-4e2d-bc2c-17121d44e4b5)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-a3f9b7-29889825-wdl5n" podUID="8277bac6-a556-4e2d-bc2c-17121d44e4b5"
Oct 30 19:45:18.291902 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:45:18.291882    4892 generic.go:356] "Generic (PLEG): container finished" podID="0bbf2f20-664b-4c26-82c8-e779d16a54e3" containerID="b5ac02294422b95bcffcea75e7978bd140d48952ff447148f9e8d0259ab6d08b" exitCode=60
Oct 30 19:45:18.292010 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:45:18.291951    4892 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-2d9861-29889825-ngvs4" event={"ID":"0bbf2f20-664b-4c26-82c8-e779d16a54e3","Type":"ContainerDied","Data":"b5ac02294422b95bcffcea75e7978bd140d48952ff447148f9e8d0259ab6d08b"}
Oct 30 19:45:18.292272 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:45:18.292259    4892 scope.go:122] "RemoveContainer" containerID="b5ac02294422b95bcffcea75e7978bd140d48952ff447148f9e8d0259ab6d08b"
Oct 30 19:45:18.292456 ip-10-0-132-82 kubenswrapper[4892]: E1030 19:45:18.292440    4892 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w0-2d9861-29889825-ngvs4_odh-ai-gateway-infra(0bbf2f20-664b-4c26-82c8-e779d16a54e3)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-2d9861-29889825-ngvs4" podUID="0bbf2f20-664b-4c26-82c8-e779d16a54e3"
Oct 30 19:45:18.300896 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:45:18.300878    4892 scope.go:122] "RemoveContainer" containerID="b355b13b771b0c32ef507933d373ad00752d168b0e946d43839bcabc29affec6"
Oct 30 19:45:18.310977 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:45:18.310944    4892 scope.go:122] "RemoveContainer" containerID="1cb3ed8747f4dbfe7467a2194a9a2c3b19e2b6c3b34498ea564a7628f0ffd361"
Oct 30 19:45:18.324627 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:45:18.324610    4892 scope.go:122] "RemoveContainer" containerID="d9002af700a71cd5d16d95d9e82dc1d38062162c81f2b72d90a75ecc18df4c5f"
Oct 30 19:45:18.486777 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:45:18.486744    4892 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-migrate-d74c1bd9-29889825-wwbj8"]
Oct 30 19:45:18.522342 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:45:18.522309    4892 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-ada8d7-29889825-mzfrg"]
Oct 30 19:45:18.527278 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:45:18.527248    4892 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-ada8d7-29889825-mzfrg"]
Oct 30 19:45:18.571587 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:45:18.571519    4892 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="3f59f302-5d8d-49a6-9cb9-2975672a6a8f" path="/var/lib/kubelet/pods/3f59f302-5d8d-49a6-9cb9-2975672a6a8f/volumes"
Oct 30 19:45:18.572132 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:45:18.572116    4892 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="4ad702ac-5c7b-4188-b918-f036e91a209f" path="/var/lib/kubelet/pods/4ad702ac-5c7b-4188-b918-f036e91a209f/volumes"
Oct 30 19:45:18.572579 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:45:18.572561    4892 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="bac2c08f-5b25-4ac6-9fb3-bf8e539c6ed8" path="/var/lib/kubelet/pods/bac2c08f-5b25-4ac6-9fb3-bf8e539c6ed8/volumes"
Oct 30 19:45:18.573815 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:45:18.573798    4892 scope.go:122] "RemoveContainer" containerID="50632fd13d588708610b7d20ea54a35a8f72d6d50d25828b76302aa5adc061d7"
Oct 30 19:45:18.729539 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:45:18.729513    4892 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-5f55cd-29889825-fb6g6"]
Oct 30 19:45:18.840120 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:45:18.840093    4892 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-a3f9b7-29889825-wdl5n"]
Oct 30 19:45:19.303331 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:45:19.303299    4892 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-b5178f-29889825-zncxf" event={"ID":"c18a2644-72a4-4b63-bf7d-c55bcb715f7c","Type":"ContainerStarted","Data":"442647a194160e1dc9de50e0648f053c68a4fcf970dee7a56b321b908399f83f"}
Oct 30 19:45:19.311516 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:45:19.311491    4892 generic.go:356] "Generic (PLEG): container finished" podID="d9801904-680f-4253-9104-04bbf36d9b35" containerID="4018e0aefbb79c880ae8a85612b0445261080cacbdbe2c1a91f14e3fdca5b7f4" exitCode=60
Oct 30 19:45:19.311618 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:45:19.311566    4892 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-d52df0-29889825-kxd49" event={"ID":"d9801904-680f-4253-9104-04bbf36d9b35","Type":"ContainerDied","Data":"4018e0aefbb79c880ae8a85612b0445261080cacbdbe2c1a91f14e3fdca5b7f4"}
Oct 30 19:45:19.311688 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:45:19.311623    4892 scope.go:122] "RemoveContainer" containerID="ce19add82cdb7592ef183f063d82701e09f4eec9e7709979ecdf3b23982adba7"
Oct 30 19:45:19.311991 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:45:19.311958    4892 scope.go:122] "RemoveContainer" containerID="4018e0aefbb79c880ae8a85612b0445261080cacbdbe2c1a91f14e3fdca5b7f4"
Oct 30 19:45:19.312327 ip-10-0-132-82 kubenswrapper[4892]: E1030 19:45:19.312305    4892 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w1-d52df0-29889825-kxd49_odh-ai-gateway-infra(d9801904-680f-4253-9104-04bbf36d9b35)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-d52df0-29889825-kxd49" podUID="d9801904-680f-4253-9104-04bbf36d9b35"
Oct 30 19:45:19.569767 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:45:19.569690    4892 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-a3f9b7-29889825-wdl5n"
Oct 30 19:45:19.575134 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:45:19.575053    4892 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-5f55cd-29889825-fb6g6"
Oct 30 19:45:19.606258 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:45:19.606235    4892 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/8277bac6-a556-4e2d-bc2c-17121d44e4b5-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/8277bac6-a556-4e2d-bc2c-17121d44e4b5-maas-api-service-ca\") pod \"8277bac6-a556-4e2d-bc2c-17121d44e4b5\" (UID: \"8277bac6-a556-4e2d-bc2c-17121d44e4b5\") "
Oct 30 19:45:19.606329 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:45:19.606306    4892 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/01eed39c-1a18-4a99-b1a6-4ffba9703c24-kube-api-access-96kbh\" (UniqueName: \"kubernetes.io/projected/01eed39c-1a18-4a99-b1a6-4ffba9703c24-kube-api-access-96kbh\") pod \"01eed39c-1a18-4a99-b1a6-4ffba9703c24\" (UID: \"01eed39c-1a18-4a99-b1a6-4ffba9703c24\") "
Oct 30 19:45:19.606432 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:45:19.606410    4892 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8277bac6-a556-4e2d-bc2c-17121d44e4b5-kube-api-access-fz6b7\" (UniqueName: \"kubernetes.io/projected/8277bac6-a556-4e2d-bc2c-17121d44e4b5-kube-api-access-fz6b7\") pod \"8277bac6-a556-4e2d-bc2c-17121d44e4b5\" (UID: \"8277bac6-a556-4e2d-bc2c-17121d44e4b5\") "
Oct 30 19:45:19.606498 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:45:19.606482    4892 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/01eed39c-1a18-4a99-b1a6-4ffba9703c24-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/01eed39c-1a18-4a99-b1a6-4ffba9703c24-maas-api-service-ca\") pod \"01eed39c-1a18-4a99-b1a6-4ffba9703c24\" (UID: \"01eed39c-1a18-4a99-b1a6-4ffba9703c24\") "
Oct 30 19:45:19.606676 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:45:19.606638    4892 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8277bac6-a556-4e2d-bc2c-17121d44e4b5-maas-api-service-ca" pod "8277bac6-a556-4e2d-bc2c-17121d44e4b5" (UID: "8277bac6-a556-4e2d-bc2c-17121d44e4b5"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 19:45:19.607319 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:45:19.607294    4892 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/01eed39c-1a18-4a99-b1a6-4ffba9703c24-maas-api-service-ca" pod "01eed39c-1a18-4a99-b1a6-4ffba9703c24" (UID: "01eed39c-1a18-4a99-b1a6-4ffba9703c24"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 19:45:19.607583 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:45:19.607353    4892 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/8277bac6-a556-4e2d-bc2c-17121d44e4b5-maas-api-service-ca\") on node \"ip-10-0-132-82.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 19:45:19.609014 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:45:19.608952    4892 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8277bac6-a556-4e2d-bc2c-17121d44e4b5-kube-api-access-fz6b7" pod "8277bac6-a556-4e2d-bc2c-17121d44e4b5" (UID: "8277bac6-a556-4e2d-bc2c-17121d44e4b5"). InnerVolumeSpecName "kube-api-access-fz6b7". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:45:19.609099 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:45:19.609073    4892 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/01eed39c-1a18-4a99-b1a6-4ffba9703c24-kube-api-access-96kbh" pod "01eed39c-1a18-4a99-b1a6-4ffba9703c24" (UID: "01eed39c-1a18-4a99-b1a6-4ffba9703c24"). InnerVolumeSpecName "kube-api-access-96kbh". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:45:19.630216 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:45:19.628356    4892 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-migrate-d74c1bd9-29889825-wwbj8"
Oct 30 19:45:19.708429 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:45:19.708403    4892 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b8655bf2-a203-4072-8894-994c2a780657-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/b8655bf2-a203-4072-8894-994c2a780657-maas-api-service-ca\") pod \"b8655bf2-a203-4072-8894-994c2a780657\" (UID: \"b8655bf2-a203-4072-8894-994c2a780657\") "
Oct 30 19:45:19.708549 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:45:19.708512    4892 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b8655bf2-a203-4072-8894-994c2a780657-kube-api-access-l7rq8\" (UniqueName: \"kubernetes.io/projected/b8655bf2-a203-4072-8894-994c2a780657-kube-api-access-l7rq8\") pod \"b8655bf2-a203-4072-8894-994c2a780657\" (UID: \"b8655bf2-a203-4072-8894-994c2a780657\") "
Oct 30 19:45:19.708745 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:45:19.708725    4892 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b8655bf2-a203-4072-8894-994c2a780657-maas-api-service-ca" pod "b8655bf2-a203-4072-8894-994c2a780657" (UID: "b8655bf2-a203-4072-8894-994c2a780657"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 19:45:19.708994 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:45:19.708977    4892 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-96kbh\" (UniqueName: \"kubernetes.io/projected/01eed39c-1a18-4a99-b1a6-4ffba9703c24-kube-api-access-96kbh\") on node \"ip-10-0-132-82.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 19:45:19.709030 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:45:19.708997    4892 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fz6b7\" (UniqueName: \"kubernetes.io/projected/8277bac6-a556-4e2d-bc2c-17121d44e4b5-kube-api-access-fz6b7\") on node \"ip-10-0-132-82.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 19:45:19.709030 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:45:19.709009    4892 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/01eed39c-1a18-4a99-b1a6-4ffba9703c24-maas-api-service-ca\") on node \"ip-10-0-132-82.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 19:45:19.709030 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:45:19.709017    4892 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/b8655bf2-a203-4072-8894-994c2a780657-maas-api-service-ca\") on node \"ip-10-0-132-82.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 19:45:19.710702 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:45:19.710676    4892 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b8655bf2-a203-4072-8894-994c2a780657-kube-api-access-l7rq8" pod "b8655bf2-a203-4072-8894-994c2a780657" (UID: "b8655bf2-a203-4072-8894-994c2a780657"). InnerVolumeSpecName "kube-api-access-l7rq8". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:45:19.771378 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:45:19.771344    4892 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-d52df0-29889825-kxd49"]
Oct 30 19:45:19.810191 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:45:19.810162    4892 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-l7rq8\" (UniqueName: \"kubernetes.io/projected/b8655bf2-a203-4072-8894-994c2a780657-kube-api-access-l7rq8\") on node \"ip-10-0-132-82.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 19:45:19.881829 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:45:19.881753    4892 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-2d9861-29889825-ngvs4"]
Oct 30 19:45:20.067512 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:45:20.067482    4892 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-2d9861-29889825-ngvs4"
Oct 30 19:45:20.118785 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:45:20.118756    4892 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/0bbf2f20-664b-4c26-82c8-e779d16a54e3-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/0bbf2f20-664b-4c26-82c8-e779d16a54e3-maas-api-service-ca\") pod \"0bbf2f20-664b-4c26-82c8-e779d16a54e3\" (UID: \"0bbf2f20-664b-4c26-82c8-e779d16a54e3\") "
Oct 30 19:45:20.118943 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:45:20.118800    4892 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0bbf2f20-664b-4c26-82c8-e779d16a54e3-kube-api-access-2r4fg\" (UniqueName: \"kubernetes.io/projected/0bbf2f20-664b-4c26-82c8-e779d16a54e3-kube-api-access-2r4fg\") pod \"0bbf2f20-664b-4c26-82c8-e779d16a54e3\" (UID: \"0bbf2f20-664b-4c26-82c8-e779d16a54e3\") "
Oct 30 19:45:20.119170 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:45:20.119148    4892 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0bbf2f20-664b-4c26-82c8-e779d16a54e3-maas-api-service-ca" pod "0bbf2f20-664b-4c26-82c8-e779d16a54e3" (UID: "0bbf2f20-664b-4c26-82c8-e779d16a54e3"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 19:45:20.119644 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:45:20.119620    4892 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/0bbf2f20-664b-4c26-82c8-e779d16a54e3-maas-api-service-ca\") on node \"ip-10-0-132-82.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 19:45:20.121280 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:45:20.121259    4892 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0bbf2f20-664b-4c26-82c8-e779d16a54e3-kube-api-access-2r4fg" pod "0bbf2f20-664b-4c26-82c8-e779d16a54e3" (UID: "0bbf2f20-664b-4c26-82c8-e779d16a54e3"). InnerVolumeSpecName "kube-api-access-2r4fg". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:45:20.220788 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:45:20.220752    4892 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2r4fg\" (UniqueName: \"kubernetes.io/projected/0bbf2f20-664b-4c26-82c8-e779d16a54e3-kube-api-access-2r4fg\") on node \"ip-10-0-132-82.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 19:45:20.322893 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:45:20.322858    4892 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-a3f9b7-29889825-wdl5n" event={"ID":"8277bac6-a556-4e2d-bc2c-17121d44e4b5","Type":"ContainerDied","Data":"a1067345d8ba62d1d11940446e94e89eb332edd964fdfa4f36d83568267d44f8"}
Oct 30 19:45:20.323395 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:45:20.322895    4892 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-a3f9b7-29889825-wdl5n"
Oct 30 19:45:20.323395 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:45:20.322993    4892 scope.go:122] "RemoveContainer" containerID="73d7b8e4edf0806f220ce5da180473e34485ccca35dd3218df6e93afb5c0fd7f"
Oct 30 19:45:20.324740 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:45:20.324714    4892 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-2d9861-29889825-ngvs4"
Oct 30 19:45:20.324880 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:45:20.324746    4892 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-2d9861-29889825-ngvs4" event={"ID":"0bbf2f20-664b-4c26-82c8-e779d16a54e3","Type":"ContainerDied","Data":"cb6b4ba662c32f4a0a7fcccf53e8517c1610388f47c1fafa5263a732e798abb8"}
Oct 30 19:45:20.326619 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:45:20.326602    4892 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-migrate-d74c1bd9-29889825-wwbj8"
Oct 30 19:45:20.326733 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:45:20.326595    4892 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-migrate-d74c1bd9-29889825-wwbj8" event={"ID":"b8655bf2-a203-4072-8894-994c2a780657","Type":"ContainerDied","Data":"03c04f553c5f2b3f5e00b290566947fb2639b0d3673ae0e3d34983d693c9471e"}
Oct 30 19:45:20.336284 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:45:20.336257    4892 generic.go:356] "Generic (PLEG): container finished" podID="c18a2644-72a4-4b63-bf7d-c55bcb715f7c" containerID="442647a194160e1dc9de50e0648f053c68a4fcf970dee7a56b321b908399f83f" exitCode=60
Oct 30 19:45:20.336384 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:45:20.336336    4892 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-b5178f-29889825-zncxf" event={"ID":"c18a2644-72a4-4b63-bf7d-c55bcb715f7c","Type":"ContainerDied","Data":"442647a194160e1dc9de50e0648f053c68a4fcf970dee7a56b321b908399f83f"}
Oct 30 19:45:20.336903 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:45:20.336811    4892 scope.go:122] "RemoveContainer" containerID="442647a194160e1dc9de50e0648f053c68a4fcf970dee7a56b321b908399f83f"
Oct 30 19:45:20.337305 ip-10-0-132-82 kubenswrapper[4892]: E1030 19:45:20.337267    4892 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w4-b5178f-29889825-zncxf_odh-ai-gateway-infra(c18a2644-72a4-4b63-bf7d-c55bcb715f7c)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-b5178f-29889825-zncxf" podUID="c18a2644-72a4-4b63-bf7d-c55bcb715f7c"
Oct 30 19:45:20.338155 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:45:20.338125    4892 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-5f55cd-29889825-fb6g6"
Oct 30 19:45:20.338254 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:45:20.338170    4892 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-5f55cd-29889825-fb6g6" event={"ID":"01eed39c-1a18-4a99-b1a6-4ffba9703c24","Type":"ContainerDied","Data":"47c1ecc95a66fe77f9b591197f067c689aaf573e5324c24a0546fa952cc1685f"}
Oct 30 19:45:20.340483 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:45:20.340462    4892 scope.go:122] "RemoveContainer" containerID="b5ac02294422b95bcffcea75e7978bd140d48952ff447148f9e8d0259ab6d08b"
Oct 30 19:45:20.368700 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:45:20.368588    4892 scope.go:122] "RemoveContainer" containerID="d45f022382b2a8d1c3bfe9f2eb11b72200a414fbdb088e77d7534cfb1784a4f0"
Oct 30 19:45:20.383166 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:45:20.383133    4892 scope.go:122] "RemoveContainer" containerID="50632fd13d588708610b7d20ea54a35a8f72d6d50d25828b76302aa5adc061d7"
Oct 30 19:45:20.406164 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:45:20.406060    4892 scope.go:122] "RemoveContainer" containerID="3df4db6b8eb03d61c0985ce02cb0cd17125b07fc10e725355ccf54397d990a92"
Oct 30 19:45:20.450865 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:45:20.450832    4892 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-b5178f-29889825-zncxf"]
Oct 30 19:45:20.502951 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:45:20.502871    4892 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-5f55cd-29889825-fb6g6"]
Oct 30 19:45:20.509526 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:45:20.509502    4892 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-5f55cd-29889825-fb6g6"]
Oct 30 19:45:20.533331 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:45:20.533308    4892 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-d52df0-29889825-kxd49"
Oct 30 19:45:20.571988 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:45:20.571944    4892 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="01eed39c-1a18-4a99-b1a6-4ffba9703c24" path="/var/lib/kubelet/pods/01eed39c-1a18-4a99-b1a6-4ffba9703c24/volumes"
Oct 30 19:45:20.627096 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:45:20.627066    4892 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d9801904-680f-4253-9104-04bbf36d9b35-kube-api-access-dv64b\" (UniqueName: \"kubernetes.io/projected/d9801904-680f-4253-9104-04bbf36d9b35-kube-api-access-dv64b\") pod \"d9801904-680f-4253-9104-04bbf36d9b35\" (UID: \"d9801904-680f-4253-9104-04bbf36d9b35\") "
Oct 30 19:45:20.627251 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:45:20.627232    4892 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d9801904-680f-4253-9104-04bbf36d9b35-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/d9801904-680f-4253-9104-04bbf36d9b35-maas-api-service-ca\") pod \"d9801904-680f-4253-9104-04bbf36d9b35\" (UID: \"d9801904-680f-4253-9104-04bbf36d9b35\") "
Oct 30 19:45:20.627568 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:45:20.627543    4892 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d9801904-680f-4253-9104-04bbf36d9b35-maas-api-service-ca" pod "d9801904-680f-4253-9104-04bbf36d9b35" (UID: "d9801904-680f-4253-9104-04bbf36d9b35"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 19:45:20.627932 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:45:20.627908    4892 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/d9801904-680f-4253-9104-04bbf36d9b35-maas-api-service-ca\") on node \"ip-10-0-132-82.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 19:45:20.629909 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:45:20.629885    4892 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d9801904-680f-4253-9104-04bbf36d9b35-kube-api-access-dv64b" pod "d9801904-680f-4253-9104-04bbf36d9b35" (UID: "d9801904-680f-4253-9104-04bbf36d9b35"). InnerVolumeSpecName "kube-api-access-dv64b". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:45:20.729733 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:45:20.729705    4892 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dv64b\" (UniqueName: \"kubernetes.io/projected/d9801904-680f-4253-9104-04bbf36d9b35-kube-api-access-dv64b\") on node \"ip-10-0-132-82.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 19:45:20.915938 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:45:20.915900    4892 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-2d9861-29889825-ngvs4"]
Oct 30 19:45:20.921539 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:45:20.921509    4892 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-2d9861-29889825-ngvs4"]
Oct 30 19:45:21.027355 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:45:21.027321    4892 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-a3f9b7-29889825-wdl5n"]
Oct 30 19:45:21.032730 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:45:21.032698    4892 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-a3f9b7-29889825-wdl5n"]
Oct 30 19:45:21.202083 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:45:21.193560    4892 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-migrate-d74c1bd9-29889825-wwbj8"]
Oct 30 19:45:21.205991 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:45:21.204757    4892 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-migrate-d74c1bd9-29889825-wwbj8"]
Oct 30 19:45:21.351415 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:45:21.351369    4892 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-d52df0-29889825-kxd49"
Oct 30 19:45:21.352014 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:45:21.351982    4892 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-d52df0-29889825-kxd49" event={"ID":"d9801904-680f-4253-9104-04bbf36d9b35","Type":"ContainerDied","Data":"8e36bf401639745d224a23b9494493fb29adc69284f513dca0150533d261873b"}
Oct 30 19:45:21.352122 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:45:21.352052    4892 scope.go:122] "RemoveContainer" containerID="4018e0aefbb79c880ae8a85612b0445261080cacbdbe2c1a91f14e3fdca5b7f4"
Oct 30 19:45:21.520556 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:45:21.520529    4892 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-d52df0-29889825-kxd49"]
Oct 30 19:45:21.523585 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:45:21.523562    4892 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-b5178f-29889825-zncxf"
Oct 30 19:45:21.539003 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:45:21.527430    4892 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-d52df0-29889825-kxd49"]
Oct 30 19:45:21.648115 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:45:21.648079    4892 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c18a2644-72a4-4b63-bf7d-c55bcb715f7c-kube-api-access-m8hqb\" (UniqueName: \"kubernetes.io/projected/c18a2644-72a4-4b63-bf7d-c55bcb715f7c-kube-api-access-m8hqb\") pod \"c18a2644-72a4-4b63-bf7d-c55bcb715f7c\" (UID: \"c18a2644-72a4-4b63-bf7d-c55bcb715f7c\") "
Oct 30 19:45:21.648309 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:45:21.648290    4892 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c18a2644-72a4-4b63-bf7d-c55bcb715f7c-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/c18a2644-72a4-4b63-bf7d-c55bcb715f7c-maas-api-service-ca\") pod \"c18a2644-72a4-4b63-bf7d-c55bcb715f7c\" (UID: \"c18a2644-72a4-4b63-bf7d-c55bcb715f7c\") "
Oct 30 19:45:21.648641 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:45:21.648617    4892 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c18a2644-72a4-4b63-bf7d-c55bcb715f7c-maas-api-service-ca" pod "c18a2644-72a4-4b63-bf7d-c55bcb715f7c" (UID: "c18a2644-72a4-4b63-bf7d-c55bcb715f7c"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 19:45:21.649910 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:45:21.649885    4892 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/c18a2644-72a4-4b63-bf7d-c55bcb715f7c-maas-api-service-ca\") on node \"ip-10-0-132-82.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 19:45:21.650574 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:45:21.650542    4892 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c18a2644-72a4-4b63-bf7d-c55bcb715f7c-kube-api-access-m8hqb" pod "c18a2644-72a4-4b63-bf7d-c55bcb715f7c" (UID: "c18a2644-72a4-4b63-bf7d-c55bcb715f7c"). InnerVolumeSpecName "kube-api-access-m8hqb". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:45:21.751292 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:45:21.751185    4892 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-m8hqb\" (UniqueName: \"kubernetes.io/projected/c18a2644-72a4-4b63-bf7d-c55bcb715f7c-kube-api-access-m8hqb\") on node \"ip-10-0-132-82.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 19:45:22.373018 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:45:22.372957    4892 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-b5178f-29889825-zncxf"
Oct 30 19:45:22.373461 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:45:22.372986    4892 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-b5178f-29889825-zncxf" event={"ID":"c18a2644-72a4-4b63-bf7d-c55bcb715f7c","Type":"ContainerDied","Data":"261a20b2bb1d731953a711a9c505a7a2567ec6dbc9549defac988a6905607bb3"}
Oct 30 19:45:22.373461 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:45:22.373122    4892 scope.go:122] "RemoveContainer" containerID="442647a194160e1dc9de50e0648f053c68a4fcf970dee7a56b321b908399f83f"
Oct 30 19:45:22.545628 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:45:22.545598    4892 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-b5178f-29889825-zncxf"]
Oct 30 19:45:22.572957 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:45:22.572925    4892 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0bbf2f20-664b-4c26-82c8-e779d16a54e3" path="/var/lib/kubelet/pods/0bbf2f20-664b-4c26-82c8-e779d16a54e3/volumes"
Oct 30 19:45:22.573466 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:45:22.573446    4892 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="8277bac6-a556-4e2d-bc2c-17121d44e4b5" path="/var/lib/kubelet/pods/8277bac6-a556-4e2d-bc2c-17121d44e4b5/volumes"
Oct 30 19:45:22.573829 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:45:22.573815    4892 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b8655bf2-a203-4072-8894-994c2a780657" path="/var/lib/kubelet/pods/b8655bf2-a203-4072-8894-994c2a780657/volumes"
Oct 30 19:45:22.574535 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:45:22.574515    4892 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d9801904-680f-4253-9104-04bbf36d9b35" path="/var/lib/kubelet/pods/d9801904-680f-4253-9104-04bbf36d9b35/volumes"
Oct 30 19:45:22.594340 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:45:22.594317    4892 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-b5178f-29889825-zncxf"]
Oct 30 19:45:23.981988 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:45:23.971955    4892 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w0-2e5977-5b974cf99f-mrcrv"]
Oct 30 19:45:23.981988 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:45:23.973421    4892 cpu_manager.go:414] "containerMap: removing container" podUID="8277bac6-a556-4e2d-bc2c-17121d44e4b5" containerName="cleanup"
Oct 30 19:45:23.981988 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:45:23.973443    4892 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8277bac6-a556-4e2d-bc2c-17121d44e4b5" containerName="cleanup"
Oct 30 19:45:23.981988 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:45:23.973463    4892 cpu_manager.go:414] "containerMap: removing container" podUID="bac2c08f-5b25-4ac6-9fb3-bf8e539c6ed8" containerName="cleanup"
Oct 30 19:45:23.981988 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:45:23.973472    4892 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="bac2c08f-5b25-4ac6-9fb3-bf8e539c6ed8" containerName="cleanup"
Oct 30 19:45:23.981988 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:45:23.973489    4892 cpu_manager.go:414] "containerMap: removing container" podUID="c18a2644-72a4-4b63-bf7d-c55bcb715f7c" containerName="cleanup"
Oct 30 19:45:23.981988 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:45:23.973496    4892 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c18a2644-72a4-4b63-bf7d-c55bcb715f7c" containerName="cleanup"
Oct 30 19:45:23.981988 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:45:23.973514    4892 cpu_manager.go:414] "containerMap: removing container" podUID="8277bac6-a556-4e2d-bc2c-17121d44e4b5" containerName="cleanup"
Oct 30 19:45:23.981988 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:45:23.973522    4892 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8277bac6-a556-4e2d-bc2c-17121d44e4b5" containerName="cleanup"
Oct 30 19:45:23.981988 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:45:23.973535    4892 cpu_manager.go:414] "containerMap: removing container" podUID="d9801904-680f-4253-9104-04bbf36d9b35" containerName="cleanup"
Oct 30 19:45:23.981988 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:45:23.973543    4892 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d9801904-680f-4253-9104-04bbf36d9b35" containerName="cleanup"
Oct 30 19:45:23.981988 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:45:23.973552    4892 cpu_manager.go:414] "containerMap: removing container" podUID="b8655bf2-a203-4072-8894-994c2a780657" containerName="cleanup"
Oct 30 19:45:23.981988 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:45:23.973559    4892 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b8655bf2-a203-4072-8894-994c2a780657" containerName="cleanup"
Oct 30 19:45:23.981988 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:45:23.973580    4892 cpu_manager.go:414] "containerMap: removing container" podUID="b8655bf2-a203-4072-8894-994c2a780657" containerName="cleanup"
Oct 30 19:45:23.981988 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:45:23.973587    4892 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b8655bf2-a203-4072-8894-994c2a780657" containerName="cleanup"
Oct 30 19:45:23.981988 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:45:23.973604    4892 cpu_manager.go:414] "containerMap: removing container" podUID="d9801904-680f-4253-9104-04bbf36d9b35" containerName="cleanup"
Oct 30 19:45:23.981988 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:45:23.973611    4892 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d9801904-680f-4253-9104-04bbf36d9b35" containerName="cleanup"
Oct 30 19:45:23.981988 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:45:23.973641    4892 cpu_manager.go:414] "containerMap: removing container" podUID="0bbf2f20-664b-4c26-82c8-e779d16a54e3" containerName="cleanup"
Oct 30 19:45:23.981988 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:45:23.973647    4892 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0bbf2f20-664b-4c26-82c8-e779d16a54e3" containerName="cleanup"
Oct 30 19:45:23.981988 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:45:23.973661    4892 cpu_manager.go:414] "containerMap: removing container" podUID="c18a2644-72a4-4b63-bf7d-c55bcb715f7c" containerName="cleanup"
Oct 30 19:45:23.981988 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:45:23.973668    4892 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c18a2644-72a4-4b63-bf7d-c55bcb715f7c" containerName="cleanup"
Oct 30 19:45:23.981988 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:45:23.973687    4892 cpu_manager.go:414] "containerMap: removing container" podUID="01eed39c-1a18-4a99-b1a6-4ffba9703c24" containerName="cleanup"
Oct 30 19:45:23.981988 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:45:23.973693    4892 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="01eed39c-1a18-4a99-b1a6-4ffba9703c24" containerName="cleanup"
Oct 30 19:45:23.981988 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:45:23.973714    4892 cpu_manager.go:414] "containerMap: removing container" podUID="0bbf2f20-664b-4c26-82c8-e779d16a54e3" containerName="cleanup"
Oct 30 19:45:23.981988 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:45:23.973721    4892 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0bbf2f20-664b-4c26-82c8-e779d16a54e3" containerName="cleanup"
Oct 30 19:45:23.981988 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:45:23.973731    4892 cpu_manager.go:414] "containerMap: removing container" podUID="0bbf2f20-664b-4c26-82c8-e779d16a54e3" containerName="cleanup"
Oct 30 19:45:23.981988 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:45:23.973737    4892 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0bbf2f20-664b-4c26-82c8-e779d16a54e3" containerName="cleanup"
Oct 30 19:45:23.981988 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:45:23.973747    4892 cpu_manager.go:414] "containerMap: removing container" podUID="01eed39c-1a18-4a99-b1a6-4ffba9703c24" containerName="cleanup"
Oct 30 19:45:23.981988 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:45:23.973754    4892 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="01eed39c-1a18-4a99-b1a6-4ffba9703c24" containerName="cleanup"
Oct 30 19:45:23.981988 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:45:23.973768    4892 cpu_manager.go:414] "containerMap: removing container" podUID="bac2c08f-5b25-4ac6-9fb3-bf8e539c6ed8" containerName="cleanup"
Oct 30 19:45:23.981988 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:45:23.973775    4892 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="bac2c08f-5b25-4ac6-9fb3-bf8e539c6ed8" containerName="cleanup"
Oct 30 19:45:23.981988 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:45:23.973908    4892 cpu_manager.go:414] "containerMap: removing container" podUID="01eed39c-1a18-4a99-b1a6-4ffba9703c24" containerName="cleanup"
Oct 30 19:45:23.981988 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:45:23.973917    4892 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="01eed39c-1a18-4a99-b1a6-4ffba9703c24" containerName="cleanup"
Oct 30 19:45:23.981988 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:45:23.974107    4892 memory_manager.go:365] "RemoveStaleState removing state" podUID="8277bac6-a556-4e2d-bc2c-17121d44e4b5" containerName="cleanup"
Oct 30 19:45:23.981988 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:45:23.974127    4892 memory_manager.go:365] "RemoveStaleState removing state" podUID="bac2c08f-5b25-4ac6-9fb3-bf8e539c6ed8" containerName="cleanup"
Oct 30 19:45:23.981988 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:45:23.974140    4892 memory_manager.go:365] "RemoveStaleState removing state" podUID="c18a2644-72a4-4b63-bf7d-c55bcb715f7c" containerName="cleanup"
Oct 30 19:45:23.981988 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:45:23.974149    4892 memory_manager.go:365] "RemoveStaleState removing state" podUID="8277bac6-a556-4e2d-bc2c-17121d44e4b5" containerName="cleanup"
Oct 30 19:45:23.981988 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:45:23.974159    4892 memory_manager.go:365] "RemoveStaleState removing state" podUID="d9801904-680f-4253-9104-04bbf36d9b35" containerName="cleanup"
Oct 30 19:45:23.981988 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:45:23.974167    4892 memory_manager.go:365] "RemoveStaleState removing state" podUID="b8655bf2-a203-4072-8894-994c2a780657" containerName="cleanup"
Oct 30 19:45:23.981988 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:45:23.974180    4892 memory_manager.go:365] "RemoveStaleState removing state" podUID="b8655bf2-a203-4072-8894-994c2a780657" containerName="cleanup"
Oct 30 19:45:23.981988 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:45:23.974191    4892 memory_manager.go:365] "RemoveStaleState removing state" podUID="d9801904-680f-4253-9104-04bbf36d9b35" containerName="cleanup"
Oct 30 19:45:23.981988 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:45:23.974204    4892 memory_manager.go:365] "RemoveStaleState removing state" podUID="bac2c08f-5b25-4ac6-9fb3-bf8e539c6ed8" containerName="cleanup"
Oct 30 19:45:23.981988 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:45:23.974212    4892 memory_manager.go:365] "RemoveStaleState removing state" podUID="0bbf2f20-664b-4c26-82c8-e779d16a54e3" containerName="cleanup"
Oct 30 19:45:23.981988 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:45:23.974221    4892 memory_manager.go:365] "RemoveStaleState removing state" podUID="c18a2644-72a4-4b63-bf7d-c55bcb715f7c" containerName="cleanup"
Oct 30 19:45:23.981988 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:45:23.974232    4892 memory_manager.go:365] "RemoveStaleState removing state" podUID="01eed39c-1a18-4a99-b1a6-4ffba9703c24" containerName="cleanup"
Oct 30 19:45:23.981988 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:45:23.974243    4892 memory_manager.go:365] "RemoveStaleState removing state" podUID="0bbf2f20-664b-4c26-82c8-e779d16a54e3" containerName="cleanup"
Oct 30 19:45:23.981988 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:45:23.974252    4892 memory_manager.go:365] "RemoveStaleState removing state" podUID="0bbf2f20-664b-4c26-82c8-e779d16a54e3" containerName="cleanup"
Oct 30 19:45:23.981988 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:45:23.974260    4892 memory_manager.go:365] "RemoveStaleState removing state" podUID="01eed39c-1a18-4a99-b1a6-4ffba9703c24" containerName="cleanup"
Oct 30 19:45:23.981988 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:45:23.974270    4892 memory_manager.go:365] "RemoveStaleState removing state" podUID="01eed39c-1a18-4a99-b1a6-4ffba9703c24" containerName="cleanup"
Oct 30 19:45:24.052278 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:45:24.037679    4892 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w0-2e5977-5b974cf99f-mrcrv"
Oct 30 19:45:24.129028 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:45:24.125468    4892 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qnp77\" (UniqueName: \"kubernetes.io/projected/f07835ed-2bb8-4dcc-8d53-9da5b4e2a14f-kube-api-access-qnp77\") pod \"payload-pre-processing-e2e-shared-a-w0-2e5977-5b974cf99f-mrcrv\" (UID: \"f07835ed-2bb8-4dcc-8d53-9da5b4e2a14f\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w0-2e5977-5b974cf99f-mrcrv"
Oct 30 19:45:24.129028 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:45:24.125593    4892 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f07835ed-2bb8-4dcc-8d53-9da5b4e2a14f-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-a-w0-2e5977-5b974cf99f-mrcrv\" (UID: \"f07835ed-2bb8-4dcc-8d53-9da5b4e2a14f\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w0-2e5977-5b974cf99f-mrcrv"
Oct 30 19:45:24.136835 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:45:24.131860    4892 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w0-2e5977-5b974cf99f-mrcrv"]
Oct 30 19:45:24.142454 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:45:24.140368    4892 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w0-2e5977-55d59bd866-nxr6c"]
Oct 30 19:45:24.142454 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:45:24.141412    4892 cpu_manager.go:414] "containerMap: removing container" podUID="c18a2644-72a4-4b63-bf7d-c55bcb715f7c" containerName="cleanup"
Oct 30 19:45:24.142454 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:45:24.141432    4892 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c18a2644-72a4-4b63-bf7d-c55bcb715f7c" containerName="cleanup"
Oct 30 19:45:24.142454 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:45:24.141453    4892 cpu_manager.go:414] "containerMap: removing container" podUID="8277bac6-a556-4e2d-bc2c-17121d44e4b5" containerName="cleanup"
Oct 30 19:45:24.142454 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:45:24.141461    4892 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8277bac6-a556-4e2d-bc2c-17121d44e4b5" containerName="cleanup"
Oct 30 19:45:24.142454 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:45:24.141486    4892 cpu_manager.go:414] "containerMap: removing container" podUID="b8655bf2-a203-4072-8894-994c2a780657" containerName="cleanup"
Oct 30 19:45:24.142454 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:45:24.141494    4892 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b8655bf2-a203-4072-8894-994c2a780657" containerName="cleanup"
Oct 30 19:45:24.142454 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:45:24.141509    4892 cpu_manager.go:414] "containerMap: removing container" podUID="d9801904-680f-4253-9104-04bbf36d9b35" containerName="cleanup"
Oct 30 19:45:24.142454 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:45:24.141517    4892 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d9801904-680f-4253-9104-04bbf36d9b35" containerName="cleanup"
Oct 30 19:45:24.142454 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:45:24.141565    4892 cpu_manager.go:414] "containerMap: removing container" podUID="bac2c08f-5b25-4ac6-9fb3-bf8e539c6ed8" containerName="cleanup"
Oct 30 19:45:24.142454 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:45:24.141572    4892 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="bac2c08f-5b25-4ac6-9fb3-bf8e539c6ed8" containerName="cleanup"
Oct 30 19:45:24.142454 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:45:24.141729    4892 memory_manager.go:365] "RemoveStaleState removing state" podUID="b8655bf2-a203-4072-8894-994c2a780657" containerName="cleanup"
Oct 30 19:45:24.142454 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:45:24.141743    4892 memory_manager.go:365] "RemoveStaleState removing state" podUID="d9801904-680f-4253-9104-04bbf36d9b35" containerName="cleanup"
Oct 30 19:45:24.142454 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:45:24.141760    4892 memory_manager.go:365] "RemoveStaleState removing state" podUID="bac2c08f-5b25-4ac6-9fb3-bf8e539c6ed8" containerName="cleanup"
Oct 30 19:45:24.142454 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:45:24.141771    4892 memory_manager.go:365] "RemoveStaleState removing state" podUID="c18a2644-72a4-4b63-bf7d-c55bcb715f7c" containerName="cleanup"
Oct 30 19:45:24.142454 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:45:24.141791    4892 memory_manager.go:365] "RemoveStaleState removing state" podUID="8277bac6-a556-4e2d-bc2c-17121d44e4b5" containerName="cleanup"
Oct 30 19:45:24.179106 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:45:24.175319    4892 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w0-2e5977-55d59bd866-nxr6c"
Oct 30 19:45:24.214206 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:45:24.195312    4892 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-shared-a-w0-2e5977\""
Oct 30 19:45:24.278175 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:45:24.204295    4892 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-shared-a-w0-2e5977-dockercfg-kjntq\""
Oct 30 19:45:24.314516 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:45:24.279069    4892 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-d5vh5\" (UniqueName: \"kubernetes.io/projected/e0fa9dd2-6f90-485c-aaff-e343ccbe428a-kube-api-access-d5vh5\") pod \"payload-processing-e2e-shared-a-w0-2e5977-55d59bd866-nxr6c\" (UID: \"e0fa9dd2-6f90-485c-aaff-e343ccbe428a\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w0-2e5977-55d59bd866-nxr6c"
Oct 30 19:45:24.314516 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:45:24.279164    4892 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qnp77\" (UniqueName: \"kubernetes.io/projected/f07835ed-2bb8-4dcc-8d53-9da5b4e2a14f-kube-api-access-qnp77\") pod \"payload-pre-processing-e2e-shared-a-w0-2e5977-5b974cf99f-mrcrv\" (UID: \"f07835ed-2bb8-4dcc-8d53-9da5b4e2a14f\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w0-2e5977-5b974cf99f-mrcrv"
Oct 30 19:45:24.314516 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:45:24.279272    4892 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f07835ed-2bb8-4dcc-8d53-9da5b4e2a14f-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-a-w0-2e5977-5b974cf99f-mrcrv\" (UID: \"f07835ed-2bb8-4dcc-8d53-9da5b4e2a14f\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w0-2e5977-5b974cf99f-mrcrv"
Oct 30 19:45:24.314516 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:45:24.279309    4892 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e0fa9dd2-6f90-485c-aaff-e343ccbe428a-plugins-config-volume\") pod \"payload-processing-e2e-shared-a-w0-2e5977-55d59bd866-nxr6c\" (UID: \"e0fa9dd2-6f90-485c-aaff-e343ccbe428a\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w0-2e5977-55d59bd866-nxr6c"
Oct 30 19:45:24.314516 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:45:24.280690    4892 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f07835ed-2bb8-4dcc-8d53-9da5b4e2a14f-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-a-w0-2e5977-5b974cf99f-mrcrv\" (UID: \"f07835ed-2bb8-4dcc-8d53-9da5b4e2a14f\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w0-2e5977-5b974cf99f-mrcrv"
Oct 30 19:45:24.358229 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:45:24.357889    4892 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w0-2e5977-55d59bd866-nxr6c"]
Oct 30 19:45:24.409600 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:45:24.402127    4892 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qnp77\" (UniqueName: \"kubernetes.io/projected/f07835ed-2bb8-4dcc-8d53-9da5b4e2a14f-kube-api-access-qnp77\") pod \"payload-pre-processing-e2e-shared-a-w0-2e5977-5b974cf99f-mrcrv\" (UID: \"f07835ed-2bb8-4dcc-8d53-9da5b4e2a14f\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w0-2e5977-5b974cf99f-mrcrv"
Oct 30 19:45:24.421310 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:45:24.421276    4892 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-d5vh5\" (UniqueName: \"kubernetes.io/projected/e0fa9dd2-6f90-485c-aaff-e343ccbe428a-kube-api-access-d5vh5\") pod \"payload-processing-e2e-shared-a-w0-2e5977-55d59bd866-nxr6c\" (UID: \"e0fa9dd2-6f90-485c-aaff-e343ccbe428a\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w0-2e5977-55d59bd866-nxr6c"
Oct 30 19:45:24.421657 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:45:24.421638    4892 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e0fa9dd2-6f90-485c-aaff-e343ccbe428a-plugins-config-volume\") pod \"payload-processing-e2e-shared-a-w0-2e5977-55d59bd866-nxr6c\" (UID: \"e0fa9dd2-6f90-485c-aaff-e343ccbe428a\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w0-2e5977-55d59bd866-nxr6c"
Oct 30 19:45:24.422983 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:45:24.422939    4892 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e0fa9dd2-6f90-485c-aaff-e343ccbe428a-plugins-config-volume\") pod \"payload-processing-e2e-shared-a-w0-2e5977-55d59bd866-nxr6c\" (UID: \"e0fa9dd2-6f90-485c-aaff-e343ccbe428a\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w0-2e5977-55d59bd866-nxr6c"
Oct 30 19:45:24.447709 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:45:24.447679    4892 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w0-2e5977-5b974cf99f-mrcrv"
Oct 30 19:45:24.486570 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:45:24.486535    4892 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-d5vh5\" (UniqueName: \"kubernetes.io/projected/e0fa9dd2-6f90-485c-aaff-e343ccbe428a-kube-api-access-d5vh5\") pod \"payload-processing-e2e-shared-a-w0-2e5977-55d59bd866-nxr6c\" (UID: \"e0fa9dd2-6f90-485c-aaff-e343ccbe428a\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w0-2e5977-55d59bd866-nxr6c"
Oct 30 19:45:24.573902 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:45:24.573837    4892 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c18a2644-72a4-4b63-bf7d-c55bcb715f7c" path="/var/lib/kubelet/pods/c18a2644-72a4-4b63-bf7d-c55bcb715f7c/volumes"
Oct 30 19:45:24.725759 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:45:24.725726    4892 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w0-2e5977-55d59bd866-nxr6c"
Oct 30 19:45:24.833826 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:45:24.833790    4892 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w0-2e5977-5b974cf99f-mrcrv"]
Oct 30 19:45:24.863481 ip-10-0-132-82 kubenswrapper[4892]: E1030 19:45:24.863431    4892 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-jrm87" containerName="extractor"
Oct 30 19:45:25.136231 ip-10-0-132-82 kubenswrapper[4892]: W1030 19:45:25.136195    4892 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode0fa9dd2_6f90_485c_aaff_e343ccbe428a.slice/crio-e60f5dbe938a9236e81d950e9cd06640f499ae6a9e201d0cd6d2e39445dbc9fe WatchSource:0}: Error finding container e60f5dbe938a9236e81d950e9cd06640f499ae6a9e201d0cd6d2e39445dbc9fe: Status 404 returned error can't find the container with id e60f5dbe938a9236e81d950e9cd06640f499ae6a9e201d0cd6d2e39445dbc9fe
Oct 30 19:45:25.139035 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:45:25.139001    4892 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w0-2e5977-55d59bd866-nxr6c"]
Oct 30 19:45:25.444537 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:45:25.444505    4892 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w0-2e5977-5b974cf99f-mrcrv" event={"ID":"f07835ed-2bb8-4dcc-8d53-9da5b4e2a14f","Type":"ContainerStarted","Data":"6bb17959ba06b84ecca4792abc36bbafdce0cc7c41775b877efabc7de0f40420"}
Oct 30 19:45:25.444687 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:45:25.444564    4892 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w0-2e5977-5b974cf99f-mrcrv" event={"ID":"f07835ed-2bb8-4dcc-8d53-9da5b4e2a14f","Type":"ContainerStarted","Data":"d2283134c0675507621f9a8df66997e141902b9b4a72c19ff72c7c07e8dba0cc"}
Oct 30 19:45:25.444687 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:45:25.444593    4892 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w0-2e5977-5b974cf99f-mrcrv"
Oct 30 19:45:25.446356 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:45:25.446332    4892 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w0-2e5977-55d59bd866-nxr6c" event={"ID":"e0fa9dd2-6f90-485c-aaff-e343ccbe428a","Type":"ContainerStarted","Data":"aa006a086c6d984fb4d0ec4d0b735358620913b45c55459cc930d37463706fb9"}
Oct 30 19:45:25.446500 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:45:25.446371    4892 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w0-2e5977-55d59bd866-nxr6c" event={"ID":"e0fa9dd2-6f90-485c-aaff-e343ccbe428a","Type":"ContainerStarted","Data":"e60f5dbe938a9236e81d950e9cd06640f499ae6a9e201d0cd6d2e39445dbc9fe"}
Oct 30 19:45:25.446500 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:45:25.446413    4892 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-shared-a-w0-2e5977-55d59bd866-nxr6c"
Oct 30 19:45:25.562923 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:45:25.562878    4892 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w0-2e5977-5b974cf99f-mrcrv" podStartSLOduration=2.562865711 podStartE2EDuration="2.562865711s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:45:23 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:45:25.559546905 +0000 UTC m=+1773.130012504" watchObservedRunningTime="2026-10-30 19:45:25.562865711 +0000 UTC m=+1773.133331310"
Oct 30 19:45:25.734117 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:45:25.733201    4892 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-shared-a-w0-2e5977-55d59bd866-nxr6c" podStartSLOduration=2.733179303 podStartE2EDuration="2.733179303s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:45:23 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:45:25.690397959 +0000 UTC m=+1773.260863560" watchObservedRunningTime="2026-10-30 19:45:25.733179303 +0000 UTC m=+1773.303644904"
Oct 30 19:45:33.064097 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:45:33.064061    4892 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-8a640f9d-8ddnv"]
Oct 30 19:45:33.108157 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:45:33.108125    4892 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-8a640f9d-8ddnv"]
Oct 30 19:45:33.585852 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:45:33.585810    4892 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-94581ab2-qvqdv"]
Oct 30 19:45:33.591931 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:45:33.591904    4892 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-94581ab2-qvqdv"
Oct 30 19:45:33.646400 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:45:33.646371    4892 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 19:45:33.646626 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:45:33.646610    4892 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-nlxp4\""
Oct 30 19:45:33.678263 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:45:33.678224    4892 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5e40fdcc-dac3-46d6-a965-f86b809e11c8-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ait-94581ab2-qvqdv\" (UID: \"5e40fdcc-dac3-46d6-a965-f86b809e11c8\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-94581ab2-qvqdv"
Oct 30 19:45:33.678439 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:45:33.678332    4892 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5e40fdcc-dac3-46d6-a965-f86b809e11c8-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ait-94581ab2-qvqdv\" (UID: \"5e40fdcc-dac3-46d6-a965-f86b809e11c8\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-94581ab2-qvqdv"
Oct 30 19:45:33.678439 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:45:33.678399    4892 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vpfsd\" (UniqueName: \"kubernetes.io/projected/5e40fdcc-dac3-46d6-a965-f86b809e11c8-kube-api-access-vpfsd\") pod \"maas-api-revoke-keys-e2e-ait-94581ab2-qvqdv\" (UID: \"5e40fdcc-dac3-46d6-a965-f86b809e11c8\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-94581ab2-qvqdv"
Oct 30 19:45:33.696104 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:45:33.696073    4892 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 30 19:45:33.696327 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:45:33.696302    4892 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 30 19:45:33.702092 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:45:33.702072    4892 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-94581ab2-qvqdv"]
Oct 30 19:45:33.780342 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:45:33.780303    4892 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5e40fdcc-dac3-46d6-a965-f86b809e11c8-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ait-94581ab2-qvqdv\" (UID: \"5e40fdcc-dac3-46d6-a965-f86b809e11c8\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-94581ab2-qvqdv"
Oct 30 19:45:33.780510 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:45:33.780396    4892 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5e40fdcc-dac3-46d6-a965-f86b809e11c8-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ait-94581ab2-qvqdv\" (UID: \"5e40fdcc-dac3-46d6-a965-f86b809e11c8\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-94581ab2-qvqdv"
Oct 30 19:45:33.780510 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:45:33.780459    4892 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vpfsd\" (UniqueName: \"kubernetes.io/projected/5e40fdcc-dac3-46d6-a965-f86b809e11c8-kube-api-access-vpfsd\") pod \"maas-api-revoke-keys-e2e-ait-94581ab2-qvqdv\" (UID: \"5e40fdcc-dac3-46d6-a965-f86b809e11c8\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-94581ab2-qvqdv"
Oct 30 19:45:33.781296 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:45:33.781236    4892 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5e40fdcc-dac3-46d6-a965-f86b809e11c8-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ait-94581ab2-qvqdv\" (UID: \"5e40fdcc-dac3-46d6-a965-f86b809e11c8\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-94581ab2-qvqdv"
Oct 30 19:45:33.783006 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:45:33.782952    4892 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5e40fdcc-dac3-46d6-a965-f86b809e11c8-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ait-94581ab2-qvqdv\" (UID: \"5e40fdcc-dac3-46d6-a965-f86b809e11c8\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-94581ab2-qvqdv"
Oct 30 19:45:33.837722 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:45:33.837660    4892 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vpfsd\" (UniqueName: \"kubernetes.io/projected/5e40fdcc-dac3-46d6-a965-f86b809e11c8-kube-api-access-vpfsd\") pod \"maas-api-revoke-keys-e2e-ait-94581ab2-qvqdv\" (UID: \"5e40fdcc-dac3-46d6-a965-f86b809e11c8\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-94581ab2-qvqdv"
Oct 30 19:45:33.907372 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:45:33.907345    4892 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-94581ab2-qvqdv"
Oct 30 19:45:34.302317 ip-10-0-132-82 kubenswrapper[4892]: W1030 19:45:34.302286    4892 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5e40fdcc_dac3_46d6_a965_f86b809e11c8.slice/crio-2172538a96c3055b1849b199dc6f498f17db8fc39384db858664b42859d8829f WatchSource:0}: Error finding container 2172538a96c3055b1849b199dc6f498f17db8fc39384db858664b42859d8829f: Status 404 returned error can't find the container with id 2172538a96c3055b1849b199dc6f498f17db8fc39384db858664b42859d8829f
Oct 30 19:45:34.306297 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:45:34.306270    4892 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-94581ab2-qvqdv"]
Oct 30 19:45:34.572131 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:45:34.572066    4892 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="1001fd5a-3158-413d-affa-26daff184f72" path="/var/lib/kubelet/pods/1001fd5a-3158-413d-affa-26daff184f72/volumes"
Oct 30 19:45:34.619914 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:45:34.619883    4892 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-94581ab2-qvqdv" event={"ID":"5e40fdcc-dac3-46d6-a965-f86b809e11c8","Type":"ContainerStarted","Data":"2172538a96c3055b1849b199dc6f498f17db8fc39384db858664b42859d8829f"}
Oct 30 19:45:35.630586 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:45:35.630496    4892 generic.go:356] "Generic (PLEG): container finished" podID="5e40fdcc-dac3-46d6-a965-f86b809e11c8" containerID="fe08b7b3aac0e21b63ef8e2a571d670ae5dfb48ee4f7c82134b5c20fa0ff6fc7" exitCode=0
Oct 30 19:45:35.630586 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:45:35.630540    4892 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-94581ab2-qvqdv" event={"ID":"5e40fdcc-dac3-46d6-a965-f86b809e11c8","Type":"ContainerDied","Data":"fe08b7b3aac0e21b63ef8e2a571d670ae5dfb48ee4f7c82134b5c20fa0ff6fc7"}
Oct 30 19:45:36.458382 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:45:36.458353    4892 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-shared-a-w0-2e5977-55d59bd866-nxr6c"
Oct 30 19:45:36.458829 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:45:36.458803    4892 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w0-2e5977-5b974cf99f-mrcrv"
Oct 30 19:45:36.819930 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:45:36.819907    4892 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-94581ab2-qvqdv"
Oct 30 19:45:36.953998 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:45:36.952817    4892 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5e40fdcc-dac3-46d6-a965-f86b809e11c8-kube-api-access-vpfsd\" (UniqueName: \"kubernetes.io/projected/5e40fdcc-dac3-46d6-a965-f86b809e11c8-kube-api-access-vpfsd\") pod \"5e40fdcc-dac3-46d6-a965-f86b809e11c8\" (UID: \"5e40fdcc-dac3-46d6-a965-f86b809e11c8\") "
Oct 30 19:45:36.953998 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:45:36.952931    4892 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/5e40fdcc-dac3-46d6-a965-f86b809e11c8-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5e40fdcc-dac3-46d6-a965-f86b809e11c8-maas-api-xks-ca\") pod \"5e40fdcc-dac3-46d6-a965-f86b809e11c8\" (UID: \"5e40fdcc-dac3-46d6-a965-f86b809e11c8\") "
Oct 30 19:45:36.953998 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:45:36.953035    4892 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/5e40fdcc-dac3-46d6-a965-f86b809e11c8-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5e40fdcc-dac3-46d6-a965-f86b809e11c8-maas-api-openshift-ca\") pod \"5e40fdcc-dac3-46d6-a965-f86b809e11c8\" (UID: \"5e40fdcc-dac3-46d6-a965-f86b809e11c8\") "
Oct 30 19:45:36.954304 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:45:36.954100    4892 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5e40fdcc-dac3-46d6-a965-f86b809e11c8-maas-api-openshift-ca" pod "5e40fdcc-dac3-46d6-a965-f86b809e11c8" (UID: "5e40fdcc-dac3-46d6-a965-f86b809e11c8"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 19:45:36.972208 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:45:36.972148    4892 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5e40fdcc-dac3-46d6-a965-f86b809e11c8-maas-api-xks-ca" pod "5e40fdcc-dac3-46d6-a965-f86b809e11c8" (UID: "5e40fdcc-dac3-46d6-a965-f86b809e11c8"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 19:45:36.977375 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:45:36.977132    4892 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5e40fdcc-dac3-46d6-a965-f86b809e11c8-kube-api-access-vpfsd" pod "5e40fdcc-dac3-46d6-a965-f86b809e11c8" (UID: "5e40fdcc-dac3-46d6-a965-f86b809e11c8"). InnerVolumeSpecName "kube-api-access-vpfsd". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:45:37.071056 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:45:37.070899    4892 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5e40fdcc-dac3-46d6-a965-f86b809e11c8-maas-api-openshift-ca\") on node \"ip-10-0-132-82.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 19:45:37.071056 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:45:37.070934    4892 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vpfsd\" (UniqueName: \"kubernetes.io/projected/5e40fdcc-dac3-46d6-a965-f86b809e11c8-kube-api-access-vpfsd\") on node \"ip-10-0-132-82.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 19:45:37.071056 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:45:37.070948    4892 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5e40fdcc-dac3-46d6-a965-f86b809e11c8-maas-api-xks-ca\") on node \"ip-10-0-132-82.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 19:45:37.659680 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:45:37.659650    4892 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-94581ab2-qvqdv"
Oct 30 19:45:37.660001 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:45:37.659676    4892 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-94581ab2-qvqdv" event={"ID":"5e40fdcc-dac3-46d6-a965-f86b809e11c8","Type":"ContainerDied","Data":"2172538a96c3055b1849b199dc6f498f17db8fc39384db858664b42859d8829f"}
Oct 30 19:45:37.660001 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:45:37.659738    4892 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="2172538a96c3055b1849b199dc6f498f17db8fc39384db858664b42859d8829f"
Oct 30 19:45:42.693188 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:45:42.693151    4892 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-d74c1bd9-dc96t"]
Oct 30 19:45:42.693877 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:45:42.693859    4892 cpu_manager.go:414] "containerMap: removing container" podUID="5e40fdcc-dac3-46d6-a965-f86b809e11c8" containerName="revoke-keys"
Oct 30 19:45:42.693921 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:45:42.693879    4892 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5e40fdcc-dac3-46d6-a965-f86b809e11c8" containerName="revoke-keys"
Oct 30 19:45:42.694040 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:45:42.694030    4892 memory_manager.go:365] "RemoveStaleState removing state" podUID="5e40fdcc-dac3-46d6-a965-f86b809e11c8" containerName="revoke-keys"
Oct 30 19:45:42.697670 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:45:42.697653    4892 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-d74c1bd9-dc96t"
Oct 30 19:45:42.752884 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:45:42.752850    4892 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0cba880a-5d89-4205-b60a-fd99cbadc021-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-migrate-d74c1bd9-dc96t\" (UID: \"0cba880a-5d89-4205-b60a-fd99cbadc021\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-d74c1bd9-dc96t"
Oct 30 19:45:42.753105 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:45:42.752905    4892 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0cba880a-5d89-4205-b60a-fd99cbadc021-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-migrate-d74c1bd9-dc96t\" (UID: \"0cba880a-5d89-4205-b60a-fd99cbadc021\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-d74c1bd9-dc96t"
Oct 30 19:45:42.753105 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:45:42.753064    4892 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-krcbv\" (UniqueName: \"kubernetes.io/projected/0cba880a-5d89-4205-b60a-fd99cbadc021-kube-api-access-krcbv\") pod \"maas-api-revoke-keys-e2e-migrate-d74c1bd9-dc96t\" (UID: \"0cba880a-5d89-4205-b60a-fd99cbadc021\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-d74c1bd9-dc96t"
Oct 30 19:45:42.803153 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:45:42.803126    4892 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-nlxp4\""
Oct 30 19:45:42.803354 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:45:42.803230    4892 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 19:45:42.803354 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:45:42.803280    4892 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 30 19:45:42.803354 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:45:42.803308    4892 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 30 19:45:42.809894 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:45:42.809870    4892 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-d74c1bd9-dc96t"]
Oct 30 19:45:42.854531 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:45:42.854506    4892 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0cba880a-5d89-4205-b60a-fd99cbadc021-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-migrate-d74c1bd9-dc96t\" (UID: \"0cba880a-5d89-4205-b60a-fd99cbadc021\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-d74c1bd9-dc96t"
Oct 30 19:45:42.854661 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:45:42.854601    4892 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0cba880a-5d89-4205-b60a-fd99cbadc021-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-migrate-d74c1bd9-dc96t\" (UID: \"0cba880a-5d89-4205-b60a-fd99cbadc021\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-d74c1bd9-dc96t"
Oct 30 19:45:42.854746 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:45:42.854729    4892 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-krcbv\" (UniqueName: \"kubernetes.io/projected/0cba880a-5d89-4205-b60a-fd99cbadc021-kube-api-access-krcbv\") pod \"maas-api-revoke-keys-e2e-migrate-d74c1bd9-dc96t\" (UID: \"0cba880a-5d89-4205-b60a-fd99cbadc021\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-d74c1bd9-dc96t"
Oct 30 19:45:42.855635 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:45:42.855612    4892 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0cba880a-5d89-4205-b60a-fd99cbadc021-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-migrate-d74c1bd9-dc96t\" (UID: \"0cba880a-5d89-4205-b60a-fd99cbadc021\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-d74c1bd9-dc96t"
Oct 30 19:45:42.856941 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:45:42.856923    4892 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0cba880a-5d89-4205-b60a-fd99cbadc021-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-migrate-d74c1bd9-dc96t\" (UID: \"0cba880a-5d89-4205-b60a-fd99cbadc021\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-d74c1bd9-dc96t"
Oct 30 19:45:42.911916 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:45:42.911890    4892 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-krcbv\" (UniqueName: \"kubernetes.io/projected/0cba880a-5d89-4205-b60a-fd99cbadc021-kube-api-access-krcbv\") pod \"maas-api-revoke-keys-e2e-migrate-d74c1bd9-dc96t\" (UID: \"0cba880a-5d89-4205-b60a-fd99cbadc021\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-d74c1bd9-dc96t"
Oct 30 19:45:43.012767 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:45:43.012705    4892 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-d74c1bd9-dc96t"
Oct 30 19:45:43.058347 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:45:43.058317    4892 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c08feb3f-fwb78"]
Oct 30 19:45:43.086648 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:45:43.086620    4892 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c08feb3f-fwb78"]
Oct 30 19:45:43.406663 ip-10-0-132-82 kubenswrapper[4892]: W1030 19:45:43.406630    4892 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0cba880a_5d89_4205_b60a_fd99cbadc021.slice/crio-34acbc6402a76e3670714e0d04dc14cf4f26e1f802fa7bbe33845542c2d69ac5 WatchSource:0}: Error finding container 34acbc6402a76e3670714e0d04dc14cf4f26e1f802fa7bbe33845542c2d69ac5: Status 404 returned error can't find the container with id 34acbc6402a76e3670714e0d04dc14cf4f26e1f802fa7bbe33845542c2d69ac5
Oct 30 19:45:43.407864 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:45:43.407841    4892 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-d74c1bd9-dc96t"]
Oct 30 19:45:43.719711 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:45:43.719633    4892 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-d74c1bd9-dc96t" event={"ID":"0cba880a-5d89-4205-b60a-fd99cbadc021","Type":"ContainerStarted","Data":"34acbc6402a76e3670714e0d04dc14cf4f26e1f802fa7bbe33845542c2d69ac5"}
Oct 30 19:45:44.573357 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:45:44.573285    4892 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="35e6d043-b9a3-4e16-bcb3-49de88dcbe75" path="/var/lib/kubelet/pods/35e6d043-b9a3-4e16-bcb3-49de88dcbe75/volumes"
Oct 30 19:45:44.730623 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:45:44.730581    4892 generic.go:356] "Generic (PLEG): container finished" podID="0cba880a-5d89-4205-b60a-fd99cbadc021" containerID="452359c17c5092139de6baf674ddbca722db515a7f9b501432124b253e3bc396" exitCode=0
Oct 30 19:45:44.731096 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:45:44.730675    4892 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-d74c1bd9-dc96t" event={"ID":"0cba880a-5d89-4205-b60a-fd99cbadc021","Type":"ContainerDied","Data":"452359c17c5092139de6baf674ddbca722db515a7f9b501432124b253e3bc396"}
Oct 30 19:45:44.902500 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:45:44.902463    4892 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-94581ab2-688dd7dd6-rxx2h"]
Oct 30 19:45:45.920957 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:45:45.920933    4892 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-d74c1bd9-dc96t"
Oct 30 19:45:46.048740 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:45:46.048705    4892 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/0cba880a-5d89-4205-b60a-fd99cbadc021-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0cba880a-5d89-4205-b60a-fd99cbadc021-maas-api-openshift-ca\") pod \"0cba880a-5d89-4205-b60a-fd99cbadc021\" (UID: \"0cba880a-5d89-4205-b60a-fd99cbadc021\") "
Oct 30 19:45:46.048937 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:45:46.048834    4892 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/0cba880a-5d89-4205-b60a-fd99cbadc021-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0cba880a-5d89-4205-b60a-fd99cbadc021-maas-api-xks-ca\") pod \"0cba880a-5d89-4205-b60a-fd99cbadc021\" (UID: \"0cba880a-5d89-4205-b60a-fd99cbadc021\") "
Oct 30 19:45:46.048937 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:45:46.048883    4892 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0cba880a-5d89-4205-b60a-fd99cbadc021-kube-api-access-krcbv\" (UniqueName: \"kubernetes.io/projected/0cba880a-5d89-4205-b60a-fd99cbadc021-kube-api-access-krcbv\") pod \"0cba880a-5d89-4205-b60a-fd99cbadc021\" (UID: \"0cba880a-5d89-4205-b60a-fd99cbadc021\") "
Oct 30 19:45:46.050241 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:45:46.050214    4892 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0cba880a-5d89-4205-b60a-fd99cbadc021-maas-api-openshift-ca" pod "0cba880a-5d89-4205-b60a-fd99cbadc021" (UID: "0cba880a-5d89-4205-b60a-fd99cbadc021"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 19:45:46.058918 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:45:46.058894    4892 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0cba880a-5d89-4205-b60a-fd99cbadc021-kube-api-access-krcbv" pod "0cba880a-5d89-4205-b60a-fd99cbadc021" (UID: "0cba880a-5d89-4205-b60a-fd99cbadc021"). InnerVolumeSpecName "kube-api-access-krcbv". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:45:46.059323 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:45:46.059293    4892 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0cba880a-5d89-4205-b60a-fd99cbadc021-maas-api-xks-ca" pod "0cba880a-5d89-4205-b60a-fd99cbadc021" (UID: "0cba880a-5d89-4205-b60a-fd99cbadc021"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 19:45:46.150805 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:45:46.150735    4892 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0cba880a-5d89-4205-b60a-fd99cbadc021-maas-api-openshift-ca\") on node \"ip-10-0-132-82.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 19:45:46.150805 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:45:46.150770    4892 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0cba880a-5d89-4205-b60a-fd99cbadc021-maas-api-xks-ca\") on node \"ip-10-0-132-82.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 19:45:46.150805 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:45:46.150785    4892 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-krcbv\" (UniqueName: \"kubernetes.io/projected/0cba880a-5d89-4205-b60a-fd99cbadc021-kube-api-access-krcbv\") on node \"ip-10-0-132-82.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 19:45:46.791994 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:45:46.790431    4892 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-d74c1bd9-dc96t" event={"ID":"0cba880a-5d89-4205-b60a-fd99cbadc021","Type":"ContainerDied","Data":"34acbc6402a76e3670714e0d04dc14cf4f26e1f802fa7bbe33845542c2d69ac5"}
Oct 30 19:45:46.791994 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:45:46.790474    4892 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-d74c1bd9-dc96t"
Oct 30 19:45:46.791994 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:45:46.790542    4892 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="34acbc6402a76e3670714e0d04dc14cf4f26e1f802fa7bbe33845542c2d69ac5"
Oct 30 19:45:49.903222 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:45:49.903164    4892 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-ait-94581ab2-688dd7dd6-rxx2h" podUID="e4da7424-8179-4b09-82cf-d2f3ed9f4894" containerName="payload-processing" containerID="cri-o://c69391ca1e932e304e80741189ad5f815e5d3d059e1057e9ef008a49916e23e3" gracePeriod=25
Oct 30 19:45:50.196086 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:45:50.196059    4892 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-94581ab2-688dd7dd6-rxx2h"
Oct 30 19:45:50.354991 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:45:50.350058    4892 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e4da7424-8179-4b09-82cf-d2f3ed9f4894-kube-api-access-9b7t8\" (UniqueName: \"kubernetes.io/projected/e4da7424-8179-4b09-82cf-d2f3ed9f4894-kube-api-access-9b7t8\") pod \"e4da7424-8179-4b09-82cf-d2f3ed9f4894\" (UID: \"e4da7424-8179-4b09-82cf-d2f3ed9f4894\") "
Oct 30 19:45:50.354991 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:45:50.350121    4892 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e4da7424-8179-4b09-82cf-d2f3ed9f4894-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e4da7424-8179-4b09-82cf-d2f3ed9f4894-plugins-config-volume\") pod \"e4da7424-8179-4b09-82cf-d2f3ed9f4894\" (UID: \"e4da7424-8179-4b09-82cf-d2f3ed9f4894\") "
Oct 30 19:45:50.354991 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:45:50.351283    4892 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e4da7424-8179-4b09-82cf-d2f3ed9f4894-plugins-config-volume" pod "e4da7424-8179-4b09-82cf-d2f3ed9f4894" (UID: "e4da7424-8179-4b09-82cf-d2f3ed9f4894"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 19:45:50.378763 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:45:50.378604    4892 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e4da7424-8179-4b09-82cf-d2f3ed9f4894-kube-api-access-9b7t8" pod "e4da7424-8179-4b09-82cf-d2f3ed9f4894" (UID: "e4da7424-8179-4b09-82cf-d2f3ed9f4894"). InnerVolumeSpecName "kube-api-access-9b7t8". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:45:50.451859 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:45:50.451779    4892 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9b7t8\" (UniqueName: \"kubernetes.io/projected/e4da7424-8179-4b09-82cf-d2f3ed9f4894-kube-api-access-9b7t8\") on node \"ip-10-0-132-82.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 19:45:50.451859 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:45:50.451815    4892 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e4da7424-8179-4b09-82cf-d2f3ed9f4894-plugins-config-volume\") on node \"ip-10-0-132-82.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 19:45:50.832928 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:45:50.832838    4892 generic.go:356] "Generic (PLEG): container finished" podID="e4da7424-8179-4b09-82cf-d2f3ed9f4894" containerID="c69391ca1e932e304e80741189ad5f815e5d3d059e1057e9ef008a49916e23e3" exitCode=0
Oct 30 19:45:50.832928 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:45:50.832890    4892 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-94581ab2-688dd7dd6-rxx2h" event={"ID":"e4da7424-8179-4b09-82cf-d2f3ed9f4894","Type":"ContainerDied","Data":"c69391ca1e932e304e80741189ad5f815e5d3d059e1057e9ef008a49916e23e3"}
Oct 30 19:45:50.833180 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:45:50.832993    4892 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-94581ab2-688dd7dd6-rxx2h" event={"ID":"e4da7424-8179-4b09-82cf-d2f3ed9f4894","Type":"ContainerDied","Data":"e1aaf116034b42bb3532cd47f3a5be2dc4204750e120ca5997f7f5c7390f2c7b"}
Oct 30 19:45:50.833180 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:45:50.832901    4892 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-94581ab2-688dd7dd6-rxx2h"
Oct 30 19:45:50.833180 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:45:50.833028    4892 scope.go:122] "RemoveContainer" containerID="c69391ca1e932e304e80741189ad5f815e5d3d059e1057e9ef008a49916e23e3"
Oct 30 19:45:50.853749 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:45:50.853726    4892 scope.go:122] "RemoveContainer" containerID="c69391ca1e932e304e80741189ad5f815e5d3d059e1057e9ef008a49916e23e3"
Oct 30 19:45:50.854137 ip-10-0-132-82 kubenswrapper[4892]: E1030 19:45:50.854110    4892 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c69391ca1e932e304e80741189ad5f815e5d3d059e1057e9ef008a49916e23e3\": container with ID starting with c69391ca1e932e304e80741189ad5f815e5d3d059e1057e9ef008a49916e23e3 not found: ID does not exist" containerID="c69391ca1e932e304e80741189ad5f815e5d3d059e1057e9ef008a49916e23e3"
Oct 30 19:45:50.854273 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:45:50.854146    4892 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c69391ca1e932e304e80741189ad5f815e5d3d059e1057e9ef008a49916e23e3"} err="failed to get container status \"c69391ca1e932e304e80741189ad5f815e5d3d059e1057e9ef008a49916e23e3\": rpc error: code = NotFound desc = could not find container \"c69391ca1e932e304e80741189ad5f815e5d3d059e1057e9ef008a49916e23e3\": container with ID starting with c69391ca1e932e304e80741189ad5f815e5d3d059e1057e9ef008a49916e23e3 not found: ID does not exist"
Oct 30 19:45:51.009497 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:45:51.009463    4892 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-94581ab2-688dd7dd6-rxx2h"]
Oct 30 19:45:51.057346 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:45:51.057315    4892 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-94581ab2-688dd7dd6-rxx2h"]
Oct 30 19:45:51.067021 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:45:51.066988    4892 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c9e9de3e-cd94p"]
Oct 30 19:45:51.071332 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:45:51.071312    4892 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c9e9de3e-cd94p"]
Oct 30 19:45:52.102677 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:45:52.102648    4892 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-affa4c07-25tmc"]
Oct 30 19:45:52.103285 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:45:52.103269    4892 cpu_manager.go:414] "containerMap: removing container" podUID="0cba880a-5d89-4205-b60a-fd99cbadc021" containerName="revoke-keys"
Oct 30 19:45:52.103336 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:45:52.103286    4892 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0cba880a-5d89-4205-b60a-fd99cbadc021" containerName="revoke-keys"
Oct 30 19:45:52.103336 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:45:52.103325    4892 cpu_manager.go:414] "containerMap: removing container" podUID="e4da7424-8179-4b09-82cf-d2f3ed9f4894" containerName="payload-processing"
Oct 30 19:45:52.103336 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:45:52.103330    4892 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e4da7424-8179-4b09-82cf-d2f3ed9f4894" containerName="payload-processing"
Oct 30 19:45:52.103445 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:45:52.103436    4892 memory_manager.go:365] "RemoveStaleState removing state" podUID="0cba880a-5d89-4205-b60a-fd99cbadc021" containerName="revoke-keys"
Oct 30 19:45:52.103484 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:45:52.103453    4892 memory_manager.go:365] "RemoveStaleState removing state" podUID="e4da7424-8179-4b09-82cf-d2f3ed9f4894" containerName="payload-processing"
Oct 30 19:45:52.108662 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:45:52.108643    4892 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-affa4c07-25tmc"
Oct 30 19:45:52.165373 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:45:52.165349    4892 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-affa4c07-25tmc"]
Oct 30 19:45:52.207440 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:45:52.207415    4892 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 30 19:45:52.207440 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:45:52.207420    4892 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 19:45:52.207622 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:45:52.207609    4892 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-nlxp4\""
Oct 30 19:45:52.207660 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:45:52.207630    4892 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 30 19:45:52.293072 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:45:52.293043    4892 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/75ab7a24-1dfb-4dd4-a1c7-e0a1c0f9209a-maas-api-xks-ca\") pod \"maas-api-revoke-sub-affa4c07-25tmc\" (UID: \"75ab7a24-1dfb-4dd4-a1c7-e0a1c0f9209a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-affa4c07-25tmc"
Oct 30 19:45:52.293232 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:45:52.293094    4892 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/75ab7a24-1dfb-4dd4-a1c7-e0a1c0f9209a-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-affa4c07-25tmc\" (UID: \"75ab7a24-1dfb-4dd4-a1c7-e0a1c0f9209a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-affa4c07-25tmc"
Oct 30 19:45:52.293232 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:45:52.293194    4892 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dzgcp\" (UniqueName: \"kubernetes.io/projected/75ab7a24-1dfb-4dd4-a1c7-e0a1c0f9209a-kube-api-access-dzgcp\") pod \"maas-api-revoke-sub-affa4c07-25tmc\" (UID: \"75ab7a24-1dfb-4dd4-a1c7-e0a1c0f9209a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-affa4c07-25tmc"
Oct 30 19:45:52.394240 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:45:52.394211    4892 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/75ab7a24-1dfb-4dd4-a1c7-e0a1c0f9209a-maas-api-xks-ca\") pod \"maas-api-revoke-sub-affa4c07-25tmc\" (UID: \"75ab7a24-1dfb-4dd4-a1c7-e0a1c0f9209a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-affa4c07-25tmc"
Oct 30 19:45:52.394395 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:45:52.394264    4892 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/75ab7a24-1dfb-4dd4-a1c7-e0a1c0f9209a-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-affa4c07-25tmc\" (UID: \"75ab7a24-1dfb-4dd4-a1c7-e0a1c0f9209a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-affa4c07-25tmc"
Oct 30 19:45:52.394395 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:45:52.394320    4892 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dzgcp\" (UniqueName: \"kubernetes.io/projected/75ab7a24-1dfb-4dd4-a1c7-e0a1c0f9209a-kube-api-access-dzgcp\") pod \"maas-api-revoke-sub-affa4c07-25tmc\" (UID: \"75ab7a24-1dfb-4dd4-a1c7-e0a1c0f9209a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-affa4c07-25tmc"
Oct 30 19:45:52.394922 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:45:52.394902    4892 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/75ab7a24-1dfb-4dd4-a1c7-e0a1c0f9209a-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-affa4c07-25tmc\" (UID: \"75ab7a24-1dfb-4dd4-a1c7-e0a1c0f9209a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-affa4c07-25tmc"
Oct 30 19:45:52.396685 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:45:52.396668    4892 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/75ab7a24-1dfb-4dd4-a1c7-e0a1c0f9209a-maas-api-xks-ca\") pod \"maas-api-revoke-sub-affa4c07-25tmc\" (UID: \"75ab7a24-1dfb-4dd4-a1c7-e0a1c0f9209a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-affa4c07-25tmc"
Oct 30 19:45:52.452175 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:45:52.452148    4892 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dzgcp\" (UniqueName: \"kubernetes.io/projected/75ab7a24-1dfb-4dd4-a1c7-e0a1c0f9209a-kube-api-access-dzgcp\") pod \"maas-api-revoke-sub-affa4c07-25tmc\" (UID: \"75ab7a24-1dfb-4dd4-a1c7-e0a1c0f9209a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-affa4c07-25tmc"
Oct 30 19:45:52.571426 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:45:52.571398    4892 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="7e48e8e5-076a-4ac3-ade6-eb7091526083" path="/var/lib/kubelet/pods/7e48e8e5-076a-4ac3-ade6-eb7091526083/volumes"
Oct 30 19:45:52.571957 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:45:52.571941    4892 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e4da7424-8179-4b09-82cf-d2f3ed9f4894" path="/var/lib/kubelet/pods/e4da7424-8179-4b09-82cf-d2f3ed9f4894/volumes"
Oct 30 19:45:52.774384 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:45:52.774315    4892 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-nlxp4\""
Oct 30 19:45:52.783410 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:45:52.783386    4892 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-affa4c07-25tmc"
Oct 30 19:45:53.163621 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:45:53.163590    4892 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-tg5xj_11db9cec-653f-4aee-8937-0cf5b367efac/console-operator/3.log"
Oct 30 19:45:53.177565 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:45:53.177536    4892 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-affa4c07-25tmc"]
Oct 30 19:45:53.200597 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:45:53.200573    4892 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-tg5xj_11db9cec-653f-4aee-8937-0cf5b367efac/console-operator/3.log"
Oct 30 19:45:53.327499 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:45:53.327428    4892 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-migrate-d74c1bd9-7c87cbbf88-2j9tg"]
Oct 30 19:45:53.453775 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:45:53.453742    4892 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-migrate-d74c1bd9-5f777c6ff-qbwsw"]
Oct 30 19:45:53.468316 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:45:53.468285    4892 scope.go:122] "RemoveContainer" containerID="6adb9a485da8f09c3db161aca29c7746a2413878183d349991a17b2e2221f027"
Oct 30 19:45:53.481424 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:45:53.481401    4892 scope.go:122] "RemoveContainer" containerID="74f9b4a40227cdd628d7f0a2a1ce04c819df41f8615bfc28d152dbb5de52b06d"
Oct 30 19:45:53.491528 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:45:53.491502    4892 scope.go:122] "RemoveContainer" containerID="7806b68e79518c76f170bf51b65616bc277e2def2c60080b6501c59c4e454f07"
Oct 30 19:45:53.503619 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:45:53.503598    4892 scope.go:122] "RemoveContainer" containerID="58655ebd49fd1ebfde56ed164df82683d25afb542ea68bf6d233a7587e6d15dd"
Oct 30 19:45:53.927171 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:45:53.927005    4892 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-affa4c07-25tmc" event={"ID":"75ab7a24-1dfb-4dd4-a1c7-e0a1c0f9209a","Type":"ContainerStarted","Data":"212e7cfe45e7685162afd76de6796d429e249e786dffc0f7d559b385f84766e0"}
Oct 30 19:45:53.927171 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:45:53.927068    4892 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-affa4c07-25tmc" event={"ID":"75ab7a24-1dfb-4dd4-a1c7-e0a1c0f9209a","Type":"ContainerStarted","Data":"853c2726feecca75ad3a1d35b13a7386d171de88a9c44439cbc8b60143e552b0"}
Oct 30 19:45:54.081016 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:45:54.074006    4892 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-affa4c07-25tmc" podStartSLOduration=2.073983038 podStartE2EDuration="2.073983038s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:45:52 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:45:54.059163006 +0000 UTC m=+1801.629628606" watchObservedRunningTime="2026-10-30 19:45:54.073983038 +0000 UTC m=+1801.644448635"
Oct 30 19:45:55.045990 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:45:55.045684    4892 generic.go:356] "Generic (PLEG): container finished" podID="75ab7a24-1dfb-4dd4-a1c7-e0a1c0f9209a" containerID="212e7cfe45e7685162afd76de6796d429e249e786dffc0f7d559b385f84766e0" exitCode=0
Oct 30 19:45:55.045990 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:45:55.045805    4892 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-affa4c07-25tmc" event={"ID":"75ab7a24-1dfb-4dd4-a1c7-e0a1c0f9209a","Type":"ContainerDied","Data":"212e7cfe45e7685162afd76de6796d429e249e786dffc0f7d559b385f84766e0"}
Oct 30 19:45:55.080297 ip-10-0-132-82 kubenswrapper[4892]: E1030 19:45:55.080255    4892 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-jrm87" containerName="extractor"
Oct 30 19:45:56.361457 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:45:56.361433    4892 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-affa4c07-25tmc"
Oct 30 19:45:56.442822 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:45:56.442797    4892 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/75ab7a24-1dfb-4dd4-a1c7-e0a1c0f9209a-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/75ab7a24-1dfb-4dd4-a1c7-e0a1c0f9209a-maas-api-xks-ca\") pod \"75ab7a24-1dfb-4dd4-a1c7-e0a1c0f9209a\" (UID: \"75ab7a24-1dfb-4dd4-a1c7-e0a1c0f9209a\") "
Oct 30 19:45:56.443002 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:45:56.442959    4892 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/75ab7a24-1dfb-4dd4-a1c7-e0a1c0f9209a-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/75ab7a24-1dfb-4dd4-a1c7-e0a1c0f9209a-maas-api-openshift-ca\") pod \"75ab7a24-1dfb-4dd4-a1c7-e0a1c0f9209a\" (UID: \"75ab7a24-1dfb-4dd4-a1c7-e0a1c0f9209a\") "
Oct 30 19:45:56.446047 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:45:56.443442    4892 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/75ab7a24-1dfb-4dd4-a1c7-e0a1c0f9209a-maas-api-openshift-ca" pod "75ab7a24-1dfb-4dd4-a1c7-e0a1c0f9209a" (UID: "75ab7a24-1dfb-4dd4-a1c7-e0a1c0f9209a"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 19:45:56.446047 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:45:56.444288    4892 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/75ab7a24-1dfb-4dd4-a1c7-e0a1c0f9209a-maas-api-openshift-ca\") on node \"ip-10-0-132-82.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 19:45:56.454535 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:45:56.454508    4892 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/75ab7a24-1dfb-4dd4-a1c7-e0a1c0f9209a-maas-api-xks-ca" pod "75ab7a24-1dfb-4dd4-a1c7-e0a1c0f9209a" (UID: "75ab7a24-1dfb-4dd4-a1c7-e0a1c0f9209a"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 19:45:56.544728 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:45:56.544699    4892 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/75ab7a24-1dfb-4dd4-a1c7-e0a1c0f9209a-kube-api-access-dzgcp\" (UniqueName: \"kubernetes.io/projected/75ab7a24-1dfb-4dd4-a1c7-e0a1c0f9209a-kube-api-access-dzgcp\") pod \"75ab7a24-1dfb-4dd4-a1c7-e0a1c0f9209a\" (UID: \"75ab7a24-1dfb-4dd4-a1c7-e0a1c0f9209a\") "
Oct 30 19:45:56.545292 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:45:56.545276    4892 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/75ab7a24-1dfb-4dd4-a1c7-e0a1c0f9209a-maas-api-xks-ca\") on node \"ip-10-0-132-82.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 19:45:56.546902 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:45:56.546874    4892 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/75ab7a24-1dfb-4dd4-a1c7-e0a1c0f9209a-kube-api-access-dzgcp" pod "75ab7a24-1dfb-4dd4-a1c7-e0a1c0f9209a" (UID: "75ab7a24-1dfb-4dd4-a1c7-e0a1c0f9209a"). InnerVolumeSpecName "kube-api-access-dzgcp". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:45:56.646205 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:45:56.646181    4892 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dzgcp\" (UniqueName: \"kubernetes.io/projected/75ab7a24-1dfb-4dd4-a1c7-e0a1c0f9209a-kube-api-access-dzgcp\") on node \"ip-10-0-132-82.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 19:45:57.064345 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:45:57.064265    4892 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-affa4c07-25tmc"
Oct 30 19:45:57.064502 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:45:57.064261    4892 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-affa4c07-25tmc" event={"ID":"75ab7a24-1dfb-4dd4-a1c7-e0a1c0f9209a","Type":"ContainerDied","Data":"853c2726feecca75ad3a1d35b13a7386d171de88a9c44439cbc8b60143e552b0"}
Oct 30 19:45:57.064502 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:45:57.064390    4892 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="853c2726feecca75ad3a1d35b13a7386d171de88a9c44439cbc8b60143e552b0"
Oct 30 19:45:58.328302 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:45:58.328257    4892 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-migrate-d74c1bd9-7c87cbbf88-2j9tg" podUID="79ae5280-77c7-451b-bd82-ade64806cdd7" containerName="payload-processing" containerID="cri-o://af8b326ce902f6443927be0567200322cbe282bd21efd5d4aab337d071585235" gracePeriod=25
Oct 30 19:45:58.454712 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:45:58.454658    4892 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-migrate-d74c1bd9-5f777c6ff-qbwsw" podUID="28480114-29bf-4642-8da2-a7e714ba3460" containerName="payload-pre-processing" containerID="cri-o://c6a83984b85dacf4ffe0b4e353826977e934b3bd8a85c4cc66d48f2b58d42aeb" gracePeriod=25
Oct 30 19:45:58.661390 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:45:58.661081    4892 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-migrate-d74c1bd9-7c87cbbf88-2j9tg"
Oct 30 19:45:58.673014 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:45:58.672557    4892 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/79ae5280-77c7-451b-bd82-ade64806cdd7-kube-api-access-vj45m\" (UniqueName: \"kubernetes.io/projected/79ae5280-77c7-451b-bd82-ade64806cdd7-kube-api-access-vj45m\") pod \"79ae5280-77c7-451b-bd82-ade64806cdd7\" (UID: \"79ae5280-77c7-451b-bd82-ade64806cdd7\") "
Oct 30 19:45:58.673014 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:45:58.672711    4892 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/79ae5280-77c7-451b-bd82-ade64806cdd7-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/79ae5280-77c7-451b-bd82-ade64806cdd7-plugins-config-volume\") pod \"79ae5280-77c7-451b-bd82-ade64806cdd7\" (UID: \"79ae5280-77c7-451b-bd82-ade64806cdd7\") "
Oct 30 19:45:58.673996 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:45:58.673338    4892 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/79ae5280-77c7-451b-bd82-ade64806cdd7-plugins-config-volume" pod "79ae5280-77c7-451b-bd82-ade64806cdd7" (UID: "79ae5280-77c7-451b-bd82-ade64806cdd7"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 19:45:58.677738 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:45:58.677711    4892 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/79ae5280-77c7-451b-bd82-ade64806cdd7-kube-api-access-vj45m" pod "79ae5280-77c7-451b-bd82-ade64806cdd7" (UID: "79ae5280-77c7-451b-bd82-ade64806cdd7"). InnerVolumeSpecName "kube-api-access-vj45m". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:45:58.749144 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:45:58.749119    4892 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-migrate-d74c1bd9-5f777c6ff-qbwsw"
Oct 30 19:45:58.774462 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:45:58.774437    4892 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/28480114-29bf-4642-8da2-a7e714ba3460-kube-api-access-jncj8\" (UniqueName: \"kubernetes.io/projected/28480114-29bf-4642-8da2-a7e714ba3460-kube-api-access-jncj8\") pod \"28480114-29bf-4642-8da2-a7e714ba3460\" (UID: \"28480114-29bf-4642-8da2-a7e714ba3460\") "
Oct 30 19:45:58.774701 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:45:58.774688    4892 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/28480114-29bf-4642-8da2-a7e714ba3460-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/28480114-29bf-4642-8da2-a7e714ba3460-plugins-config-volume\") pod \"28480114-29bf-4642-8da2-a7e714ba3460\" (UID: \"28480114-29bf-4642-8da2-a7e714ba3460\") "
Oct 30 19:45:58.775192 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:45:58.775120    4892 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/28480114-29bf-4642-8da2-a7e714ba3460-plugins-config-volume" pod "28480114-29bf-4642-8da2-a7e714ba3460" (UID: "28480114-29bf-4642-8da2-a7e714ba3460"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 19:45:58.775192 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:45:58.775213    4892 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vj45m\" (UniqueName: \"kubernetes.io/projected/79ae5280-77c7-451b-bd82-ade64806cdd7-kube-api-access-vj45m\") on node \"ip-10-0-132-82.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 19:45:58.775192 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:45:58.775231    4892 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/28480114-29bf-4642-8da2-a7e714ba3460-plugins-config-volume\") on node \"ip-10-0-132-82.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 19:45:58.775192 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:45:58.775243    4892 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/79ae5280-77c7-451b-bd82-ade64806cdd7-plugins-config-volume\") on node \"ip-10-0-132-82.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 19:45:58.777019 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:45:58.776956    4892 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/28480114-29bf-4642-8da2-a7e714ba3460-kube-api-access-jncj8" pod "28480114-29bf-4642-8da2-a7e714ba3460" (UID: "28480114-29bf-4642-8da2-a7e714ba3460"). InnerVolumeSpecName "kube-api-access-jncj8". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:45:58.876682 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:45:58.876600    4892 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jncj8\" (UniqueName: \"kubernetes.io/projected/28480114-29bf-4642-8da2-a7e714ba3460-kube-api-access-jncj8\") on node \"ip-10-0-132-82.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 19:45:59.086028 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:45:59.085989    4892 generic.go:356] "Generic (PLEG): container finished" podID="28480114-29bf-4642-8da2-a7e714ba3460" containerID="c6a83984b85dacf4ffe0b4e353826977e934b3bd8a85c4cc66d48f2b58d42aeb" exitCode=0
Oct 30 19:45:59.086207 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:45:59.086094    4892 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-migrate-d74c1bd9-5f777c6ff-qbwsw"
Oct 30 19:45:59.086207 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:45:59.086111    4892 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-migrate-d74c1bd9-5f777c6ff-qbwsw" event={"ID":"28480114-29bf-4642-8da2-a7e714ba3460","Type":"ContainerDied","Data":"c6a83984b85dacf4ffe0b4e353826977e934b3bd8a85c4cc66d48f2b58d42aeb"}
Oct 30 19:45:59.086207 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:45:59.086162    4892 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-migrate-d74c1bd9-5f777c6ff-qbwsw" event={"ID":"28480114-29bf-4642-8da2-a7e714ba3460","Type":"ContainerDied","Data":"a3acfc7ccdc5bac38140ceeda6c8549cf3efb74ef09bede38d8a76e191e5fec7"}
Oct 30 19:45:59.086207 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:45:59.086194    4892 scope.go:122] "RemoveContainer" containerID="c6a83984b85dacf4ffe0b4e353826977e934b3bd8a85c4cc66d48f2b58d42aeb"
Oct 30 19:45:59.092930 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:45:59.092311    4892 generic.go:356] "Generic (PLEG): container finished" podID="79ae5280-77c7-451b-bd82-ade64806cdd7" containerID="af8b326ce902f6443927be0567200322cbe282bd21efd5d4aab337d071585235" exitCode=0
Oct 30 19:45:59.092930 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:45:59.092401    4892 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-migrate-d74c1bd9-7c87cbbf88-2j9tg"
Oct 30 19:45:59.093535 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:45:59.092431    4892 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-migrate-d74c1bd9-7c87cbbf88-2j9tg" event={"ID":"79ae5280-77c7-451b-bd82-ade64806cdd7","Type":"ContainerDied","Data":"af8b326ce902f6443927be0567200322cbe282bd21efd5d4aab337d071585235"}
Oct 30 19:45:59.093629 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:45:59.093569    4892 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-migrate-d74c1bd9-7c87cbbf88-2j9tg" event={"ID":"79ae5280-77c7-451b-bd82-ade64806cdd7","Type":"ContainerDied","Data":"308ff9019c9f212fb1f96eb28ac25a246d143c053660d531cc08bbc739fb0c32"}
Oct 30 19:45:59.102044 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:45:59.102025    4892 scope.go:122] "RemoveContainer" containerID="c6a83984b85dacf4ffe0b4e353826977e934b3bd8a85c4cc66d48f2b58d42aeb"
Oct 30 19:45:59.102327 ip-10-0-132-82 kubenswrapper[4892]: E1030 19:45:59.102303    4892 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c6a83984b85dacf4ffe0b4e353826977e934b3bd8a85c4cc66d48f2b58d42aeb\": container with ID starting with c6a83984b85dacf4ffe0b4e353826977e934b3bd8a85c4cc66d48f2b58d42aeb not found: ID does not exist" containerID="c6a83984b85dacf4ffe0b4e353826977e934b3bd8a85c4cc66d48f2b58d42aeb"
Oct 30 19:45:59.102432 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:45:59.102338    4892 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c6a83984b85dacf4ffe0b4e353826977e934b3bd8a85c4cc66d48f2b58d42aeb"} err="failed to get container status \"c6a83984b85dacf4ffe0b4e353826977e934b3bd8a85c4cc66d48f2b58d42aeb\": rpc error: code = NotFound desc = could not find container \"c6a83984b85dacf4ffe0b4e353826977e934b3bd8a85c4cc66d48f2b58d42aeb\": container with ID starting with c6a83984b85dacf4ffe0b4e353826977e934b3bd8a85c4cc66d48f2b58d42aeb not found: ID does not exist"
Oct 30 19:45:59.102432 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:45:59.102362    4892 scope.go:122] "RemoveContainer" containerID="af8b326ce902f6443927be0567200322cbe282bd21efd5d4aab337d071585235"
Oct 30 19:45:59.112552 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:45:59.112536    4892 scope.go:122] "RemoveContainer" containerID="af8b326ce902f6443927be0567200322cbe282bd21efd5d4aab337d071585235"
Oct 30 19:45:59.112796 ip-10-0-132-82 kubenswrapper[4892]: E1030 19:45:59.112772    4892 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"af8b326ce902f6443927be0567200322cbe282bd21efd5d4aab337d071585235\": container with ID starting with af8b326ce902f6443927be0567200322cbe282bd21efd5d4aab337d071585235 not found: ID does not exist" containerID="af8b326ce902f6443927be0567200322cbe282bd21efd5d4aab337d071585235"
Oct 30 19:45:59.112845 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:45:59.112804    4892 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"af8b326ce902f6443927be0567200322cbe282bd21efd5d4aab337d071585235"} err="failed to get container status \"af8b326ce902f6443927be0567200322cbe282bd21efd5d4aab337d071585235\": rpc error: code = NotFound desc = could not find container \"af8b326ce902f6443927be0567200322cbe282bd21efd5d4aab337d071585235\": container with ID starting with af8b326ce902f6443927be0567200322cbe282bd21efd5d4aab337d071585235 not found: ID does not exist"
Oct 30 19:45:59.252236 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:45:59.252203    4892 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-migrate-d74c1bd9-5f777c6ff-qbwsw"]
Oct 30 19:45:59.257255 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:45:59.257234    4892 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-migrate-d74c1bd9-5f777c6ff-qbwsw"]
Oct 30 19:45:59.369809 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:45:59.369776    4892 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-migrate-d74c1bd9-7c87cbbf88-2j9tg"]
Oct 30 19:45:59.377134 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:45:59.377112    4892 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-migrate-d74c1bd9-7c87cbbf88-2j9tg"]
Oct 30 19:46:00.573575 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:46:00.573538    4892 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="28480114-29bf-4642-8da2-a7e714ba3460" path="/var/lib/kubelet/pods/28480114-29bf-4642-8da2-a7e714ba3460/volumes"
Oct 30 19:46:00.573988 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:46:00.573958    4892 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="79ae5280-77c7-451b-bd82-ade64806cdd7" path="/var/lib/kubelet/pods/79ae5280-77c7-451b-bd82-ade64806cdd7/volumes"
Oct 30 19:46:02.080525 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:46:02.080491    4892 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-9d428be1-54gl7"]
Oct 30 19:46:02.129124 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:46:02.129091    4892 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-9d428be1-54gl7"]
Oct 30 19:46:02.572114 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:46:02.572082    4892 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b225bbb1-5af0-4000-91b6-687f9f5f69f4" path="/var/lib/kubelet/pods/b225bbb1-5af0-4000-91b6-687f9f5f69f4/volumes"
Oct 30 19:46:05.382269 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:46:05.382233    4892 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-fsfvk"]
Oct 30 19:46:11.072354 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:46:11.072321    4892 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ad0fb289-f9ffv"]
Oct 30 19:46:11.117451 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:46:11.117425    4892 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ad0fb289-f9ffv"]
Oct 30 19:46:12.573043 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:46:12.573013    4892 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="5702f32d-9d4a-4176-86f7-fd358beb49bd" path="/var/lib/kubelet/pods/5702f32d-9d4a-4176-86f7-fd358beb49bd/volumes"
Oct 30 19:46:15.061525 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:46:15.061468    4892 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c6271da6-fkqzd"]
Oct 30 19:46:15.104535 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:46:15.104503    4892 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c6271da6-fkqzd"]
Oct 30 19:46:16.572582 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:46:16.572538    4892 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="04f8161f-571c-4656-b577-0a30c8a52252" path="/var/lib/kubelet/pods/04f8161f-571c-4656-b577-0a30c8a52252/volumes"
Oct 30 19:46:20.059558 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:46:20.059521    4892 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5bcd350f-qg8dm"]
Oct 30 19:46:20.108595 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:46:20.108558    4892 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5bcd350f-qg8dm"]
Oct 30 19:46:20.571148 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:46:20.571109    4892 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ba863759-2880-439a-baf3-e4a9ef0bdda0" path="/var/lib/kubelet/pods/ba863759-2880-439a-baf3-e4a9ef0bdda0/volumes"
Oct 30 19:46:24.845141 ip-10-0-132-82 kubenswrapper[4892]: E1030 19:46:24.845104    4892 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-jrm87" containerName="extractor"
Oct 30 19:46:25.689401 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:46:25.689353    4892 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-94581ab2-qvqdv"]
Oct 30 19:46:25.739675 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:46:25.739646    4892 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-94581ab2-qvqdv"]
Oct 30 19:46:26.571656 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:46:26.571622    4892 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="5e40fdcc-dac3-46d6-a965-f86b809e11c8" path="/var/lib/kubelet/pods/5e40fdcc-dac3-46d6-a965-f86b809e11c8/volumes"
Oct 30 19:46:27.033583 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:46:27.033547    4892 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-derive-8a3c7851-openshift-default-7845b86bf7-xx8vl"]
Oct 30 19:46:27.034408 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:46:27.034379    4892 cpu_manager.go:414] "containerMap: removing container" podUID="75ab7a24-1dfb-4dd4-a1c7-e0a1c0f9209a" containerName="revoke-keys"
Oct 30 19:46:27.034537 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:46:27.034410    4892 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="75ab7a24-1dfb-4dd4-a1c7-e0a1c0f9209a" containerName="revoke-keys"
Oct 30 19:46:27.034537 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:46:27.034433    4892 cpu_manager.go:414] "containerMap: removing container" podUID="28480114-29bf-4642-8da2-a7e714ba3460" containerName="payload-pre-processing"
Oct 30 19:46:27.034537 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:46:27.034443    4892 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="28480114-29bf-4642-8da2-a7e714ba3460" containerName="payload-pre-processing"
Oct 30 19:46:27.034537 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:46:27.034491    4892 cpu_manager.go:414] "containerMap: removing container" podUID="79ae5280-77c7-451b-bd82-ade64806cdd7" containerName="payload-processing"
Oct 30 19:46:27.034537 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:46:27.034498    4892 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="79ae5280-77c7-451b-bd82-ade64806cdd7" containerName="payload-processing"
Oct 30 19:46:27.034802 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:46:27.034675    4892 memory_manager.go:365] "RemoveStaleState removing state" podUID="79ae5280-77c7-451b-bd82-ade64806cdd7" containerName="payload-processing"
Oct 30 19:46:27.034802 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:46:27.034701    4892 memory_manager.go:365] "RemoveStaleState removing state" podUID="75ab7a24-1dfb-4dd4-a1c7-e0a1c0f9209a" containerName="revoke-keys"
Oct 30 19:46:27.034802 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:46:27.034712    4892 memory_manager.go:365] "RemoveStaleState removing state" podUID="28480114-29bf-4642-8da2-a7e714ba3460" containerName="payload-pre-processing"
Oct 30 19:46:27.041517 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:46:27.041495    4892 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-derive-8a3c7851-openshift-default-7845b86bf7-xx8vl"
Oct 30 19:46:27.175988 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:46:27.170541    4892 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-derive-8a3c7851-openshift-default-dockercfg-lcmgz\""
Oct 30 19:46:27.175988 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:46:27.172513    4892 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-h5gnr\" (UniqueName: \"kubernetes.io/projected/9c98da0e-489b-4333-b450-704beaf2f325-kube-api-access-h5gnr\") pod \"e2e-derive-8a3c7851-openshift-default-7845b86bf7-xx8vl\" (UID: \"9c98da0e-489b-4333-b450-704beaf2f325\") " pod="openshift-ingress/e2e-derive-8a3c7851-openshift-default-7845b86bf7-xx8vl"
Oct 30 19:46:27.175988 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:46:27.172558    4892 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9c98da0e-489b-4333-b450-704beaf2f325-workload-certs\") pod \"e2e-derive-8a3c7851-openshift-default-7845b86bf7-xx8vl\" (UID: \"9c98da0e-489b-4333-b450-704beaf2f325\") " pod="openshift-ingress/e2e-derive-8a3c7851-openshift-default-7845b86bf7-xx8vl"
Oct 30 19:46:27.175988 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:46:27.172640    4892 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9c98da0e-489b-4333-b450-704beaf2f325-workload-socket\") pod \"e2e-derive-8a3c7851-openshift-default-7845b86bf7-xx8vl\" (UID: \"9c98da0e-489b-4333-b450-704beaf2f325\") " pod="openshift-ingress/e2e-derive-8a3c7851-openshift-default-7845b86bf7-xx8vl"
Oct 30 19:46:27.175988 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:46:27.172682    4892 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9c98da0e-489b-4333-b450-704beaf2f325-istio-envoy\") pod \"e2e-derive-8a3c7851-openshift-default-7845b86bf7-xx8vl\" (UID: \"9c98da0e-489b-4333-b450-704beaf2f325\") " pod="openshift-ingress/e2e-derive-8a3c7851-openshift-default-7845b86bf7-xx8vl"
Oct 30 19:46:27.175988 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:46:27.172713    4892 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9c98da0e-489b-4333-b450-704beaf2f325-istio-podinfo\") pod \"e2e-derive-8a3c7851-openshift-default-7845b86bf7-xx8vl\" (UID: \"9c98da0e-489b-4333-b450-704beaf2f325\") " pod="openshift-ingress/e2e-derive-8a3c7851-openshift-default-7845b86bf7-xx8vl"
Oct 30 19:46:27.175988 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:46:27.172816    4892 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9c98da0e-489b-4333-b450-704beaf2f325-istio-token\") pod \"e2e-derive-8a3c7851-openshift-default-7845b86bf7-xx8vl\" (UID: \"9c98da0e-489b-4333-b450-704beaf2f325\") " pod="openshift-ingress/e2e-derive-8a3c7851-openshift-default-7845b86bf7-xx8vl"
Oct 30 19:46:27.175988 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:46:27.172882    4892 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9c98da0e-489b-4333-b450-704beaf2f325-istio-data\") pod \"e2e-derive-8a3c7851-openshift-default-7845b86bf7-xx8vl\" (UID: \"9c98da0e-489b-4333-b450-704beaf2f325\") " pod="openshift-ingress/e2e-derive-8a3c7851-openshift-default-7845b86bf7-xx8vl"
Oct 30 19:46:27.175988 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:46:27.172915    4892 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9c98da0e-489b-4333-b450-704beaf2f325-istiod-ca-cert\") pod \"e2e-derive-8a3c7851-openshift-default-7845b86bf7-xx8vl\" (UID: \"9c98da0e-489b-4333-b450-704beaf2f325\") " pod="openshift-ingress/e2e-derive-8a3c7851-openshift-default-7845b86bf7-xx8vl"
Oct 30 19:46:27.175988 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:46:27.172996    4892 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9c98da0e-489b-4333-b450-704beaf2f325-credential-socket\") pod \"e2e-derive-8a3c7851-openshift-default-7845b86bf7-xx8vl\" (UID: \"9c98da0e-489b-4333-b450-704beaf2f325\") " pod="openshift-ingress/e2e-derive-8a3c7851-openshift-default-7845b86bf7-xx8vl"
Oct 30 19:46:27.232626 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:46:27.220878    4892 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-derive-8a3c7851-openshift-default-7845b86bf7-xx8vl"]
Oct 30 19:46:27.344498 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:46:27.314759    4892 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9c98da0e-489b-4333-b450-704beaf2f325-istio-token\") pod \"e2e-derive-8a3c7851-openshift-default-7845b86bf7-xx8vl\" (UID: \"9c98da0e-489b-4333-b450-704beaf2f325\") " pod="openshift-ingress/e2e-derive-8a3c7851-openshift-default-7845b86bf7-xx8vl"
Oct 30 19:46:27.344498 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:46:27.314844    4892 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9c98da0e-489b-4333-b450-704beaf2f325-istio-data\") pod \"e2e-derive-8a3c7851-openshift-default-7845b86bf7-xx8vl\" (UID: \"9c98da0e-489b-4333-b450-704beaf2f325\") " pod="openshift-ingress/e2e-derive-8a3c7851-openshift-default-7845b86bf7-xx8vl"
Oct 30 19:46:27.344498 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:46:27.314877    4892 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9c98da0e-489b-4333-b450-704beaf2f325-istiod-ca-cert\") pod \"e2e-derive-8a3c7851-openshift-default-7845b86bf7-xx8vl\" (UID: \"9c98da0e-489b-4333-b450-704beaf2f325\") " pod="openshift-ingress/e2e-derive-8a3c7851-openshift-default-7845b86bf7-xx8vl"
Oct 30 19:46:27.344498 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:46:27.314934    4892 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9c98da0e-489b-4333-b450-704beaf2f325-credential-socket\") pod \"e2e-derive-8a3c7851-openshift-default-7845b86bf7-xx8vl\" (UID: \"9c98da0e-489b-4333-b450-704beaf2f325\") " pod="openshift-ingress/e2e-derive-8a3c7851-openshift-default-7845b86bf7-xx8vl"
Oct 30 19:46:27.344498 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:46:27.315011    4892 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-h5gnr\" (UniqueName: \"kubernetes.io/projected/9c98da0e-489b-4333-b450-704beaf2f325-kube-api-access-h5gnr\") pod \"e2e-derive-8a3c7851-openshift-default-7845b86bf7-xx8vl\" (UID: \"9c98da0e-489b-4333-b450-704beaf2f325\") " pod="openshift-ingress/e2e-derive-8a3c7851-openshift-default-7845b86bf7-xx8vl"
Oct 30 19:46:27.344498 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:46:27.315044    4892 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9c98da0e-489b-4333-b450-704beaf2f325-workload-certs\") pod \"e2e-derive-8a3c7851-openshift-default-7845b86bf7-xx8vl\" (UID: \"9c98da0e-489b-4333-b450-704beaf2f325\") " pod="openshift-ingress/e2e-derive-8a3c7851-openshift-default-7845b86bf7-xx8vl"
Oct 30 19:46:27.344498 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:46:27.315126    4892 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9c98da0e-489b-4333-b450-704beaf2f325-workload-socket\") pod \"e2e-derive-8a3c7851-openshift-default-7845b86bf7-xx8vl\" (UID: \"9c98da0e-489b-4333-b450-704beaf2f325\") " pod="openshift-ingress/e2e-derive-8a3c7851-openshift-default-7845b86bf7-xx8vl"
Oct 30 19:46:27.344498 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:46:27.315177    4892 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9c98da0e-489b-4333-b450-704beaf2f325-istio-envoy\") pod \"e2e-derive-8a3c7851-openshift-default-7845b86bf7-xx8vl\" (UID: \"9c98da0e-489b-4333-b450-704beaf2f325\") " pod="openshift-ingress/e2e-derive-8a3c7851-openshift-default-7845b86bf7-xx8vl"
Oct 30 19:46:27.344498 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:46:27.315206    4892 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9c98da0e-489b-4333-b450-704beaf2f325-istio-podinfo\") pod \"e2e-derive-8a3c7851-openshift-default-7845b86bf7-xx8vl\" (UID: \"9c98da0e-489b-4333-b450-704beaf2f325\") " pod="openshift-ingress/e2e-derive-8a3c7851-openshift-default-7845b86bf7-xx8vl"
Oct 30 19:46:27.355490 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:46:27.346685    4892 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9c98da0e-489b-4333-b450-704beaf2f325-credential-socket\") pod \"e2e-derive-8a3c7851-openshift-default-7845b86bf7-xx8vl\" (UID: \"9c98da0e-489b-4333-b450-704beaf2f325\") " pod="openshift-ingress/e2e-derive-8a3c7851-openshift-default-7845b86bf7-xx8vl"
Oct 30 19:46:27.355490 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:46:27.347387    4892 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9c98da0e-489b-4333-b450-704beaf2f325-istio-data\") pod \"e2e-derive-8a3c7851-openshift-default-7845b86bf7-xx8vl\" (UID: \"9c98da0e-489b-4333-b450-704beaf2f325\") " pod="openshift-ingress/e2e-derive-8a3c7851-openshift-default-7845b86bf7-xx8vl"
Oct 30 19:46:27.355490 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:46:27.348074    4892 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9c98da0e-489b-4333-b450-704beaf2f325-istiod-ca-cert\") pod \"e2e-derive-8a3c7851-openshift-default-7845b86bf7-xx8vl\" (UID: \"9c98da0e-489b-4333-b450-704beaf2f325\") " pod="openshift-ingress/e2e-derive-8a3c7851-openshift-default-7845b86bf7-xx8vl"
Oct 30 19:46:27.355490 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:46:27.348303    4892 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9c98da0e-489b-4333-b450-704beaf2f325-workload-certs\") pod \"e2e-derive-8a3c7851-openshift-default-7845b86bf7-xx8vl\" (UID: \"9c98da0e-489b-4333-b450-704beaf2f325\") " pod="openshift-ingress/e2e-derive-8a3c7851-openshift-default-7845b86bf7-xx8vl"
Oct 30 19:46:27.355490 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:46:27.348732    4892 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9c98da0e-489b-4333-b450-704beaf2f325-workload-socket\") pod \"e2e-derive-8a3c7851-openshift-default-7845b86bf7-xx8vl\" (UID: \"9c98da0e-489b-4333-b450-704beaf2f325\") " pod="openshift-ingress/e2e-derive-8a3c7851-openshift-default-7845b86bf7-xx8vl"
Oct 30 19:46:27.465865 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:46:27.465832    4892 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-h5gnr\" (UniqueName: \"kubernetes.io/projected/9c98da0e-489b-4333-b450-704beaf2f325-kube-api-access-h5gnr\") pod \"e2e-derive-8a3c7851-openshift-default-7845b86bf7-xx8vl\" (UID: \"9c98da0e-489b-4333-b450-704beaf2f325\") " pod="openshift-ingress/e2e-derive-8a3c7851-openshift-default-7845b86bf7-xx8vl"
Oct 30 19:46:27.472953 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:46:27.472884    4892 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9c98da0e-489b-4333-b450-704beaf2f325-istio-podinfo\") pod \"e2e-derive-8a3c7851-openshift-default-7845b86bf7-xx8vl\" (UID: \"9c98da0e-489b-4333-b450-704beaf2f325\") " pod="openshift-ingress/e2e-derive-8a3c7851-openshift-default-7845b86bf7-xx8vl"
Oct 30 19:46:27.481114 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:46:27.481081    4892 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9c98da0e-489b-4333-b450-704beaf2f325-istio-token\") pod \"e2e-derive-8a3c7851-openshift-default-7845b86bf7-xx8vl\" (UID: \"9c98da0e-489b-4333-b450-704beaf2f325\") " pod="openshift-ingress/e2e-derive-8a3c7851-openshift-default-7845b86bf7-xx8vl"
Oct 30 19:46:27.497988 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:46:27.486217    4892 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9c98da0e-489b-4333-b450-704beaf2f325-istio-envoy\") pod \"e2e-derive-8a3c7851-openshift-default-7845b86bf7-xx8vl\" (UID: \"9c98da0e-489b-4333-b450-704beaf2f325\") " pod="openshift-ingress/e2e-derive-8a3c7851-openshift-default-7845b86bf7-xx8vl"
Oct 30 19:46:27.699001 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:46:27.697371    4892 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-derive-8a3c7851-openshift-default-7845b86bf7-xx8vl"
Oct 30 19:46:28.180744 ip-10-0-132-82 kubenswrapper[4892]: W1030 19:46:28.180709    4892 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9c98da0e_489b_4333_b450_704beaf2f325.slice/crio-ca2c1dbf03b0c840fb882c9c8ef4807899d5e4d89d547e6029662cd2d9a57d7f WatchSource:0}: Error finding container ca2c1dbf03b0c840fb882c9c8ef4807899d5e4d89d547e6029662cd2d9a57d7f: Status 404 returned error can't find the container with id ca2c1dbf03b0c840fb882c9c8ef4807899d5e4d89d547e6029662cd2d9a57d7f
Oct 30 19:46:28.182125 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:46:28.182099    4892 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-derive-8a3c7851-openshift-default-7845b86bf7-xx8vl"]
Oct 30 19:46:28.183681 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:46:28.183647    4892 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892000Ki","pods":"250"}
Oct 30 19:46:28.183809 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:46:28.183710    4892 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892000Ki","pods":"250"}
Oct 30 19:46:28.183809 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:46:28.183738    4892 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892000Ki","pods":"250"}
Oct 30 19:46:28.493919 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:46:28.493820    4892 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-derive-8a3c7851-openshift-default-7845b86bf7-xx8vl" event={"ID":"9c98da0e-489b-4333-b450-704beaf2f325","Type":"ContainerStarted","Data":"9b069f96d6daf9fc90a38ed7626db9a2a20aa656f12734127a85f4d468ed67cb"}
Oct 30 19:46:28.493919 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:46:28.493885    4892 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-derive-8a3c7851-openshift-default-7845b86bf7-xx8vl" event={"ID":"9c98da0e-489b-4333-b450-704beaf2f325","Type":"ContainerStarted","Data":"ca2c1dbf03b0c840fb882c9c8ef4807899d5e4d89d547e6029662cd2d9a57d7f"}
Oct 30 19:46:28.659727 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:46:28.659670    4892 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-derive-8a3c7851-openshift-default-7845b86bf7-xx8vl" podStartSLOduration=2.659651159 podStartE2EDuration="2.659651159s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:46:26 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:46:28.654179077 +0000 UTC m=+1836.224644676" watchObservedRunningTime="2026-10-30 19:46:28.659651159 +0000 UTC m=+1836.230116757"
Oct 30 19:46:28.708687 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:46:28.708658    4892 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-derive-8a3c7851-openshift-default-7845b86bf7-xx8vl"
Oct 30 19:46:29.055770 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:46:29.055740    4892 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e4655384-zbxsh"]
Oct 30 19:46:29.069200 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:46:29.069162    4892 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e4655384-zbxsh"]
Oct 30 19:46:29.713199 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:46:29.713171    4892 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-derive-8a3c7851-openshift-default-7845b86bf7-xx8vl"
Oct 30 19:46:30.560993 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:46:30.542220    4892 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-derive-8a3c7851-openshift-default-7845b86bf7-xx8vl"
Oct 30 19:46:30.560993 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:46:30.546305    4892 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-derive-8a3c7851-openshift-default-7845b86bf7-xx8vl"
Oct 30 19:46:30.577014 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:46:30.575189    4892 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="784612e5-1d50-4667-abdb-cf3bce669205" path="/var/lib/kubelet/pods/784612e5-1d50-4667-abdb-cf3bce669205/volumes"
Oct 30 19:46:32.104420 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:46:32.104342    4892 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-d74c1bd9-dc96t"]
Oct 30 19:46:32.164418 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:46:32.164386    4892 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-d74c1bd9-dc96t"]
Oct 30 19:46:32.586437 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:46:32.586404    4892 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0cba880a-5d89-4205-b60a-fd99cbadc021" path="/var/lib/kubelet/pods/0cba880a-5d89-4205-b60a-fd99cbadc021/volumes"
Oct 30 19:46:43.063144 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:46:43.063095    4892 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-83faf4fb-wr9j2"]
Oct 30 19:46:43.100083 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:46:43.100042    4892 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-83faf4fb-wr9j2"]
Oct 30 19:46:44.065638 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:46:44.065582    4892 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-68c9e294-bwx5n"]
Oct 30 19:46:44.100119 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:46:44.100085    4892 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-68c9e294-bwx5n"]
Oct 30 19:46:44.573695 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:46:44.573654    4892 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="402433ad-7d20-46f4-8ed6-c0071f0053ce" path="/var/lib/kubelet/pods/402433ad-7d20-46f4-8ed6-c0071f0053ce/volumes"
Oct 30 19:46:44.574170 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:46:44.574155    4892 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="8082afb5-3a19-4038-8987-fcadf0f420b2" path="/var/lib/kubelet/pods/8082afb5-3a19-4038-8987-fcadf0f420b2/volumes"
Oct 30 19:46:45.003685 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:46:45.003648    4892 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-fsfvk"]
Oct 30 19:46:51.717162 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:46:51.717121    4892 patch_prober.go:29] interesting pod/e2e-shared-a-w3-cf9b6f-openshift-default-56db6769f7-xl4xv container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.144:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 30 19:46:51.717653 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:46:51.717199    4892 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-shared-a-w3-cf9b6f-openshift-default-56db6769f7-xl4xv" podUID="fd212b75-a818-45a3-946a-864a230b7707" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.144:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:46:53.670058 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:46:53.670023    4892 scope.go:122] "RemoveContainer" containerID="faf912cff9432e5db97615f7ffa6751fe6473e86d1274ddb4bbccf55c8a8b3b0"
Oct 30 19:46:53.685489 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:46:53.685471    4892 scope.go:122] "RemoveContainer" containerID="d0fd81f5a45ed327e9929fc61728405a8f866e0f8524dc02ef54489ec55c3958"
Oct 30 19:46:53.695295 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:46:53.695277    4892 scope.go:122] "RemoveContainer" containerID="b85fea48a051823d697f6c6ce1cf4b097cc971f1062d085c42dedb359afebfb3"
Oct 30 19:46:53.708554 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:46:53.708531    4892 scope.go:122] "RemoveContainer" containerID="b54f48d792fedb3c7c4ae8c1ad5c8155dd52757a3ec0d2bc22a8db9e3739ff07"
Oct 30 19:46:53.718446 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:46:53.718420    4892 scope.go:122] "RemoveContainer" containerID="f9c43cddc7ff1f77c9beb4e96f00348f5744821ba2e165f98d4dac055b950571"
Oct 30 19:46:53.755579 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:46:53.755547    4892 generic.go:356] "Generic (PLEG): container finished" podID="fd212b75-a818-45a3-946a-864a230b7707" containerID="658c14d7f26cca7441ed312ad29eb8f102c041b9138faff109a3707e3f723c75" exitCode=0
Oct 30 19:46:53.755684 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:46:53.755581    4892 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w3-cf9b6f-openshift-default-56db6769f7-xl4xv" event={"ID":"fd212b75-a818-45a3-946a-864a230b7707","Type":"ContainerDied","Data":"658c14d7f26cca7441ed312ad29eb8f102c041b9138faff109a3707e3f723c75"}
Oct 30 19:46:53.756066 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:46:53.756047    4892 scope.go:122] "RemoveContainer" containerID="658c14d7f26cca7441ed312ad29eb8f102c041b9138faff109a3707e3f723c75"
Oct 30 19:46:53.757719 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:46:53.757691    4892 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892000Ki","pods":"250"}
Oct 30 19:46:53.757779 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:46:53.757761    4892 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892000Ki","pods":"250"}
Oct 30 19:46:53.757814 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:46:53.757793    4892 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892000Ki","pods":"250"}
Oct 30 19:46:54.771443 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:46:54.771404    4892 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w3-cf9b6f-openshift-default-56db6769f7-xl4xv" event={"ID":"fd212b75-a818-45a3-946a-864a230b7707","Type":"ContainerStarted","Data":"a54da5a3cea214225b82ca4f392b05e4f74757f52c2e6374de65120122176f86"}
Oct 30 19:46:54.874085 ip-10-0-132-82 kubenswrapper[4892]: E1030 19:46:54.874050    4892 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-jrm87" containerName="extractor"
Oct 30 19:46:55.064397 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:46:55.064293    4892 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b6c91eb9-rp7vq"]
Oct 30 19:46:55.069168 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:46:55.069137    4892 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b6c91eb9-rp7vq"]
Oct 30 19:46:55.387118 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:46:55.387086    4892 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-a-w3-cf9b6f-openshift-default-56db6769f7-xl4xv"
Oct 30 19:46:55.387311 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:46:55.387228    4892 patch_prober.go:29] interesting pod/e2e-shared-a-w3-cf9b6f-openshift-default-56db6769f7-xl4xv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.144:15021/healthz/ready\": dial tcp 10.132.0.144:15021: connect: connection refused" start-of-body=
Oct 30 19:46:55.387311 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:46:55.387283    4892 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w3-cf9b6f-openshift-default-56db6769f7-xl4xv" podUID="fd212b75-a818-45a3-946a-864a230b7707" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.144:15021/healthz/ready\": dial tcp 10.132.0.144:15021: connect: connection refused"
Oct 30 19:46:56.388022 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:46:56.387988    4892 patch_prober.go:29] interesting pod/e2e-shared-a-w3-cf9b6f-openshift-default-56db6769f7-xl4xv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.144:15021/healthz/ready\": dial tcp 10.132.0.144:15021: connect: connection refused" start-of-body=
Oct 30 19:46:56.388445 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:46:56.388041    4892 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w3-cf9b6f-openshift-default-56db6769f7-xl4xv" podUID="fd212b75-a818-45a3-946a-864a230b7707" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.144:15021/healthz/ready\": dial tcp 10.132.0.144:15021: connect: connection refused"
Oct 30 19:46:56.572611 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:46:56.572579    4892 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="01f7d0ee-1cb4-4e90-8f2c-846b45c3ed68" path="/var/lib/kubelet/pods/01f7d0ee-1cb4-4e90-8f2c-846b45c3ed68/volumes"
Oct 30 19:46:57.061138 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:46:57.061092    4892 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-75b6bc4a-ffcq9"]
Oct 30 19:46:57.106942 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:46:57.106913    4892 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-75b6bc4a-ffcq9"]
Oct 30 19:46:57.387723 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:46:57.387695    4892 patch_prober.go:29] interesting pod/e2e-shared-a-w3-cf9b6f-openshift-default-56db6769f7-xl4xv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.144:15021/healthz/ready\": dial tcp 10.132.0.144:15021: connect: connection refused" start-of-body=
Oct 30 19:46:57.387889 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:46:57.387749    4892 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w3-cf9b6f-openshift-default-56db6769f7-xl4xv" podUID="fd212b75-a818-45a3-946a-864a230b7707" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.144:15021/healthz/ready\": dial tcp 10.132.0.144:15021: connect: connection refused"
Oct 30 19:46:58.386950 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:46:58.386912    4892 patch_prober.go:29] interesting pod/e2e-shared-a-w3-cf9b6f-openshift-default-56db6769f7-xl4xv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.144:15021/healthz/ready\": dial tcp 10.132.0.144:15021: connect: connection refused" start-of-body=
Oct 30 19:46:58.387372 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:46:58.386986    4892 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w3-cf9b6f-openshift-default-56db6769f7-xl4xv" podUID="fd212b75-a818-45a3-946a-864a230b7707" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.144:15021/healthz/ready\": dial tcp 10.132.0.144:15021: connect: connection refused"
Oct 30 19:46:58.573208 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:46:58.573174    4892 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="91c01247-b2cd-497d-835b-fdd57fdf08d0" path="/var/lib/kubelet/pods/91c01247-b2cd-497d-835b-fdd57fdf08d0/volumes"
Oct 30 19:46:59.387693 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:46:59.387660    4892 patch_prober.go:29] interesting pod/e2e-shared-a-w3-cf9b6f-openshift-default-56db6769f7-xl4xv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.144:15021/healthz/ready\": dial tcp 10.132.0.144:15021: connect: connection refused" start-of-body=
Oct 30 19:46:59.388119 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:46:59.387739    4892 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w3-cf9b6f-openshift-default-56db6769f7-xl4xv" podUID="fd212b75-a818-45a3-946a-864a230b7707" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.144:15021/healthz/ready\": dial tcp 10.132.0.144:15021: connect: connection refused"
Oct 30 19:47:00.055712 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:47:00.055672    4892 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e52034d3-rjjj5"]
Oct 30 19:47:00.101667 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:47:00.101626    4892 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e52034d3-rjjj5"]
Oct 30 19:47:00.391849 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:47:00.391818    4892 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-a-w3-cf9b6f-openshift-default-56db6769f7-xl4xv"
Oct 30 19:47:00.392303 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:47:00.392288    4892 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-a-w3-cf9b6f-openshift-default-56db6769f7-xl4xv"
Oct 30 19:47:00.392879 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:47:00.392859    4892 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-a-w3-cf9b6f-openshift-default-56db6769f7-xl4xv"
Oct 30 19:47:00.570254 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:47:00.570220    4892 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="8a1b4524-f816-4eab-b88d-599a7689f138" path="/var/lib/kubelet/pods/8a1b4524-f816-4eab-b88d-599a7689f138/volumes"
Oct 30 19:47:03.828981 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:47:03.828943    4892 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-2pqkg"]
Oct 30 19:47:03.835409 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:47:03.835387    4892 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-2pqkg"
Oct 30 19:47:03.878301 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:47:03.878273    4892 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-sj8lf"]
Oct 30 19:47:03.882814 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:47:03.882797    4892 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-sj8lf"
Oct 30 19:47:03.935954 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:47:03.935926    4892 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-2pqkg"]
Oct 30 19:47:03.953521 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:47:03.953486    4892 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/efd3e1ce-d3c6-419d-9b5f-dee2907b2cb6-istio-data\") pod \"maas-default-gateway-openshift-default-6db586dd6c-2pqkg\" (UID: \"efd3e1ce-d3c6-419d-9b5f-dee2907b2cb6\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-2pqkg"
Oct 30 19:47:03.953656 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:47:03.953534    4892 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/efd3e1ce-d3c6-419d-9b5f-dee2907b2cb6-credential-socket\") pod \"maas-default-gateway-openshift-default-6db586dd6c-2pqkg\" (UID: \"efd3e1ce-d3c6-419d-9b5f-dee2907b2cb6\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-2pqkg"
Oct 30 19:47:03.953656 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:47:03.953627    4892 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/efd3e1ce-d3c6-419d-9b5f-dee2907b2cb6-istio-podinfo\") pod \"maas-default-gateway-openshift-default-6db586dd6c-2pqkg\" (UID: \"efd3e1ce-d3c6-419d-9b5f-dee2907b2cb6\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-2pqkg"
Oct 30 19:47:03.953796 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:47:03.953653    4892 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/aaea6d2c-ad01-47ba-9aa4-8f90ffff6901-workload-socket\") pod \"maas-default-gateway-openshift-default-6db586dd6c-sj8lf\" (UID: \"aaea6d2c-ad01-47ba-9aa4-8f90ffff6901\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-sj8lf"
Oct 30 19:47:03.953796 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:47:03.953693    4892 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6gbvt\" (UniqueName: \"kubernetes.io/projected/aaea6d2c-ad01-47ba-9aa4-8f90ffff6901-kube-api-access-6gbvt\") pod \"maas-default-gateway-openshift-default-6db586dd6c-sj8lf\" (UID: \"aaea6d2c-ad01-47ba-9aa4-8f90ffff6901\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-sj8lf"
Oct 30 19:47:03.953796 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:47:03.953747    4892 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/efd3e1ce-d3c6-419d-9b5f-dee2907b2cb6-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-6db586dd6c-2pqkg\" (UID: \"efd3e1ce-d3c6-419d-9b5f-dee2907b2cb6\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-2pqkg"
Oct 30 19:47:03.954005 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:47:03.953811    4892 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/aaea6d2c-ad01-47ba-9aa4-8f90ffff6901-istio-envoy\") pod \"maas-default-gateway-openshift-default-6db586dd6c-sj8lf\" (UID: \"aaea6d2c-ad01-47ba-9aa4-8f90ffff6901\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-sj8lf"
Oct 30 19:47:03.954005 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:47:03.953845    4892 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/efd3e1ce-d3c6-419d-9b5f-dee2907b2cb6-workload-socket\") pod \"maas-default-gateway-openshift-default-6db586dd6c-2pqkg\" (UID: \"efd3e1ce-d3c6-419d-9b5f-dee2907b2cb6\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-2pqkg"
Oct 30 19:47:03.954005 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:47:03.953884    4892 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-w95lm\" (UniqueName: \"kubernetes.io/projected/efd3e1ce-d3c6-419d-9b5f-dee2907b2cb6-kube-api-access-w95lm\") pod \"maas-default-gateway-openshift-default-6db586dd6c-2pqkg\" (UID: \"efd3e1ce-d3c6-419d-9b5f-dee2907b2cb6\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-2pqkg"
Oct 30 19:47:03.954173 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:47:03.954014    4892 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/aaea6d2c-ad01-47ba-9aa4-8f90ffff6901-istio-data\") pod \"maas-default-gateway-openshift-default-6db586dd6c-sj8lf\" (UID: \"aaea6d2c-ad01-47ba-9aa4-8f90ffff6901\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-sj8lf"
Oct 30 19:47:03.954173 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:47:03.954091    4892 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/efd3e1ce-d3c6-419d-9b5f-dee2907b2cb6-istio-envoy\") pod \"maas-default-gateway-openshift-default-6db586dd6c-2pqkg\" (UID: \"efd3e1ce-d3c6-419d-9b5f-dee2907b2cb6\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-2pqkg"
Oct 30 19:47:03.954173 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:47:03.954114    4892 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/aaea6d2c-ad01-47ba-9aa4-8f90ffff6901-workload-certs\") pod \"maas-default-gateway-openshift-default-6db586dd6c-sj8lf\" (UID: \"aaea6d2c-ad01-47ba-9aa4-8f90ffff6901\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-sj8lf"
Oct 30 19:47:03.954173 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:47:03.954148    4892 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/efd3e1ce-d3c6-419d-9b5f-dee2907b2cb6-workload-certs\") pod \"maas-default-gateway-openshift-default-6db586dd6c-2pqkg\" (UID: \"efd3e1ce-d3c6-419d-9b5f-dee2907b2cb6\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-2pqkg"
Oct 30 19:47:03.954386 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:47:03.954194    4892 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/aaea6d2c-ad01-47ba-9aa4-8f90ffff6901-istio-podinfo\") pod \"maas-default-gateway-openshift-default-6db586dd6c-sj8lf\" (UID: \"aaea6d2c-ad01-47ba-9aa4-8f90ffff6901\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-sj8lf"
Oct 30 19:47:03.954386 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:47:03.954236    4892 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/efd3e1ce-d3c6-419d-9b5f-dee2907b2cb6-istio-token\") pod \"maas-default-gateway-openshift-default-6db586dd6c-2pqkg\" (UID: \"efd3e1ce-d3c6-419d-9b5f-dee2907b2cb6\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-2pqkg"
Oct 30 19:47:03.954386 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:47:03.954267    4892 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/aaea6d2c-ad01-47ba-9aa4-8f90ffff6901-credential-socket\") pod \"maas-default-gateway-openshift-default-6db586dd6c-sj8lf\" (UID: \"aaea6d2c-ad01-47ba-9aa4-8f90ffff6901\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-sj8lf"
Oct 30 19:47:03.954386 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:47:03.954297    4892 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/aaea6d2c-ad01-47ba-9aa4-8f90ffff6901-istio-token\") pod \"maas-default-gateway-openshift-default-6db586dd6c-sj8lf\" (UID: \"aaea6d2c-ad01-47ba-9aa4-8f90ffff6901\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-sj8lf"
Oct 30 19:47:03.954386 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:47:03.954337    4892 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/aaea6d2c-ad01-47ba-9aa4-8f90ffff6901-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-6db586dd6c-sj8lf\" (UID: \"aaea6d2c-ad01-47ba-9aa4-8f90ffff6901\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-sj8lf"
Oct 30 19:47:03.995765 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:47:03.995724    4892 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-sj8lf"]
Oct 30 19:47:04.055623 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:47:04.055571    4892 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/aaea6d2c-ad01-47ba-9aa4-8f90ffff6901-istio-data\") pod \"maas-default-gateway-openshift-default-6db586dd6c-sj8lf\" (UID: \"aaea6d2c-ad01-47ba-9aa4-8f90ffff6901\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-sj8lf"
Oct 30 19:47:04.055828 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:47:04.055667    4892 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/efd3e1ce-d3c6-419d-9b5f-dee2907b2cb6-istio-envoy\") pod \"maas-default-gateway-openshift-default-6db586dd6c-2pqkg\" (UID: \"efd3e1ce-d3c6-419d-9b5f-dee2907b2cb6\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-2pqkg"
Oct 30 19:47:04.055828 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:47:04.055694    4892 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/aaea6d2c-ad01-47ba-9aa4-8f90ffff6901-workload-certs\") pod \"maas-default-gateway-openshift-default-6db586dd6c-sj8lf\" (UID: \"aaea6d2c-ad01-47ba-9aa4-8f90ffff6901\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-sj8lf"
Oct 30 19:47:04.055828 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:47:04.055726    4892 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/efd3e1ce-d3c6-419d-9b5f-dee2907b2cb6-workload-certs\") pod \"maas-default-gateway-openshift-default-6db586dd6c-2pqkg\" (UID: \"efd3e1ce-d3c6-419d-9b5f-dee2907b2cb6\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-2pqkg"
Oct 30 19:47:04.055828 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:47:04.055773    4892 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/aaea6d2c-ad01-47ba-9aa4-8f90ffff6901-istio-podinfo\") pod \"maas-default-gateway-openshift-default-6db586dd6c-sj8lf\" (UID: \"aaea6d2c-ad01-47ba-9aa4-8f90ffff6901\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-sj8lf"
Oct 30 19:47:04.055828 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:47:04.055820    4892 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/efd3e1ce-d3c6-419d-9b5f-dee2907b2cb6-istio-token\") pod \"maas-default-gateway-openshift-default-6db586dd6c-2pqkg\" (UID: \"efd3e1ce-d3c6-419d-9b5f-dee2907b2cb6\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-2pqkg"
Oct 30 19:47:04.056190 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:47:04.055851    4892 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/aaea6d2c-ad01-47ba-9aa4-8f90ffff6901-credential-socket\") pod \"maas-default-gateway-openshift-default-6db586dd6c-sj8lf\" (UID: \"aaea6d2c-ad01-47ba-9aa4-8f90ffff6901\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-sj8lf"
Oct 30 19:47:04.056190 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:47:04.055884    4892 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/aaea6d2c-ad01-47ba-9aa4-8f90ffff6901-istio-token\") pod \"maas-default-gateway-openshift-default-6db586dd6c-sj8lf\" (UID: \"aaea6d2c-ad01-47ba-9aa4-8f90ffff6901\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-sj8lf"
Oct 30 19:47:04.056190 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:47:04.055927    4892 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/aaea6d2c-ad01-47ba-9aa4-8f90ffff6901-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-6db586dd6c-sj8lf\" (UID: \"aaea6d2c-ad01-47ba-9aa4-8f90ffff6901\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-sj8lf"
Oct 30 19:47:04.056190 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:47:04.055952    4892 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/efd3e1ce-d3c6-419d-9b5f-dee2907b2cb6-istio-data\") pod \"maas-default-gateway-openshift-default-6db586dd6c-2pqkg\" (UID: \"efd3e1ce-d3c6-419d-9b5f-dee2907b2cb6\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-2pqkg"
Oct 30 19:47:04.056190 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:47:04.055987    4892 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/efd3e1ce-d3c6-419d-9b5f-dee2907b2cb6-credential-socket\") pod \"maas-default-gateway-openshift-default-6db586dd6c-2pqkg\" (UID: \"efd3e1ce-d3c6-419d-9b5f-dee2907b2cb6\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-2pqkg"
Oct 30 19:47:04.056190 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:47:04.056052    4892 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/efd3e1ce-d3c6-419d-9b5f-dee2907b2cb6-istio-podinfo\") pod \"maas-default-gateway-openshift-default-6db586dd6c-2pqkg\" (UID: \"efd3e1ce-d3c6-419d-9b5f-dee2907b2cb6\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-2pqkg"
Oct 30 19:47:04.056190 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:47:04.056068    4892 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/aaea6d2c-ad01-47ba-9aa4-8f90ffff6901-workload-socket\") pod \"maas-default-gateway-openshift-default-6db586dd6c-sj8lf\" (UID: \"aaea6d2c-ad01-47ba-9aa4-8f90ffff6901\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-sj8lf"
Oct 30 19:47:04.056190 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:47:04.056094    4892 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6gbvt\" (UniqueName: \"kubernetes.io/projected/aaea6d2c-ad01-47ba-9aa4-8f90ffff6901-kube-api-access-6gbvt\") pod \"maas-default-gateway-openshift-default-6db586dd6c-sj8lf\" (UID: \"aaea6d2c-ad01-47ba-9aa4-8f90ffff6901\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-sj8lf"
Oct 30 19:47:04.056190 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:47:04.056133    4892 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/efd3e1ce-d3c6-419d-9b5f-dee2907b2cb6-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-6db586dd6c-2pqkg\" (UID: \"efd3e1ce-d3c6-419d-9b5f-dee2907b2cb6\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-2pqkg"
Oct 30 19:47:04.056659 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:47:04.056195    4892 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/aaea6d2c-ad01-47ba-9aa4-8f90ffff6901-istio-envoy\") pod \"maas-default-gateway-openshift-default-6db586dd6c-sj8lf\" (UID: \"aaea6d2c-ad01-47ba-9aa4-8f90ffff6901\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-sj8lf"
Oct 30 19:47:04.056659 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:47:04.056230    4892 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/efd3e1ce-d3c6-419d-9b5f-dee2907b2cb6-workload-socket\") pod \"maas-default-gateway-openshift-default-6db586dd6c-2pqkg\" (UID: \"efd3e1ce-d3c6-419d-9b5f-dee2907b2cb6\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-2pqkg"
Oct 30 19:47:04.056659 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:47:04.056272    4892 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-w95lm\" (UniqueName: \"kubernetes.io/projected/efd3e1ce-d3c6-419d-9b5f-dee2907b2cb6-kube-api-access-w95lm\") pod \"maas-default-gateway-openshift-default-6db586dd6c-2pqkg\" (UID: \"efd3e1ce-d3c6-419d-9b5f-dee2907b2cb6\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-2pqkg"
Oct 30 19:47:04.057004 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:47:04.056953    4892 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/aaea6d2c-ad01-47ba-9aa4-8f90ffff6901-istio-data\") pod \"maas-default-gateway-openshift-default-6db586dd6c-sj8lf\" (UID: \"aaea6d2c-ad01-47ba-9aa4-8f90ffff6901\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-sj8lf"
Oct 30 19:47:04.057334 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:47:04.057309    4892 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/efd3e1ce-d3c6-419d-9b5f-dee2907b2cb6-istio-data\") pod \"maas-default-gateway-openshift-default-6db586dd6c-2pqkg\" (UID: \"efd3e1ce-d3c6-419d-9b5f-dee2907b2cb6\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-2pqkg"
Oct 30 19:47:04.057543 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:47:04.057523    4892 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/aaea6d2c-ad01-47ba-9aa4-8f90ffff6901-credential-socket\") pod \"maas-default-gateway-openshift-default-6db586dd6c-sj8lf\" (UID: \"aaea6d2c-ad01-47ba-9aa4-8f90ffff6901\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-sj8lf"
Oct 30 19:47:04.057631 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:47:04.057525    4892 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/aaea6d2c-ad01-47ba-9aa4-8f90ffff6901-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-6db586dd6c-sj8lf\" (UID: \"aaea6d2c-ad01-47ba-9aa4-8f90ffff6901\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-sj8lf"
Oct 30 19:47:04.057906 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:47:04.057884    4892 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/efd3e1ce-d3c6-419d-9b5f-dee2907b2cb6-credential-socket\") pod \"maas-default-gateway-openshift-default-6db586dd6c-2pqkg\" (UID: \"efd3e1ce-d3c6-419d-9b5f-dee2907b2cb6\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-2pqkg"
Oct 30 19:47:04.058113 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:47:04.058088    4892 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/aaea6d2c-ad01-47ba-9aa4-8f90ffff6901-workload-certs\") pod \"maas-default-gateway-openshift-default-6db586dd6c-sj8lf\" (UID: \"aaea6d2c-ad01-47ba-9aa4-8f90ffff6901\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-sj8lf"
Oct 30 19:47:04.058367 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:47:04.058344    4892 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/efd3e1ce-d3c6-419d-9b5f-dee2907b2cb6-workload-certs\") pod \"maas-default-gateway-openshift-default-6db586dd6c-2pqkg\" (UID: \"efd3e1ce-d3c6-419d-9b5f-dee2907b2cb6\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-2pqkg"
Oct 30 19:47:04.058603 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:47:04.058573    4892 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/aaea6d2c-ad01-47ba-9aa4-8f90ffff6901-workload-socket\") pod \"maas-default-gateway-openshift-default-6db586dd6c-sj8lf\" (UID: \"aaea6d2c-ad01-47ba-9aa4-8f90ffff6901\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-sj8lf"
Oct 30 19:47:04.058709 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:47:04.058578    4892 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/efd3e1ce-d3c6-419d-9b5f-dee2907b2cb6-workload-socket\") pod \"maas-default-gateway-openshift-default-6db586dd6c-2pqkg\" (UID: \"efd3e1ce-d3c6-419d-9b5f-dee2907b2cb6\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-2pqkg"
Oct 30 19:47:04.059399 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:47:04.059378    4892 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/efd3e1ce-d3c6-419d-9b5f-dee2907b2cb6-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-6db586dd6c-2pqkg\" (UID: \"efd3e1ce-d3c6-419d-9b5f-dee2907b2cb6\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-2pqkg"
Oct 30 19:47:04.060923 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:47:04.060896    4892 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/aaea6d2c-ad01-47ba-9aa4-8f90ffff6901-istio-podinfo\") pod \"maas-default-gateway-openshift-default-6db586dd6c-sj8lf\" (UID: \"aaea6d2c-ad01-47ba-9aa4-8f90ffff6901\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-sj8lf"
Oct 30 19:47:04.061419 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:47:04.061399    4892 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/efd3e1ce-d3c6-419d-9b5f-dee2907b2cb6-istio-podinfo\") pod \"maas-default-gateway-openshift-default-6db586dd6c-2pqkg\" (UID: \"efd3e1ce-d3c6-419d-9b5f-dee2907b2cb6\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-2pqkg"
Oct 30 19:47:04.061703 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:47:04.061681    4892 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/aaea6d2c-ad01-47ba-9aa4-8f90ffff6901-istio-envoy\") pod \"maas-default-gateway-openshift-default-6db586dd6c-sj8lf\" (UID: \"aaea6d2c-ad01-47ba-9aa4-8f90ffff6901\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-sj8lf"
Oct 30 19:47:04.062205 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:47:04.062181    4892 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/efd3e1ce-d3c6-419d-9b5f-dee2907b2cb6-istio-envoy\") pod \"maas-default-gateway-openshift-default-6db586dd6c-2pqkg\" (UID: \"efd3e1ce-d3c6-419d-9b5f-dee2907b2cb6\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-2pqkg"
Oct 30 19:47:04.120524 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:47:04.120407    4892 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/aaea6d2c-ad01-47ba-9aa4-8f90ffff6901-istio-token\") pod \"maas-default-gateway-openshift-default-6db586dd6c-sj8lf\" (UID: \"aaea6d2c-ad01-47ba-9aa4-8f90ffff6901\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-sj8lf"
Oct 30 19:47:04.124247 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:47:04.124086    4892 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6gbvt\" (UniqueName: \"kubernetes.io/projected/aaea6d2c-ad01-47ba-9aa4-8f90ffff6901-kube-api-access-6gbvt\") pod \"maas-default-gateway-openshift-default-6db586dd6c-sj8lf\" (UID: \"aaea6d2c-ad01-47ba-9aa4-8f90ffff6901\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-sj8lf"
Oct 30 19:47:04.124247 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:47:04.124217    4892 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-w95lm\" (UniqueName: \"kubernetes.io/projected/efd3e1ce-d3c6-419d-9b5f-dee2907b2cb6-kube-api-access-w95lm\") pod \"maas-default-gateway-openshift-default-6db586dd6c-2pqkg\" (UID: \"efd3e1ce-d3c6-419d-9b5f-dee2907b2cb6\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-2pqkg"
Oct 30 19:47:04.124730 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:47:04.124705    4892 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/efd3e1ce-d3c6-419d-9b5f-dee2907b2cb6-istio-token\") pod \"maas-default-gateway-openshift-default-6db586dd6c-2pqkg\" (UID: \"efd3e1ce-d3c6-419d-9b5f-dee2907b2cb6\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-2pqkg"
Oct 30 19:47:04.150954 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:47:04.150933    4892 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-2pqkg"
Oct 30 19:47:04.199856 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:47:04.199821    4892 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-sj8lf"
Oct 30 19:47:04.720560 ip-10-0-132-82 kubenswrapper[4892]: W1030 19:47:04.713635    4892 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podaaea6d2c_ad01_47ba_9aa4_8f90ffff6901.slice/crio-d80b11f24bb9e8334d3e0bea09aeb14f6402f857fca73e10dc635ccc1075e0f4 WatchSource:0}: Error finding container d80b11f24bb9e8334d3e0bea09aeb14f6402f857fca73e10dc635ccc1075e0f4: Status 404 returned error can't find the container with id d80b11f24bb9e8334d3e0bea09aeb14f6402f857fca73e10dc635ccc1075e0f4
Oct 30 19:47:04.740880 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:47:04.738461    4892 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-2pqkg"]
Oct 30 19:47:04.740880 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:47:04.739586    4892 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892000Ki","pods":"250"}
Oct 30 19:47:04.740880 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:47:04.739651    4892 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892000Ki","pods":"250"}
Oct 30 19:47:04.740880 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:47:04.739690    4892 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892000Ki","pods":"250"}
Oct 30 19:47:04.741377 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:47:04.741339    4892 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892000Ki","pods":"250"}
Oct 30 19:47:04.741613 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:47:04.741539    4892 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892000Ki","pods":"250"}
Oct 30 19:47:04.741743 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:47:04.741726    4892 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892000Ki","pods":"250"}
Oct 30 19:47:04.745693 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:47:04.745670    4892 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-sj8lf"]
Oct 30 19:47:04.931933 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:47:04.931900    4892 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w3-c75ed4-545bd7675b-lj8lp"]
Oct 30 19:47:05.107128 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:47:05.084996    4892 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w3-c75ed4-545bd7675b-lj8lp"
Oct 30 19:47:05.173931 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:47:05.173884    4892 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w3-c75ed4-545bd7675b-lj8lp"]
Oct 30 19:47:05.229990 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:47:05.204649    4892 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-shared-b-w3-c75ed4\""
Oct 30 19:47:05.320668 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:47:05.214509    4892 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-shared-b-w3-c75ed4-dockercfg-9rfxt\""
Oct 30 19:47:05.320890 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:47:05.263109    4892 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-k7k2w\" (UniqueName: \"kubernetes.io/projected/83a6bdb5-d704-47cf-bdc4-202978ec386b-kube-api-access-k7k2w\") pod \"payload-processing-e2e-shared-b-w3-c75ed4-545bd7675b-lj8lp\" (UID: \"83a6bdb5-d704-47cf-bdc4-202978ec386b\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w3-c75ed4-545bd7675b-lj8lp"
Oct 30 19:47:05.321121 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:47:05.321099    4892 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/83a6bdb5-d704-47cf-bdc4-202978ec386b-plugins-config-volume\") pod \"payload-processing-e2e-shared-b-w3-c75ed4-545bd7675b-lj8lp\" (UID: \"83a6bdb5-d704-47cf-bdc4-202978ec386b\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w3-c75ed4-545bd7675b-lj8lp"
Oct 30 19:47:05.395330 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:47:05.395300    4892 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w3-c75ed4-b45dd89bc-pd54p"]
Oct 30 19:47:05.413402 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:47:05.405045    4892 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-c75ed4-b45dd89bc-pd54p"
Oct 30 19:47:05.433119 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:47:05.432935    4892 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-55ccq\" (UniqueName: \"kubernetes.io/projected/6003eaae-92d0-41c3-ab2d-118574a64955-kube-api-access-55ccq\") pod \"payload-pre-processing-e2e-shared-b-w3-c75ed4-b45dd89bc-pd54p\" (UID: \"6003eaae-92d0-41c3-ab2d-118574a64955\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-c75ed4-b45dd89bc-pd54p"
Oct 30 19:47:05.433474 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:47:05.433451    4892 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-k7k2w\" (UniqueName: \"kubernetes.io/projected/83a6bdb5-d704-47cf-bdc4-202978ec386b-kube-api-access-k7k2w\") pod \"payload-processing-e2e-shared-b-w3-c75ed4-545bd7675b-lj8lp\" (UID: \"83a6bdb5-d704-47cf-bdc4-202978ec386b\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w3-c75ed4-545bd7675b-lj8lp"
Oct 30 19:47:05.433626 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:47:05.433611    4892 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/83a6bdb5-d704-47cf-bdc4-202978ec386b-plugins-config-volume\") pod \"payload-processing-e2e-shared-b-w3-c75ed4-545bd7675b-lj8lp\" (UID: \"83a6bdb5-d704-47cf-bdc4-202978ec386b\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w3-c75ed4-545bd7675b-lj8lp"
Oct 30 19:47:05.433817 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:47:05.433796    4892 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6003eaae-92d0-41c3-ab2d-118574a64955-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-b-w3-c75ed4-b45dd89bc-pd54p\" (UID: \"6003eaae-92d0-41c3-ab2d-118574a64955\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-c75ed4-b45dd89bc-pd54p"
Oct 30 19:47:05.434743 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:47:05.434699    4892 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/83a6bdb5-d704-47cf-bdc4-202978ec386b-plugins-config-volume\") pod \"payload-processing-e2e-shared-b-w3-c75ed4-545bd7675b-lj8lp\" (UID: \"83a6bdb5-d704-47cf-bdc4-202978ec386b\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w3-c75ed4-545bd7675b-lj8lp"
Oct 30 19:47:05.444245 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:47:05.444208    4892 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-sj8lf" event={"ID":"aaea6d2c-ad01-47ba-9aa4-8f90ffff6901","Type":"ContainerStarted","Data":"d80b11f24bb9e8334d3e0bea09aeb14f6402f857fca73e10dc635ccc1075e0f4"}
Oct 30 19:47:05.455376 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:47:05.454210    4892 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w3-c75ed4-b45dd89bc-pd54p"]
Oct 30 19:47:05.456801 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:47:05.456755    4892 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-2pqkg" event={"ID":"efd3e1ce-d3c6-419d-9b5f-dee2907b2cb6","Type":"ContainerStarted","Data":"a5d98329863df4e28dcf48117e12471018be2032b86db49fa33d698b7deb4651"}
Oct 30 19:47:05.495941 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:47:05.495878    4892 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-k7k2w\" (UniqueName: \"kubernetes.io/projected/83a6bdb5-d704-47cf-bdc4-202978ec386b-kube-api-access-k7k2w\") pod \"payload-processing-e2e-shared-b-w3-c75ed4-545bd7675b-lj8lp\" (UID: \"83a6bdb5-d704-47cf-bdc4-202978ec386b\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w3-c75ed4-545bd7675b-lj8lp"
Oct 30 19:47:05.536930 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:47:05.536389    4892 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6003eaae-92d0-41c3-ab2d-118574a64955-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-b-w3-c75ed4-b45dd89bc-pd54p\" (UID: \"6003eaae-92d0-41c3-ab2d-118574a64955\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-c75ed4-b45dd89bc-pd54p"
Oct 30 19:47:05.536930 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:47:05.536580    4892 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-55ccq\" (UniqueName: \"kubernetes.io/projected/6003eaae-92d0-41c3-ab2d-118574a64955-kube-api-access-55ccq\") pod \"payload-pre-processing-e2e-shared-b-w3-c75ed4-b45dd89bc-pd54p\" (UID: \"6003eaae-92d0-41c3-ab2d-118574a64955\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-c75ed4-b45dd89bc-pd54p"
Oct 30 19:47:05.537440 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:47:05.537416    4892 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6003eaae-92d0-41c3-ab2d-118574a64955-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-b-w3-c75ed4-b45dd89bc-pd54p\" (UID: \"6003eaae-92d0-41c3-ab2d-118574a64955\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-c75ed4-b45dd89bc-pd54p"
Oct 30 19:47:05.594273 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:47:05.594193    4892 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-55ccq\" (UniqueName: \"kubernetes.io/projected/6003eaae-92d0-41c3-ab2d-118574a64955-kube-api-access-55ccq\") pod \"payload-pre-processing-e2e-shared-b-w3-c75ed4-b45dd89bc-pd54p\" (UID: \"6003eaae-92d0-41c3-ab2d-118574a64955\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-c75ed4-b45dd89bc-pd54p"
Oct 30 19:47:05.761555 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:47:05.761523    4892 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-c75ed4-b45dd89bc-pd54p"
Oct 30 19:47:05.774831 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:47:05.774803    4892 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w3-c75ed4-545bd7675b-lj8lp"
Oct 30 19:47:06.184678 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:47:06.184390    4892 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w3-c75ed4-b45dd89bc-pd54p"]
Oct 30 19:47:06.262000 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:47:06.257863    4892 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w3-c75ed4-545bd7675b-lj8lp"]
Oct 30 19:47:06.496733 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:47:06.496634    4892 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-c75ed4-b45dd89bc-pd54p" event={"ID":"6003eaae-92d0-41c3-ab2d-118574a64955","Type":"ContainerStarted","Data":"325e4ec21ace3346672609aa23bcb8f70e4dcf65f46ccf3bfaabc66dc581230f"}
Oct 30 19:47:06.496733 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:47:06.496699    4892 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-c75ed4-b45dd89bc-pd54p" event={"ID":"6003eaae-92d0-41c3-ab2d-118574a64955","Type":"ContainerStarted","Data":"d20addaec98073821a1b39389acaca002338be0bd9c5a99ae2ea9e484e8903e0"}
Oct 30 19:47:06.496733 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:47:06.496731    4892 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-c75ed4-b45dd89bc-pd54p"
Oct 30 19:47:06.498819 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:47:06.498789    4892 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w3-c75ed4-545bd7675b-lj8lp" event={"ID":"83a6bdb5-d704-47cf-bdc4-202978ec386b","Type":"ContainerStarted","Data":"d66c5248b5b79ab3eb73d7a4637c56df358b9891acc0b9ef45da901ad23c581f"}
Oct 30 19:47:06.498995 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:47:06.498840    4892 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w3-c75ed4-545bd7675b-lj8lp" event={"ID":"83a6bdb5-d704-47cf-bdc4-202978ec386b","Type":"ContainerStarted","Data":"023e4d1ab631941e492cbaf4fb8f1e8f564bba62ca3788a02f7cced9eef25a12"}
Oct 30 19:47:06.499195 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:47:06.499177    4892 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-shared-b-w3-c75ed4-545bd7675b-lj8lp"
Oct 30 19:47:06.501036 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:47:06.501013    4892 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-sj8lf" event={"ID":"aaea6d2c-ad01-47ba-9aa4-8f90ffff6901","Type":"ContainerStarted","Data":"f7d551895cf5eb0329c0857dfbb29fe3bab5f3901961469267f575fad9f849e4"}
Oct 30 19:47:06.503309 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:47:06.503286    4892 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-2pqkg" event={"ID":"efd3e1ce-d3c6-419d-9b5f-dee2907b2cb6","Type":"ContainerStarted","Data":"2e8b20742702850f9d506aa439396f9c00dba206ad60eef9d015213953bd4fbd"}
Oct 30 19:47:06.849038 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:47:06.832721    4892 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-c75ed4-b45dd89bc-pd54p" podStartSLOduration=1.832702039 podStartE2EDuration="1.832702039s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:47:05 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:47:06.666353084 +0000 UTC m=+1874.236818684" watchObservedRunningTime="2026-10-30 19:47:06.832702039 +0000 UTC m=+1874.403167639"
Oct 30 19:47:06.849038 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:47:06.847991    4892 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-2pqkg" podStartSLOduration=3.847953675 podStartE2EDuration="3.847953675s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:47:03 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:47:06.832367709 +0000 UTC m=+1874.402833308" watchObservedRunningTime="2026-10-30 19:47:06.847953675 +0000 UTC m=+1874.418419278"
Oct 30 19:47:06.999893 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:47:06.999829    4892 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-sj8lf" podStartSLOduration=3.999810373 podStartE2EDuration="3.999810373s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:47:03 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:47:06.996626547 +0000 UTC m=+1874.567092149" watchObservedRunningTime="2026-10-30 19:47:06.999810373 +0000 UTC m=+1874.570275970"
Oct 30 19:47:07.154997 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:47:07.153295    4892 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-2pqkg"
Oct 30 19:47:07.154997 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:47:07.154166    4892 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6db586dd6c-2pqkg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.161:15021/healthz/ready\": dial tcp 10.132.0.161:15021: connect: connection refused" start-of-body=
Oct 30 19:47:07.154997 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:47:07.154213    4892 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-2pqkg" podUID="efd3e1ce-d3c6-419d-9b5f-dee2907b2cb6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.161:15021/healthz/ready\": dial tcp 10.132.0.161:15021: connect: connection refused"
Oct 30 19:47:07.167988 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:47:07.166286    4892 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-shared-b-w3-c75ed4-545bd7675b-lj8lp" podStartSLOduration=3.166267117 podStartE2EDuration="3.166267117s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:47:04 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:47:07.121432193 +0000 UTC m=+1874.691897807" watchObservedRunningTime="2026-10-30 19:47:07.166267117 +0000 UTC m=+1874.736732717"
Oct 30 19:47:07.200435 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:47:07.200408    4892 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-sj8lf"
Oct 30 19:47:07.201713 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:47:07.201689    4892 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6db586dd6c-sj8lf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.162:15021/healthz/ready\": dial tcp 10.132.0.162:15021: connect: connection refused" start-of-body=
Oct 30 19:47:07.201811 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:47:07.201738    4892 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-sj8lf" podUID="aaea6d2c-ad01-47ba-9aa4-8f90ffff6901" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.162:15021/healthz/ready\": dial tcp 10.132.0.162:15021: connect: connection refused"
Oct 30 19:47:08.095389 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:47:08.095349    4892 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ccb90673-knzsq"]
Oct 30 19:47:08.130989 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:47:08.124072    4892 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ccb90673-knzsq"]
Oct 30 19:47:08.170996 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:47:08.167160    4892 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6db586dd6c-2pqkg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.161:15021/healthz/ready\": dial tcp 10.132.0.161:15021: connect: connection refused" start-of-body=
Oct 30 19:47:08.170996 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:47:08.167217    4892 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-2pqkg" podUID="efd3e1ce-d3c6-419d-9b5f-dee2907b2cb6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.161:15021/healthz/ready\": dial tcp 10.132.0.161:15021: connect: connection refused"
Oct 30 19:47:08.206994 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:47:08.203342    4892 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6db586dd6c-sj8lf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.162:15021/healthz/ready\": dial tcp 10.132.0.162:15021: connect: connection refused" start-of-body=
Oct 30 19:47:08.206994 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:47:08.203418    4892 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-sj8lf" podUID="aaea6d2c-ad01-47ba-9aa4-8f90ffff6901" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.162:15021/healthz/ready\": dial tcp 10.132.0.162:15021: connect: connection refused"
Oct 30 19:47:08.594906 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:47:08.594870    4892 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="6cdcce3e-9dc9-4d19-a0cf-fac60ebf288e" path="/var/lib/kubelet/pods/6cdcce3e-9dc9-4d19-a0cf-fac60ebf288e/volumes"
Oct 30 19:47:09.158991 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:47:09.154783    4892 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6db586dd6c-2pqkg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.161:15021/healthz/ready\": dial tcp 10.132.0.161:15021: connect: connection refused" start-of-body=
Oct 30 19:47:09.158991 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:47:09.154844    4892 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-2pqkg" podUID="efd3e1ce-d3c6-419d-9b5f-dee2907b2cb6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.161:15021/healthz/ready\": dial tcp 10.132.0.161:15021: connect: connection refused"
Oct 30 19:47:09.200382 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:47:09.200339    4892 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6db586dd6c-sj8lf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.162:15021/healthz/ready\": dial tcp 10.132.0.162:15021: connect: connection refused" start-of-body=
Oct 30 19:47:09.200547 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:47:09.200402    4892 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-sj8lf" podUID="aaea6d2c-ad01-47ba-9aa4-8f90ffff6901" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.162:15021/healthz/ready\": dial tcp 10.132.0.162:15021: connect: connection refused"
Oct 30 19:47:10.151991 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:47:10.151277    4892 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6db586dd6c-2pqkg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.161:15021/healthz/ready\": dial tcp 10.132.0.161:15021: connect: connection refused" start-of-body=
Oct 30 19:47:10.151991 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:47:10.151333    4892 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-2pqkg" podUID="efd3e1ce-d3c6-419d-9b5f-dee2907b2cb6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.161:15021/healthz/ready\": dial tcp 10.132.0.161:15021: connect: connection refused"
Oct 30 19:47:10.200188 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:47:10.200145    4892 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6db586dd6c-sj8lf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.162:15021/healthz/ready\": dial tcp 10.132.0.162:15021: connect: connection refused" start-of-body=
Oct 30 19:47:10.200374 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:47:10.200203    4892 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-sj8lf" podUID="aaea6d2c-ad01-47ba-9aa4-8f90ffff6901" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.162:15021/healthz/ready\": dial tcp 10.132.0.162:15021: connect: connection refused"
Oct 30 19:47:11.157991 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:47:11.151296    4892 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6db586dd6c-2pqkg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.161:15021/healthz/ready\": dial tcp 10.132.0.161:15021: connect: connection refused" start-of-body=
Oct 30 19:47:11.157991 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:47:11.151356    4892 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-2pqkg" podUID="efd3e1ce-d3c6-419d-9b5f-dee2907b2cb6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.161:15021/healthz/ready\": dial tcp 10.132.0.161:15021: connect: connection refused"
Oct 30 19:47:11.200282 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:47:11.200250    4892 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6db586dd6c-sj8lf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.162:15021/healthz/ready\": dial tcp 10.132.0.162:15021: connect: connection refused" start-of-body=
Oct 30 19:47:11.200531 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:47:11.200509    4892 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-sj8lf" podUID="aaea6d2c-ad01-47ba-9aa4-8f90ffff6901" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.162:15021/healthz/ready\": dial tcp 10.132.0.162:15021: connect: connection refused"
Oct 30 19:47:12.155988 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:47:12.154332    4892 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6db586dd6c-2pqkg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.161:15021/healthz/ready\": dial tcp 10.132.0.161:15021: connect: connection refused" start-of-body=
Oct 30 19:47:12.155988 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:47:12.154392    4892 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-2pqkg" podUID="efd3e1ce-d3c6-419d-9b5f-dee2907b2cb6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.161:15021/healthz/ready\": dial tcp 10.132.0.161:15021: connect: connection refused"
Oct 30 19:47:12.206985 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:47:12.200361    4892 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6db586dd6c-sj8lf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.162:15021/healthz/ready\": dial tcp 10.132.0.162:15021: connect: connection refused" start-of-body=
Oct 30 19:47:12.206985 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:47:12.200427    4892 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-sj8lf" podUID="aaea6d2c-ad01-47ba-9aa4-8f90ffff6901" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.162:15021/healthz/ready\": dial tcp 10.132.0.162:15021: connect: connection refused"
Oct 30 19:47:13.152118 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:47:13.152083    4892 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6db586dd6c-2pqkg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.161:15021/healthz/ready\": dial tcp 10.132.0.161:15021: connect: connection refused" start-of-body=
Oct 30 19:47:13.152319 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:47:13.152159    4892 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-2pqkg" podUID="efd3e1ce-d3c6-419d-9b5f-dee2907b2cb6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.161:15021/healthz/ready\": dial tcp 10.132.0.161:15021: connect: connection refused"
Oct 30 19:47:13.200731 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:47:13.200700    4892 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6db586dd6c-sj8lf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.162:15021/healthz/ready\": dial tcp 10.132.0.162:15021: connect: connection refused" start-of-body=
Oct 30 19:47:13.200885 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:47:13.200753    4892 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-sj8lf" podUID="aaea6d2c-ad01-47ba-9aa4-8f90ffff6901" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.162:15021/healthz/ready\": dial tcp 10.132.0.162:15021: connect: connection refused"
Oct 30 19:47:14.060119 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:47:14.060075    4892 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b7f56eab-tg2gm"]
Oct 30 19:47:14.105297 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:47:14.105244    4892 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b7f56eab-tg2gm"]
Oct 30 19:47:14.585125 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:47:14.585087    4892 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e7cd4cc4-e0fa-4b6b-8895-8be85ddb31c5" path="/var/lib/kubelet/pods/e7cd4cc4-e0fa-4b6b-8895-8be85ddb31c5/volumes"
Oct 30 19:47:14.732772 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:47:14.732718    4892 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-2pqkg"
Oct 30 19:47:14.733565 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:47:14.733534    4892 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-2pqkg"
Oct 30 19:47:14.734953 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:47:14.734928    4892 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-2pqkg"
Oct 30 19:47:14.812947 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:47:14.812917    4892 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-sj8lf"
Oct 30 19:47:14.813559 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:47:14.813537    4892 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-sj8lf"
Oct 30 19:47:14.814095 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:47:14.814080    4892 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-sj8lf"
Oct 30 19:47:17.556943 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:47:17.556905    4892 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-shared-b-w3-c75ed4-545bd7675b-lj8lp"
Oct 30 19:47:17.558147 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:47:17.558128    4892 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-c75ed4-b45dd89bc-pd54p"
Oct 30 19:47:22.051401 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:47:22.051366    4892 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e426ecf3-gfxx4"]
Oct 30 19:47:22.101642 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:47:22.101610    4892 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e426ecf3-gfxx4"]
Oct 30 19:47:22.570679 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:47:22.570636    4892 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="6f5154a2-9a70-4ccc-8584-a2ce72ba6564" path="/var/lib/kubelet/pods/6f5154a2-9a70-4ccc-8584-a2ce72ba6564/volumes"
Oct 30 19:47:23.085576 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:47:23.085540    4892 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-b-w0-6ac304-openshift-default-8cdbdbc67-rgbzh"]
Oct 30 19:47:23.091425 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:47:23.091405    4892 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w0-6ac304-openshift-default-8cdbdbc67-rgbzh"
Oct 30 19:47:23.126991 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:47:23.122638    4892 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0378dcfe-8d0a-43bb-808d-01a939801130-istio-token\") pod \"e2e-shared-b-w0-6ac304-openshift-default-8cdbdbc67-rgbzh\" (UID: \"0378dcfe-8d0a-43bb-808d-01a939801130\") " pod="openshift-ingress/e2e-shared-b-w0-6ac304-openshift-default-8cdbdbc67-rgbzh"
Oct 30 19:47:23.126991 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:47:23.122699    4892 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0378dcfe-8d0a-43bb-808d-01a939801130-istio-data\") pod \"e2e-shared-b-w0-6ac304-openshift-default-8cdbdbc67-rgbzh\" (UID: \"0378dcfe-8d0a-43bb-808d-01a939801130\") " pod="openshift-ingress/e2e-shared-b-w0-6ac304-openshift-default-8cdbdbc67-rgbzh"
Oct 30 19:47:23.126991 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:47:23.122746    4892 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0378dcfe-8d0a-43bb-808d-01a939801130-workload-socket\") pod \"e2e-shared-b-w0-6ac304-openshift-default-8cdbdbc67-rgbzh\" (UID: \"0378dcfe-8d0a-43bb-808d-01a939801130\") " pod="openshift-ingress/e2e-shared-b-w0-6ac304-openshift-default-8cdbdbc67-rgbzh"
Oct 30 19:47:23.126991 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:47:23.122822    4892 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0378dcfe-8d0a-43bb-808d-01a939801130-istiod-ca-cert\") pod \"e2e-shared-b-w0-6ac304-openshift-default-8cdbdbc67-rgbzh\" (UID: \"0378dcfe-8d0a-43bb-808d-01a939801130\") " pod="openshift-ingress/e2e-shared-b-w0-6ac304-openshift-default-8cdbdbc67-rgbzh"
Oct 30 19:47:23.126991 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:47:23.122865    4892 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0378dcfe-8d0a-43bb-808d-01a939801130-credential-socket\") pod \"e2e-shared-b-w0-6ac304-openshift-default-8cdbdbc67-rgbzh\" (UID: \"0378dcfe-8d0a-43bb-808d-01a939801130\") " pod="openshift-ingress/e2e-shared-b-w0-6ac304-openshift-default-8cdbdbc67-rgbzh"
Oct 30 19:47:23.126991 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:47:23.122892    4892 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0378dcfe-8d0a-43bb-808d-01a939801130-workload-certs\") pod \"e2e-shared-b-w0-6ac304-openshift-default-8cdbdbc67-rgbzh\" (UID: \"0378dcfe-8d0a-43bb-808d-01a939801130\") " pod="openshift-ingress/e2e-shared-b-w0-6ac304-openshift-default-8cdbdbc67-rgbzh"
Oct 30 19:47:23.126991 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:47:23.122943    4892 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0378dcfe-8d0a-43bb-808d-01a939801130-istio-podinfo\") pod \"e2e-shared-b-w0-6ac304-openshift-default-8cdbdbc67-rgbzh\" (UID: \"0378dcfe-8d0a-43bb-808d-01a939801130\") " pod="openshift-ingress/e2e-shared-b-w0-6ac304-openshift-default-8cdbdbc67-rgbzh"
Oct 30 19:47:23.126991 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:47:23.123044    4892 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-n2brw\" (UniqueName: \"kubernetes.io/projected/0378dcfe-8d0a-43bb-808d-01a939801130-kube-api-access-n2brw\") pod \"e2e-shared-b-w0-6ac304-openshift-default-8cdbdbc67-rgbzh\" (UID: \"0378dcfe-8d0a-43bb-808d-01a939801130\") " pod="openshift-ingress/e2e-shared-b-w0-6ac304-openshift-default-8cdbdbc67-rgbzh"
Oct 30 19:47:23.126991 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:47:23.123116    4892 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0378dcfe-8d0a-43bb-808d-01a939801130-istio-envoy\") pod \"e2e-shared-b-w0-6ac304-openshift-default-8cdbdbc67-rgbzh\" (UID: \"0378dcfe-8d0a-43bb-808d-01a939801130\") " pod="openshift-ingress/e2e-shared-b-w0-6ac304-openshift-default-8cdbdbc67-rgbzh"
Oct 30 19:47:23.146053 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:47:23.146019    4892 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-shared-b-w0-6ac304-openshift-default-dockercfg-5bgmz\""
Oct 30 19:47:23.202514 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:47:23.202483    4892 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w0-6ac304-openshift-default-8cdbdbc67-rgbzh"]
Oct 30 19:47:23.258415 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:47:23.226934    4892 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0378dcfe-8d0a-43bb-808d-01a939801130-istio-token\") pod \"e2e-shared-b-w0-6ac304-openshift-default-8cdbdbc67-rgbzh\" (UID: \"0378dcfe-8d0a-43bb-808d-01a939801130\") " pod="openshift-ingress/e2e-shared-b-w0-6ac304-openshift-default-8cdbdbc67-rgbzh"
Oct 30 19:47:23.354159 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:47:23.354080    4892 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0378dcfe-8d0a-43bb-808d-01a939801130-istio-data\") pod \"e2e-shared-b-w0-6ac304-openshift-default-8cdbdbc67-rgbzh\" (UID: \"0378dcfe-8d0a-43bb-808d-01a939801130\") " pod="openshift-ingress/e2e-shared-b-w0-6ac304-openshift-default-8cdbdbc67-rgbzh"
Oct 30 19:47:23.355620 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:47:23.355593    4892 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0378dcfe-8d0a-43bb-808d-01a939801130-workload-socket\") pod \"e2e-shared-b-w0-6ac304-openshift-default-8cdbdbc67-rgbzh\" (UID: \"0378dcfe-8d0a-43bb-808d-01a939801130\") " pod="openshift-ingress/e2e-shared-b-w0-6ac304-openshift-default-8cdbdbc67-rgbzh"
Oct 30 19:47:23.356861 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:47:23.356840    4892 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0378dcfe-8d0a-43bb-808d-01a939801130-istiod-ca-cert\") pod \"e2e-shared-b-w0-6ac304-openshift-default-8cdbdbc67-rgbzh\" (UID: \"0378dcfe-8d0a-43bb-808d-01a939801130\") " pod="openshift-ingress/e2e-shared-b-w0-6ac304-openshift-default-8cdbdbc67-rgbzh"
Oct 30 19:47:23.357789 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:47:23.357769    4892 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0378dcfe-8d0a-43bb-808d-01a939801130-credential-socket\") pod \"e2e-shared-b-w0-6ac304-openshift-default-8cdbdbc67-rgbzh\" (UID: \"0378dcfe-8d0a-43bb-808d-01a939801130\") " pod="openshift-ingress/e2e-shared-b-w0-6ac304-openshift-default-8cdbdbc67-rgbzh"
Oct 30 19:47:23.358279 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:47:23.358261    4892 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0378dcfe-8d0a-43bb-808d-01a939801130-workload-certs\") pod \"e2e-shared-b-w0-6ac304-openshift-default-8cdbdbc67-rgbzh\" (UID: \"0378dcfe-8d0a-43bb-808d-01a939801130\") " pod="openshift-ingress/e2e-shared-b-w0-6ac304-openshift-default-8cdbdbc67-rgbzh"
Oct 30 19:47:23.370413 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:47:23.370386    4892 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0378dcfe-8d0a-43bb-808d-01a939801130-istio-podinfo\") pod \"e2e-shared-b-w0-6ac304-openshift-default-8cdbdbc67-rgbzh\" (UID: \"0378dcfe-8d0a-43bb-808d-01a939801130\") " pod="openshift-ingress/e2e-shared-b-w0-6ac304-openshift-default-8cdbdbc67-rgbzh"
Oct 30 19:47:23.374272 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:47:23.374245    4892 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-n2brw\" (UniqueName: \"kubernetes.io/projected/0378dcfe-8d0a-43bb-808d-01a939801130-kube-api-access-n2brw\") pod \"e2e-shared-b-w0-6ac304-openshift-default-8cdbdbc67-rgbzh\" (UID: \"0378dcfe-8d0a-43bb-808d-01a939801130\") " pod="openshift-ingress/e2e-shared-b-w0-6ac304-openshift-default-8cdbdbc67-rgbzh"
Oct 30 19:47:23.374805 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:47:23.358219    4892 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0378dcfe-8d0a-43bb-808d-01a939801130-istio-data\") pod \"e2e-shared-b-w0-6ac304-openshift-default-8cdbdbc67-rgbzh\" (UID: \"0378dcfe-8d0a-43bb-808d-01a939801130\") " pod="openshift-ingress/e2e-shared-b-w0-6ac304-openshift-default-8cdbdbc67-rgbzh"
Oct 30 19:47:23.374957 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:47:23.370236    4892 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0378dcfe-8d0a-43bb-808d-01a939801130-workload-certs\") pod \"e2e-shared-b-w0-6ac304-openshift-default-8cdbdbc67-rgbzh\" (UID: \"0378dcfe-8d0a-43bb-808d-01a939801130\") " pod="openshift-ingress/e2e-shared-b-w0-6ac304-openshift-default-8cdbdbc67-rgbzh"
Oct 30 19:47:23.375073 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:47:23.356086    4892 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0378dcfe-8d0a-43bb-808d-01a939801130-workload-socket\") pod \"e2e-shared-b-w0-6ac304-openshift-default-8cdbdbc67-rgbzh\" (UID: \"0378dcfe-8d0a-43bb-808d-01a939801130\") " pod="openshift-ingress/e2e-shared-b-w0-6ac304-openshift-default-8cdbdbc67-rgbzh"
Oct 30 19:47:23.375162 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:47:23.357684    4892 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0378dcfe-8d0a-43bb-808d-01a939801130-istiod-ca-cert\") pod \"e2e-shared-b-w0-6ac304-openshift-default-8cdbdbc67-rgbzh\" (UID: \"0378dcfe-8d0a-43bb-808d-01a939801130\") " pod="openshift-ingress/e2e-shared-b-w0-6ac304-openshift-default-8cdbdbc67-rgbzh"
Oct 30 19:47:23.375251 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:47:23.358164    4892 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0378dcfe-8d0a-43bb-808d-01a939801130-credential-socket\") pod \"e2e-shared-b-w0-6ac304-openshift-default-8cdbdbc67-rgbzh\" (UID: \"0378dcfe-8d0a-43bb-808d-01a939801130\") " pod="openshift-ingress/e2e-shared-b-w0-6ac304-openshift-default-8cdbdbc67-rgbzh"
Oct 30 19:47:23.375470 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:47:23.375449    4892 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0378dcfe-8d0a-43bb-808d-01a939801130-istio-envoy\") pod \"e2e-shared-b-w0-6ac304-openshift-default-8cdbdbc67-rgbzh\" (UID: \"0378dcfe-8d0a-43bb-808d-01a939801130\") " pod="openshift-ingress/e2e-shared-b-w0-6ac304-openshift-default-8cdbdbc67-rgbzh"
Oct 30 19:47:23.433989 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:47:23.430246    4892 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0378dcfe-8d0a-43bb-808d-01a939801130-istio-podinfo\") pod \"e2e-shared-b-w0-6ac304-openshift-default-8cdbdbc67-rgbzh\" (UID: \"0378dcfe-8d0a-43bb-808d-01a939801130\") " pod="openshift-ingress/e2e-shared-b-w0-6ac304-openshift-default-8cdbdbc67-rgbzh"
Oct 30 19:47:23.461816 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:47:23.461741    4892 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0378dcfe-8d0a-43bb-808d-01a939801130-istio-token\") pod \"e2e-shared-b-w0-6ac304-openshift-default-8cdbdbc67-rgbzh\" (UID: \"0378dcfe-8d0a-43bb-808d-01a939801130\") " pod="openshift-ingress/e2e-shared-b-w0-6ac304-openshift-default-8cdbdbc67-rgbzh"
Oct 30 19:47:23.462698 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:47:23.462675    4892 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0378dcfe-8d0a-43bb-808d-01a939801130-istio-envoy\") pod \"e2e-shared-b-w0-6ac304-openshift-default-8cdbdbc67-rgbzh\" (UID: \"0378dcfe-8d0a-43bb-808d-01a939801130\") " pod="openshift-ingress/e2e-shared-b-w0-6ac304-openshift-default-8cdbdbc67-rgbzh"
Oct 30 19:47:23.477996 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:47:23.476645    4892 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-n2brw\" (UniqueName: \"kubernetes.io/projected/0378dcfe-8d0a-43bb-808d-01a939801130-kube-api-access-n2brw\") pod \"e2e-shared-b-w0-6ac304-openshift-default-8cdbdbc67-rgbzh\" (UID: \"0378dcfe-8d0a-43bb-808d-01a939801130\") " pod="openshift-ingress/e2e-shared-b-w0-6ac304-openshift-default-8cdbdbc67-rgbzh"
Oct 30 19:47:23.713552 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:47:23.713105    4892 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w0-6ac304-openshift-default-8cdbdbc67-rgbzh"
Oct 30 19:47:24.232568 ip-10-0-132-82 kubenswrapper[4892]: W1030 19:47:24.232539    4892 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0378dcfe_8d0a_43bb_808d_01a939801130.slice/crio-0007b628d99c4db118be1064bae47abfc9d7dcd2fd0670ba91d16353b95520ee WatchSource:0}: Error finding container 0007b628d99c4db118be1064bae47abfc9d7dcd2fd0670ba91d16353b95520ee: Status 404 returned error can't find the container with id 0007b628d99c4db118be1064bae47abfc9d7dcd2fd0670ba91d16353b95520ee
Oct 30 19:47:24.235057 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:47:24.235035    4892 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w0-6ac304-openshift-default-8cdbdbc67-rgbzh"]
Oct 30 19:47:24.235437 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:47:24.235404    4892 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892000Ki","pods":"250"}
Oct 30 19:47:24.235526 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:47:24.235471    4892 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892000Ki","pods":"250"}
Oct 30 19:47:24.235526 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:47:24.235501    4892 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892000Ki","pods":"250"}
Oct 30 19:47:24.876649 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:47:24.875749    4892 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w0-6ac304-openshift-default-8cdbdbc67-rgbzh" event={"ID":"0378dcfe-8d0a-43bb-808d-01a939801130","Type":"ContainerStarted","Data":"b29ccaad7d805db2c688141f390e9322ab08c52aae531c10ef5dad204e09e713"}
Oct 30 19:47:24.876649 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:47:24.875819    4892 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w0-6ac304-openshift-default-8cdbdbc67-rgbzh" event={"ID":"0378dcfe-8d0a-43bb-808d-01a939801130","Type":"ContainerStarted","Data":"0007b628d99c4db118be1064bae47abfc9d7dcd2fd0670ba91d16353b95520ee"}
Oct 30 19:47:24.918083 ip-10-0-132-82 kubenswrapper[4892]: E1030 19:47:24.918043    4892 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-jrm87" containerName="extractor"
Oct 30 19:47:25.092044 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:47:25.091987    4892 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-b-w0-6ac304-openshift-default-8cdbdbc67-rgbzh" podStartSLOduration=2.09195667 podStartE2EDuration="2.09195667s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:47:23 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:47:25.041127313 +0000 UTC m=+1892.611592911" watchObservedRunningTime="2026-10-30 19:47:25.09195667 +0000 UTC m=+1892.662422270"
Oct 30 19:47:25.716326 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:47:25.716296    4892 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-b-w0-6ac304-openshift-default-8cdbdbc67-rgbzh"
Oct 30 19:47:25.742379 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:47:25.742348    4892 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-b-w0-6ac304-openshift-default-8cdbdbc67-rgbzh"
Oct 30 19:47:25.948036 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:47:25.947955    4892 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-b-w0-6ac304-openshift-default-8cdbdbc67-rgbzh"
Oct 30 19:47:25.955279 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:47:25.955246    4892 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-b-w0-6ac304-openshift-default-8cdbdbc67-rgbzh"
Oct 30 19:47:27.047623 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:47:27.047592    4892 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-10bbe06e-wwv8x"]
Oct 30 19:47:27.095900 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:47:27.095860    4892 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-10bbe06e-wwv8x"]
Oct 30 19:47:28.573313 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:47:28.573270    4892 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="da70be07-9754-47d1-95bc-a5970bf9ecae" path="/var/lib/kubelet/pods/da70be07-9754-47d1-95bc-a5970bf9ecae/volumes"
Oct 30 19:47:29.572995 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:47:29.548109    4892 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-dd9b8d79-867946f667-plb8n"]
Oct 30 19:47:29.696361 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:47:29.675500    4892 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-dd9b8d79-867946f667-plb8n"
Oct 30 19:47:29.727117 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:47:29.727084    4892 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-dd9b8d79-867946f667-plb8n"]
Oct 30 19:47:29.739387 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:47:29.739149    4892 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-mt-dd9b8d79\""
Oct 30 19:47:29.740608 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:47:29.740529    4892 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-dd9b8d79-67c8d6d46-m4wc6"]
Oct 30 19:47:29.745422 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:47:29.745402    4892 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-dd9b8d79-67c8d6d46-m4wc6"
Oct 30 19:47:29.749119 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:47:29.749097    4892 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-dd9b8d79-67c8d6d46-m4wc6"]
Oct 30 19:47:29.772653 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:47:29.772614    4892 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f7e053ee-836a-4b0e-b2d9-88e3af83ca7b-plugins-config-volume\") pod \"payload-pre-processing-e2e-mt-dd9b8d79-867946f667-plb8n\" (UID: \"f7e053ee-836a-4b0e-b2d9-88e3af83ca7b\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-dd9b8d79-867946f667-plb8n"
Oct 30 19:47:29.772759 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:47:29.772708    4892 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wbdcr\" (UniqueName: \"kubernetes.io/projected/f7e053ee-836a-4b0e-b2d9-88e3af83ca7b-kube-api-access-wbdcr\") pod \"payload-pre-processing-e2e-mt-dd9b8d79-867946f667-plb8n\" (UID: \"f7e053ee-836a-4b0e-b2d9-88e3af83ca7b\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-dd9b8d79-867946f667-plb8n"
Oct 30 19:47:29.791069 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:47:29.791046    4892 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-mt-dd9b8d79-dockercfg-zmmg6\""
Oct 30 19:47:29.874186 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:47:29.874098    4892 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3c568859-9035-498d-b347-170ab8d1b2e7-plugins-config-volume\") pod \"payload-processing-e2e-mt-dd9b8d79-67c8d6d46-m4wc6\" (UID: \"3c568859-9035-498d-b347-170ab8d1b2e7\") " pod="openshift-ingress/payload-processing-e2e-mt-dd9b8d79-67c8d6d46-m4wc6"
Oct 30 19:47:29.874342 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:47:29.874293    4892 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f7e053ee-836a-4b0e-b2d9-88e3af83ca7b-plugins-config-volume\") pod \"payload-pre-processing-e2e-mt-dd9b8d79-867946f667-plb8n\" (UID: \"f7e053ee-836a-4b0e-b2d9-88e3af83ca7b\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-dd9b8d79-867946f667-plb8n"
Oct 30 19:47:29.874409 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:47:29.874369    4892 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-f9hcn\" (UniqueName: \"kubernetes.io/projected/3c568859-9035-498d-b347-170ab8d1b2e7-kube-api-access-f9hcn\") pod \"payload-processing-e2e-mt-dd9b8d79-67c8d6d46-m4wc6\" (UID: \"3c568859-9035-498d-b347-170ab8d1b2e7\") " pod="openshift-ingress/payload-processing-e2e-mt-dd9b8d79-67c8d6d46-m4wc6"
Oct 30 19:47:29.874467 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:47:29.874418    4892 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wbdcr\" (UniqueName: \"kubernetes.io/projected/f7e053ee-836a-4b0e-b2d9-88e3af83ca7b-kube-api-access-wbdcr\") pod \"payload-pre-processing-e2e-mt-dd9b8d79-867946f667-plb8n\" (UID: \"f7e053ee-836a-4b0e-b2d9-88e3af83ca7b\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-dd9b8d79-867946f667-plb8n"
Oct 30 19:47:29.875179 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:47:29.875154    4892 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f7e053ee-836a-4b0e-b2d9-88e3af83ca7b-plugins-config-volume\") pod \"payload-pre-processing-e2e-mt-dd9b8d79-867946f667-plb8n\" (UID: \"f7e053ee-836a-4b0e-b2d9-88e3af83ca7b\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-dd9b8d79-867946f667-plb8n"
Oct 30 19:47:29.941514 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:47:29.941488    4892 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wbdcr\" (UniqueName: \"kubernetes.io/projected/f7e053ee-836a-4b0e-b2d9-88e3af83ca7b-kube-api-access-wbdcr\") pod \"payload-pre-processing-e2e-mt-dd9b8d79-867946f667-plb8n\" (UID: \"f7e053ee-836a-4b0e-b2d9-88e3af83ca7b\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-dd9b8d79-867946f667-plb8n"
Oct 30 19:47:29.976008 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:47:29.975955    4892 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3c568859-9035-498d-b347-170ab8d1b2e7-plugins-config-volume\") pod \"payload-processing-e2e-mt-dd9b8d79-67c8d6d46-m4wc6\" (UID: \"3c568859-9035-498d-b347-170ab8d1b2e7\") " pod="openshift-ingress/payload-processing-e2e-mt-dd9b8d79-67c8d6d46-m4wc6"
Oct 30 19:47:29.976179 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:47:29.976160    4892 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-f9hcn\" (UniqueName: \"kubernetes.io/projected/3c568859-9035-498d-b347-170ab8d1b2e7-kube-api-access-f9hcn\") pod \"payload-processing-e2e-mt-dd9b8d79-67c8d6d46-m4wc6\" (UID: \"3c568859-9035-498d-b347-170ab8d1b2e7\") " pod="openshift-ingress/payload-processing-e2e-mt-dd9b8d79-67c8d6d46-m4wc6"
Oct 30 19:47:29.976762 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:47:29.976742    4892 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3c568859-9035-498d-b347-170ab8d1b2e7-plugins-config-volume\") pod \"payload-processing-e2e-mt-dd9b8d79-67c8d6d46-m4wc6\" (UID: \"3c568859-9035-498d-b347-170ab8d1b2e7\") " pod="openshift-ingress/payload-processing-e2e-mt-dd9b8d79-67c8d6d46-m4wc6"
Oct 30 19:47:30.035756 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:47:30.035724    4892 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-f9hcn\" (UniqueName: \"kubernetes.io/projected/3c568859-9035-498d-b347-170ab8d1b2e7-kube-api-access-f9hcn\") pod \"payload-processing-e2e-mt-dd9b8d79-67c8d6d46-m4wc6\" (UID: \"3c568859-9035-498d-b347-170ab8d1b2e7\") " pod="openshift-ingress/payload-processing-e2e-mt-dd9b8d79-67c8d6d46-m4wc6"
Oct 30 19:47:30.057562 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:47:30.057531    4892 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-dd9b8d79-867946f667-plb8n"
Oct 30 19:47:30.065246 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:47:30.065224    4892 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-dd9b8d79-67c8d6d46-m4wc6"
Oct 30 19:47:30.448009 ip-10-0-132-82 kubenswrapper[4892]: W1030 19:47:30.447980    4892 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf7e053ee_836a_4b0e_b2d9_88e3af83ca7b.slice/crio-2d9c145d33e835327bdcd0a2ad4b23ee4165cd350b5d8807cf3a90504ffa99f0 WatchSource:0}: Error finding container 2d9c145d33e835327bdcd0a2ad4b23ee4165cd350b5d8807cf3a90504ffa99f0: Status 404 returned error can't find the container with id 2d9c145d33e835327bdcd0a2ad4b23ee4165cd350b5d8807cf3a90504ffa99f0
Oct 30 19:47:30.449321 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:47:30.449286    4892 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-dd9b8d79-867946f667-plb8n"]
Oct 30 19:47:30.462326 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:47:30.462297    4892 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-dd9b8d79-867946f667-plb8n" event={"ID":"f7e053ee-836a-4b0e-b2d9-88e3af83ca7b","Type":"ContainerStarted","Data":"2d9c145d33e835327bdcd0a2ad4b23ee4165cd350b5d8807cf3a90504ffa99f0"}
Oct 30 19:47:30.498347 ip-10-0-132-82 kubenswrapper[4892]: W1030 19:47:30.498283    4892 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3c568859_9035_498d_b347_170ab8d1b2e7.slice/crio-6f9e3fca6bfd90c98163099996b6566e8093a6c945018767e05b026c2cc89076 WatchSource:0}: Error finding container 6f9e3fca6bfd90c98163099996b6566e8093a6c945018767e05b026c2cc89076: Status 404 returned error can't find the container with id 6f9e3fca6bfd90c98163099996b6566e8093a6c945018767e05b026c2cc89076
Oct 30 19:47:30.498548 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:47:30.498526    4892 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-dd9b8d79-67c8d6d46-m4wc6"]
Oct 30 19:47:31.473060 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:47:31.473021    4892 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-dd9b8d79-67c8d6d46-m4wc6" event={"ID":"3c568859-9035-498d-b347-170ab8d1b2e7","Type":"ContainerStarted","Data":"37908c15ba0247793b89d486084e09149301e2174a8dd2d1818e4daa1fdf60ae"}
Oct 30 19:47:31.473528 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:47:31.473131    4892 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-dd9b8d79-67c8d6d46-m4wc6" event={"ID":"3c568859-9035-498d-b347-170ab8d1b2e7","Type":"ContainerStarted","Data":"6f9e3fca6bfd90c98163099996b6566e8093a6c945018767e05b026c2cc89076"}
Oct 30 19:47:31.473528 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:47:31.473309    4892 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-mt-dd9b8d79-67c8d6d46-m4wc6"
Oct 30 19:47:31.474995 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:47:31.474952    4892 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-dd9b8d79-867946f667-plb8n" event={"ID":"f7e053ee-836a-4b0e-b2d9-88e3af83ca7b","Type":"ContainerStarted","Data":"14decd769a728ad5e92f5c606283d140be79657786e6905ffd6151f105b2f1fd"}
Oct 30 19:47:31.475109 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:47:31.475020    4892 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-mt-dd9b8d79-867946f667-plb8n"
Oct 30 19:47:31.612269 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:47:31.612210    4892 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-mt-dd9b8d79-67c8d6d46-m4wc6" podStartSLOduration=2.6121973990000003 podStartE2EDuration="2.612197399s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:47:29 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:47:31.608900709 +0000 UTC m=+1899.179366310" watchObservedRunningTime="2026-10-30 19:47:31.612197399 +0000 UTC m=+1899.182663030"
Oct 30 19:47:33.497340 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:47:33.496028    4892 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-mt-dd9b8d79-867946f667-plb8n" podStartSLOduration=4.4960002150000005 podStartE2EDuration="4.496000215s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:47:29 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:47:31.763646727 +0000 UTC m=+1899.334112327" watchObservedRunningTime="2026-10-30 19:47:33.496000215 +0000 UTC m=+1901.066465837"
Oct 30 19:47:33.512991 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:47:33.499184    4892 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-7c2ecd3d-pft4g"]
Oct 30 19:47:33.518996 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:47:33.516229    4892 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7c2ecd3d-pft4g"
Oct 30 19:47:33.577989 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:47:33.575652    4892 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 30 19:47:33.582899 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:47:33.578842    4892 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-7c2ecd3d-pft4g"]
Oct 30 19:47:33.646016 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:47:33.643878    4892 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 19:47:33.646016 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:47:33.644042    4892 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 30 19:47:33.646016 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:47:33.644701    4892 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-nlxp4\""
Oct 30 19:47:33.646529 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:47:33.646504    4892 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rm82k\" (UniqueName: \"kubernetes.io/projected/e21b494a-695e-4dd9-a55c-fa48b8f0957b-kube-api-access-rm82k\") pod \"maas-api-revoke-sub-7c2ecd3d-pft4g\" (UID: \"e21b494a-695e-4dd9-a55c-fa48b8f0957b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7c2ecd3d-pft4g"
Oct 30 19:47:33.646663 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:47:33.646642    4892 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e21b494a-695e-4dd9-a55c-fa48b8f0957b-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-7c2ecd3d-pft4g\" (UID: \"e21b494a-695e-4dd9-a55c-fa48b8f0957b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7c2ecd3d-pft4g"
Oct 30 19:47:33.646761 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:47:33.646745    4892 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e21b494a-695e-4dd9-a55c-fa48b8f0957b-maas-api-xks-ca\") pod \"maas-api-revoke-sub-7c2ecd3d-pft4g\" (UID: \"e21b494a-695e-4dd9-a55c-fa48b8f0957b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7c2ecd3d-pft4g"
Oct 30 19:47:33.747747 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:47:33.747666    4892 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rm82k\" (UniqueName: \"kubernetes.io/projected/e21b494a-695e-4dd9-a55c-fa48b8f0957b-kube-api-access-rm82k\") pod \"maas-api-revoke-sub-7c2ecd3d-pft4g\" (UID: \"e21b494a-695e-4dd9-a55c-fa48b8f0957b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7c2ecd3d-pft4g"
Oct 30 19:47:33.747936 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:47:33.747833    4892 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e21b494a-695e-4dd9-a55c-fa48b8f0957b-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-7c2ecd3d-pft4g\" (UID: \"e21b494a-695e-4dd9-a55c-fa48b8f0957b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7c2ecd3d-pft4g"
Oct 30 19:47:33.748030 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:47:33.747950    4892 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e21b494a-695e-4dd9-a55c-fa48b8f0957b-maas-api-xks-ca\") pod \"maas-api-revoke-sub-7c2ecd3d-pft4g\" (UID: \"e21b494a-695e-4dd9-a55c-fa48b8f0957b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7c2ecd3d-pft4g"
Oct 30 19:47:33.748620 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:47:33.748593    4892 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e21b494a-695e-4dd9-a55c-fa48b8f0957b-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-7c2ecd3d-pft4g\" (UID: \"e21b494a-695e-4dd9-a55c-fa48b8f0957b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7c2ecd3d-pft4g"
Oct 30 19:47:33.750799 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:47:33.750777    4892 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e21b494a-695e-4dd9-a55c-fa48b8f0957b-maas-api-xks-ca\") pod \"maas-api-revoke-sub-7c2ecd3d-pft4g\" (UID: \"e21b494a-695e-4dd9-a55c-fa48b8f0957b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7c2ecd3d-pft4g"
Oct 30 19:47:33.804315 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:47:33.804290    4892 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rm82k\" (UniqueName: \"kubernetes.io/projected/e21b494a-695e-4dd9-a55c-fa48b8f0957b-kube-api-access-rm82k\") pod \"maas-api-revoke-sub-7c2ecd3d-pft4g\" (UID: \"e21b494a-695e-4dd9-a55c-fa48b8f0957b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7c2ecd3d-pft4g"
Oct 30 19:47:33.883803 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:47:33.883747    4892 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7c2ecd3d-pft4g"
Oct 30 19:47:34.268161 ip-10-0-132-82 kubenswrapper[4892]: W1030 19:47:34.268126    4892 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode21b494a_695e_4dd9_a55c_fa48b8f0957b.slice/crio-37f77a56ae471a0c7d23187e6651f94ce26623887f58acc38e668decbfd281f6 WatchSource:0}: Error finding container 37f77a56ae471a0c7d23187e6651f94ce26623887f58acc38e668decbfd281f6: Status 404 returned error can't find the container with id 37f77a56ae471a0c7d23187e6651f94ce26623887f58acc38e668decbfd281f6
Oct 30 19:47:34.268347 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:47:34.268326    4892 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-7c2ecd3d-pft4g"]
Oct 30 19:47:34.632926 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:47:34.632894    4892 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7c2ecd3d-pft4g" event={"ID":"e21b494a-695e-4dd9-a55c-fa48b8f0957b","Type":"ContainerStarted","Data":"3828be61c54deca37236da277e803386906529cc06b5aafd70d7f10a0f4a9e2f"}
Oct 30 19:47:34.633346 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:47:34.632949    4892 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7c2ecd3d-pft4g" event={"ID":"e21b494a-695e-4dd9-a55c-fa48b8f0957b","Type":"ContainerStarted","Data":"37f77a56ae471a0c7d23187e6651f94ce26623887f58acc38e668decbfd281f6"}
Oct 30 19:47:34.753004 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:47:34.750369    4892 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7c2ecd3d-pft4g" podStartSLOduration=1.750352705 podStartE2EDuration="1.750352705s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:47:33 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:47:34.745496735 +0000 UTC m=+1902.315962369" watchObservedRunningTime="2026-10-30 19:47:34.750352705 +0000 UTC m=+1902.320818310"
Oct 30 19:47:35.643672 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:47:35.643634    4892 generic.go:356] "Generic (PLEG): container finished" podID="e21b494a-695e-4dd9-a55c-fa48b8f0957b" containerID="3828be61c54deca37236da277e803386906529cc06b5aafd70d7f10a0f4a9e2f" exitCode=0
Oct 30 19:47:35.644106 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:47:35.643713    4892 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7c2ecd3d-pft4g" event={"ID":"e21b494a-695e-4dd9-a55c-fa48b8f0957b","Type":"ContainerDied","Data":"3828be61c54deca37236da277e803386906529cc06b5aafd70d7f10a0f4a9e2f"}
Oct 30 19:47:36.846514 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:47:36.846487    4892 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7c2ecd3d-pft4g"
Oct 30 19:47:36.917480 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:47:36.917413    4892 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e21b494a-695e-4dd9-a55c-fa48b8f0957b-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e21b494a-695e-4dd9-a55c-fa48b8f0957b-maas-api-openshift-ca\") pod \"e21b494a-695e-4dd9-a55c-fa48b8f0957b\" (UID: \"e21b494a-695e-4dd9-a55c-fa48b8f0957b\") "
Oct 30 19:47:36.917480 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:47:36.917467    4892 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e21b494a-695e-4dd9-a55c-fa48b8f0957b-kube-api-access-rm82k\" (UniqueName: \"kubernetes.io/projected/e21b494a-695e-4dd9-a55c-fa48b8f0957b-kube-api-access-rm82k\") pod \"e21b494a-695e-4dd9-a55c-fa48b8f0957b\" (UID: \"e21b494a-695e-4dd9-a55c-fa48b8f0957b\") "
Oct 30 19:47:36.917667 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:47:36.917503    4892 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/e21b494a-695e-4dd9-a55c-fa48b8f0957b-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e21b494a-695e-4dd9-a55c-fa48b8f0957b-maas-api-xks-ca\") pod \"e21b494a-695e-4dd9-a55c-fa48b8f0957b\" (UID: \"e21b494a-695e-4dd9-a55c-fa48b8f0957b\") "
Oct 30 19:47:36.917795 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:47:36.917773    4892 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e21b494a-695e-4dd9-a55c-fa48b8f0957b-maas-api-openshift-ca" pod "e21b494a-695e-4dd9-a55c-fa48b8f0957b" (UID: "e21b494a-695e-4dd9-a55c-fa48b8f0957b"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 19:47:36.918135 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:47:36.918117    4892 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e21b494a-695e-4dd9-a55c-fa48b8f0957b-maas-api-openshift-ca\") on node \"ip-10-0-132-82.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 19:47:36.919718 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:47:36.919695    4892 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e21b494a-695e-4dd9-a55c-fa48b8f0957b-maas-api-xks-ca" pod "e21b494a-695e-4dd9-a55c-fa48b8f0957b" (UID: "e21b494a-695e-4dd9-a55c-fa48b8f0957b"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 19:47:36.919834 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:47:36.919704    4892 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e21b494a-695e-4dd9-a55c-fa48b8f0957b-kube-api-access-rm82k" pod "e21b494a-695e-4dd9-a55c-fa48b8f0957b" (UID: "e21b494a-695e-4dd9-a55c-fa48b8f0957b"). InnerVolumeSpecName "kube-api-access-rm82k". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:47:37.020150 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:47:37.020110    4892 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rm82k\" (UniqueName: \"kubernetes.io/projected/e21b494a-695e-4dd9-a55c-fa48b8f0957b-kube-api-access-rm82k\") on node \"ip-10-0-132-82.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 19:47:37.020313 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:47:37.020154    4892 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e21b494a-695e-4dd9-a55c-fa48b8f0957b-maas-api-xks-ca\") on node \"ip-10-0-132-82.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 19:47:37.672259 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:47:37.672228    4892 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7c2ecd3d-pft4g"
Oct 30 19:47:37.672433 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:47:37.672259    4892 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7c2ecd3d-pft4g" event={"ID":"e21b494a-695e-4dd9-a55c-fa48b8f0957b","Type":"ContainerDied","Data":"37f77a56ae471a0c7d23187e6651f94ce26623887f58acc38e668decbfd281f6"}
Oct 30 19:47:37.672433 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:47:37.672322    4892 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="37f77a56ae471a0c7d23187e6651f94ce26623887f58acc38e668decbfd281f6"
Oct 30 19:47:39.061911 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:47:39.061875    4892 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5c8ee9bf-j4wgj"]
Oct 30 19:47:39.103143 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:47:39.103106    4892 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5c8ee9bf-j4wgj"]
Oct 30 19:47:39.581989 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:47:39.569849    4892 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w0-6ac304-79859bfff5-zx2hb"]
Oct 30 19:47:39.581989 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:47:39.570872    4892 cpu_manager.go:414] "containerMap: removing container" podUID="e21b494a-695e-4dd9-a55c-fa48b8f0957b" containerName="revoke-keys"
Oct 30 19:47:39.581989 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:47:39.570892    4892 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e21b494a-695e-4dd9-a55c-fa48b8f0957b" containerName="revoke-keys"
Oct 30 19:47:39.581989 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:47:39.574335    4892 memory_manager.go:365] "RemoveStaleState removing state" podUID="e21b494a-695e-4dd9-a55c-fa48b8f0957b" containerName="revoke-keys"
Oct 30 19:47:39.637000 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:47:39.631939    4892 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w0-6ac304-79859bfff5-zx2hb"
Oct 30 19:47:39.751004 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:47:39.736360    4892 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f1d57e1b-e316-48ac-a1a5-a9520a9e068e-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-b-w0-6ac304-79859bfff5-zx2hb\" (UID: \"f1d57e1b-e316-48ac-a1a5-a9520a9e068e\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w0-6ac304-79859bfff5-zx2hb"
Oct 30 19:47:39.751004 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:47:39.736578    4892 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4qqhq\" (UniqueName: \"kubernetes.io/projected/f1d57e1b-e316-48ac-a1a5-a9520a9e068e-kube-api-access-4qqhq\") pod \"payload-pre-processing-e2e-shared-b-w0-6ac304-79859bfff5-zx2hb\" (UID: \"f1d57e1b-e316-48ac-a1a5-a9520a9e068e\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w0-6ac304-79859bfff5-zx2hb"
Oct 30 19:47:39.765999 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:47:39.751993    4892 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-shared-b-w0-6ac304\""
Oct 30 19:47:39.765999 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:47:39.752094    4892 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-shared-b-w0-6ac304-dockercfg-4n76h\""
Oct 30 19:47:39.924991 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:47:39.855088    4892 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4qqhq\" (UniqueName: \"kubernetes.io/projected/f1d57e1b-e316-48ac-a1a5-a9520a9e068e-kube-api-access-4qqhq\") pod \"payload-pre-processing-e2e-shared-b-w0-6ac304-79859bfff5-zx2hb\" (UID: \"f1d57e1b-e316-48ac-a1a5-a9520a9e068e\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w0-6ac304-79859bfff5-zx2hb"
Oct 30 19:47:39.948108 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:47:39.947126    4892 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f1d57e1b-e316-48ac-a1a5-a9520a9e068e-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-b-w0-6ac304-79859bfff5-zx2hb\" (UID: \"f1d57e1b-e316-48ac-a1a5-a9520a9e068e\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w0-6ac304-79859bfff5-zx2hb"
Oct 30 19:47:39.948386 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:47:39.948343    4892 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f1d57e1b-e316-48ac-a1a5-a9520a9e068e-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-b-w0-6ac304-79859bfff5-zx2hb\" (UID: \"f1d57e1b-e316-48ac-a1a5-a9520a9e068e\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w0-6ac304-79859bfff5-zx2hb"
Oct 30 19:47:40.022988 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:47:40.010811    4892 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4qqhq\" (UniqueName: \"kubernetes.io/projected/f1d57e1b-e316-48ac-a1a5-a9520a9e068e-kube-api-access-4qqhq\") pod \"payload-pre-processing-e2e-shared-b-w0-6ac304-79859bfff5-zx2hb\" (UID: \"f1d57e1b-e316-48ac-a1a5-a9520a9e068e\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w0-6ac304-79859bfff5-zx2hb"
Oct 30 19:47:40.087031 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:47:40.086989    4892 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w0-6ac304-79859bfff5-zx2hb"]
Oct 30 19:47:40.201052 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:47:40.198744    4892 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w0-6ac304-846999954d-tq2ft"]
Oct 30 19:47:40.205929 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:47:40.205331    4892 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w0-6ac304-846999954d-tq2ft"
Oct 30 19:47:40.225267 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:47:40.225238    4892 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w0-6ac304-846999954d-tq2ft"]
Oct 30 19:47:40.290581 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:47:40.290544    4892 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w0-6ac304-79859bfff5-zx2hb"
Oct 30 19:47:40.383022 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:47:40.382508    4892 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cs7ph\" (UniqueName: \"kubernetes.io/projected/12863020-d0ea-419d-aba2-816627ab0431-kube-api-access-cs7ph\") pod \"payload-processing-e2e-shared-b-w0-6ac304-846999954d-tq2ft\" (UID: \"12863020-d0ea-419d-aba2-816627ab0431\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w0-6ac304-846999954d-tq2ft"
Oct 30 19:47:40.383022 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:47:40.382663    4892 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/12863020-d0ea-419d-aba2-816627ab0431-plugins-config-volume\") pod \"payload-processing-e2e-shared-b-w0-6ac304-846999954d-tq2ft\" (UID: \"12863020-d0ea-419d-aba2-816627ab0431\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w0-6ac304-846999954d-tq2ft"
Oct 30 19:47:40.484830 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:47:40.484746    4892 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cs7ph\" (UniqueName: \"kubernetes.io/projected/12863020-d0ea-419d-aba2-816627ab0431-kube-api-access-cs7ph\") pod \"payload-processing-e2e-shared-b-w0-6ac304-846999954d-tq2ft\" (UID: \"12863020-d0ea-419d-aba2-816627ab0431\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w0-6ac304-846999954d-tq2ft"
Oct 30 19:47:40.485014 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:47:40.484906    4892 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/12863020-d0ea-419d-aba2-816627ab0431-plugins-config-volume\") pod \"payload-processing-e2e-shared-b-w0-6ac304-846999954d-tq2ft\" (UID: \"12863020-d0ea-419d-aba2-816627ab0431\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w0-6ac304-846999954d-tq2ft"
Oct 30 19:47:40.485551 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:47:40.485533    4892 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/12863020-d0ea-419d-aba2-816627ab0431-plugins-config-volume\") pod \"payload-processing-e2e-shared-b-w0-6ac304-846999954d-tq2ft\" (UID: \"12863020-d0ea-419d-aba2-816627ab0431\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w0-6ac304-846999954d-tq2ft"
Oct 30 19:47:40.541629 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:47:40.541605    4892 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-cs7ph\" (UniqueName: \"kubernetes.io/projected/12863020-d0ea-419d-aba2-816627ab0431-kube-api-access-cs7ph\") pod \"payload-processing-e2e-shared-b-w0-6ac304-846999954d-tq2ft\" (UID: \"12863020-d0ea-419d-aba2-816627ab0431\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w0-6ac304-846999954d-tq2ft"
Oct 30 19:47:40.574204 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:47:40.574174    4892 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="7291702b-9653-4a58-a2c2-9c67afd8ed0a" path="/var/lib/kubelet/pods/7291702b-9653-4a58-a2c2-9c67afd8ed0a/volumes"
Oct 30 19:47:40.686646 ip-10-0-132-82 kubenswrapper[4892]: W1030 19:47:40.686615    4892 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf1d57e1b_e316_48ac_a1a5_a9520a9e068e.slice/crio-855f82b370d62be0b08f4ede7f7d9b77357e518e814795500de05e18d379a3e4 WatchSource:0}: Error finding container 855f82b370d62be0b08f4ede7f7d9b77357e518e814795500de05e18d379a3e4: Status 404 returned error can't find the container with id 855f82b370d62be0b08f4ede7f7d9b77357e518e814795500de05e18d379a3e4
Oct 30 19:47:40.687088 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:47:40.687063    4892 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w0-6ac304-79859bfff5-zx2hb"]
Oct 30 19:47:40.814048 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:47:40.814021    4892 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w0-6ac304-79859bfff5-zx2hb" event={"ID":"f1d57e1b-e316-48ac-a1a5-a9520a9e068e","Type":"ContainerStarted","Data":"45e7e4c2c487385d9b7c19a9b2ffafa7dea4d76bc721247e55757afa40a29171"}
Oct 30 19:47:40.814180 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:47:40.814067    4892 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w0-6ac304-79859bfff5-zx2hb" event={"ID":"f1d57e1b-e316-48ac-a1a5-a9520a9e068e","Type":"ContainerStarted","Data":"855f82b370d62be0b08f4ede7f7d9b77357e518e814795500de05e18d379a3e4"}
Oct 30 19:47:40.814180 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:47:40.814138    4892 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w0-6ac304-79859bfff5-zx2hb"
Oct 30 19:47:40.824424 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:47:40.824408    4892 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w0-6ac304-846999954d-tq2ft"
Oct 30 19:47:40.931546 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:47:40.931488    4892 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w0-6ac304-79859bfff5-zx2hb" podStartSLOduration=1.931469257 podStartE2EDuration="1.931469257s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:47:39 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:47:40.925852906 +0000 UTC m=+1908.496318506" watchObservedRunningTime="2026-10-30 19:47:40.931469257 +0000 UTC m=+1908.501934857"
Oct 30 19:47:41.241240 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:47:41.241212    4892 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w0-6ac304-846999954d-tq2ft"]
Oct 30 19:47:41.824607 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:47:41.824573    4892 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w0-6ac304-846999954d-tq2ft" event={"ID":"12863020-d0ea-419d-aba2-816627ab0431","Type":"ContainerStarted","Data":"52a6070c0003df14be74d3b52ca40a184ef6a0d76edc4de151480c1d25789884"}
Oct 30 19:47:41.824788 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:47:41.824631    4892 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w0-6ac304-846999954d-tq2ft" event={"ID":"12863020-d0ea-419d-aba2-816627ab0431","Type":"ContainerStarted","Data":"e67a3656d34f7514aec2aef0e041e24f35c31ff4ca2c95f2c91ff0849610a2db"}
Oct 30 19:47:41.824788 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:47:41.824657    4892 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-shared-b-w0-6ac304-846999954d-tq2ft"
Oct 30 19:47:41.944258 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:47:41.944202    4892 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-shared-b-w0-6ac304-846999954d-tq2ft" podStartSLOduration=2.943393821 podStartE2EDuration="2.943393821s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:47:39 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:47:41.936748301 +0000 UTC m=+1909.507213901" watchObservedRunningTime="2026-10-30 19:47:41.943393821 +0000 UTC m=+1909.513859422"
Oct 30 19:47:42.485671 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:47:42.485645    4892 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-mt-dd9b8d79-867946f667-plb8n"
Oct 30 19:47:42.486078 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:47:42.485708    4892 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-mt-dd9b8d79-67c8d6d46-m4wc6"
Oct 30 19:47:43.077990 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:47:43.064121    4892 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f64d0dcb-2whv4"]
Oct 30 19:47:43.077990 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:47:43.071907    4892 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f64d0dcb-2whv4"]
Oct 30 19:47:43.649230 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:47:43.641605    4892 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-mt-5bf13ff3-openshift-default-7558bbbc65-9j7w6"]
Oct 30 19:47:43.649230 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:47:43.648987    4892 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-5bf13ff3-openshift-default-7558bbbc65-9j7w6"
Oct 30 19:47:43.788280 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:47:43.782528    4892 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-mt-5bf13ff3-openshift-default-7558bbbc65-9j7w6"]
Oct 30 19:47:43.818054 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:47:43.816089    4892 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-mt-5bf13ff3-openshift-default-dockercfg-tq4cl\""
Oct 30 19:47:43.878990 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:47:43.877936    4892 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/474b0b88-a062-4ee2-9994-022ae1253f28-workload-socket\") pod \"e2e-mt-5bf13ff3-openshift-default-7558bbbc65-9j7w6\" (UID: \"474b0b88-a062-4ee2-9994-022ae1253f28\") " pod="openshift-ingress/e2e-mt-5bf13ff3-openshift-default-7558bbbc65-9j7w6"
Oct 30 19:47:43.878990 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:47:43.878047    4892 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/474b0b88-a062-4ee2-9994-022ae1253f28-istio-data\") pod \"e2e-mt-5bf13ff3-openshift-default-7558bbbc65-9j7w6\" (UID: \"474b0b88-a062-4ee2-9994-022ae1253f28\") " pod="openshift-ingress/e2e-mt-5bf13ff3-openshift-default-7558bbbc65-9j7w6"
Oct 30 19:47:43.878990 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:47:43.878185    4892 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/474b0b88-a062-4ee2-9994-022ae1253f28-istio-envoy\") pod \"e2e-mt-5bf13ff3-openshift-default-7558bbbc65-9j7w6\" (UID: \"474b0b88-a062-4ee2-9994-022ae1253f28\") " pod="openshift-ingress/e2e-mt-5bf13ff3-openshift-default-7558bbbc65-9j7w6"
Oct 30 19:47:43.878990 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:47:43.878351    4892 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5vffk\" (UniqueName: \"kubernetes.io/projected/474b0b88-a062-4ee2-9994-022ae1253f28-kube-api-access-5vffk\") pod \"e2e-mt-5bf13ff3-openshift-default-7558bbbc65-9j7w6\" (UID: \"474b0b88-a062-4ee2-9994-022ae1253f28\") " pod="openshift-ingress/e2e-mt-5bf13ff3-openshift-default-7558bbbc65-9j7w6"
Oct 30 19:47:43.878990 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:47:43.878416    4892 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/474b0b88-a062-4ee2-9994-022ae1253f28-istiod-ca-cert\") pod \"e2e-mt-5bf13ff3-openshift-default-7558bbbc65-9j7w6\" (UID: \"474b0b88-a062-4ee2-9994-022ae1253f28\") " pod="openshift-ingress/e2e-mt-5bf13ff3-openshift-default-7558bbbc65-9j7w6"
Oct 30 19:47:43.878990 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:47:43.878463    4892 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/474b0b88-a062-4ee2-9994-022ae1253f28-credential-socket\") pod \"e2e-mt-5bf13ff3-openshift-default-7558bbbc65-9j7w6\" (UID: \"474b0b88-a062-4ee2-9994-022ae1253f28\") " pod="openshift-ingress/e2e-mt-5bf13ff3-openshift-default-7558bbbc65-9j7w6"
Oct 30 19:47:43.878990 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:47:43.878557    4892 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/474b0b88-a062-4ee2-9994-022ae1253f28-istio-token\") pod \"e2e-mt-5bf13ff3-openshift-default-7558bbbc65-9j7w6\" (UID: \"474b0b88-a062-4ee2-9994-022ae1253f28\") " pod="openshift-ingress/e2e-mt-5bf13ff3-openshift-default-7558bbbc65-9j7w6"
Oct 30 19:47:43.878990 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:47:43.878586    4892 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/474b0b88-a062-4ee2-9994-022ae1253f28-istio-podinfo\") pod \"e2e-mt-5bf13ff3-openshift-default-7558bbbc65-9j7w6\" (UID: \"474b0b88-a062-4ee2-9994-022ae1253f28\") " pod="openshift-ingress/e2e-mt-5bf13ff3-openshift-default-7558bbbc65-9j7w6"
Oct 30 19:47:43.878990 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:47:43.878651    4892 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/474b0b88-a062-4ee2-9994-022ae1253f28-workload-certs\") pod \"e2e-mt-5bf13ff3-openshift-default-7558bbbc65-9j7w6\" (UID: \"474b0b88-a062-4ee2-9994-022ae1253f28\") " pod="openshift-ingress/e2e-mt-5bf13ff3-openshift-default-7558bbbc65-9j7w6"
Oct 30 19:47:44.003244 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:47:44.002228    4892 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5vffk\" (UniqueName: \"kubernetes.io/projected/474b0b88-a062-4ee2-9994-022ae1253f28-kube-api-access-5vffk\") pod \"e2e-mt-5bf13ff3-openshift-default-7558bbbc65-9j7w6\" (UID: \"474b0b88-a062-4ee2-9994-022ae1253f28\") " pod="openshift-ingress/e2e-mt-5bf13ff3-openshift-default-7558bbbc65-9j7w6"
Oct 30 19:47:44.003244 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:47:44.002314    4892 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/474b0b88-a062-4ee2-9994-022ae1253f28-istiod-ca-cert\") pod \"e2e-mt-5bf13ff3-openshift-default-7558bbbc65-9j7w6\" (UID: \"474b0b88-a062-4ee2-9994-022ae1253f28\") " pod="openshift-ingress/e2e-mt-5bf13ff3-openshift-default-7558bbbc65-9j7w6"
Oct 30 19:47:44.003244 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:47:44.002362    4892 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/474b0b88-a062-4ee2-9994-022ae1253f28-credential-socket\") pod \"e2e-mt-5bf13ff3-openshift-default-7558bbbc65-9j7w6\" (UID: \"474b0b88-a062-4ee2-9994-022ae1253f28\") " pod="openshift-ingress/e2e-mt-5bf13ff3-openshift-default-7558bbbc65-9j7w6"
Oct 30 19:47:44.003244 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:47:44.002452    4892 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/474b0b88-a062-4ee2-9994-022ae1253f28-istio-token\") pod \"e2e-mt-5bf13ff3-openshift-default-7558bbbc65-9j7w6\" (UID: \"474b0b88-a062-4ee2-9994-022ae1253f28\") " pod="openshift-ingress/e2e-mt-5bf13ff3-openshift-default-7558bbbc65-9j7w6"
Oct 30 19:47:44.003244 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:47:44.002480    4892 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/474b0b88-a062-4ee2-9994-022ae1253f28-istio-podinfo\") pod \"e2e-mt-5bf13ff3-openshift-default-7558bbbc65-9j7w6\" (UID: \"474b0b88-a062-4ee2-9994-022ae1253f28\") " pod="openshift-ingress/e2e-mt-5bf13ff3-openshift-default-7558bbbc65-9j7w6"
Oct 30 19:47:44.003244 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:47:44.002558    4892 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/474b0b88-a062-4ee2-9994-022ae1253f28-workload-certs\") pod \"e2e-mt-5bf13ff3-openshift-default-7558bbbc65-9j7w6\" (UID: \"474b0b88-a062-4ee2-9994-022ae1253f28\") " pod="openshift-ingress/e2e-mt-5bf13ff3-openshift-default-7558bbbc65-9j7w6"
Oct 30 19:47:44.003244 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:47:44.002624    4892 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/474b0b88-a062-4ee2-9994-022ae1253f28-workload-socket\") pod \"e2e-mt-5bf13ff3-openshift-default-7558bbbc65-9j7w6\" (UID: \"474b0b88-a062-4ee2-9994-022ae1253f28\") " pod="openshift-ingress/e2e-mt-5bf13ff3-openshift-default-7558bbbc65-9j7w6"
Oct 30 19:47:44.003244 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:47:44.002654    4892 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/474b0b88-a062-4ee2-9994-022ae1253f28-istio-data\") pod \"e2e-mt-5bf13ff3-openshift-default-7558bbbc65-9j7w6\" (UID: \"474b0b88-a062-4ee2-9994-022ae1253f28\") " pod="openshift-ingress/e2e-mt-5bf13ff3-openshift-default-7558bbbc65-9j7w6"
Oct 30 19:47:44.003244 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:47:44.002789    4892 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/474b0b88-a062-4ee2-9994-022ae1253f28-istio-envoy\") pod \"e2e-mt-5bf13ff3-openshift-default-7558bbbc65-9j7w6\" (UID: \"474b0b88-a062-4ee2-9994-022ae1253f28\") " pod="openshift-ingress/e2e-mt-5bf13ff3-openshift-default-7558bbbc65-9j7w6"
Oct 30 19:47:44.005040 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:47:44.005009    4892 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/474b0b88-a062-4ee2-9994-022ae1253f28-istiod-ca-cert\") pod \"e2e-mt-5bf13ff3-openshift-default-7558bbbc65-9j7w6\" (UID: \"474b0b88-a062-4ee2-9994-022ae1253f28\") " pod="openshift-ingress/e2e-mt-5bf13ff3-openshift-default-7558bbbc65-9j7w6"
Oct 30 19:47:44.005303 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:47:44.005279    4892 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/474b0b88-a062-4ee2-9994-022ae1253f28-credential-socket\") pod \"e2e-mt-5bf13ff3-openshift-default-7558bbbc65-9j7w6\" (UID: \"474b0b88-a062-4ee2-9994-022ae1253f28\") " pod="openshift-ingress/e2e-mt-5bf13ff3-openshift-default-7558bbbc65-9j7w6"
Oct 30 19:47:44.005548 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:47:44.005524    4892 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/474b0b88-a062-4ee2-9994-022ae1253f28-workload-certs\") pod \"e2e-mt-5bf13ff3-openshift-default-7558bbbc65-9j7w6\" (UID: \"474b0b88-a062-4ee2-9994-022ae1253f28\") " pod="openshift-ingress/e2e-mt-5bf13ff3-openshift-default-7558bbbc65-9j7w6"
Oct 30 19:47:44.005806 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:47:44.005781    4892 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/474b0b88-a062-4ee2-9994-022ae1253f28-workload-socket\") pod \"e2e-mt-5bf13ff3-openshift-default-7558bbbc65-9j7w6\" (UID: \"474b0b88-a062-4ee2-9994-022ae1253f28\") " pod="openshift-ingress/e2e-mt-5bf13ff3-openshift-default-7558bbbc65-9j7w6"
Oct 30 19:47:44.006109 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:47:44.006081    4892 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/474b0b88-a062-4ee2-9994-022ae1253f28-istio-data\") pod \"e2e-mt-5bf13ff3-openshift-default-7558bbbc65-9j7w6\" (UID: \"474b0b88-a062-4ee2-9994-022ae1253f28\") " pod="openshift-ingress/e2e-mt-5bf13ff3-openshift-default-7558bbbc65-9j7w6"
Oct 30 19:47:44.181034 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:47:44.169505    4892 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/474b0b88-a062-4ee2-9994-022ae1253f28-istio-token\") pod \"e2e-mt-5bf13ff3-openshift-default-7558bbbc65-9j7w6\" (UID: \"474b0b88-a062-4ee2-9994-022ae1253f28\") " pod="openshift-ingress/e2e-mt-5bf13ff3-openshift-default-7558bbbc65-9j7w6"
Oct 30 19:47:44.181034 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:47:44.172088    4892 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5vffk\" (UniqueName: \"kubernetes.io/projected/474b0b88-a062-4ee2-9994-022ae1253f28-kube-api-access-5vffk\") pod \"e2e-mt-5bf13ff3-openshift-default-7558bbbc65-9j7w6\" (UID: \"474b0b88-a062-4ee2-9994-022ae1253f28\") " pod="openshift-ingress/e2e-mt-5bf13ff3-openshift-default-7558bbbc65-9j7w6"
Oct 30 19:47:44.187995 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:47:44.181477    4892 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/474b0b88-a062-4ee2-9994-022ae1253f28-istio-envoy\") pod \"e2e-mt-5bf13ff3-openshift-default-7558bbbc65-9j7w6\" (UID: \"474b0b88-a062-4ee2-9994-022ae1253f28\") " pod="openshift-ingress/e2e-mt-5bf13ff3-openshift-default-7558bbbc65-9j7w6"
Oct 30 19:47:44.225993 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:47:44.223743    4892 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/474b0b88-a062-4ee2-9994-022ae1253f28-istio-podinfo\") pod \"e2e-mt-5bf13ff3-openshift-default-7558bbbc65-9j7w6\" (UID: \"474b0b88-a062-4ee2-9994-022ae1253f28\") " pod="openshift-ingress/e2e-mt-5bf13ff3-openshift-default-7558bbbc65-9j7w6"
Oct 30 19:47:44.321061 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:47:44.320945    4892 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-5bf13ff3-openshift-default-7558bbbc65-9j7w6"
Oct 30 19:47:44.582226 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:47:44.582128    4892 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="6350b173-e45f-43ef-8ec6-ee217764b552" path="/var/lib/kubelet/pods/6350b173-e45f-43ef-8ec6-ee217764b552/volumes"
Oct 30 19:47:44.868517 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:47:44.868491    4892 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-mt-5bf13ff3-openshift-default-7558bbbc65-9j7w6"]
Oct 30 19:47:44.869612 ip-10-0-132-82 kubenswrapper[4892]: W1030 19:47:44.869570    4892 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod474b0b88_a062_4ee2_9994_022ae1253f28.slice/crio-5fa4d52cf29ae2ba35de6aa200d4d3030384b985a2ecb8536516b0d149ef8fdf WatchSource:0}: Error finding container 5fa4d52cf29ae2ba35de6aa200d4d3030384b985a2ecb8536516b0d149ef8fdf: Status 404 returned error can't find the container with id 5fa4d52cf29ae2ba35de6aa200d4d3030384b985a2ecb8536516b0d149ef8fdf
Oct 30 19:47:44.874035 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:47:44.873312    4892 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892000Ki","pods":"250"}
Oct 30 19:47:44.874035 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:47:44.873406    4892 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892000Ki","pods":"250"}
Oct 30 19:47:44.874035 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:47:44.873472    4892 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892000Ki","pods":"250"}
Oct 30 19:47:45.017202 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:47:45.017167    4892 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-5bf13ff3-openshift-default-7558bbbc65-9j7w6" event={"ID":"474b0b88-a062-4ee2-9994-022ae1253f28","Type":"ContainerStarted","Data":"191522a582678a8a642c14a3c468d7a6d308a6de40f6275658f2e3176f86eeec"}
Oct 30 19:47:45.017390 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:47:45.017232    4892 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-5bf13ff3-openshift-default-7558bbbc65-9j7w6" event={"ID":"474b0b88-a062-4ee2-9994-022ae1253f28","Type":"ContainerStarted","Data":"5fa4d52cf29ae2ba35de6aa200d4d3030384b985a2ecb8536516b0d149ef8fdf"}
Oct 30 19:47:45.146497 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:47:45.143595    4892 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-mt-5bf13ff3-openshift-default-7558bbbc65-9j7w6" podStartSLOduration=2.143572513 podStartE2EDuration="2.143572513s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:47:43 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:47:45.132055699 +0000 UTC m=+1912.702521299" watchObservedRunningTime="2026-10-30 19:47:45.143572513 +0000 UTC m=+1912.714038113"
Oct 30 19:47:45.328987 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:47:45.323289    4892 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-mt-5bf13ff3-openshift-default-7558bbbc65-9j7w6"
Oct 30 19:47:45.328987 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:47:45.323463    4892 patch_prober.go:29] interesting pod/e2e-mt-5bf13ff3-openshift-default-7558bbbc65-9j7w6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.171:15021/healthz/ready\": dial tcp 10.132.0.171:15021: connect: connection refused" start-of-body=
Oct 30 19:47:45.328987 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:47:45.323509    4892 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-5bf13ff3-openshift-default-7558bbbc65-9j7w6" podUID="474b0b88-a062-4ee2-9994-022ae1253f28" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.171:15021/healthz/ready\": dial tcp 10.132.0.171:15021: connect: connection refused"
Oct 30 19:47:46.326646 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:47:46.326615    4892 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-mt-5bf13ff3-openshift-default-7558bbbc65-9j7w6"
Oct 30 19:47:47.038021 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:47:47.037992    4892 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-mt-5bf13ff3-openshift-default-7558bbbc65-9j7w6"
Oct 30 19:47:47.038956 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:47:47.038929    4892 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-mt-5bf13ff3-openshift-default-7558bbbc65-9j7w6"
Oct 30 19:47:47.091329 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:47:47.091306    4892 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-fsfvk"]
Oct 30 19:47:49.059991 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:47:49.056273    4892 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-9a514ead-qjp69"]
Oct 30 19:47:49.099068 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:47:49.099028    4892 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-9a514ead-qjp69"]
Oct 30 19:47:50.620991 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:47:50.619184    4892 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="29c286de-ac5b-4dea-ba6c-565c16549eb9" path="/var/lib/kubelet/pods/29c286de-ac5b-4dea-ba6c-565c16549eb9/volumes"
Oct 30 19:47:51.404991 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:47:51.401047    4892 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-b5178f/distinct-2-w4-b5178f-kserve-69ddf8b665-9pdth" podUID="fb71d961-8500-4367-b48c-88d385962bd5" containerName="main" probeResult="failure" output="Get \"https://10.132.0.90:8000/ready\": context deadline exceeded"
Oct 30 19:47:51.555992 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:47:51.555530    4892 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-5f55cd/simulator-w6-5f55cd-kserve-5ff86b7447-h9tm7" podUID="3cb47a99-8be1-4756-b31a-43b58cc6b8dc" containerName="main" probeResult="failure" output="Get \"https://10.132.0.58:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:47:51.835683 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:47:51.835498    4892 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-ada8d7/simulator-w3-ada8d7-kserve-7d575cfcb5-5txgn" podUID="10a5c7c8-6146-47b0-9643-bb137cf10bf5" containerName="main" probeResult="failure" output="Get \"https://10.132.0.55:8000/ready\": context deadline exceeded"
Oct 30 19:47:51.846094 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:47:51.843128    4892 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w0-6ac304-79859bfff5-zx2hb"
Oct 30 19:47:52.299311 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:47:52.298246    4892 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w3-ada8d7/simulator-w3-ada8d7-kserve-7d575cfcb5-5txgn" podUID="10a5c7c8-6146-47b0-9643-bb137cf10bf5" containerName="main" probeResult="failure" output="Get \"https://10.132.0.55:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:47:52.890323 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:47:52.890294    4892 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-shared-b-w0-6ac304-846999954d-tq2ft"
Oct 30 19:47:53.813617 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:47:53.813581    4892 scope.go:122] "RemoveContainer" containerID="ea31cff2af7f01d1ae598debaa94044ecd4f592d721b2f9fb1c03680fbbd0ce4"
Oct 30 19:47:53.824662 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:47:53.824637    4892 scope.go:122] "RemoveContainer" containerID="8fe8c13c1c21d567999b067fc5db434423acabd867d95ed42721a952e4a1bf0c"
Oct 30 19:47:53.866163 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:47:53.866121    4892 scope.go:122] "RemoveContainer" containerID="ea66e3dd065608341cb2d2078371492a8e29e4a87a8186af9bfbb628c054625a"
Oct 30 19:47:53.880952 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:47:53.880932    4892 scope.go:122] "RemoveContainer" containerID="76f9ef7be90050521c9d8afc8f22c2ade61432de01a3e692c2c9472b84b9b37d"
Oct 30 19:47:53.893726 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:47:53.893707    4892 scope.go:122] "RemoveContainer" containerID="4a9bd6c20c8e83e9f8336d1780cb332f7c0ec994b21816fc3647492dedba1fe4"
Oct 30 19:47:53.905073 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:47:53.905047    4892 scope.go:122] "RemoveContainer" containerID="9e7c61fd70be68775d576d62909ff5f820377f78eec5bd232c62e01619567652"
Oct 30 19:47:53.915828 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:47:53.915812    4892 scope.go:122] "RemoveContainer" containerID="baddb70af190f55295bf86157eeafe35e4d47dd9bfd69b16b77a34deaf496f45"
Oct 30 19:47:53.925722 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:47:53.925704    4892 scope.go:122] "RemoveContainer" containerID="a1a9300f93a1c00790cb4fb07f5030fda406b554b943efe483ca4378a987bd60"
Oct 30 19:47:53.972190 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:47:53.972162    4892 scope.go:122] "RemoveContainer" containerID="9ed7b544758691d118d999a1c4cfb63507a634c6e25c3011fde791f7c05092d9"
Oct 30 19:47:54.873218 ip-10-0-132-82 kubenswrapper[4892]: E1030 19:47:54.873171    4892 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-jrm87" containerName="extractor"
Oct 30 19:47:58.054563 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:47:58.054518    4892 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-981da6fa-ff42h"]
Oct 30 19:47:58.101229 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:47:58.101200    4892 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-981da6fa-ff42h"]
Oct 30 19:47:58.574857 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:47:58.574821    4892 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="dfe23e43-6b96-4557-90f0-10f4bf7866e7" path="/var/lib/kubelet/pods/dfe23e43-6b96-4557-90f0-10f4bf7866e7/volumes"
Oct 30 19:48:00.052071 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:48:00.051923    4892 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-90599b65-8lzkg"]
Oct 30 19:48:00.101720 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:48:00.101687    4892 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-90599b65-8lzkg"]
Oct 30 19:48:00.446763 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:48:00.446729    4892 generic.go:356] "Generic (PLEG): container finished" podID="9c98da0e-489b-4333-b450-704beaf2f325" containerID="9b069f96d6daf9fc90a38ed7626db9a2a20aa656f12734127a85f4d468ed67cb" exitCode=0
Oct 30 19:48:00.446914 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:48:00.446803    4892 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-derive-8a3c7851-openshift-default-7845b86bf7-xx8vl" event={"ID":"9c98da0e-489b-4333-b450-704beaf2f325","Type":"ContainerDied","Data":"9b069f96d6daf9fc90a38ed7626db9a2a20aa656f12734127a85f4d468ed67cb"}
Oct 30 19:48:00.447242 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:48:00.447218    4892 scope.go:122] "RemoveContainer" containerID="9b069f96d6daf9fc90a38ed7626db9a2a20aa656f12734127a85f4d468ed67cb"
Oct 30 19:48:00.449044 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:48:00.449012    4892 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892000Ki","pods":"250"}
Oct 30 19:48:00.449131 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:48:00.449101    4892 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892000Ki","pods":"250"}
Oct 30 19:48:00.449176 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:48:00.449149    4892 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892000Ki","pods":"250"}
Oct 30 19:48:00.546700 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:48:00.546667    4892 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-derive-8a3c7851-openshift-default-7845b86bf7-xx8vl"
Oct 30 19:48:00.581997 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:48:00.580008    4892 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="2409c047-ff6e-47a7-ac84-b05f825dac37" path="/var/lib/kubelet/pods/2409c047-ff6e-47a7-ac84-b05f825dac37/volumes"
Oct 30 19:48:01.469378 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:48:01.469278    4892 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-derive-8a3c7851-openshift-default-7845b86bf7-xx8vl" event={"ID":"9c98da0e-489b-4333-b450-704beaf2f325","Type":"ContainerStarted","Data":"9d7be60ad01352d352c6af18b909548df3c56c93ca1db09aa276ee8eedea5ec1"}
Oct 30 19:48:01.708299 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:48:01.708262    4892 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-derive-8a3c7851-openshift-default-7845b86bf7-xx8vl"
Oct 30 19:48:01.709848 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:48:01.709818    4892 patch_prober.go:29] interesting pod/e2e-derive-8a3c7851-openshift-default-7845b86bf7-xx8vl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.160:15021/healthz/ready\": dial tcp 10.132.0.160:15021: connect: connection refused" start-of-body=
Oct 30 19:48:01.710018 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:48:01.709869    4892 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-derive-8a3c7851-openshift-default-7845b86bf7-xx8vl" podUID="9c98da0e-489b-4333-b450-704beaf2f325" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.160:15021/healthz/ready\": dial tcp 10.132.0.160:15021: connect: connection refused"
Oct 30 19:48:02.383777 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:48:02.383729    4892 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-5bf13ff3-7b94c6cdf9-8ll9x"]
Oct 30 19:48:02.498107 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:48:02.493690    4892 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-5bf13ff3-7b94c6cdf9-8ll9x"]
Oct 30 19:48:02.498107 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:48:02.493810    4892 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-5bf13ff3-7b94c6cdf9-8ll9x"
Oct 30 19:48:02.580054 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:48:02.577559    4892 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-mt-5bf13ff3\""
Oct 30 19:48:02.580054 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:48:02.577716    4892 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-mt-5bf13ff3-dockercfg-5mpp5\""
Oct 30 19:48:02.580054 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:48:02.579122    4892 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pvdmw\" (UniqueName: \"kubernetes.io/projected/3189293d-967f-4896-8fce-94249776053b-kube-api-access-pvdmw\") pod \"payload-pre-processing-e2e-mt-5bf13ff3-7b94c6cdf9-8ll9x\" (UID: \"3189293d-967f-4896-8fce-94249776053b\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-5bf13ff3-7b94c6cdf9-8ll9x"
Oct 30 19:48:02.580054 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:48:02.579314    4892 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3189293d-967f-4896-8fce-94249776053b-plugins-config-volume\") pod \"payload-pre-processing-e2e-mt-5bf13ff3-7b94c6cdf9-8ll9x\" (UID: \"3189293d-967f-4896-8fce-94249776053b\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-5bf13ff3-7b94c6cdf9-8ll9x"
Oct 30 19:48:02.689046 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:48:02.684621    4892 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3189293d-967f-4896-8fce-94249776053b-plugins-config-volume\") pod \"payload-pre-processing-e2e-mt-5bf13ff3-7b94c6cdf9-8ll9x\" (UID: \"3189293d-967f-4896-8fce-94249776053b\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-5bf13ff3-7b94c6cdf9-8ll9x"
Oct 30 19:48:02.689046 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:48:02.684863    4892 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pvdmw\" (UniqueName: \"kubernetes.io/projected/3189293d-967f-4896-8fce-94249776053b-kube-api-access-pvdmw\") pod \"payload-pre-processing-e2e-mt-5bf13ff3-7b94c6cdf9-8ll9x\" (UID: \"3189293d-967f-4896-8fce-94249776053b\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-5bf13ff3-7b94c6cdf9-8ll9x"
Oct 30 19:48:02.828987 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:48:02.718957    4892 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3189293d-967f-4896-8fce-94249776053b-plugins-config-volume\") pod \"payload-pre-processing-e2e-mt-5bf13ff3-7b94c6cdf9-8ll9x\" (UID: \"3189293d-967f-4896-8fce-94249776053b\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-5bf13ff3-7b94c6cdf9-8ll9x"
Oct 30 19:48:02.948722 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:48:02.948590    4892 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-2d9861/premium-w0-2d9861-kserve-67945c6848-4n6kz" podUID="798988b2-c769-4f73-a7c8-ca15de0beef1" containerName="main" probeResult="failure" output="Get \"https://10.132.0.62:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:48:03.142160 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:48:03.135085    4892 patch_prober.go:29] interesting pod/e2e-derive-8a3c7851-openshift-default-7845b86bf7-xx8vl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.160:15021/healthz/ready\": dial tcp 10.132.0.160:15021: connect: connection refused" start-of-body=
Oct 30 19:48:03.142160 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:48:03.135145    4892 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-derive-8a3c7851-openshift-default-7845b86bf7-xx8vl" podUID="9c98da0e-489b-4333-b450-704beaf2f325" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.160:15021/healthz/ready\": dial tcp 10.132.0.160:15021: connect: connection refused"
Oct 30 19:48:03.145777 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:48:03.145745    4892 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-5bf13ff3-5554f7696-czdq8"]
Oct 30 19:48:03.202514 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:48:03.202437    4892 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-5bf13ff3-5554f7696-czdq8"
Oct 30 19:48:03.225109 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:48:03.225053    4892 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-5f55cd/premium-w6-5f55cd-kserve-7f785c544b-pcvhb" podUID="191593a2-e3a9-4ce4-a1c6-c089723b0237" containerName="main" probeResult="failure" output="Get \"https://10.132.0.60:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:48:03.298615 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:48:03.298584    4892 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fhl6r\" (UniqueName: \"kubernetes.io/projected/0647888c-d306-43e3-b54c-132157f9da65-kube-api-access-fhl6r\") pod \"payload-processing-e2e-mt-5bf13ff3-5554f7696-czdq8\" (UID: \"0647888c-d306-43e3-b54c-132157f9da65\") " pod="openshift-ingress/payload-processing-e2e-mt-5bf13ff3-5554f7696-czdq8"
Oct 30 19:48:03.299477 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:48:03.299457    4892 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0647888c-d306-43e3-b54c-132157f9da65-plugins-config-volume\") pod \"payload-processing-e2e-mt-5bf13ff3-5554f7696-czdq8\" (UID: \"0647888c-d306-43e3-b54c-132157f9da65\") " pod="openshift-ingress/payload-processing-e2e-mt-5bf13ff3-5554f7696-czdq8"
Oct 30 19:48:03.301928 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:48:03.299850    4892 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-5bf13ff3-5554f7696-czdq8"]
Oct 30 19:48:03.302209 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:48:03.301224    4892 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-pvdmw\" (UniqueName: \"kubernetes.io/projected/3189293d-967f-4896-8fce-94249776053b-kube-api-access-pvdmw\") pod \"payload-pre-processing-e2e-mt-5bf13ff3-7b94c6cdf9-8ll9x\" (UID: \"3189293d-967f-4896-8fce-94249776053b\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-5bf13ff3-7b94c6cdf9-8ll9x"
Oct 30 19:48:03.320516 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:48:03.320491    4892 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-5bf13ff3-7b94c6cdf9-8ll9x"
Oct 30 19:48:03.405108 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:48:03.403493    4892 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0647888c-d306-43e3-b54c-132157f9da65-plugins-config-volume\") pod \"payload-processing-e2e-mt-5bf13ff3-5554f7696-czdq8\" (UID: \"0647888c-d306-43e3-b54c-132157f9da65\") " pod="openshift-ingress/payload-processing-e2e-mt-5bf13ff3-5554f7696-czdq8"
Oct 30 19:48:03.405108 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:48:03.403775    4892 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fhl6r\" (UniqueName: \"kubernetes.io/projected/0647888c-d306-43e3-b54c-132157f9da65-kube-api-access-fhl6r\") pod \"payload-processing-e2e-mt-5bf13ff3-5554f7696-czdq8\" (UID: \"0647888c-d306-43e3-b54c-132157f9da65\") " pod="openshift-ingress/payload-processing-e2e-mt-5bf13ff3-5554f7696-czdq8"
Oct 30 19:48:03.406447 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:48:03.405384    4892 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0647888c-d306-43e3-b54c-132157f9da65-plugins-config-volume\") pod \"payload-processing-e2e-mt-5bf13ff3-5554f7696-czdq8\" (UID: \"0647888c-d306-43e3-b54c-132157f9da65\") " pod="openshift-ingress/payload-processing-e2e-mt-5bf13ff3-5554f7696-czdq8"
Oct 30 19:48:03.464071 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:48:03.460808    4892 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fhl6r\" (UniqueName: \"kubernetes.io/projected/0647888c-d306-43e3-b54c-132157f9da65-kube-api-access-fhl6r\") pod \"payload-processing-e2e-mt-5bf13ff3-5554f7696-czdq8\" (UID: \"0647888c-d306-43e3-b54c-132157f9da65\") " pod="openshift-ingress/payload-processing-e2e-mt-5bf13ff3-5554f7696-czdq8"
Oct 30 19:48:03.615334 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:48:03.615301    4892 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-5bf13ff3-5554f7696-czdq8"
Oct 30 19:48:03.709144 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:48:03.708547    4892 patch_prober.go:29] interesting pod/e2e-derive-8a3c7851-openshift-default-7845b86bf7-xx8vl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.160:15021/healthz/ready\": dial tcp 10.132.0.160:15021: connect: connection refused" start-of-body=
Oct 30 19:48:03.709144 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:48:03.708597    4892 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-derive-8a3c7851-openshift-default-7845b86bf7-xx8vl" podUID="9c98da0e-489b-4333-b450-704beaf2f325" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.160:15021/healthz/ready\": dial tcp 10.132.0.160:15021: connect: connection refused"
Oct 30 19:48:03.779751 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:48:03.779471    4892 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-5bf13ff3-7b94c6cdf9-8ll9x"]
Oct 30 19:48:04.058562 ip-10-0-132-82 kubenswrapper[4892]: W1030 19:48:04.058520    4892 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0647888c_d306_43e3_b54c_132157f9da65.slice/crio-3c5108899ea98ed6f3049cdae925895ee46abd470c0f51b80c72ec9e26436edf WatchSource:0}: Error finding container 3c5108899ea98ed6f3049cdae925895ee46abd470c0f51b80c72ec9e26436edf: Status 404 returned error can't find the container with id 3c5108899ea98ed6f3049cdae925895ee46abd470c0f51b80c72ec9e26436edf
Oct 30 19:48:04.061370 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:48:04.061346    4892 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-5bf13ff3-5554f7696-czdq8"]
Oct 30 19:48:04.708262 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:48:04.708216    4892 patch_prober.go:29] interesting pod/e2e-derive-8a3c7851-openshift-default-7845b86bf7-xx8vl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.160:15021/healthz/ready\": dial tcp 10.132.0.160:15021: connect: connection refused" start-of-body=
Oct 30 19:48:04.708948 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:48:04.708287    4892 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-derive-8a3c7851-openshift-default-7845b86bf7-xx8vl" podUID="9c98da0e-489b-4333-b450-704beaf2f325" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.160:15021/healthz/ready\": dial tcp 10.132.0.160:15021: connect: connection refused"
Oct 30 19:48:04.794068 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:48:04.794022    4892 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-5bf13ff3-5554f7696-czdq8" event={"ID":"0647888c-d306-43e3-b54c-132157f9da65","Type":"ContainerStarted","Data":"928aba9079baa7ff6d25bc5701642c3afc1bdc35af7ced24ee284b862d11680b"}
Oct 30 19:48:04.794263 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:48:04.794093    4892 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-5bf13ff3-5554f7696-czdq8" event={"ID":"0647888c-d306-43e3-b54c-132157f9da65","Type":"ContainerStarted","Data":"3c5108899ea98ed6f3049cdae925895ee46abd470c0f51b80c72ec9e26436edf"}
Oct 30 19:48:04.796251 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:48:04.796222    4892 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-5bf13ff3-7b94c6cdf9-8ll9x" event={"ID":"3189293d-967f-4896-8fce-94249776053b","Type":"ContainerStarted","Data":"1999bb58215eba07d37bd2ccb42867d25dde775138fd6bdb4133e8034c38c641"}
Oct 30 19:48:04.796394 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:48:04.796272    4892 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-5bf13ff3-7b94c6cdf9-8ll9x" event={"ID":"3189293d-967f-4896-8fce-94249776053b","Type":"ContainerStarted","Data":"04fb3444dc4125a75ec2e227eb8315d15c8a2aad4db1b6b9e23f3a19b14c5210"}
Oct 30 19:48:04.796901 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:48:04.796878    4892 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-mt-5bf13ff3-7b94c6cdf9-8ll9x"
Oct 30 19:48:04.915261 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:48:04.915199    4892 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-mt-5bf13ff3-5554f7696-czdq8" podStartSLOduration=2.91518067 podStartE2EDuration="2.91518067s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:48:02 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:48:04.912254392 +0000 UTC m=+1932.482719993" watchObservedRunningTime="2026-10-30 19:48:04.91518067 +0000 UTC m=+1932.485646270"
Oct 30 19:48:05.120385 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:48:05.120270    4892 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-mt-5bf13ff3-7b94c6cdf9-8ll9x" podStartSLOduration=3.12024738 podStartE2EDuration="3.12024738s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:48:02 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:48:05.062282972 +0000 UTC m=+1932.632748571" watchObservedRunningTime="2026-10-30 19:48:05.12024738 +0000 UTC m=+1932.690712983"
Oct 30 19:48:05.713990 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:48:05.708314    4892 patch_prober.go:29] interesting pod/e2e-derive-8a3c7851-openshift-default-7845b86bf7-xx8vl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.160:15021/healthz/ready\": dial tcp 10.132.0.160:15021: connect: connection refused" start-of-body=
Oct 30 19:48:05.713990 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:48:05.708378    4892 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-derive-8a3c7851-openshift-default-7845b86bf7-xx8vl" podUID="9c98da0e-489b-4333-b450-704beaf2f325" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.160:15021/healthz/ready\": dial tcp 10.132.0.160:15021: connect: connection refused"
Oct 30 19:48:05.853613 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:48:05.853509    4892 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-mt-5bf13ff3-5554f7696-czdq8"
Oct 30 19:48:06.708928 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:48:06.708893    4892 patch_prober.go:29] interesting pod/e2e-derive-8a3c7851-openshift-default-7845b86bf7-xx8vl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.160:15021/healthz/ready\": dial tcp 10.132.0.160:15021: connect: connection refused" start-of-body=
Oct 30 19:48:06.709137 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:48:06.708952    4892 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-derive-8a3c7851-openshift-default-7845b86bf7-xx8vl" podUID="9c98da0e-489b-4333-b450-704beaf2f325" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.160:15021/healthz/ready\": dial tcp 10.132.0.160:15021: connect: connection refused"
Oct 30 19:48:07.813677 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:48:07.813645    4892 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-derive-8a3c7851-openshift-default-7845b86bf7-xx8vl"
Oct 30 19:48:07.814550 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:48:07.814523    4892 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-derive-8a3c7851-openshift-default-7845b86bf7-xx8vl"
Oct 30 19:48:07.815705 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:48:07.815679    4892 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-derive-8a3c7851-openshift-default-7845b86bf7-xx8vl"
Oct 30 19:48:08.053420 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:48:08.053388    4892 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-454a9e34-dtlzm"]
Oct 30 19:48:08.058545 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:48:08.058522    4892 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-454a9e34-dtlzm"]
Oct 30 19:48:08.571907 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:48:08.571876    4892 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="90541e2c-7c60-4919-830e-73341e6c4cfe" path="/var/lib/kubelet/pods/90541e2c-7c60-4919-830e-73341e6c4cfe/volumes"
Oct 30 19:48:16.880896 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:48:16.880862    4892 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-mt-5bf13ff3-5554f7696-czdq8"
Oct 30 19:48:16.881525 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:48:16.881004    4892 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-mt-5bf13ff3-7b94c6cdf9-8ll9x"
Oct 30 19:48:19.049791 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:48:19.049752    4892 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ff93af8f-7pjx4"]
Oct 30 19:48:19.099210 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:48:19.099183    4892 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ff93af8f-7pjx4"]
Oct 30 19:48:19.409047 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:48:19.409014    4892 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-a3f9b7-openshift-default-7d6d8b8fb5-m5zsm"]
Oct 30 19:48:19.409277 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:48:19.409242    4892 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w2-a3f9b7-openshift-default-7d6d8b8fb5-m5zsm" podUID="02bc8b7f-a2b1-4994-85cf-436cfc0d221c" containerName="istio-proxy" containerID="cri-o://40cfcef57396e51e49572d8e71e1b1f64c7664b57dd6596ec8842905d10b3ae0" gracePeriod=30
Oct 30 19:48:20.575509 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:48:20.575471    4892 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="398bf4dc-e72f-46de-93fe-32dfeb6574d3" path="/var/lib/kubelet/pods/398bf4dc-e72f-46de-93fe-32dfeb6574d3/volumes"
Oct 30 19:48:21.053287 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:48:21.053253    4892 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-7415809c-75wmh"]
Oct 30 19:48:21.099854 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:48:21.099823    4892 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-7415809c-75wmh"]
Oct 30 19:48:22.574183 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:48:22.574153    4892 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a1a081b7-23cf-4337-9ad4-cb512b27fa7f" path="/var/lib/kubelet/pods/a1a081b7-23cf-4337-9ad4-cb512b27fa7f/volumes"
Oct 30 19:48:24.448604 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:48:24.445813    4892 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-fsfvk"]
Oct 30 19:48:24.550152 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:48:24.550104    4892 patch_prober.go:29] interesting pod/e2e-worker-w2-a3f9b7-openshift-default-7d6d8b8fb5-m5zsm container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.106:15021/healthz/ready\": dial tcp 10.132.0.106:15021: connect: connection refused" start-of-body=
Oct 30 19:48:24.550368 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:48:24.550181    4892 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w2-a3f9b7-openshift-default-7d6d8b8fb5-m5zsm" podUID="02bc8b7f-a2b1-4994-85cf-436cfc0d221c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.106:15021/healthz/ready\": dial tcp 10.132.0.106:15021: connect: connection refused"
Oct 30 19:48:24.847394 ip-10-0-132-82 kubenswrapper[4892]: E1030 19:48:24.847302    4892 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-jrm87" containerName="extractor"
Oct 30 19:48:25.047239 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:48:25.047199    4892 generic.go:356] "Generic (PLEG): container finished" podID="02bc8b7f-a2b1-4994-85cf-436cfc0d221c" containerID="40cfcef57396e51e49572d8e71e1b1f64c7664b57dd6596ec8842905d10b3ae0" exitCode=0
Oct 30 19:48:25.047400 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:48:25.047267    4892 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-a3f9b7-openshift-default-7d6d8b8fb5-m5zsm" event={"ID":"02bc8b7f-a2b1-4994-85cf-436cfc0d221c","Type":"ContainerDied","Data":"40cfcef57396e51e49572d8e71e1b1f64c7664b57dd6596ec8842905d10b3ae0"}
Oct 30 19:48:25.512940 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:48:25.512915    4892 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-a3f9b7-openshift-default-7d6d8b8fb5-m5zsm"
Oct 30 19:48:25.639917 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:48:25.639883    4892 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/02bc8b7f-a2b1-4994-85cf-436cfc0d221c-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/02bc8b7f-a2b1-4994-85cf-436cfc0d221c-workload-certs\") pod \"02bc8b7f-a2b1-4994-85cf-436cfc0d221c\" (UID: \"02bc8b7f-a2b1-4994-85cf-436cfc0d221c\") "
Oct 30 19:48:25.640098 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:48:25.639940    4892 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/02bc8b7f-a2b1-4994-85cf-436cfc0d221c-istio-token\" (UniqueName: \"kubernetes.io/projected/02bc8b7f-a2b1-4994-85cf-436cfc0d221c-istio-token\") pod \"02bc8b7f-a2b1-4994-85cf-436cfc0d221c\" (UID: \"02bc8b7f-a2b1-4994-85cf-436cfc0d221c\") "
Oct 30 19:48:25.640098 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:48:25.640039    4892 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/02bc8b7f-a2b1-4994-85cf-436cfc0d221c-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/02bc8b7f-a2b1-4994-85cf-436cfc0d221c-istio-podinfo\") pod \"02bc8b7f-a2b1-4994-85cf-436cfc0d221c\" (UID: \"02bc8b7f-a2b1-4994-85cf-436cfc0d221c\") "
Oct 30 19:48:25.640098 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:48:25.640064    4892 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/02bc8b7f-a2b1-4994-85cf-436cfc0d221c-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/02bc8b7f-a2b1-4994-85cf-436cfc0d221c-istiod-ca-cert\") pod \"02bc8b7f-a2b1-4994-85cf-436cfc0d221c\" (UID: \"02bc8b7f-a2b1-4994-85cf-436cfc0d221c\") "
Oct 30 19:48:25.640098 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:48:25.640080    4892 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/02bc8b7f-a2b1-4994-85cf-436cfc0d221c-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/02bc8b7f-a2b1-4994-85cf-436cfc0d221c-istio-envoy\") pod \"02bc8b7f-a2b1-4994-85cf-436cfc0d221c\" (UID: \"02bc8b7f-a2b1-4994-85cf-436cfc0d221c\") "
Oct 30 19:48:25.640284 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:48:25.640195    4892 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/02bc8b7f-a2b1-4994-85cf-436cfc0d221c-workload-certs" pod "02bc8b7f-a2b1-4994-85cf-436cfc0d221c" (UID: "02bc8b7f-a2b1-4994-85cf-436cfc0d221c"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:48:25.640284 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:48:25.640244    4892 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/02bc8b7f-a2b1-4994-85cf-436cfc0d221c-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/02bc8b7f-a2b1-4994-85cf-436cfc0d221c-istio-data\") pod \"02bc8b7f-a2b1-4994-85cf-436cfc0d221c\" (UID: \"02bc8b7f-a2b1-4994-85cf-436cfc0d221c\") "
Oct 30 19:48:25.640388 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:48:25.640364    4892 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/02bc8b7f-a2b1-4994-85cf-436cfc0d221c-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/02bc8b7f-a2b1-4994-85cf-436cfc0d221c-credential-socket\") pod \"02bc8b7f-a2b1-4994-85cf-436cfc0d221c\" (UID: \"02bc8b7f-a2b1-4994-85cf-436cfc0d221c\") "
Oct 30 19:48:25.640445 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:48:25.640418    4892 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/02bc8b7f-a2b1-4994-85cf-436cfc0d221c-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/02bc8b7f-a2b1-4994-85cf-436cfc0d221c-workload-socket\") pod \"02bc8b7f-a2b1-4994-85cf-436cfc0d221c\" (UID: \"02bc8b7f-a2b1-4994-85cf-436cfc0d221c\") "
Oct 30 19:48:25.640499 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:48:25.640447    4892 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/02bc8b7f-a2b1-4994-85cf-436cfc0d221c-kube-api-access-wzxs6\" (UniqueName: \"kubernetes.io/projected/02bc8b7f-a2b1-4994-85cf-436cfc0d221c-kube-api-access-wzxs6\") pod \"02bc8b7f-a2b1-4994-85cf-436cfc0d221c\" (UID: \"02bc8b7f-a2b1-4994-85cf-436cfc0d221c\") "
Oct 30 19:48:25.640499 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:48:25.640481    4892 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/02bc8b7f-a2b1-4994-85cf-436cfc0d221c-istio-data" pod "02bc8b7f-a2b1-4994-85cf-436cfc0d221c" (UID: "02bc8b7f-a2b1-4994-85cf-436cfc0d221c"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:48:25.640601 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:48:25.640531    4892 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/02bc8b7f-a2b1-4994-85cf-436cfc0d221c-istiod-ca-cert" pod "02bc8b7f-a2b1-4994-85cf-436cfc0d221c" (UID: "02bc8b7f-a2b1-4994-85cf-436cfc0d221c"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 19:48:25.640693 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:48:25.640669    4892 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/02bc8b7f-a2b1-4994-85cf-436cfc0d221c-credential-socket" pod "02bc8b7f-a2b1-4994-85cf-436cfc0d221c" (UID: "02bc8b7f-a2b1-4994-85cf-436cfc0d221c"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:48:25.640763 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:48:25.640688    4892 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/02bc8b7f-a2b1-4994-85cf-436cfc0d221c-workload-socket" pod "02bc8b7f-a2b1-4994-85cf-436cfc0d221c" (UID: "02bc8b7f-a2b1-4994-85cf-436cfc0d221c"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:48:25.641212 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:48:25.641191    4892 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/02bc8b7f-a2b1-4994-85cf-436cfc0d221c-istiod-ca-cert\") on node \"ip-10-0-132-82.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 19:48:25.641212 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:48:25.641213    4892 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/02bc8b7f-a2b1-4994-85cf-436cfc0d221c-istio-data\") on node \"ip-10-0-132-82.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 19:48:25.641359 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:48:25.641225    4892 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/02bc8b7f-a2b1-4994-85cf-436cfc0d221c-credential-socket\") on node \"ip-10-0-132-82.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 19:48:25.641359 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:48:25.641238    4892 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/02bc8b7f-a2b1-4994-85cf-436cfc0d221c-workload-socket\") on node \"ip-10-0-132-82.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 19:48:25.641359 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:48:25.641251    4892 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/02bc8b7f-a2b1-4994-85cf-436cfc0d221c-workload-certs\") on node \"ip-10-0-132-82.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 19:48:25.642683 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:48:25.642657    4892 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/02bc8b7f-a2b1-4994-85cf-436cfc0d221c-istio-podinfo" pod "02bc8b7f-a2b1-4994-85cf-436cfc0d221c" (UID: "02bc8b7f-a2b1-4994-85cf-436cfc0d221c"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 19:48:25.642866 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:48:25.642848    4892 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/02bc8b7f-a2b1-4994-85cf-436cfc0d221c-istio-token" pod "02bc8b7f-a2b1-4994-85cf-436cfc0d221c" (UID: "02bc8b7f-a2b1-4994-85cf-436cfc0d221c"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:48:25.643149 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:48:25.643132    4892 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/02bc8b7f-a2b1-4994-85cf-436cfc0d221c-istio-envoy" pod "02bc8b7f-a2b1-4994-85cf-436cfc0d221c" (UID: "02bc8b7f-a2b1-4994-85cf-436cfc0d221c"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:48:25.643437 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:48:25.643413    4892 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/02bc8b7f-a2b1-4994-85cf-436cfc0d221c-kube-api-access-wzxs6" pod "02bc8b7f-a2b1-4994-85cf-436cfc0d221c" (UID: "02bc8b7f-a2b1-4994-85cf-436cfc0d221c"). InnerVolumeSpecName "kube-api-access-wzxs6". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:48:25.742506 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:48:25.742414    4892 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/02bc8b7f-a2b1-4994-85cf-436cfc0d221c-istio-podinfo\") on node \"ip-10-0-132-82.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 19:48:25.742506 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:48:25.742447    4892 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/02bc8b7f-a2b1-4994-85cf-436cfc0d221c-istio-envoy\") on node \"ip-10-0-132-82.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 19:48:25.742506 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:48:25.742460    4892 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wzxs6\" (UniqueName: \"kubernetes.io/projected/02bc8b7f-a2b1-4994-85cf-436cfc0d221c-kube-api-access-wzxs6\") on node \"ip-10-0-132-82.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 19:48:25.742506 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:48:25.742475    4892 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/02bc8b7f-a2b1-4994-85cf-436cfc0d221c-istio-token\") on node \"ip-10-0-132-82.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 19:48:26.111044 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:48:26.110436    4892 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-a3f9b7-openshift-default-7d6d8b8fb5-m5zsm" event={"ID":"02bc8b7f-a2b1-4994-85cf-436cfc0d221c","Type":"ContainerDied","Data":"f21c2d24bf38805f245ee8c8db8cf43f9f7ab0437a725940104fb9157cd96e35"}
Oct 30 19:48:26.111044 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:48:26.110520    4892 scope.go:122] "RemoveContainer" containerID="40cfcef57396e51e49572d8e71e1b1f64c7664b57dd6596ec8842905d10b3ae0"
Oct 30 19:48:26.111044 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:48:26.110728    4892 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-a3f9b7-openshift-default-7d6d8b8fb5-m5zsm"
Oct 30 19:48:26.412205 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:48:26.412122    4892 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-a3f9b7-openshift-default-7d6d8b8fb5-m5zsm"]
Oct 30 19:48:26.502994 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:48:26.501171    4892 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w2-a3f9b7-openshift-default-7d6d8b8fb5-m5zsm"]
Oct 30 19:48:26.682044 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:48:26.679233    4892 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="02bc8b7f-a2b1-4994-85cf-436cfc0d221c" path="/var/lib/kubelet/pods/02bc8b7f-a2b1-4994-85cf-436cfc0d221c/volumes"
Oct 30 19:48:28.835898 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:48:28.835855    4892 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-ada8d7/simulator-w3-ada8d7-kserve-7d575cfcb5-5txgn" podUID="10a5c7c8-6146-47b0-9643-bb137cf10bf5" containerName="main" probeResult="failure" output="Get \"https://10.132.0.55:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:48:29.160083 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:48:29.160035    4892 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-5f55cd/premium-w6-5f55cd-kserve-7f785c544b-pcvhb" podUID="191593a2-e3a9-4ce4-a1c6-c089723b0237" containerName="main" probeResult="failure" output="Get \"https://10.132.0.60:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:48:29.428926 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:48:29.428830    4892 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-2d9861/distinct-2-w0-2d9861-kserve-679c6b56dd-nmhbm" podUID="65b0586a-6593-48cc-adf8-8ff48417ece4" containerName="main" probeResult="failure" output="Get \"https://10.132.0.133:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:48:29.509164 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:48:29.509109    4892 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-686689/distinct-2-w5-686689-kserve-5ff578b4cd-rkc6s" podUID="a8829732-ac6f-4238-b8ce-fc781084668b" containerName="main" probeResult="failure" output="Get \"https://10.132.0.87:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:48:29.613556 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:48:29.613516    4892 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-d52df0/distinct-2-w1-d52df0-kserve-7846558464-xrkdc" podUID="9c614ab6-30e2-4fe1-8609-d5e2dadc198a" containerName="main" probeResult="failure" output="Get \"https://10.132.0.83:8000/ready\": context deadline exceeded"
Oct 30 19:48:29.938992 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:48:29.931098    4892 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6db586dd6c-lqrjf container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.145:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 30 19:48:29.938992 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:48:29.931168    4892 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-lqrjf" podUID="c5510c1c-0da3-4b39-95ee-a9195c32e990" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.145:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:48:30.168111 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:48:30.168064    4892 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-5f55cd/premium-w6-5f55cd-kserve-7f785c544b-pcvhb" podUID="191593a2-e3a9-4ce4-a1c6-c089723b0237" containerName="main" probeResult="failure" output="Get \"https://10.132.0.60:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:48:30.169075 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:48:30.169052    4892 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w6-5f55cd/premium-w6-5f55cd-kserve-7f785c544b-pcvhb"
Oct 30 19:48:30.252139 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:48:30.252052    4892 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6db586dd6c-lgwz5 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.140:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 30 19:48:30.252381 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:48:30.252351    4892 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-lgwz5" podUID="c63df543-fcb1-4bee-9d52-c50286ac6f91" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.140:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:48:30.426122 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:48:30.426079    4892 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-b5178f/distinct-2-w4-b5178f-kserve-69ddf8b665-9pdth" podUID="fb71d961-8500-4367-b48c-88d385962bd5" containerName="main" probeResult="failure" output="Get \"https://10.132.0.90:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:48:30.516999 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:48:30.516796    4892 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-686689/distinct-2-w5-686689-kserve-5ff578b4cd-rkc6s" podUID="a8829732-ac6f-4238-b8ce-fc781084668b" containerName="main" probeResult="failure" output="Get \"https://10.132.0.87:8000/ready\": context deadline exceeded"
Oct 30 19:48:30.549422 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:48:30.549381    4892 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w5-686689/distinct-2-w5-686689-kserve-5ff578b4cd-rkc6s"
Oct 30 19:48:30.635828 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:48:30.635047    4892 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-d52df0/distinct-2-w1-d52df0-kserve-7846558464-xrkdc" podUID="9c614ab6-30e2-4fe1-8609-d5e2dadc198a" containerName="main" probeResult="failure" output="Get \"https://10.132.0.83:8000/ready\": context deadline exceeded"
Oct 30 19:48:30.930766 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:48:30.930729    4892 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-d52df0/distinct-2-w1-d52df0-kserve-7846558464-xrkdc"
Oct 30 19:48:30.931426 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:48:30.931392    4892 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w6-5f55cd/premium-w6-5f55cd-kserve-7f785c544b-pcvhb"
Oct 30 19:48:30.959522 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:48:30.959490    4892 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w5-686689/distinct-2-w5-686689-kserve-5ff578b4cd-rkc6s"
Oct 30 19:48:30.972561 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:48:30.972520    4892 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-d52df0/simulator-w1-d52df0-kserve-67f74bc577-q5bjz" podUID="81b7bb84-1003-4452-b7e0-b4aaba6fefa4" containerName="main" probeResult="failure" output="Get \"https://10.132.0.57:8000/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:48:30.982997 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:48:30.982901    4892 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-d52df0/distinct-2-w1-d52df0-kserve-7846558464-xrkdc"
Oct 30 19:48:32.050934 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:48:32.050893    4892 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b16664de-zhmcs"]
Oct 30 19:48:32.099718 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:48:32.099689    4892 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b16664de-zhmcs"]
Oct 30 19:48:32.572160 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:48:32.572129    4892 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="545e8fd9-b409-4437-9ed0-c227942ca7da" path="/var/lib/kubelet/pods/545e8fd9-b409-4437-9ed0-c227942ca7da/volumes"
Oct 30 19:48:32.706852 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:48:32.706817    4892 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-8a3c7851-z798v"]
Oct 30 19:48:32.707558 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:48:32.707544    4892 cpu_manager.go:414] "containerMap: removing container" podUID="02bc8b7f-a2b1-4994-85cf-436cfc0d221c" containerName="istio-proxy"
Oct 30 19:48:32.707616 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:48:32.707560    4892 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="02bc8b7f-a2b1-4994-85cf-436cfc0d221c" containerName="istio-proxy"
Oct 30 19:48:32.707700 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:48:32.707691    4892 memory_manager.go:365] "RemoveStaleState removing state" podUID="02bc8b7f-a2b1-4994-85cf-436cfc0d221c" containerName="istio-proxy"
Oct 30 19:48:32.712917 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:48:32.712900    4892 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-8a3c7851-z798v"
Oct 30 19:48:32.735510 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:48:32.735486    4892 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7nndc\" (UniqueName: \"kubernetes.io/projected/5b7dcbb7-54cd-4d78-af9b-f1d8ccf00d3f-kube-api-access-7nndc\") pod \"maas-api-revoke-keys-e2e-derive-8a3c7851-z798v\" (UID: \"5b7dcbb7-54cd-4d78-af9b-f1d8ccf00d3f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-8a3c7851-z798v"
Oct 30 19:48:32.735642 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:48:32.735535    4892 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5b7dcbb7-54cd-4d78-af9b-f1d8ccf00d3f-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-derive-8a3c7851-z798v\" (UID: \"5b7dcbb7-54cd-4d78-af9b-f1d8ccf00d3f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-8a3c7851-z798v"
Oct 30 19:48:32.735729 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:48:32.735712    4892 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5b7dcbb7-54cd-4d78-af9b-f1d8ccf00d3f-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-derive-8a3c7851-z798v\" (UID: \"5b7dcbb7-54cd-4d78-af9b-f1d8ccf00d3f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-8a3c7851-z798v"
Oct 30 19:48:32.769310 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:48:32.769284    4892 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-8a3c7851-z798v"]
Oct 30 19:48:32.812541 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:48:32.812503    4892 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 19:48:32.812541 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:48:32.812524    4892 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 30 19:48:32.813874 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:48:32.813861    4892 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-nlxp4\""
Oct 30 19:48:32.813935 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:48:32.813878    4892 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 30 19:48:32.837681 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:48:32.837618    4892 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7nndc\" (UniqueName: \"kubernetes.io/projected/5b7dcbb7-54cd-4d78-af9b-f1d8ccf00d3f-kube-api-access-7nndc\") pod \"maas-api-revoke-keys-e2e-derive-8a3c7851-z798v\" (UID: \"5b7dcbb7-54cd-4d78-af9b-f1d8ccf00d3f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-8a3c7851-z798v"
Oct 30 19:48:32.837681 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:48:32.837662    4892 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5b7dcbb7-54cd-4d78-af9b-f1d8ccf00d3f-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-derive-8a3c7851-z798v\" (UID: \"5b7dcbb7-54cd-4d78-af9b-f1d8ccf00d3f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-8a3c7851-z798v"
Oct 30 19:48:32.837866 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:48:32.837774    4892 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5b7dcbb7-54cd-4d78-af9b-f1d8ccf00d3f-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-derive-8a3c7851-z798v\" (UID: \"5b7dcbb7-54cd-4d78-af9b-f1d8ccf00d3f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-8a3c7851-z798v"
Oct 30 19:48:32.838468 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:48:32.838451    4892 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5b7dcbb7-54cd-4d78-af9b-f1d8ccf00d3f-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-derive-8a3c7851-z798v\" (UID: \"5b7dcbb7-54cd-4d78-af9b-f1d8ccf00d3f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-8a3c7851-z798v"
Oct 30 19:48:32.840310 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:48:32.840287    4892 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5b7dcbb7-54cd-4d78-af9b-f1d8ccf00d3f-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-derive-8a3c7851-z798v\" (UID: \"5b7dcbb7-54cd-4d78-af9b-f1d8ccf00d3f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-8a3c7851-z798v"
Oct 30 19:48:32.893668 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:48:32.893643    4892 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7nndc\" (UniqueName: \"kubernetes.io/projected/5b7dcbb7-54cd-4d78-af9b-f1d8ccf00d3f-kube-api-access-7nndc\") pod \"maas-api-revoke-keys-e2e-derive-8a3c7851-z798v\" (UID: \"5b7dcbb7-54cd-4d78-af9b-f1d8ccf00d3f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-8a3c7851-z798v"
Oct 30 19:48:33.027137 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:48:33.027110    4892 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-8a3c7851-z798v"
Oct 30 19:48:33.408404 ip-10-0-132-82 kubenswrapper[4892]: W1030 19:48:33.408376    4892 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5b7dcbb7_54cd_4d78_af9b_f1d8ccf00d3f.slice/crio-9a851232d7f8e126d1c01ab82ec9c6358cb2877961324efea3c408c9b95cf087 WatchSource:0}: Error finding container 9a851232d7f8e126d1c01ab82ec9c6358cb2877961324efea3c408c9b95cf087: Status 404 returned error can't find the container with id 9a851232d7f8e126d1c01ab82ec9c6358cb2877961324efea3c408c9b95cf087
Oct 30 19:48:33.409179 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:48:33.409156    4892 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-8a3c7851-z798v"]
Oct 30 19:48:33.492660 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:48:33.492628    4892 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-8a3c7851-z798v" event={"ID":"5b7dcbb7-54cd-4d78-af9b-f1d8ccf00d3f","Type":"ContainerStarted","Data":"9a851232d7f8e126d1c01ab82ec9c6358cb2877961324efea3c408c9b95cf087"}
Oct 30 19:48:34.506031 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:48:34.505946    4892 generic.go:356] "Generic (PLEG): container finished" podID="5b7dcbb7-54cd-4d78-af9b-f1d8ccf00d3f" containerID="fd9edaa409d311d255f364d9a1c76b454663df1dc342150af3006b900660fca2" exitCode=0
Oct 30 19:48:34.506360 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:48:34.506032    4892 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-8a3c7851-z798v" event={"ID":"5b7dcbb7-54cd-4d78-af9b-f1d8ccf00d3f","Type":"ContainerDied","Data":"fd9edaa409d311d255f364d9a1c76b454663df1dc342150af3006b900660fca2"}
Oct 30 19:48:36.030986 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:48:36.030934    4892 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-8a3c7851-z798v"
Oct 30 19:48:36.056223 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:48:36.056184    4892 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-9837c7af-m8n4b"]
Oct 30 19:48:36.099196 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:48:36.099166    4892 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-9837c7af-m8n4b"]
Oct 30 19:48:36.141928 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:48:36.141853    4892 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5b7dcbb7-54cd-4d78-af9b-f1d8ccf00d3f-kube-api-access-7nndc\" (UniqueName: \"kubernetes.io/projected/5b7dcbb7-54cd-4d78-af9b-f1d8ccf00d3f-kube-api-access-7nndc\") pod \"5b7dcbb7-54cd-4d78-af9b-f1d8ccf00d3f\" (UID: \"5b7dcbb7-54cd-4d78-af9b-f1d8ccf00d3f\") "
Oct 30 19:48:36.141928 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:48:36.141956    4892 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/5b7dcbb7-54cd-4d78-af9b-f1d8ccf00d3f-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5b7dcbb7-54cd-4d78-af9b-f1d8ccf00d3f-maas-api-xks-ca\") pod \"5b7dcbb7-54cd-4d78-af9b-f1d8ccf00d3f\" (UID: \"5b7dcbb7-54cd-4d78-af9b-f1d8ccf00d3f\") "
Oct 30 19:48:36.142278 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:48:36.142222    4892 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/5b7dcbb7-54cd-4d78-af9b-f1d8ccf00d3f-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5b7dcbb7-54cd-4d78-af9b-f1d8ccf00d3f-maas-api-openshift-ca\") pod \"5b7dcbb7-54cd-4d78-af9b-f1d8ccf00d3f\" (UID: \"5b7dcbb7-54cd-4d78-af9b-f1d8ccf00d3f\") "
Oct 30 19:48:36.142812 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:48:36.142650    4892 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5b7dcbb7-54cd-4d78-af9b-f1d8ccf00d3f-maas-api-openshift-ca" pod "5b7dcbb7-54cd-4d78-af9b-f1d8ccf00d3f" (UID: "5b7dcbb7-54cd-4d78-af9b-f1d8ccf00d3f"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 19:48:36.143189 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:48:36.143164    4892 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5b7dcbb7-54cd-4d78-af9b-f1d8ccf00d3f-maas-api-openshift-ca\") on node \"ip-10-0-132-82.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 19:48:36.149076 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:48:36.144567    4892 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5b7dcbb7-54cd-4d78-af9b-f1d8ccf00d3f-kube-api-access-7nndc" pod "5b7dcbb7-54cd-4d78-af9b-f1d8ccf00d3f" (UID: "5b7dcbb7-54cd-4d78-af9b-f1d8ccf00d3f"). InnerVolumeSpecName "kube-api-access-7nndc". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:48:36.149076 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:48:36.147076    4892 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5b7dcbb7-54cd-4d78-af9b-f1d8ccf00d3f-maas-api-xks-ca" pod "5b7dcbb7-54cd-4d78-af9b-f1d8ccf00d3f" (UID: "5b7dcbb7-54cd-4d78-af9b-f1d8ccf00d3f"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 19:48:36.245249 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:48:36.245147    4892 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7nndc\" (UniqueName: \"kubernetes.io/projected/5b7dcbb7-54cd-4d78-af9b-f1d8ccf00d3f-kube-api-access-7nndc\") on node \"ip-10-0-132-82.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 19:48:36.245249 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:48:36.245179    4892 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5b7dcbb7-54cd-4d78-af9b-f1d8ccf00d3f-maas-api-xks-ca\") on node \"ip-10-0-132-82.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 19:48:36.607073 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:48:36.571392    4892 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-8a3c7851-z798v"
Oct 30 19:48:36.675273 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:48:36.674323    4892 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ead14343-b5b5-4c17-94af-9790ed4c8c08" path="/var/lib/kubelet/pods/ead14343-b5b5-4c17-94af-9790ed4c8c08/volumes"
Oct 30 19:48:36.675273 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:48:36.675260    4892 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-8a3c7851-z798v" event={"ID":"5b7dcbb7-54cd-4d78-af9b-f1d8ccf00d3f","Type":"ContainerDied","Data":"9a851232d7f8e126d1c01ab82ec9c6358cb2877961324efea3c408c9b95cf087"}
Oct 30 19:48:36.675273 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:48:36.675292    4892 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="9a851232d7f8e126d1c01ab82ec9c6358cb2877961324efea3c408c9b95cf087"
Oct 30 19:48:51.049152 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:48:51.049120    4892 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a429595a-fnjb5"]
Oct 30 19:48:51.098790 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:48:51.098762    4892 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a429595a-fnjb5"]
Oct 30 19:48:52.574459 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:48:52.574421    4892 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="65aca2dc-d914-46c8-9b84-20a864873a1d" path="/var/lib/kubelet/pods/65aca2dc-d914-46c8-9b84-20a864873a1d/volumes"
Oct 30 19:48:54.052151 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:48:54.052112    4892 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-9ea0f852-lt2xc"]
Oct 30 19:48:54.083944 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:48:54.083916    4892 scope.go:122] "RemoveContainer" containerID="d2ab1aefae7164e84473c64f8603a5a085c6bca68bb389a79efee3798941d177"
Oct 30 19:48:54.095480 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:48:54.095460    4892 scope.go:122] "RemoveContainer" containerID="ac9164aca539e0063140988dfbb316258f71d0216817fee8518d905b8db147c2"
Oct 30 19:48:54.098105 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:48:54.098079    4892 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-9ea0f852-lt2xc"]
Oct 30 19:48:54.106422 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:48:54.106402    4892 scope.go:122] "RemoveContainer" containerID="f0b5d9951889dbe6bc52d2bb56f2024fcf194176c7e04edbd4ef96c6edcb262d"
Oct 30 19:48:54.116519 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:48:54.116502    4892 scope.go:122] "RemoveContainer" containerID="810822b52e5f41f6c0fc2f4dc1ad71e32ba41392583e4936720f0152313d922c"
Oct 30 19:48:54.126582 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:48:54.126564    4892 scope.go:122] "RemoveContainer" containerID="d754ba5106e4070ba151b76294bc6b69c5bb4fa6703b29c8759278e4337167e8"
Oct 30 19:48:54.136115 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:48:54.136095    4892 scope.go:122] "RemoveContainer" containerID="dab73619cd618c9218820cd33f34d457cd7d234c2bcc6032495817dc06b615ea"
Oct 30 19:48:54.145695 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:48:54.145670    4892 scope.go:122] "RemoveContainer" containerID="d7d4927145a878ada4803a8ab02820b95d3237cc5c16601b2037826b0317c5a4"
Oct 30 19:48:54.158503 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:48:54.158479    4892 scope.go:122] "RemoveContainer" containerID="4dfa5a2a4510e0889a1cbc0f9862d3e92a5a6fc1b8cb500c5b5cb7ab06052a7d"
Oct 30 19:48:54.572276 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:48:54.572243    4892 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f2c3a6a5-3357-4220-a7c2-75c96d152aa0" path="/var/lib/kubelet/pods/f2c3a6a5-3357-4220-a7c2-75c96d152aa0/volumes"
Oct 30 19:48:54.852334 ip-10-0-132-82 kubenswrapper[4892]: E1030 19:48:54.852238    4892 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-jrm87" containerName="extractor"
Oct 30 19:49:09.720378 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:49:09.720344    4892 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w5-686689-openshift-default-5fbf5787bf-9tcts"]
Oct 30 19:49:09.720781 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:49:09.720547    4892 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w5-686689-openshift-default-5fbf5787bf-9tcts" podUID="1354d401-f0d6-4252-82d1-47fad799e04c" containerName="istio-proxy" containerID="cri-o://dee7dad7ea49c8478eb999fd610e045c82b71102bbb3e4b9470719bdbd699220" gracePeriod=30
Oct 30 19:49:09.770204 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:49:09.770164    4892 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w5-686689-openshift-default-5fbf5787bf-7b547"]
Oct 30 19:49:09.770381 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:49:09.770347    4892 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w5-686689-openshift-default-5fbf5787bf-7b547" podUID="098876de-0b44-493b-a9da-45280afe2cdf" containerName="istio-proxy" containerID="cri-o://ef2b5070ac0677e6a68b0547d89df8e41c1a7d13082ad109f63614b9a9d6c507" gracePeriod=30
Oct 30 19:49:09.774602 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:49:09.774577    4892 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w5-686689-openshift-default-5fbf5787bf-jrt7h"]
Oct 30 19:49:09.774796 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:49:09.774762    4892 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w5-686689-openshift-default-5fbf5787bf-jrt7h" podUID="fa668020-72be-46a7-aa53-e8249130c4f1" containerName="istio-proxy" containerID="cri-o://708e6d3035d20872700663b9fa3f2578cb766d96a3df70f687bb61988f9c56a7" gracePeriod=30
Oct 30 19:49:09.819089 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:49:09.819063    4892 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w5-686689-openshift-default-5fbf5787bf-nsql4"]
Oct 30 19:49:09.819262 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:49:09.819243    4892 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w5-686689-openshift-default-5fbf5787bf-nsql4" podUID="dffc80d1-bd47-4493-8cd9-87530034d485" containerName="istio-proxy" containerID="cri-o://997549f2768f85294570e473388022acab09b95ff50b2d4b15b7a7f3bd5503aa" gracePeriod=30
Oct 30 19:49:12.951675 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:49:12.951641    4892 patch_prober.go:29] interesting pod/e2e-worker-w5-686689-openshift-default-5fbf5787bf-jrt7h container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Oct 30 19:49:12.952180 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:49:12.951699    4892 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w5-686689-openshift-default-5fbf5787bf-jrt7h" podUID="fa668020-72be-46a7-aa53-e8249130c4f1" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Oct 30 19:49:14.915975 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:49:14.915918    4892 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2e1c40f5-mh9t2"]
Oct 30 19:49:14.917064 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:49:14.917042    4892 cpu_manager.go:414] "containerMap: removing container" podUID="5b7dcbb7-54cd-4d78-af9b-f1d8ccf00d3f" containerName="revoke-keys"
Oct 30 19:49:14.917123 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:49:14.917067    4892 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5b7dcbb7-54cd-4d78-af9b-f1d8ccf00d3f" containerName="revoke-keys"
Oct 30 19:49:14.917308 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:49:14.917294    4892 memory_manager.go:365] "RemoveStaleState removing state" podUID="5b7dcbb7-54cd-4d78-af9b-f1d8ccf00d3f" containerName="revoke-keys"
Oct 30 19:49:14.921484 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:49:14.921462    4892 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2e1c40f5-mh9t2"
Oct 30 19:49:14.992782 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:49:14.992702    4892 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4dzlp\" (UniqueName: \"kubernetes.io/projected/257ea0f4-ca30-42ea-ae62-c4975741b48b-kube-api-access-4dzlp\") pod \"maas-api-revoke-sub-2e1c40f5-mh9t2\" (UID: \"257ea0f4-ca30-42ea-ae62-c4975741b48b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2e1c40f5-mh9t2"
Oct 30 19:49:14.997845 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:49:14.993164    4892 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/257ea0f4-ca30-42ea-ae62-c4975741b48b-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-2e1c40f5-mh9t2\" (UID: \"257ea0f4-ca30-42ea-ae62-c4975741b48b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2e1c40f5-mh9t2"
Oct 30 19:49:14.997845 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:49:14.993337    4892 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/257ea0f4-ca30-42ea-ae62-c4975741b48b-maas-api-xks-ca\") pod \"maas-api-revoke-sub-2e1c40f5-mh9t2\" (UID: \"257ea0f4-ca30-42ea-ae62-c4975741b48b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2e1c40f5-mh9t2"
Oct 30 19:49:14.997845 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:49:14.997640    4892 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2e1c40f5-mh9t2"]
Oct 30 19:49:15.040037 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:49:15.026915    4892 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 19:49:15.040037 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:49:15.027088    4892 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-nlxp4\""
Oct 30 19:49:15.040037 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:49:15.027141    4892 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 30 19:49:15.040037 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:49:15.027208    4892 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 30 19:49:15.055216 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:49:15.055175    4892 patch_prober.go:29] interesting pod/e2e-worker-w5-686689-openshift-default-5fbf5787bf-7b547 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.114:15021/healthz/ready\": dial tcp 10.132.0.114:15021: connect: connection refused" start-of-body=
Oct 30 19:49:15.055424 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:49:15.055267    4892 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w5-686689-openshift-default-5fbf5787bf-7b547" podUID="098876de-0b44-493b-a9da-45280afe2cdf" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.114:15021/healthz/ready\": dial tcp 10.132.0.114:15021: connect: connection refused"
Oct 30 19:49:15.095765 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:49:15.094795    4892 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/257ea0f4-ca30-42ea-ae62-c4975741b48b-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-2e1c40f5-mh9t2\" (UID: \"257ea0f4-ca30-42ea-ae62-c4975741b48b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2e1c40f5-mh9t2"
Oct 30 19:49:15.095765 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:49:15.094927    4892 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/257ea0f4-ca30-42ea-ae62-c4975741b48b-maas-api-xks-ca\") pod \"maas-api-revoke-sub-2e1c40f5-mh9t2\" (UID: \"257ea0f4-ca30-42ea-ae62-c4975741b48b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2e1c40f5-mh9t2"
Oct 30 19:49:15.095765 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:49:15.095045    4892 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4dzlp\" (UniqueName: \"kubernetes.io/projected/257ea0f4-ca30-42ea-ae62-c4975741b48b-kube-api-access-4dzlp\") pod \"maas-api-revoke-sub-2e1c40f5-mh9t2\" (UID: \"257ea0f4-ca30-42ea-ae62-c4975741b48b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2e1c40f5-mh9t2"
Oct 30 19:49:15.095765 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:49:15.095717    4892 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/257ea0f4-ca30-42ea-ae62-c4975741b48b-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-2e1c40f5-mh9t2\" (UID: \"257ea0f4-ca30-42ea-ae62-c4975741b48b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2e1c40f5-mh9t2"
Oct 30 19:49:15.097727 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:49:15.097704    4892 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/257ea0f4-ca30-42ea-ae62-c4975741b48b-maas-api-xks-ca\") pod \"maas-api-revoke-sub-2e1c40f5-mh9t2\" (UID: \"257ea0f4-ca30-42ea-ae62-c4975741b48b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2e1c40f5-mh9t2"
Oct 30 19:49:15.117573 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:49:15.117506    4892 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-686689-openshift-default-5fbf5787bf-9tcts"
Oct 30 19:49:15.164162 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:49:15.163071    4892 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4dzlp\" (UniqueName: \"kubernetes.io/projected/257ea0f4-ca30-42ea-ae62-c4975741b48b-kube-api-access-4dzlp\") pod \"maas-api-revoke-sub-2e1c40f5-mh9t2\" (UID: \"257ea0f4-ca30-42ea-ae62-c4975741b48b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2e1c40f5-mh9t2"
Oct 30 19:49:15.196171 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:49:15.196136    4892 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1354d401-f0d6-4252-82d1-47fad799e04c-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1354d401-f0d6-4252-82d1-47fad799e04c-istio-data\") pod \"1354d401-f0d6-4252-82d1-47fad799e04c\" (UID: \"1354d401-f0d6-4252-82d1-47fad799e04c\") "
Oct 30 19:49:15.196317 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:49:15.196245    4892 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1354d401-f0d6-4252-82d1-47fad799e04c-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1354d401-f0d6-4252-82d1-47fad799e04c-istio-envoy\") pod \"1354d401-f0d6-4252-82d1-47fad799e04c\" (UID: \"1354d401-f0d6-4252-82d1-47fad799e04c\") "
Oct 30 19:49:15.196464 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:49:15.196428    4892 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1354d401-f0d6-4252-82d1-47fad799e04c-istio-data" pod "1354d401-f0d6-4252-82d1-47fad799e04c" (UID: "1354d401-f0d6-4252-82d1-47fad799e04c"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:49:15.196533 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:49:15.196446    4892 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1354d401-f0d6-4252-82d1-47fad799e04c-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1354d401-f0d6-4252-82d1-47fad799e04c-credential-socket\") pod \"1354d401-f0d6-4252-82d1-47fad799e04c\" (UID: \"1354d401-f0d6-4252-82d1-47fad799e04c\") "
Oct 30 19:49:15.196607 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:49:15.196545    4892 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1354d401-f0d6-4252-82d1-47fad799e04c-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1354d401-f0d6-4252-82d1-47fad799e04c-workload-certs\") pod \"1354d401-f0d6-4252-82d1-47fad799e04c\" (UID: \"1354d401-f0d6-4252-82d1-47fad799e04c\") "
Oct 30 19:49:15.196607 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:49:15.196589    4892 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/1354d401-f0d6-4252-82d1-47fad799e04c-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1354d401-f0d6-4252-82d1-47fad799e04c-istio-podinfo\") pod \"1354d401-f0d6-4252-82d1-47fad799e04c\" (UID: \"1354d401-f0d6-4252-82d1-47fad799e04c\") "
Oct 30 19:49:15.196716 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:49:15.196652    4892 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1354d401-f0d6-4252-82d1-47fad799e04c-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1354d401-f0d6-4252-82d1-47fad799e04c-workload-socket\") pod \"1354d401-f0d6-4252-82d1-47fad799e04c\" (UID: \"1354d401-f0d6-4252-82d1-47fad799e04c\") "
Oct 30 19:49:15.196716 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:49:15.196693    4892 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1354d401-f0d6-4252-82d1-47fad799e04c-credential-socket" pod "1354d401-f0d6-4252-82d1-47fad799e04c" (UID: "1354d401-f0d6-4252-82d1-47fad799e04c"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:49:15.196835 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:49:15.196731    4892 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/1354d401-f0d6-4252-82d1-47fad799e04c-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1354d401-f0d6-4252-82d1-47fad799e04c-istiod-ca-cert\") pod \"1354d401-f0d6-4252-82d1-47fad799e04c\" (UID: \"1354d401-f0d6-4252-82d1-47fad799e04c\") "
Oct 30 19:49:15.196835 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:49:15.196762    4892 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1354d401-f0d6-4252-82d1-47fad799e04c-kube-api-access-tnmv5\" (UniqueName: \"kubernetes.io/projected/1354d401-f0d6-4252-82d1-47fad799e04c-kube-api-access-tnmv5\") pod \"1354d401-f0d6-4252-82d1-47fad799e04c\" (UID: \"1354d401-f0d6-4252-82d1-47fad799e04c\") "
Oct 30 19:49:15.196950 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:49:15.196866    4892 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1354d401-f0d6-4252-82d1-47fad799e04c-istio-token\" (UniqueName: \"kubernetes.io/projected/1354d401-f0d6-4252-82d1-47fad799e04c-istio-token\") pod \"1354d401-f0d6-4252-82d1-47fad799e04c\" (UID: \"1354d401-f0d6-4252-82d1-47fad799e04c\") "
Oct 30 19:49:15.196950 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:49:15.196939    4892 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1354d401-f0d6-4252-82d1-47fad799e04c-workload-certs" pod "1354d401-f0d6-4252-82d1-47fad799e04c" (UID: "1354d401-f0d6-4252-82d1-47fad799e04c"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:49:15.197159 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:49:15.197125    4892 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1354d401-f0d6-4252-82d1-47fad799e04c-workload-socket" pod "1354d401-f0d6-4252-82d1-47fad799e04c" (UID: "1354d401-f0d6-4252-82d1-47fad799e04c"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:49:15.197655 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:49:15.197630    4892 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1354d401-f0d6-4252-82d1-47fad799e04c-istio-data\") on node \"ip-10-0-132-82.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 19:49:15.197748 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:49:15.197660    4892 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1354d401-f0d6-4252-82d1-47fad799e04c-credential-socket\") on node \"ip-10-0-132-82.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 19:49:15.197748 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:49:15.197675    4892 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1354d401-f0d6-4252-82d1-47fad799e04c-workload-certs\") on node \"ip-10-0-132-82.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 19:49:15.197748 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:49:15.197690    4892 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1354d401-f0d6-4252-82d1-47fad799e04c-workload-socket\") on node \"ip-10-0-132-82.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 19:49:15.198126 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:49:15.198048    4892 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1354d401-f0d6-4252-82d1-47fad799e04c-istiod-ca-cert" pod "1354d401-f0d6-4252-82d1-47fad799e04c" (UID: "1354d401-f0d6-4252-82d1-47fad799e04c"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 19:49:15.212194 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:49:15.212085    4892 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1354d401-f0d6-4252-82d1-47fad799e04c-istio-envoy" pod "1354d401-f0d6-4252-82d1-47fad799e04c" (UID: "1354d401-f0d6-4252-82d1-47fad799e04c"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:49:15.214067 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:49:15.213925    4892 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/1354d401-f0d6-4252-82d1-47fad799e04c-istio-podinfo" pod "1354d401-f0d6-4252-82d1-47fad799e04c" (UID: "1354d401-f0d6-4252-82d1-47fad799e04c"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 19:49:15.214144 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:49:15.214114    4892 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1354d401-f0d6-4252-82d1-47fad799e04c-istio-token" pod "1354d401-f0d6-4252-82d1-47fad799e04c" (UID: "1354d401-f0d6-4252-82d1-47fad799e04c"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:49:15.214897 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:49:15.214812    4892 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1354d401-f0d6-4252-82d1-47fad799e04c-kube-api-access-tnmv5" pod "1354d401-f0d6-4252-82d1-47fad799e04c" (UID: "1354d401-f0d6-4252-82d1-47fad799e04c"). InnerVolumeSpecName "kube-api-access-tnmv5". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:49:15.228277 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:49:15.228203    4892 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-686689-openshift-default-5fbf5787bf-7b547"
Oct 30 19:49:15.234038 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:49:15.234015    4892 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-686689-openshift-default-5fbf5787bf-jrt7h"
Oct 30 19:49:15.283279 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:49:15.283250    4892 generic.go:356] "Generic (PLEG): container finished" podID="fa668020-72be-46a7-aa53-e8249130c4f1" containerID="708e6d3035d20872700663b9fa3f2578cb766d96a3df70f687bb61988f9c56a7" exitCode=0
Oct 30 19:49:15.283438 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:49:15.283394    4892 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-686689-openshift-default-5fbf5787bf-jrt7h" event={"ID":"fa668020-72be-46a7-aa53-e8249130c4f1","Type":"ContainerDied","Data":"708e6d3035d20872700663b9fa3f2578cb766d96a3df70f687bb61988f9c56a7"}
Oct 30 19:49:15.283514 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:49:15.283457    4892 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-686689-openshift-default-5fbf5787bf-jrt7h" event={"ID":"fa668020-72be-46a7-aa53-e8249130c4f1","Type":"ContainerDied","Data":"c8f34947636c0237d3bb41844f7545ded494ee81bd036089b50da280252418b8"}
Oct 30 19:49:15.283514 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:49:15.283478    4892 scope.go:122] "RemoveContainer" containerID="708e6d3035d20872700663b9fa3f2578cb766d96a3df70f687bb61988f9c56a7"
Oct 30 19:49:15.284278 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:49:15.284255    4892 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-686689-openshift-default-5fbf5787bf-jrt7h"
Oct 30 19:49:15.290314 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:49:15.286810    4892 generic.go:356] "Generic (PLEG): container finished" podID="1354d401-f0d6-4252-82d1-47fad799e04c" containerID="dee7dad7ea49c8478eb999fd610e045c82b71102bbb3e4b9470719bdbd699220" exitCode=0
Oct 30 19:49:15.290314 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:49:15.286898    4892 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-686689-openshift-default-5fbf5787bf-9tcts" event={"ID":"1354d401-f0d6-4252-82d1-47fad799e04c","Type":"ContainerDied","Data":"dee7dad7ea49c8478eb999fd610e045c82b71102bbb3e4b9470719bdbd699220"}
Oct 30 19:49:15.290314 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:49:15.286941    4892 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-686689-openshift-default-5fbf5787bf-9tcts" event={"ID":"1354d401-f0d6-4252-82d1-47fad799e04c","Type":"ContainerDied","Data":"cd85568748275a5cc1a4e70ca5dbfd7403ab164c5e8715ac0020b2a1014973c5"}
Oct 30 19:49:15.290314 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:49:15.287047    4892 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-686689-openshift-default-5fbf5787bf-9tcts"
Oct 30 19:49:15.291446 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:49:15.291418    4892 generic.go:356] "Generic (PLEG): container finished" podID="dffc80d1-bd47-4493-8cd9-87530034d485" containerID="997549f2768f85294570e473388022acab09b95ff50b2d4b15b7a7f3bd5503aa" exitCode=0
Oct 30 19:49:15.291544 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:49:15.291461    4892 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-686689-openshift-default-5fbf5787bf-nsql4" event={"ID":"dffc80d1-bd47-4493-8cd9-87530034d485","Type":"ContainerDied","Data":"997549f2768f85294570e473388022acab09b95ff50b2d4b15b7a7f3bd5503aa"}
Oct 30 19:49:15.293911 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:49:15.293890    4892 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-686689-openshift-default-5fbf5787bf-nsql4"
Oct 30 19:49:15.298290 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:49:15.298269    4892 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/fa668020-72be-46a7-aa53-e8249130c4f1-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/fa668020-72be-46a7-aa53-e8249130c4f1-credential-socket\") pod \"fa668020-72be-46a7-aa53-e8249130c4f1\" (UID: \"fa668020-72be-46a7-aa53-e8249130c4f1\") "
Oct 30 19:49:15.298456 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:49:15.298438    4892 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/fa668020-72be-46a7-aa53-e8249130c4f1-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/fa668020-72be-46a7-aa53-e8249130c4f1-istio-data\") pod \"fa668020-72be-46a7-aa53-e8249130c4f1\" (UID: \"fa668020-72be-46a7-aa53-e8249130c4f1\") "
Oct 30 19:49:15.298579 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:49:15.298555    4892 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fa668020-72be-46a7-aa53-e8249130c4f1-credential-socket" pod "fa668020-72be-46a7-aa53-e8249130c4f1" (UID: "fa668020-72be-46a7-aa53-e8249130c4f1"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:49:15.298667 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:49:15.298561    4892 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/fa668020-72be-46a7-aa53-e8249130c4f1-kube-api-access-f7478\" (UniqueName: \"kubernetes.io/projected/fa668020-72be-46a7-aa53-e8249130c4f1-kube-api-access-f7478\") pod \"fa668020-72be-46a7-aa53-e8249130c4f1\" (UID: \"fa668020-72be-46a7-aa53-e8249130c4f1\") "
Oct 30 19:49:15.298667 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:49:15.298648    4892 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/fa668020-72be-46a7-aa53-e8249130c4f1-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/fa668020-72be-46a7-aa53-e8249130c4f1-workload-certs\") pod \"fa668020-72be-46a7-aa53-e8249130c4f1\" (UID: \"fa668020-72be-46a7-aa53-e8249130c4f1\") "
Oct 30 19:49:15.298796 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:49:15.298776    4892 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/098876de-0b44-493b-a9da-45280afe2cdf-kube-api-access-h4jt6\" (UniqueName: \"kubernetes.io/projected/098876de-0b44-493b-a9da-45280afe2cdf-kube-api-access-h4jt6\") pod \"098876de-0b44-493b-a9da-45280afe2cdf\" (UID: \"098876de-0b44-493b-a9da-45280afe2cdf\") "
Oct 30 19:49:15.298860 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:49:15.298835    4892 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/098876de-0b44-493b-a9da-45280afe2cdf-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/098876de-0b44-493b-a9da-45280afe2cdf-istio-data\") pod \"098876de-0b44-493b-a9da-45280afe2cdf\" (UID: \"098876de-0b44-493b-a9da-45280afe2cdf\") "
Oct 30 19:49:15.298914 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:49:15.298877    4892 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/098876de-0b44-493b-a9da-45280afe2cdf-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/098876de-0b44-493b-a9da-45280afe2cdf-workload-certs\") pod \"098876de-0b44-493b-a9da-45280afe2cdf\" (UID: \"098876de-0b44-493b-a9da-45280afe2cdf\") "
Oct 30 19:49:15.298914 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:49:15.298905    4892 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/098876de-0b44-493b-a9da-45280afe2cdf-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/098876de-0b44-493b-a9da-45280afe2cdf-istiod-ca-cert\") pod \"098876de-0b44-493b-a9da-45280afe2cdf\" (UID: \"098876de-0b44-493b-a9da-45280afe2cdf\") "
Oct 30 19:49:15.299047 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:49:15.298936    4892 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/fa668020-72be-46a7-aa53-e8249130c4f1-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/fa668020-72be-46a7-aa53-e8249130c4f1-istiod-ca-cert\") pod \"fa668020-72be-46a7-aa53-e8249130c4f1\" (UID: \"fa668020-72be-46a7-aa53-e8249130c4f1\") "
Oct 30 19:49:15.299047 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:49:15.299031    4892 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/098876de-0b44-493b-a9da-45280afe2cdf-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/098876de-0b44-493b-a9da-45280afe2cdf-istio-envoy\") pod \"098876de-0b44-493b-a9da-45280afe2cdf\" (UID: \"098876de-0b44-493b-a9da-45280afe2cdf\") "
Oct 30 19:49:15.299227 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:49:15.299076    4892 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/fa668020-72be-46a7-aa53-e8249130c4f1-istio-token\" (UniqueName: \"kubernetes.io/projected/fa668020-72be-46a7-aa53-e8249130c4f1-istio-token\") pod \"fa668020-72be-46a7-aa53-e8249130c4f1\" (UID: \"fa668020-72be-46a7-aa53-e8249130c4f1\") "
Oct 30 19:49:15.299227 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:49:15.299137    4892 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/098876de-0b44-493b-a9da-45280afe2cdf-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/098876de-0b44-493b-a9da-45280afe2cdf-workload-socket\") pod \"098876de-0b44-493b-a9da-45280afe2cdf\" (UID: \"098876de-0b44-493b-a9da-45280afe2cdf\") "
Oct 30 19:49:15.299405 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:49:15.299387    4892 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/fa668020-72be-46a7-aa53-e8249130c4f1-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/fa668020-72be-46a7-aa53-e8249130c4f1-workload-socket\") pod \"fa668020-72be-46a7-aa53-e8249130c4f1\" (UID: \"fa668020-72be-46a7-aa53-e8249130c4f1\") "
Oct 30 19:49:15.300177 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:49:15.299578    4892 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/098876de-0b44-493b-a9da-45280afe2cdf-istiod-ca-cert" pod "098876de-0b44-493b-a9da-45280afe2cdf" (UID: "098876de-0b44-493b-a9da-45280afe2cdf"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 19:49:15.300177 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:49:15.299614    4892 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/fa668020-72be-46a7-aa53-e8249130c4f1-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/fa668020-72be-46a7-aa53-e8249130c4f1-istio-podinfo\") pod \"fa668020-72be-46a7-aa53-e8249130c4f1\" (UID: \"fa668020-72be-46a7-aa53-e8249130c4f1\") "
Oct 30 19:49:15.300177 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:49:15.299751    4892 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/098876de-0b44-493b-a9da-45280afe2cdf-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/098876de-0b44-493b-a9da-45280afe2cdf-istio-podinfo\") pod \"098876de-0b44-493b-a9da-45280afe2cdf\" (UID: \"098876de-0b44-493b-a9da-45280afe2cdf\") "
Oct 30 19:49:15.300177 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:49:15.299790    4892 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/098876de-0b44-493b-a9da-45280afe2cdf-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/098876de-0b44-493b-a9da-45280afe2cdf-credential-socket\") pod \"098876de-0b44-493b-a9da-45280afe2cdf\" (UID: \"098876de-0b44-493b-a9da-45280afe2cdf\") "
Oct 30 19:49:15.300177 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:49:15.299795    4892 generic.go:356] "Generic (PLEG): container finished" podID="098876de-0b44-493b-a9da-45280afe2cdf" containerID="ef2b5070ac0677e6a68b0547d89df8e41c1a7d13082ad109f63614b9a9d6c507" exitCode=0
Oct 30 19:49:15.300177 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:49:15.299818    4892 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/fa668020-72be-46a7-aa53-e8249130c4f1-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/fa668020-72be-46a7-aa53-e8249130c4f1-istio-envoy\") pod \"fa668020-72be-46a7-aa53-e8249130c4f1\" (UID: \"fa668020-72be-46a7-aa53-e8249130c4f1\") "
Oct 30 19:49:15.300177 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:49:15.299867    4892 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/098876de-0b44-493b-a9da-45280afe2cdf-istio-token\" (UniqueName: \"kubernetes.io/projected/098876de-0b44-493b-a9da-45280afe2cdf-istio-token\") pod \"098876de-0b44-493b-a9da-45280afe2cdf\" (UID: \"098876de-0b44-493b-a9da-45280afe2cdf\") "
Oct 30 19:49:15.300177 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:49:15.299867    4892 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fa668020-72be-46a7-aa53-e8249130c4f1-workload-certs" pod "fa668020-72be-46a7-aa53-e8249130c4f1" (UID: "fa668020-72be-46a7-aa53-e8249130c4f1"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:49:15.300177 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:49:15.299927    4892 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-686689-openshift-default-5fbf5787bf-7b547" event={"ID":"098876de-0b44-493b-a9da-45280afe2cdf","Type":"ContainerDied","Data":"ef2b5070ac0677e6a68b0547d89df8e41c1a7d13082ad109f63614b9a9d6c507"}
Oct 30 19:49:15.300177 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:49:15.300032    4892 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-686689-openshift-default-5fbf5787bf-7b547" event={"ID":"098876de-0b44-493b-a9da-45280afe2cdf","Type":"ContainerDied","Data":"c56a324eb3f4c166a59bb3aa1c32c0d3a36e213fc254f9f5025a61e6454b8a55"}
Oct 30 19:49:15.300177 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:49:15.300093    4892 scope.go:122] "RemoveContainer" containerID="708e6d3035d20872700663b9fa3f2578cb766d96a3df70f687bb61988f9c56a7"
Oct 30 19:49:15.300177 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:49:15.300171    4892 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-686689-openshift-default-5fbf5787bf-7b547"
Oct 30 19:49:15.301056 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:49:15.300437    4892 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/fa668020-72be-46a7-aa53-e8249130c4f1-credential-socket\") on node \"ip-10-0-132-82.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 19:49:15.301056 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:49:15.300456    4892 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/fa668020-72be-46a7-aa53-e8249130c4f1-workload-certs\") on node \"ip-10-0-132-82.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 19:49:15.301056 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:49:15.300468    4892 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1354d401-f0d6-4252-82d1-47fad799e04c-istio-podinfo\") on node \"ip-10-0-132-82.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 19:49:15.301056 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:49:15.300476    4892 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1354d401-f0d6-4252-82d1-47fad799e04c-istiod-ca-cert\") on node \"ip-10-0-132-82.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 19:49:15.301056 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:49:15.300485    4892 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tnmv5\" (UniqueName: \"kubernetes.io/projected/1354d401-f0d6-4252-82d1-47fad799e04c-kube-api-access-tnmv5\") on node \"ip-10-0-132-82.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 19:49:15.301056 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:49:15.300498    4892 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1354d401-f0d6-4252-82d1-47fad799e04c-istio-token\") on node \"ip-10-0-132-82.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 19:49:15.301056 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:49:15.300511    4892 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/098876de-0b44-493b-a9da-45280afe2cdf-istiod-ca-cert\") on node \"ip-10-0-132-82.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 19:49:15.301056 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:49:15.300520    4892 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1354d401-f0d6-4252-82d1-47fad799e04c-istio-envoy\") on node \"ip-10-0-132-82.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 19:49:15.301056 ip-10-0-132-82 kubenswrapper[4892]: E1030 19:49:15.300550    4892 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"708e6d3035d20872700663b9fa3f2578cb766d96a3df70f687bb61988f9c56a7\": container with ID starting with 708e6d3035d20872700663b9fa3f2578cb766d96a3df70f687bb61988f9c56a7 not found: ID does not exist" containerID="708e6d3035d20872700663b9fa3f2578cb766d96a3df70f687bb61988f9c56a7"
Oct 30 19:49:15.301056 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:49:15.300581    4892 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"708e6d3035d20872700663b9fa3f2578cb766d96a3df70f687bb61988f9c56a7"} err="failed to get container status \"708e6d3035d20872700663b9fa3f2578cb766d96a3df70f687bb61988f9c56a7\": rpc error: code = NotFound desc = could not find container \"708e6d3035d20872700663b9fa3f2578cb766d96a3df70f687bb61988f9c56a7\": container with ID starting with 708e6d3035d20872700663b9fa3f2578cb766d96a3df70f687bb61988f9c56a7 not found: ID does not exist"
Oct 30 19:49:15.301056 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:49:15.300618    4892 scope.go:122] "RemoveContainer" containerID="dee7dad7ea49c8478eb999fd610e045c82b71102bbb3e4b9470719bdbd699220"
Oct 30 19:49:15.301056 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:49:15.300911    4892 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/098876de-0b44-493b-a9da-45280afe2cdf-istio-data" pod "098876de-0b44-493b-a9da-45280afe2cdf" (UID: "098876de-0b44-493b-a9da-45280afe2cdf"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:49:15.301759 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:49:15.301086    4892 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2e1c40f5-mh9t2"
Oct 30 19:49:15.301759 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:49:15.301548    4892 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/098876de-0b44-493b-a9da-45280afe2cdf-workload-certs" pod "098876de-0b44-493b-a9da-45280afe2cdf" (UID: "098876de-0b44-493b-a9da-45280afe2cdf"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:49:15.301865 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:49:15.301761    4892 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fa668020-72be-46a7-aa53-e8249130c4f1-workload-socket" pod "fa668020-72be-46a7-aa53-e8249130c4f1" (UID: "fa668020-72be-46a7-aa53-e8249130c4f1"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:49:15.302068 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:49:15.302028    4892 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fa668020-72be-46a7-aa53-e8249130c4f1-istio-data" pod "fa668020-72be-46a7-aa53-e8249130c4f1" (UID: "fa668020-72be-46a7-aa53-e8249130c4f1"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:49:15.302174 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:49:15.302148    4892 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/fa668020-72be-46a7-aa53-e8249130c4f1-istiod-ca-cert" pod "fa668020-72be-46a7-aa53-e8249130c4f1" (UID: "fa668020-72be-46a7-aa53-e8249130c4f1"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 19:49:15.303614 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:49:15.303589    4892 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/098876de-0b44-493b-a9da-45280afe2cdf-istio-token" pod "098876de-0b44-493b-a9da-45280afe2cdf" (UID: "098876de-0b44-493b-a9da-45280afe2cdf"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:49:15.304395 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:49:15.304373    4892 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/098876de-0b44-493b-a9da-45280afe2cdf-credential-socket" pod "098876de-0b44-493b-a9da-45280afe2cdf" (UID: "098876de-0b44-493b-a9da-45280afe2cdf"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:49:15.304541 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:49:15.304412    4892 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/098876de-0b44-493b-a9da-45280afe2cdf-workload-socket" pod "098876de-0b44-493b-a9da-45280afe2cdf" (UID: "098876de-0b44-493b-a9da-45280afe2cdf"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:49:15.304659 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:49:15.304429    4892 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/098876de-0b44-493b-a9da-45280afe2cdf-kube-api-access-h4jt6" pod "098876de-0b44-493b-a9da-45280afe2cdf" (UID: "098876de-0b44-493b-a9da-45280afe2cdf"). InnerVolumeSpecName "kube-api-access-h4jt6". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:49:15.309114 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:49:15.309095    4892 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/098876de-0b44-493b-a9da-45280afe2cdf-istio-podinfo" pod "098876de-0b44-493b-a9da-45280afe2cdf" (UID: "098876de-0b44-493b-a9da-45280afe2cdf"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 19:49:15.309217 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:49:15.309153    4892 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fa668020-72be-46a7-aa53-e8249130c4f1-istio-envoy" pod "fa668020-72be-46a7-aa53-e8249130c4f1" (UID: "fa668020-72be-46a7-aa53-e8249130c4f1"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:49:15.310276 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:49:15.310251    4892 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fa668020-72be-46a7-aa53-e8249130c4f1-kube-api-access-f7478" pod "fa668020-72be-46a7-aa53-e8249130c4f1" (UID: "fa668020-72be-46a7-aa53-e8249130c4f1"). InnerVolumeSpecName "kube-api-access-f7478". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:49:15.323952 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:49:15.323933    4892 scope.go:122] "RemoveContainer" containerID="dee7dad7ea49c8478eb999fd610e045c82b71102bbb3e4b9470719bdbd699220"
Oct 30 19:49:15.324382 ip-10-0-132-82 kubenswrapper[4892]: E1030 19:49:15.324361    4892 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"dee7dad7ea49c8478eb999fd610e045c82b71102bbb3e4b9470719bdbd699220\": container with ID starting with dee7dad7ea49c8478eb999fd610e045c82b71102bbb3e4b9470719bdbd699220 not found: ID does not exist" containerID="dee7dad7ea49c8478eb999fd610e045c82b71102bbb3e4b9470719bdbd699220"
Oct 30 19:49:15.324473 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:49:15.324393    4892 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"dee7dad7ea49c8478eb999fd610e045c82b71102bbb3e4b9470719bdbd699220"} err="failed to get container status \"dee7dad7ea49c8478eb999fd610e045c82b71102bbb3e4b9470719bdbd699220\": rpc error: code = NotFound desc = could not find container \"dee7dad7ea49c8478eb999fd610e045c82b71102bbb3e4b9470719bdbd699220\": container with ID starting with dee7dad7ea49c8478eb999fd610e045c82b71102bbb3e4b9470719bdbd699220 not found: ID does not exist"
Oct 30 19:49:15.324473 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:49:15.324419    4892 scope.go:122] "RemoveContainer" containerID="ef2b5070ac0677e6a68b0547d89df8e41c1a7d13082ad109f63614b9a9d6c507"
Oct 30 19:49:15.326470 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:49:15.326441    4892 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/fa668020-72be-46a7-aa53-e8249130c4f1-istio-podinfo" pod "fa668020-72be-46a7-aa53-e8249130c4f1" (UID: "fa668020-72be-46a7-aa53-e8249130c4f1"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 19:49:15.326579 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:49:15.326500    4892 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fa668020-72be-46a7-aa53-e8249130c4f1-istio-token" pod "fa668020-72be-46a7-aa53-e8249130c4f1" (UID: "fa668020-72be-46a7-aa53-e8249130c4f1"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:49:15.326837 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:49:15.326812    4892 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/098876de-0b44-493b-a9da-45280afe2cdf-istio-envoy" pod "098876de-0b44-493b-a9da-45280afe2cdf" (UID: "098876de-0b44-493b-a9da-45280afe2cdf"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:49:15.338834 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:49:15.338813    4892 scope.go:122] "RemoveContainer" containerID="ef2b5070ac0677e6a68b0547d89df8e41c1a7d13082ad109f63614b9a9d6c507"
Oct 30 19:49:15.339190 ip-10-0-132-82 kubenswrapper[4892]: E1030 19:49:15.339167    4892 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ef2b5070ac0677e6a68b0547d89df8e41c1a7d13082ad109f63614b9a9d6c507\": container with ID starting with ef2b5070ac0677e6a68b0547d89df8e41c1a7d13082ad109f63614b9a9d6c507 not found: ID does not exist" containerID="ef2b5070ac0677e6a68b0547d89df8e41c1a7d13082ad109f63614b9a9d6c507"
Oct 30 19:49:15.339275 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:49:15.339200    4892 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ef2b5070ac0677e6a68b0547d89df8e41c1a7d13082ad109f63614b9a9d6c507"} err="failed to get container status \"ef2b5070ac0677e6a68b0547d89df8e41c1a7d13082ad109f63614b9a9d6c507\": rpc error: code = NotFound desc = could not find container \"ef2b5070ac0677e6a68b0547d89df8e41c1a7d13082ad109f63614b9a9d6c507\": container with ID starting with ef2b5070ac0677e6a68b0547d89df8e41c1a7d13082ad109f63614b9a9d6c507 not found: ID does not exist"
Oct 30 19:49:15.408005 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:49:15.404947    4892 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-f7478\" (UniqueName: \"kubernetes.io/projected/fa668020-72be-46a7-aa53-e8249130c4f1-kube-api-access-f7478\") on node \"ip-10-0-132-82.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 19:49:15.408005 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:49:15.405003    4892 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-h4jt6\" (UniqueName: \"kubernetes.io/projected/098876de-0b44-493b-a9da-45280afe2cdf-kube-api-access-h4jt6\") on node \"ip-10-0-132-82.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 19:49:15.408005 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:49:15.405020    4892 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/098876de-0b44-493b-a9da-45280afe2cdf-istio-data\") on node \"ip-10-0-132-82.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 19:49:15.408005 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:49:15.405033    4892 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/098876de-0b44-493b-a9da-45280afe2cdf-workload-certs\") on node \"ip-10-0-132-82.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 19:49:15.408005 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:49:15.405046    4892 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/fa668020-72be-46a7-aa53-e8249130c4f1-istiod-ca-cert\") on node \"ip-10-0-132-82.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 19:49:15.408005 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:49:15.405059    4892 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/098876de-0b44-493b-a9da-45280afe2cdf-istio-envoy\") on node \"ip-10-0-132-82.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 19:49:15.408005 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:49:15.405071    4892 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/fa668020-72be-46a7-aa53-e8249130c4f1-istio-token\") on node \"ip-10-0-132-82.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 19:49:15.408005 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:49:15.405082    4892 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/098876de-0b44-493b-a9da-45280afe2cdf-workload-socket\") on node \"ip-10-0-132-82.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 19:49:15.408005 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:49:15.405094    4892 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/fa668020-72be-46a7-aa53-e8249130c4f1-workload-socket\") on node \"ip-10-0-132-82.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 19:49:15.408005 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:49:15.405106    4892 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/fa668020-72be-46a7-aa53-e8249130c4f1-istio-podinfo\") on node \"ip-10-0-132-82.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 19:49:15.408005 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:49:15.405117    4892 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/098876de-0b44-493b-a9da-45280afe2cdf-istio-podinfo\") on node \"ip-10-0-132-82.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 19:49:15.408005 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:49:15.405130    4892 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/098876de-0b44-493b-a9da-45280afe2cdf-credential-socket\") on node \"ip-10-0-132-82.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 19:49:15.408005 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:49:15.405144    4892 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/fa668020-72be-46a7-aa53-e8249130c4f1-istio-envoy\") on node \"ip-10-0-132-82.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 19:49:15.408005 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:49:15.405155    4892 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/098876de-0b44-493b-a9da-45280afe2cdf-istio-token\") on node \"ip-10-0-132-82.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 19:49:15.408005 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:49:15.405169    4892 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/fa668020-72be-46a7-aa53-e8249130c4f1-istio-data\") on node \"ip-10-0-132-82.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 19:49:15.517755 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:49:15.511509    4892 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/dffc80d1-bd47-4493-8cd9-87530034d485-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/dffc80d1-bd47-4493-8cd9-87530034d485-istio-podinfo\") pod \"dffc80d1-bd47-4493-8cd9-87530034d485\" (UID: \"dffc80d1-bd47-4493-8cd9-87530034d485\") "
Oct 30 19:49:15.517755 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:49:15.511790    4892 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/dffc80d1-bd47-4493-8cd9-87530034d485-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/dffc80d1-bd47-4493-8cd9-87530034d485-istio-envoy\") pod \"dffc80d1-bd47-4493-8cd9-87530034d485\" (UID: \"dffc80d1-bd47-4493-8cd9-87530034d485\") "
Oct 30 19:49:15.517755 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:49:15.511892    4892 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/dffc80d1-bd47-4493-8cd9-87530034d485-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/dffc80d1-bd47-4493-8cd9-87530034d485-credential-socket\") pod \"dffc80d1-bd47-4493-8cd9-87530034d485\" (UID: \"dffc80d1-bd47-4493-8cd9-87530034d485\") "
Oct 30 19:49:15.517755 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:49:15.511920    4892 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/dffc80d1-bd47-4493-8cd9-87530034d485-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/dffc80d1-bd47-4493-8cd9-87530034d485-workload-certs\") pod \"dffc80d1-bd47-4493-8cd9-87530034d485\" (UID: \"dffc80d1-bd47-4493-8cd9-87530034d485\") "
Oct 30 19:49:15.517755 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:49:15.512022    4892 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/dffc80d1-bd47-4493-8cd9-87530034d485-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/dffc80d1-bd47-4493-8cd9-87530034d485-workload-socket\") pod \"dffc80d1-bd47-4493-8cd9-87530034d485\" (UID: \"dffc80d1-bd47-4493-8cd9-87530034d485\") "
Oct 30 19:49:15.517755 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:49:15.512050    4892 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/dffc80d1-bd47-4493-8cd9-87530034d485-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/dffc80d1-bd47-4493-8cd9-87530034d485-istio-data\") pod \"dffc80d1-bd47-4493-8cd9-87530034d485\" (UID: \"dffc80d1-bd47-4493-8cd9-87530034d485\") "
Oct 30 19:49:15.517755 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:49:15.512078    4892 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/dffc80d1-bd47-4493-8cd9-87530034d485-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/dffc80d1-bd47-4493-8cd9-87530034d485-istiod-ca-cert\") pod \"dffc80d1-bd47-4493-8cd9-87530034d485\" (UID: \"dffc80d1-bd47-4493-8cd9-87530034d485\") "
Oct 30 19:49:15.517755 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:49:15.512108    4892 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/dffc80d1-bd47-4493-8cd9-87530034d485-kube-api-access-6c89v\" (UniqueName: \"kubernetes.io/projected/dffc80d1-bd47-4493-8cd9-87530034d485-kube-api-access-6c89v\") pod \"dffc80d1-bd47-4493-8cd9-87530034d485\" (UID: \"dffc80d1-bd47-4493-8cd9-87530034d485\") "
Oct 30 19:49:15.517755 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:49:15.512226    4892 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/dffc80d1-bd47-4493-8cd9-87530034d485-istio-token\" (UniqueName: \"kubernetes.io/projected/dffc80d1-bd47-4493-8cd9-87530034d485-istio-token\") pod \"dffc80d1-bd47-4493-8cd9-87530034d485\" (UID: \"dffc80d1-bd47-4493-8cd9-87530034d485\") "
Oct 30 19:49:15.523802 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:49:15.519887    4892 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/dffc80d1-bd47-4493-8cd9-87530034d485-workload-certs" pod "dffc80d1-bd47-4493-8cd9-87530034d485" (UID: "dffc80d1-bd47-4493-8cd9-87530034d485"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:49:15.523802 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:49:15.521198    4892 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/dffc80d1-bd47-4493-8cd9-87530034d485-istiod-ca-cert" pod "dffc80d1-bd47-4493-8cd9-87530034d485" (UID: "dffc80d1-bd47-4493-8cd9-87530034d485"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 19:49:15.523802 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:49:15.521293    4892 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/dffc80d1-bd47-4493-8cd9-87530034d485-credential-socket" pod "dffc80d1-bd47-4493-8cd9-87530034d485" (UID: "dffc80d1-bd47-4493-8cd9-87530034d485"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:49:15.523802 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:49:15.521509    4892 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/dffc80d1-bd47-4493-8cd9-87530034d485-workload-socket" pod "dffc80d1-bd47-4493-8cd9-87530034d485" (UID: "dffc80d1-bd47-4493-8cd9-87530034d485"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:49:15.523802 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:49:15.521796    4892 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/dffc80d1-bd47-4493-8cd9-87530034d485-istio-data" pod "dffc80d1-bd47-4493-8cd9-87530034d485" (UID: "dffc80d1-bd47-4493-8cd9-87530034d485"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:49:15.524713 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:49:15.524690    4892 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/dffc80d1-bd47-4493-8cd9-87530034d485-kube-api-access-6c89v" pod "dffc80d1-bd47-4493-8cd9-87530034d485" (UID: "dffc80d1-bd47-4493-8cd9-87530034d485"). InnerVolumeSpecName "kube-api-access-6c89v". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:49:15.526754 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:49:15.526655    4892 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/dffc80d1-bd47-4493-8cd9-87530034d485-istio-envoy" pod "dffc80d1-bd47-4493-8cd9-87530034d485" (UID: "dffc80d1-bd47-4493-8cd9-87530034d485"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:49:15.558098 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:49:15.558043    4892 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/dffc80d1-bd47-4493-8cd9-87530034d485-istio-podinfo" pod "dffc80d1-bd47-4493-8cd9-87530034d485" (UID: "dffc80d1-bd47-4493-8cd9-87530034d485"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 19:49:15.573896 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:49:15.573076    4892 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/dffc80d1-bd47-4493-8cd9-87530034d485-istio-token" pod "dffc80d1-bd47-4493-8cd9-87530034d485" (UID: "dffc80d1-bd47-4493-8cd9-87530034d485"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:49:15.614999 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:49:15.614952    4892 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/dffc80d1-bd47-4493-8cd9-87530034d485-istio-envoy\") on node \"ip-10-0-132-82.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 19:49:15.615102 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:49:15.615007    4892 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/dffc80d1-bd47-4493-8cd9-87530034d485-credential-socket\") on node \"ip-10-0-132-82.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 19:49:15.615102 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:49:15.615026    4892 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/dffc80d1-bd47-4493-8cd9-87530034d485-workload-certs\") on node \"ip-10-0-132-82.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 19:49:15.615102 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:49:15.615040    4892 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/dffc80d1-bd47-4493-8cd9-87530034d485-workload-socket\") on node \"ip-10-0-132-82.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 19:49:15.615102 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:49:15.615054    4892 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/dffc80d1-bd47-4493-8cd9-87530034d485-istio-data\") on node \"ip-10-0-132-82.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 19:49:15.615102 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:49:15.615067    4892 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/dffc80d1-bd47-4493-8cd9-87530034d485-istiod-ca-cert\") on node \"ip-10-0-132-82.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 19:49:15.615102 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:49:15.615084    4892 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6c89v\" (UniqueName: \"kubernetes.io/projected/dffc80d1-bd47-4493-8cd9-87530034d485-kube-api-access-6c89v\") on node \"ip-10-0-132-82.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 19:49:15.615102 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:49:15.615099    4892 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/dffc80d1-bd47-4493-8cd9-87530034d485-istio-token\") on node \"ip-10-0-132-82.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 19:49:15.615322 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:49:15.615114    4892 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/dffc80d1-bd47-4493-8cd9-87530034d485-istio-podinfo\") on node \"ip-10-0-132-82.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 19:49:15.650236 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:49:15.650203    4892 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w5-686689-openshift-default-5fbf5787bf-9tcts"]
Oct 30 19:49:15.655104 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:49:15.655076    4892 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w5-686689-openshift-default-5fbf5787bf-9tcts"]
Oct 30 19:49:15.771914 ip-10-0-132-82 kubenswrapper[4892]: W1030 19:49:15.771883    4892 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod257ea0f4_ca30_42ea_ae62_c4975741b48b.slice/crio-e1164f6abe0d094936a259566f0da8c3b6701fbcbc5be4eb0428324edba0e0c2 WatchSource:0}: Error finding container e1164f6abe0d094936a259566f0da8c3b6701fbcbc5be4eb0428324edba0e0c2: Status 404 returned error can't find the container with id e1164f6abe0d094936a259566f0da8c3b6701fbcbc5be4eb0428324edba0e0c2
Oct 30 19:49:15.772691 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:49:15.772663    4892 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2e1c40f5-mh9t2"]
Oct 30 19:49:15.988935 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:49:15.983310    4892 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w5-686689-openshift-default-5fbf5787bf-jrt7h"]
Oct 30 19:49:15.995783 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:49:15.995754    4892 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w5-686689-openshift-default-5fbf5787bf-jrt7h"]
Oct 30 19:49:16.054032 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:49:16.054002    4892 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3884c8de-l6x8q"]
Oct 30 19:49:16.058153 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:49:16.058120    4892 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3884c8de-l6x8q"]
Oct 30 19:49:16.104387 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:49:16.104319    4892 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w5-686689-openshift-default-5fbf5787bf-7b547"]
Oct 30 19:49:16.108209 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:49:16.108185    4892 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w5-686689-openshift-default-5fbf5787bf-7b547"]
Oct 30 19:49:16.193822 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:49:16.193792    4892 patch_prober.go:29] interesting pod/e2e-worker-w5-686689-openshift-default-5fbf5787bf-nsql4 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.120:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 30 19:49:16.193953 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:49:16.193869    4892 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w5-686689-openshift-default-5fbf5787bf-nsql4" podUID="dffc80d1-bd47-4493-8cd9-87530034d485" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.120:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:49:16.311378 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:49:16.311351    4892 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-686689-openshift-default-5fbf5787bf-nsql4"
Oct 30 19:49:16.311511 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:49:16.311396    4892 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-686689-openshift-default-5fbf5787bf-nsql4" event={"ID":"dffc80d1-bd47-4493-8cd9-87530034d485","Type":"ContainerDied","Data":"cdfbbc85084b57edab9bf71c947c433a2b6793b930e52c0c4a0a0310f5235c69"}
Oct 30 19:49:16.311511 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:49:16.311448    4892 scope.go:122] "RemoveContainer" containerID="997549f2768f85294570e473388022acab09b95ff50b2d4b15b7a7f3bd5503aa"
Oct 30 19:49:16.315891 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:49:16.315866    4892 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2e1c40f5-mh9t2" event={"ID":"257ea0f4-ca30-42ea-ae62-c4975741b48b","Type":"ContainerStarted","Data":"d32a232be94572370539917d7c91e7532b89869bbc3e1bb63275f2c44283ab6e"}
Oct 30 19:49:16.316033 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:49:16.315910    4892 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2e1c40f5-mh9t2" event={"ID":"257ea0f4-ca30-42ea-ae62-c4975741b48b","Type":"ContainerStarted","Data":"e1164f6abe0d094936a259566f0da8c3b6701fbcbc5be4eb0428324edba0e0c2"}
Oct 30 19:49:16.488997 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:49:16.484756    4892 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2e1c40f5-mh9t2" podStartSLOduration=2.484734987 podStartE2EDuration="2.484734987s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:49:14 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:49:16.478628082 +0000 UTC m=+2004.049093693" watchObservedRunningTime="2026-10-30 19:49:16.484734987 +0000 UTC m=+2004.055200604"
Oct 30 19:49:16.572912 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:49:16.572871    4892 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="098876de-0b44-493b-a9da-45280afe2cdf" path="/var/lib/kubelet/pods/098876de-0b44-493b-a9da-45280afe2cdf/volumes"
Oct 30 19:49:16.573412 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:49:16.573394    4892 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="1354d401-f0d6-4252-82d1-47fad799e04c" path="/var/lib/kubelet/pods/1354d401-f0d6-4252-82d1-47fad799e04c/volumes"
Oct 30 19:49:16.573808 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:49:16.573792    4892 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ef9ff75d-23ce-4a8c-9271-0dd12f292e00" path="/var/lib/kubelet/pods/ef9ff75d-23ce-4a8c-9271-0dd12f292e00/volumes"
Oct 30 19:49:16.574548 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:49:16.574529    4892 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="fa668020-72be-46a7-aa53-e8249130c4f1" path="/var/lib/kubelet/pods/fa668020-72be-46a7-aa53-e8249130c4f1/volumes"
Oct 30 19:49:16.592325 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:49:16.592296    4892 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w5-686689-openshift-default-5fbf5787bf-nsql4"]
Oct 30 19:49:16.599952 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:49:16.599918    4892 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w5-686689-openshift-default-5fbf5787bf-nsql4"]
Oct 30 19:49:17.326947 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:49:17.326915    4892 generic.go:356] "Generic (PLEG): container finished" podID="257ea0f4-ca30-42ea-ae62-c4975741b48b" containerID="d32a232be94572370539917d7c91e7532b89869bbc3e1bb63275f2c44283ab6e" exitCode=0
Oct 30 19:49:17.327399 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:49:17.327070    4892 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2e1c40f5-mh9t2" event={"ID":"257ea0f4-ca30-42ea-ae62-c4975741b48b","Type":"ContainerDied","Data":"d32a232be94572370539917d7c91e7532b89869bbc3e1bb63275f2c44283ab6e"}
Oct 30 19:49:18.512389 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:49:18.512366    4892 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2e1c40f5-mh9t2"
Oct 30 19:49:18.573916 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:49:18.573876    4892 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="dffc80d1-bd47-4493-8cd9-87530034d485" path="/var/lib/kubelet/pods/dffc80d1-bd47-4493-8cd9-87530034d485/volumes"
Oct 30 19:49:18.579558 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:49:18.579534    4892 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/257ea0f4-ca30-42ea-ae62-c4975741b48b-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/257ea0f4-ca30-42ea-ae62-c4975741b48b-maas-api-openshift-ca\") pod \"257ea0f4-ca30-42ea-ae62-c4975741b48b\" (UID: \"257ea0f4-ca30-42ea-ae62-c4975741b48b\") "
Oct 30 19:49:18.579678 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:49:18.579623    4892 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/257ea0f4-ca30-42ea-ae62-c4975741b48b-kube-api-access-4dzlp\" (UniqueName: \"kubernetes.io/projected/257ea0f4-ca30-42ea-ae62-c4975741b48b-kube-api-access-4dzlp\") pod \"257ea0f4-ca30-42ea-ae62-c4975741b48b\" (UID: \"257ea0f4-ca30-42ea-ae62-c4975741b48b\") "
Oct 30 19:49:18.579737 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:49:18.579679    4892 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/257ea0f4-ca30-42ea-ae62-c4975741b48b-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/257ea0f4-ca30-42ea-ae62-c4975741b48b-maas-api-xks-ca\") pod \"257ea0f4-ca30-42ea-ae62-c4975741b48b\" (UID: \"257ea0f4-ca30-42ea-ae62-c4975741b48b\") "
Oct 30 19:49:18.579843 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:49:18.579821    4892 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/257ea0f4-ca30-42ea-ae62-c4975741b48b-maas-api-openshift-ca" pod "257ea0f4-ca30-42ea-ae62-c4975741b48b" (UID: "257ea0f4-ca30-42ea-ae62-c4975741b48b"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 19:49:18.580163 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:49:18.580142    4892 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/257ea0f4-ca30-42ea-ae62-c4975741b48b-maas-api-openshift-ca\") on node \"ip-10-0-132-82.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 19:49:18.582052 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:49:18.582030    4892 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/257ea0f4-ca30-42ea-ae62-c4975741b48b-maas-api-xks-ca" pod "257ea0f4-ca30-42ea-ae62-c4975741b48b" (UID: "257ea0f4-ca30-42ea-ae62-c4975741b48b"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 19:49:18.582138 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:49:18.582076    4892 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/257ea0f4-ca30-42ea-ae62-c4975741b48b-kube-api-access-4dzlp" pod "257ea0f4-ca30-42ea-ae62-c4975741b48b" (UID: "257ea0f4-ca30-42ea-ae62-c4975741b48b"). InnerVolumeSpecName "kube-api-access-4dzlp". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:49:18.681171 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:49:18.681139    4892 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4dzlp\" (UniqueName: \"kubernetes.io/projected/257ea0f4-ca30-42ea-ae62-c4975741b48b-kube-api-access-4dzlp\") on node \"ip-10-0-132-82.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 19:49:18.681171 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:49:18.681167    4892 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/257ea0f4-ca30-42ea-ae62-c4975741b48b-maas-api-xks-ca\") on node \"ip-10-0-132-82.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 19:49:19.350507 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:49:19.350473    4892 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2e1c40f5-mh9t2"
Oct 30 19:49:19.350507 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:49:19.350500    4892 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2e1c40f5-mh9t2" event={"ID":"257ea0f4-ca30-42ea-ae62-c4975741b48b","Type":"ContainerDied","Data":"e1164f6abe0d094936a259566f0da8c3b6701fbcbc5be4eb0428324edba0e0c2"}
Oct 30 19:49:19.350715 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:49:19.350552    4892 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="e1164f6abe0d094936a259566f0da8c3b6701fbcbc5be4eb0428324edba0e0c2"
Oct 30 19:49:22.060031 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:49:22.059996    4892 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-fsfvk"]
Oct 30 19:49:24.863566 ip-10-0-132-82 kubenswrapper[4892]: E1030 19:49:24.863528    4892 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-jrm87" containerName="extractor"
Oct 30 19:49:25.062499 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:49:25.062461    4892 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-ada8d7-openshift-default-5f8b49bdfc-pkd8j"]
Oct 30 19:49:25.062715 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:49:25.062694    4892 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w3-ada8d7-openshift-default-5f8b49bdfc-pkd8j" podUID="fcc8f06d-659a-42e9-9ee8-4390a2591f5b" containerName="istio-proxy" containerID="cri-o://be881d28f681b44ed2009fa0469ff21b153a84e2544d23cd03188c1351c5c990" gracePeriod=30
Oct 30 19:49:26.618135 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:49:26.618101    4892 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-df789477-gkkmz"]
Oct 30 19:49:26.618757 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:49:26.618742    4892 cpu_manager.go:414] "containerMap: removing container" podUID="dffc80d1-bd47-4493-8cd9-87530034d485" containerName="istio-proxy"
Oct 30 19:49:26.618801 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:49:26.618758    4892 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="dffc80d1-bd47-4493-8cd9-87530034d485" containerName="istio-proxy"
Oct 30 19:49:26.618801 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:49:26.618772    4892 cpu_manager.go:414] "containerMap: removing container" podUID="fa668020-72be-46a7-aa53-e8249130c4f1" containerName="istio-proxy"
Oct 30 19:49:26.618801 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:49:26.618777    4892 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="fa668020-72be-46a7-aa53-e8249130c4f1" containerName="istio-proxy"
Oct 30 19:49:26.618801 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:49:26.618785    4892 cpu_manager.go:414] "containerMap: removing container" podUID="098876de-0b44-493b-a9da-45280afe2cdf" containerName="istio-proxy"
Oct 30 19:49:26.618801 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:49:26.618790    4892 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="098876de-0b44-493b-a9da-45280afe2cdf" containerName="istio-proxy"
Oct 30 19:49:26.618801 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:49:26.618803    4892 cpu_manager.go:414] "containerMap: removing container" podUID="257ea0f4-ca30-42ea-ae62-c4975741b48b" containerName="revoke-keys"
Oct 30 19:49:26.619031 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:49:26.618807    4892 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="257ea0f4-ca30-42ea-ae62-c4975741b48b" containerName="revoke-keys"
Oct 30 19:49:26.619031 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:49:26.618821    4892 cpu_manager.go:414] "containerMap: removing container" podUID="1354d401-f0d6-4252-82d1-47fad799e04c" containerName="istio-proxy"
Oct 30 19:49:26.619031 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:49:26.618825    4892 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1354d401-f0d6-4252-82d1-47fad799e04c" containerName="istio-proxy"
Oct 30 19:49:26.619031 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:49:26.618954    4892 memory_manager.go:365] "RemoveStaleState removing state" podUID="dffc80d1-bd47-4493-8cd9-87530034d485" containerName="istio-proxy"
Oct 30 19:49:26.619031 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:49:26.618980    4892 memory_manager.go:365] "RemoveStaleState removing state" podUID="fa668020-72be-46a7-aa53-e8249130c4f1" containerName="istio-proxy"
Oct 30 19:49:26.619031 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:49:26.618987    4892 memory_manager.go:365] "RemoveStaleState removing state" podUID="098876de-0b44-493b-a9da-45280afe2cdf" containerName="istio-proxy"
Oct 30 19:49:26.619031 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:49:26.618995    4892 memory_manager.go:365] "RemoveStaleState removing state" podUID="257ea0f4-ca30-42ea-ae62-c4975741b48b" containerName="revoke-keys"
Oct 30 19:49:26.619031 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:49:26.619002    4892 memory_manager.go:365] "RemoveStaleState removing state" podUID="1354d401-f0d6-4252-82d1-47fad799e04c" containerName="istio-proxy"
Oct 30 19:49:26.623746 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:49:26.623730    4892 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-df789477-gkkmz"
Oct 30 19:49:26.678594 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:49:26.678561    4892 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-df789477-gkkmz"]
Oct 30 19:49:26.701537 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:49:26.701500    4892 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zwc6d\" (UniqueName: \"kubernetes.io/projected/1419be25-72fa-4db5-a92b-44972189a2cb-kube-api-access-zwc6d\") pod \"maas-api-revoke-keys-e2e-ait-df789477-gkkmz\" (UID: \"1419be25-72fa-4db5-a92b-44972189a2cb\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-df789477-gkkmz"
Oct 30 19:49:26.701702 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:49:26.701572    4892 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1419be25-72fa-4db5-a92b-44972189a2cb-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ait-df789477-gkkmz\" (UID: \"1419be25-72fa-4db5-a92b-44972189a2cb\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-df789477-gkkmz"
Oct 30 19:49:26.701797 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:49:26.701782    4892 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1419be25-72fa-4db5-a92b-44972189a2cb-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ait-df789477-gkkmz\" (UID: \"1419be25-72fa-4db5-a92b-44972189a2cb\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-df789477-gkkmz"
Oct 30 19:49:26.722664 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:49:26.722638    4892 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 19:49:26.722780 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:49:26.722664    4892 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 30 19:49:26.722780 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:49:26.722705    4892 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 30 19:49:26.726056 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:49:26.726026    4892 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-nlxp4\""
Oct 30 19:49:26.802561 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:49:26.802531    4892 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1419be25-72fa-4db5-a92b-44972189a2cb-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ait-df789477-gkkmz\" (UID: \"1419be25-72fa-4db5-a92b-44972189a2cb\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-df789477-gkkmz"
Oct 30 19:49:26.802727 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:49:26.802578    4892 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zwc6d\" (UniqueName: \"kubernetes.io/projected/1419be25-72fa-4db5-a92b-44972189a2cb-kube-api-access-zwc6d\") pod \"maas-api-revoke-keys-e2e-ait-df789477-gkkmz\" (UID: \"1419be25-72fa-4db5-a92b-44972189a2cb\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-df789477-gkkmz"
Oct 30 19:49:26.802727 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:49:26.802630    4892 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1419be25-72fa-4db5-a92b-44972189a2cb-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ait-df789477-gkkmz\" (UID: \"1419be25-72fa-4db5-a92b-44972189a2cb\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-df789477-gkkmz"
Oct 30 19:49:26.803380 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:49:26.803351    4892 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1419be25-72fa-4db5-a92b-44972189a2cb-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ait-df789477-gkkmz\" (UID: \"1419be25-72fa-4db5-a92b-44972189a2cb\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-df789477-gkkmz"
Oct 30 19:49:26.805136 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:49:26.805116    4892 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1419be25-72fa-4db5-a92b-44972189a2cb-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ait-df789477-gkkmz\" (UID: \"1419be25-72fa-4db5-a92b-44972189a2cb\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-df789477-gkkmz"
Oct 30 19:49:26.859178 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:49:26.859151    4892 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zwc6d\" (UniqueName: \"kubernetes.io/projected/1419be25-72fa-4db5-a92b-44972189a2cb-kube-api-access-zwc6d\") pod \"maas-api-revoke-keys-e2e-ait-df789477-gkkmz\" (UID: \"1419be25-72fa-4db5-a92b-44972189a2cb\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-df789477-gkkmz"
Oct 30 19:49:26.961689 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:49:26.961662    4892 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-df789477-gkkmz"
Oct 30 19:49:27.359402 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:49:27.359368    4892 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-df789477-gkkmz"]
Oct 30 19:49:27.361119 ip-10-0-132-82 kubenswrapper[4892]: W1030 19:49:27.361083    4892 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1419be25_72fa_4db5_a92b_44972189a2cb.slice/crio-4ebc1c6b4b93f674697c1c052fcfde86b67d2d50f42256e0b81932d047beb8c4 WatchSource:0}: Error finding container 4ebc1c6b4b93f674697c1c052fcfde86b67d2d50f42256e0b81932d047beb8c4: Status 404 returned error can't find the container with id 4ebc1c6b4b93f674697c1c052fcfde86b67d2d50f42256e0b81932d047beb8c4
Oct 30 19:49:27.500203 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:49:27.500163    4892 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-df789477-gkkmz" event={"ID":"1419be25-72fa-4db5-a92b-44972189a2cb","Type":"ContainerStarted","Data":"4ebc1c6b4b93f674697c1c052fcfde86b67d2d50f42256e0b81932d047beb8c4"}
Oct 30 19:49:28.510164 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:49:28.510120    4892 generic.go:356] "Generic (PLEG): container finished" podID="1419be25-72fa-4db5-a92b-44972189a2cb" containerID="a671086f7f53ba5ac31f7400543a8d3182bbdb8df5fb97503ee4c07c04bc7eb8" exitCode=0
Oct 30 19:49:28.510529 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:49:28.510168    4892 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-df789477-gkkmz" event={"ID":"1419be25-72fa-4db5-a92b-44972189a2cb","Type":"ContainerDied","Data":"a671086f7f53ba5ac31f7400543a8d3182bbdb8df5fb97503ee4c07c04bc7eb8"}
Oct 30 19:49:29.695555 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:49:29.695523    4892 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-df789477-gkkmz"
Oct 30 19:49:29.768121 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:49:29.768093    4892 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1419be25-72fa-4db5-a92b-44972189a2cb-kube-api-access-zwc6d\" (UniqueName: \"kubernetes.io/projected/1419be25-72fa-4db5-a92b-44972189a2cb-kube-api-access-zwc6d\") pod \"1419be25-72fa-4db5-a92b-44972189a2cb\" (UID: \"1419be25-72fa-4db5-a92b-44972189a2cb\") "
Oct 30 19:49:29.768295 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:49:29.768202    4892 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/1419be25-72fa-4db5-a92b-44972189a2cb-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1419be25-72fa-4db5-a92b-44972189a2cb-maas-api-xks-ca\") pod \"1419be25-72fa-4db5-a92b-44972189a2cb\" (UID: \"1419be25-72fa-4db5-a92b-44972189a2cb\") "
Oct 30 19:49:29.768430 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:49:29.768413    4892 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/1419be25-72fa-4db5-a92b-44972189a2cb-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1419be25-72fa-4db5-a92b-44972189a2cb-maas-api-openshift-ca\") pod \"1419be25-72fa-4db5-a92b-44972189a2cb\" (UID: \"1419be25-72fa-4db5-a92b-44972189a2cb\") "
Oct 30 19:49:29.768764 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:49:29.768740    4892 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1419be25-72fa-4db5-a92b-44972189a2cb-maas-api-openshift-ca" pod "1419be25-72fa-4db5-a92b-44972189a2cb" (UID: "1419be25-72fa-4db5-a92b-44972189a2cb"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 19:49:29.769065 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:49:29.769048    4892 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1419be25-72fa-4db5-a92b-44972189a2cb-maas-api-openshift-ca\") on node \"ip-10-0-132-82.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 19:49:29.770340 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:49:29.770317    4892 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/1419be25-72fa-4db5-a92b-44972189a2cb-maas-api-xks-ca" pod "1419be25-72fa-4db5-a92b-44972189a2cb" (UID: "1419be25-72fa-4db5-a92b-44972189a2cb"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 19:49:29.770455 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:49:29.770441    4892 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1419be25-72fa-4db5-a92b-44972189a2cb-kube-api-access-zwc6d" pod "1419be25-72fa-4db5-a92b-44972189a2cb" (UID: "1419be25-72fa-4db5-a92b-44972189a2cb"). InnerVolumeSpecName "kube-api-access-zwc6d". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:49:29.870695 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:49:29.870650    4892 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zwc6d\" (UniqueName: \"kubernetes.io/projected/1419be25-72fa-4db5-a92b-44972189a2cb-kube-api-access-zwc6d\") on node \"ip-10-0-132-82.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 19:49:29.870695 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:49:29.870693    4892 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1419be25-72fa-4db5-a92b-44972189a2cb-maas-api-xks-ca\") on node \"ip-10-0-132-82.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 19:49:30.345844 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:49:30.345818    4892 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-ada8d7-openshift-default-5f8b49bdfc-pkd8j"
Oct 30 19:49:30.487541 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:49:30.487501    4892 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/fcc8f06d-659a-42e9-9ee8-4390a2591f5b-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/fcc8f06d-659a-42e9-9ee8-4390a2591f5b-istio-envoy\") pod \"fcc8f06d-659a-42e9-9ee8-4390a2591f5b\" (UID: \"fcc8f06d-659a-42e9-9ee8-4390a2591f5b\") "
Oct 30 19:49:30.487737 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:49:30.487551    4892 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/fcc8f06d-659a-42e9-9ee8-4390a2591f5b-istio-token\" (UniqueName: \"kubernetes.io/projected/fcc8f06d-659a-42e9-9ee8-4390a2591f5b-istio-token\") pod \"fcc8f06d-659a-42e9-9ee8-4390a2591f5b\" (UID: \"fcc8f06d-659a-42e9-9ee8-4390a2591f5b\") "
Oct 30 19:49:30.487737 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:49:30.487587    4892 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/fcc8f06d-659a-42e9-9ee8-4390a2591f5b-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/fcc8f06d-659a-42e9-9ee8-4390a2591f5b-workload-socket\") pod \"fcc8f06d-659a-42e9-9ee8-4390a2591f5b\" (UID: \"fcc8f06d-659a-42e9-9ee8-4390a2591f5b\") "
Oct 30 19:49:30.487737 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:49:30.487616    4892 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/fcc8f06d-659a-42e9-9ee8-4390a2591f5b-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/fcc8f06d-659a-42e9-9ee8-4390a2591f5b-workload-certs\") pod \"fcc8f06d-659a-42e9-9ee8-4390a2591f5b\" (UID: \"fcc8f06d-659a-42e9-9ee8-4390a2591f5b\") "
Oct 30 19:49:30.487737 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:49:30.487682    4892 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/fcc8f06d-659a-42e9-9ee8-4390a2591f5b-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/fcc8f06d-659a-42e9-9ee8-4390a2591f5b-istio-data\") pod \"fcc8f06d-659a-42e9-9ee8-4390a2591f5b\" (UID: \"fcc8f06d-659a-42e9-9ee8-4390a2591f5b\") "
Oct 30 19:49:30.487737 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:49:30.487723    4892 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/fcc8f06d-659a-42e9-9ee8-4390a2591f5b-kube-api-access-x9k2k\" (UniqueName: \"kubernetes.io/projected/fcc8f06d-659a-42e9-9ee8-4390a2591f5b-kube-api-access-x9k2k\") pod \"fcc8f06d-659a-42e9-9ee8-4390a2591f5b\" (UID: \"fcc8f06d-659a-42e9-9ee8-4390a2591f5b\") "
Oct 30 19:49:30.488109 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:49:30.487747    4892 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/fcc8f06d-659a-42e9-9ee8-4390a2591f5b-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/fcc8f06d-659a-42e9-9ee8-4390a2591f5b-credential-socket\") pod \"fcc8f06d-659a-42e9-9ee8-4390a2591f5b\" (UID: \"fcc8f06d-659a-42e9-9ee8-4390a2591f5b\") "
Oct 30 19:49:30.488109 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:49:30.487799    4892 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/fcc8f06d-659a-42e9-9ee8-4390a2591f5b-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/fcc8f06d-659a-42e9-9ee8-4390a2591f5b-istiod-ca-cert\") pod \"fcc8f06d-659a-42e9-9ee8-4390a2591f5b\" (UID: \"fcc8f06d-659a-42e9-9ee8-4390a2591f5b\") "
Oct 30 19:49:30.488109 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:49:30.487930    4892 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/fcc8f06d-659a-42e9-9ee8-4390a2591f5b-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/fcc8f06d-659a-42e9-9ee8-4390a2591f5b-istio-podinfo\") pod \"fcc8f06d-659a-42e9-9ee8-4390a2591f5b\" (UID: \"fcc8f06d-659a-42e9-9ee8-4390a2591f5b\") "
Oct 30 19:49:30.491002 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:49:30.490921    4892 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/fcc8f06d-659a-42e9-9ee8-4390a2591f5b-istiod-ca-cert" pod "fcc8f06d-659a-42e9-9ee8-4390a2591f5b" (UID: "fcc8f06d-659a-42e9-9ee8-4390a2591f5b"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 19:49:30.491127 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:49:30.491059    4892 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/fcc8f06d-659a-42e9-9ee8-4390a2591f5b-istio-podinfo" pod "fcc8f06d-659a-42e9-9ee8-4390a2591f5b" (UID: "fcc8f06d-659a-42e9-9ee8-4390a2591f5b"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 19:49:30.491408 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:49:30.491240    4892 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fcc8f06d-659a-42e9-9ee8-4390a2591f5b-istio-data" pod "fcc8f06d-659a-42e9-9ee8-4390a2591f5b" (UID: "fcc8f06d-659a-42e9-9ee8-4390a2591f5b"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:49:30.491507 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:49:30.491368    4892 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fcc8f06d-659a-42e9-9ee8-4390a2591f5b-workload-certs" pod "fcc8f06d-659a-42e9-9ee8-4390a2591f5b" (UID: "fcc8f06d-659a-42e9-9ee8-4390a2591f5b"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:49:30.491507 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:49:30.491379    4892 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fcc8f06d-659a-42e9-9ee8-4390a2591f5b-workload-socket" pod "fcc8f06d-659a-42e9-9ee8-4390a2591f5b" (UID: "fcc8f06d-659a-42e9-9ee8-4390a2591f5b"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:49:30.491507 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:49:30.491430    4892 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fcc8f06d-659a-42e9-9ee8-4390a2591f5b-credential-socket" pod "fcc8f06d-659a-42e9-9ee8-4390a2591f5b" (UID: "fcc8f06d-659a-42e9-9ee8-4390a2591f5b"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:49:30.496722 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:49:30.496697    4892 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fcc8f06d-659a-42e9-9ee8-4390a2591f5b-istio-token" pod "fcc8f06d-659a-42e9-9ee8-4390a2591f5b" (UID: "fcc8f06d-659a-42e9-9ee8-4390a2591f5b"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:49:30.496847 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:49:30.496814    4892 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fcc8f06d-659a-42e9-9ee8-4390a2591f5b-istio-envoy" pod "fcc8f06d-659a-42e9-9ee8-4390a2591f5b" (UID: "fcc8f06d-659a-42e9-9ee8-4390a2591f5b"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:49:30.498818 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:49:30.498785    4892 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fcc8f06d-659a-42e9-9ee8-4390a2591f5b-kube-api-access-x9k2k" pod "fcc8f06d-659a-42e9-9ee8-4390a2591f5b" (UID: "fcc8f06d-659a-42e9-9ee8-4390a2591f5b"). InnerVolumeSpecName "kube-api-access-x9k2k". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:49:30.539162 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:49:30.537584    4892 generic.go:356] "Generic (PLEG): container finished" podID="fcc8f06d-659a-42e9-9ee8-4390a2591f5b" containerID="be881d28f681b44ed2009fa0469ff21b153a84e2544d23cd03188c1351c5c990" exitCode=0
Oct 30 19:49:30.539162 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:49:30.537633    4892 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-ada8d7-openshift-default-5f8b49bdfc-pkd8j"
Oct 30 19:49:30.539162 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:49:30.537676    4892 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-ada8d7-openshift-default-5f8b49bdfc-pkd8j" event={"ID":"fcc8f06d-659a-42e9-9ee8-4390a2591f5b","Type":"ContainerDied","Data":"be881d28f681b44ed2009fa0469ff21b153a84e2544d23cd03188c1351c5c990"}
Oct 30 19:49:30.539162 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:49:30.537747    4892 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-ada8d7-openshift-default-5f8b49bdfc-pkd8j" event={"ID":"fcc8f06d-659a-42e9-9ee8-4390a2591f5b","Type":"ContainerDied","Data":"a122e028e90ced031b5a7154a9b4dffe9025cb8ed9fea71a815712d597041556"}
Oct 30 19:49:30.539162 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:49:30.537780    4892 scope.go:122] "RemoveContainer" containerID="be881d28f681b44ed2009fa0469ff21b153a84e2544d23cd03188c1351c5c990"
Oct 30 19:49:30.543784 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:49:30.543742    4892 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-df789477-gkkmz" event={"ID":"1419be25-72fa-4db5-a92b-44972189a2cb","Type":"ContainerDied","Data":"4ebc1c6b4b93f674697c1c052fcfde86b67d2d50f42256e0b81932d047beb8c4"}
Oct 30 19:49:30.543917 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:49:30.543802    4892 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="4ebc1c6b4b93f674697c1c052fcfde86b67d2d50f42256e0b81932d047beb8c4"
Oct 30 19:49:30.543917 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:49:30.543819    4892 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-df789477-gkkmz"
Oct 30 19:49:30.565767 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:49:30.565743    4892 scope.go:122] "RemoveContainer" containerID="be881d28f681b44ed2009fa0469ff21b153a84e2544d23cd03188c1351c5c990"
Oct 30 19:49:30.566201 ip-10-0-132-82 kubenswrapper[4892]: E1030 19:49:30.566175    4892 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"be881d28f681b44ed2009fa0469ff21b153a84e2544d23cd03188c1351c5c990\": container with ID starting with be881d28f681b44ed2009fa0469ff21b153a84e2544d23cd03188c1351c5c990 not found: ID does not exist" containerID="be881d28f681b44ed2009fa0469ff21b153a84e2544d23cd03188c1351c5c990"
Oct 30 19:49:30.566297 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:49:30.566211    4892 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"be881d28f681b44ed2009fa0469ff21b153a84e2544d23cd03188c1351c5c990"} err="failed to get container status \"be881d28f681b44ed2009fa0469ff21b153a84e2544d23cd03188c1351c5c990\": rpc error: code = NotFound desc = could not find container \"be881d28f681b44ed2009fa0469ff21b153a84e2544d23cd03188c1351c5c990\": container with ID starting with be881d28f681b44ed2009fa0469ff21b153a84e2544d23cd03188c1351c5c990 not found: ID does not exist"
Oct 30 19:49:30.589806 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:49:30.589727    4892 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/fcc8f06d-659a-42e9-9ee8-4390a2591f5b-workload-certs\") on node \"ip-10-0-132-82.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 19:49:30.589806 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:49:30.589762    4892 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/fcc8f06d-659a-42e9-9ee8-4390a2591f5b-istio-data\") on node \"ip-10-0-132-82.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 19:49:30.589806 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:49:30.589779    4892 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-x9k2k\" (UniqueName: \"kubernetes.io/projected/fcc8f06d-659a-42e9-9ee8-4390a2591f5b-kube-api-access-x9k2k\") on node \"ip-10-0-132-82.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 19:49:30.589806 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:49:30.589794    4892 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/fcc8f06d-659a-42e9-9ee8-4390a2591f5b-credential-socket\") on node \"ip-10-0-132-82.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 19:49:30.590072 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:49:30.589807    4892 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/fcc8f06d-659a-42e9-9ee8-4390a2591f5b-istiod-ca-cert\") on node \"ip-10-0-132-82.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 19:49:30.590072 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:49:30.589822    4892 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/fcc8f06d-659a-42e9-9ee8-4390a2591f5b-istio-podinfo\") on node \"ip-10-0-132-82.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 19:49:30.590072 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:49:30.589835    4892 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/fcc8f06d-659a-42e9-9ee8-4390a2591f5b-istio-envoy\") on node \"ip-10-0-132-82.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 19:49:30.590072 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:49:30.589850    4892 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/fcc8f06d-659a-42e9-9ee8-4390a2591f5b-istio-token\") on node \"ip-10-0-132-82.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 19:49:30.590072 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:49:30.589864    4892 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/fcc8f06d-659a-42e9-9ee8-4390a2591f5b-workload-socket\") on node \"ip-10-0-132-82.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 19:49:30.755316 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:49:30.755274    4892 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-ada8d7-openshift-default-5f8b49bdfc-pkd8j"]
Oct 30 19:49:30.760327 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:49:30.760301    4892 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w3-ada8d7-openshift-default-5f8b49bdfc-pkd8j"]
Oct 30 19:49:32.570813 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:49:32.570738    4892 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="fcc8f06d-659a-42e9-9ee8-4390a2591f5b" path="/var/lib/kubelet/pods/fcc8f06d-659a-42e9-9ee8-4390a2591f5b/volumes"
Oct 30 19:49:33.574506 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:49:33.574472    4892 generic.go:356] "Generic (PLEG): container finished" podID="0378dcfe-8d0a-43bb-808d-01a939801130" containerID="b29ccaad7d805db2c688141f390e9322ab08c52aae531c10ef5dad204e09e713" exitCode=0
Oct 30 19:49:33.575019 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:49:33.574547    4892 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w0-6ac304-openshift-default-8cdbdbc67-rgbzh" event={"ID":"0378dcfe-8d0a-43bb-808d-01a939801130","Type":"ContainerDied","Data":"b29ccaad7d805db2c688141f390e9322ab08c52aae531c10ef5dad204e09e713"}
Oct 30 19:49:33.575019 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:49:33.574954    4892 scope.go:122] "RemoveContainer" containerID="b29ccaad7d805db2c688141f390e9322ab08c52aae531c10ef5dad204e09e713"
Oct 30 19:49:33.576633 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:49:33.576599    4892 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892000Ki","pods":"250"}
Oct 30 19:49:33.576748 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:49:33.576690    4892 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892000Ki","pods":"250"}
Oct 30 19:49:33.576748 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:49:33.576737    4892 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892000Ki","pods":"250"}
Oct 30 19:49:34.585477 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:49:34.585445    4892 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w0-6ac304-openshift-default-8cdbdbc67-rgbzh" event={"ID":"0378dcfe-8d0a-43bb-808d-01a939801130","Type":"ContainerStarted","Data":"343ea86dd91bab14b3b2cdc43890a7dfe95c1eb3bc38abe0a14217acd0eb9d7d"}
Oct 30 19:49:34.714358 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:49:34.714330    4892 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-b-w0-6ac304-openshift-default-8cdbdbc67-rgbzh"
Oct 30 19:49:34.714709 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:49:34.714669    4892 patch_prober.go:29] interesting pod/e2e-shared-b-w0-6ac304-openshift-default-8cdbdbc67-rgbzh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.165:15021/healthz/ready\": dial tcp 10.132.0.165:15021: connect: connection refused" start-of-body=
Oct 30 19:49:34.714853 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:49:34.714789    4892 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w0-6ac304-openshift-default-8cdbdbc67-rgbzh" podUID="0378dcfe-8d0a-43bb-808d-01a939801130" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.165:15021/healthz/ready\": dial tcp 10.132.0.165:15021: connect: connection refused"
Oct 30 19:49:35.725246 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:49:35.724337    4892 patch_prober.go:29] interesting pod/e2e-shared-b-w0-6ac304-openshift-default-8cdbdbc67-rgbzh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.165:15021/healthz/ready\": dial tcp 10.132.0.165:15021: connect: connection refused" start-of-body=
Oct 30 19:49:35.725246 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:49:35.724401    4892 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w0-6ac304-openshift-default-8cdbdbc67-rgbzh" podUID="0378dcfe-8d0a-43bb-808d-01a939801130" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.165:15021/healthz/ready\": dial tcp 10.132.0.165:15021: connect: connection refused"
Oct 30 19:49:36.715497 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:49:36.715462    4892 patch_prober.go:29] interesting pod/e2e-shared-b-w0-6ac304-openshift-default-8cdbdbc67-rgbzh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.165:15021/healthz/ready\": dial tcp 10.132.0.165:15021: connect: connection refused" start-of-body=
Oct 30 19:49:36.715688 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:49:36.715516    4892 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w0-6ac304-openshift-default-8cdbdbc67-rgbzh" podUID="0378dcfe-8d0a-43bb-808d-01a939801130" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.165:15021/healthz/ready\": dial tcp 10.132.0.165:15021: connect: connection refused"
Oct 30 19:49:37.715581 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:49:37.715549    4892 patch_prober.go:29] interesting pod/e2e-shared-b-w0-6ac304-openshift-default-8cdbdbc67-rgbzh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.165:15021/healthz/ready\": dial tcp 10.132.0.165:15021: connect: connection refused" start-of-body=
Oct 30 19:49:37.716044 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:49:37.715601    4892 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w0-6ac304-openshift-default-8cdbdbc67-rgbzh" podUID="0378dcfe-8d0a-43bb-808d-01a939801130" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.165:15021/healthz/ready\": dial tcp 10.132.0.165:15021: connect: connection refused"
Oct 30 19:49:38.715253 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:49:38.715220    4892 patch_prober.go:29] interesting pod/e2e-shared-b-w0-6ac304-openshift-default-8cdbdbc67-rgbzh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.165:15021/healthz/ready\": dial tcp 10.132.0.165:15021: connect: connection refused" start-of-body=
Oct 30 19:49:38.715425 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:49:38.715277    4892 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w0-6ac304-openshift-default-8cdbdbc67-rgbzh" podUID="0378dcfe-8d0a-43bb-808d-01a939801130" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.165:15021/healthz/ready\": dial tcp 10.132.0.165:15021: connect: connection refused"
Oct 30 19:49:39.715103 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:49:39.715066    4892 patch_prober.go:29] interesting pod/e2e-shared-b-w0-6ac304-openshift-default-8cdbdbc67-rgbzh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.165:15021/healthz/ready\": dial tcp 10.132.0.165:15021: connect: connection refused" start-of-body=
Oct 30 19:49:39.715556 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:49:39.715127    4892 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w0-6ac304-openshift-default-8cdbdbc67-rgbzh" podUID="0378dcfe-8d0a-43bb-808d-01a939801130" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.165:15021/healthz/ready\": dial tcp 10.132.0.165:15021: connect: connection refused"
Oct 30 19:49:40.719752 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:49:40.719722    4892 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-b-w0-6ac304-openshift-default-8cdbdbc67-rgbzh"
Oct 30 19:49:40.720204 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:49:40.720138    4892 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-b-w0-6ac304-openshift-default-8cdbdbc67-rgbzh"
Oct 30 19:49:40.720712 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:49:40.720693    4892 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-b-w0-6ac304-openshift-default-8cdbdbc67-rgbzh"
Oct 30 19:49:40.750082 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:49:40.750059    4892 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-d52df0-openshift-default-6d5f887b45-2f4xc"]
Oct 30 19:49:40.750324 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:49:40.750298    4892 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w1-d52df0-openshift-default-6d5f887b45-2f4xc" podUID="6befc94b-40e5-42a4-adf5-392b5ecde123" containerName="istio-proxy" containerID="cri-o://46e079b04871b67635b433adadb16a2bbd7b3117558a7b1d0bb392e0be15de8c" gracePeriod=30
Oct 30 19:49:40.798120 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:49:40.798089    4892 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-d52df0-openshift-default-6d5f887b45-lxbtl"]
Oct 30 19:49:40.798324 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:49:40.798280    4892 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w1-d52df0-openshift-default-6d5f887b45-lxbtl" podUID="dc4774a0-3da8-4c65-a0b3-887a476ab465" containerName="istio-proxy" containerID="cri-o://440d694500bb30cdd0530699bd2d413223f8c1d1f41cccf42775b61856b664e4" gracePeriod=30
Oct 30 19:49:42.513215 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:49:42.513180    4892 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-b5178f-openshift-default-c8fbdff8c-wgm46"]
Oct 30 19:49:42.513678 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:49:42.513363    4892 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w4-b5178f-openshift-default-c8fbdff8c-wgm46" podUID="0bdf7217-4cc3-4710-b13e-ffcee12febf0" containerName="istio-proxy" containerID="cri-o://3dd4fa0d82cb56258cb9c53b4db22f21fb21a22a306f2079be7486ee58300540" gracePeriod=30
Oct 30 19:49:44.177856 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:49:44.177813    4892 patch_prober.go:29] interesting pod/e2e-worker-w1-d52df0-openshift-default-6d5f887b45-2f4xc container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Oct 30 19:49:44.178277 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:49:44.177885    4892 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w1-d52df0-openshift-default-6d5f887b45-2f4xc" podUID="6befc94b-40e5-42a4-adf5-392b5ecde123" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Oct 30 19:49:45.295200 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:49:45.295163    4892 patch_prober.go:29] interesting pod/e2e-worker-w1-d52df0-openshift-default-6d5f887b45-lxbtl container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Oct 30 19:49:45.295593 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:49:45.295220    4892 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w1-d52df0-openshift-default-6d5f887b45-lxbtl" podUID="dc4774a0-3da8-4c65-a0b3-887a476ab465" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Oct 30 19:49:46.106411 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:49:46.106353    4892 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-d52df0-openshift-default-6d5f887b45-2f4xc"
Oct 30 19:49:46.136437 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:49:46.136411    4892 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-d52df0-openshift-default-6d5f887b45-lxbtl"
Oct 30 19:49:46.188175 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:49:46.188147    4892 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/dc4774a0-3da8-4c65-a0b3-887a476ab465-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/dc4774a0-3da8-4c65-a0b3-887a476ab465-istio-podinfo\") pod \"dc4774a0-3da8-4c65-a0b3-887a476ab465\" (UID: \"dc4774a0-3da8-4c65-a0b3-887a476ab465\") "
Oct 30 19:49:46.188350 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:49:46.188215    4892 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/dc4774a0-3da8-4c65-a0b3-887a476ab465-istio-token\" (UniqueName: \"kubernetes.io/projected/dc4774a0-3da8-4c65-a0b3-887a476ab465-istio-token\") pod \"dc4774a0-3da8-4c65-a0b3-887a476ab465\" (UID: \"dc4774a0-3da8-4c65-a0b3-887a476ab465\") "
Oct 30 19:49:46.188350 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:49:46.188287    4892 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6befc94b-40e5-42a4-adf5-392b5ecde123-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6befc94b-40e5-42a4-adf5-392b5ecde123-workload-socket\") pod \"6befc94b-40e5-42a4-adf5-392b5ecde123\" (UID: \"6befc94b-40e5-42a4-adf5-392b5ecde123\") "
Oct 30 19:49:46.188350 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:49:46.188319    4892 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/dc4774a0-3da8-4c65-a0b3-887a476ab465-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/dc4774a0-3da8-4c65-a0b3-887a476ab465-istiod-ca-cert\") pod \"dc4774a0-3da8-4c65-a0b3-887a476ab465\" (UID: \"dc4774a0-3da8-4c65-a0b3-887a476ab465\") "
Oct 30 19:49:46.188350 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:49:46.188340    4892 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/6befc94b-40e5-42a4-adf5-392b5ecde123-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6befc94b-40e5-42a4-adf5-392b5ecde123-istiod-ca-cert\") pod \"6befc94b-40e5-42a4-adf5-392b5ecde123\" (UID: \"6befc94b-40e5-42a4-adf5-392b5ecde123\") "
Oct 30 19:49:46.188573 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:49:46.188369    4892 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/dc4774a0-3da8-4c65-a0b3-887a476ab465-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/dc4774a0-3da8-4c65-a0b3-887a476ab465-istio-envoy\") pod \"dc4774a0-3da8-4c65-a0b3-887a476ab465\" (UID: \"dc4774a0-3da8-4c65-a0b3-887a476ab465\") "
Oct 30 19:49:46.188573 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:49:46.188406    4892 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/dc4774a0-3da8-4c65-a0b3-887a476ab465-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/dc4774a0-3da8-4c65-a0b3-887a476ab465-workload-certs\") pod \"dc4774a0-3da8-4c65-a0b3-887a476ab465\" (UID: \"dc4774a0-3da8-4c65-a0b3-887a476ab465\") "
Oct 30 19:49:46.188573 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:49:46.188472    4892 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/dc4774a0-3da8-4c65-a0b3-887a476ab465-kube-api-access-72lrd\" (UniqueName: \"kubernetes.io/projected/dc4774a0-3da8-4c65-a0b3-887a476ab465-kube-api-access-72lrd\") pod \"dc4774a0-3da8-4c65-a0b3-887a476ab465\" (UID: \"dc4774a0-3da8-4c65-a0b3-887a476ab465\") "
Oct 30 19:49:46.188573 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:49:46.188515    4892 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/6befc94b-40e5-42a4-adf5-392b5ecde123-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6befc94b-40e5-42a4-adf5-392b5ecde123-istio-podinfo\") pod \"6befc94b-40e5-42a4-adf5-392b5ecde123\" (UID: \"6befc94b-40e5-42a4-adf5-392b5ecde123\") "
Oct 30 19:49:46.188573 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:49:46.188541    4892 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6befc94b-40e5-42a4-adf5-392b5ecde123-istio-token\" (UniqueName: \"kubernetes.io/projected/6befc94b-40e5-42a4-adf5-392b5ecde123-istio-token\") pod \"6befc94b-40e5-42a4-adf5-392b5ecde123\" (UID: \"6befc94b-40e5-42a4-adf5-392b5ecde123\") "
Oct 30 19:49:46.188838 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:49:46.188594    4892 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6befc94b-40e5-42a4-adf5-392b5ecde123-kube-api-access-z7f72\" (UniqueName: \"kubernetes.io/projected/6befc94b-40e5-42a4-adf5-392b5ecde123-kube-api-access-z7f72\") pod \"6befc94b-40e5-42a4-adf5-392b5ecde123\" (UID: \"6befc94b-40e5-42a4-adf5-392b5ecde123\") "
Oct 30 19:49:46.188838 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:49:46.188606    4892 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6befc94b-40e5-42a4-adf5-392b5ecde123-workload-socket" pod "6befc94b-40e5-42a4-adf5-392b5ecde123" (UID: "6befc94b-40e5-42a4-adf5-392b5ecde123"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:49:46.188838 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:49:46.188644    4892 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6befc94b-40e5-42a4-adf5-392b5ecde123-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6befc94b-40e5-42a4-adf5-392b5ecde123-credential-socket\") pod \"6befc94b-40e5-42a4-adf5-392b5ecde123\" (UID: \"6befc94b-40e5-42a4-adf5-392b5ecde123\") "
Oct 30 19:49:46.188838 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:49:46.188692    4892 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/dc4774a0-3da8-4c65-a0b3-887a476ab465-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/dc4774a0-3da8-4c65-a0b3-887a476ab465-credential-socket\") pod \"dc4774a0-3da8-4c65-a0b3-887a476ab465\" (UID: \"dc4774a0-3da8-4c65-a0b3-887a476ab465\") "
Oct 30 19:49:46.188838 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:49:46.188724    4892 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/dc4774a0-3da8-4c65-a0b3-887a476ab465-workload-certs" pod "dc4774a0-3da8-4c65-a0b3-887a476ab465" (UID: "dc4774a0-3da8-4c65-a0b3-887a476ab465"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:49:46.188838 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:49:46.188755    4892 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/dc4774a0-3da8-4c65-a0b3-887a476ab465-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/dc4774a0-3da8-4c65-a0b3-887a476ab465-istio-data\") pod \"dc4774a0-3da8-4c65-a0b3-887a476ab465\" (UID: \"dc4774a0-3da8-4c65-a0b3-887a476ab465\") "
Oct 30 19:49:46.188838 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:49:46.188792    4892 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6befc94b-40e5-42a4-adf5-392b5ecde123-istiod-ca-cert" pod "6befc94b-40e5-42a4-adf5-392b5ecde123" (UID: "6befc94b-40e5-42a4-adf5-392b5ecde123"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 19:49:46.188838 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:49:46.188809    4892 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6befc94b-40e5-42a4-adf5-392b5ecde123-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6befc94b-40e5-42a4-adf5-392b5ecde123-istio-envoy\") pod \"6befc94b-40e5-42a4-adf5-392b5ecde123\" (UID: \"6befc94b-40e5-42a4-adf5-392b5ecde123\") "
Oct 30 19:49:46.189311 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:49:46.188864    4892 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6befc94b-40e5-42a4-adf5-392b5ecde123-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6befc94b-40e5-42a4-adf5-392b5ecde123-workload-certs\") pod \"6befc94b-40e5-42a4-adf5-392b5ecde123\" (UID: \"6befc94b-40e5-42a4-adf5-392b5ecde123\") "
Oct 30 19:49:46.189311 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:49:46.188951    4892 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/dc4774a0-3da8-4c65-a0b3-887a476ab465-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/dc4774a0-3da8-4c65-a0b3-887a476ab465-workload-socket\") pod \"dc4774a0-3da8-4c65-a0b3-887a476ab465\" (UID: \"dc4774a0-3da8-4c65-a0b3-887a476ab465\") "
Oct 30 19:49:46.189311 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:49:46.189043    4892 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6befc94b-40e5-42a4-adf5-392b5ecde123-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6befc94b-40e5-42a4-adf5-392b5ecde123-istio-data\") pod \"6befc94b-40e5-42a4-adf5-392b5ecde123\" (UID: \"6befc94b-40e5-42a4-adf5-392b5ecde123\") "
Oct 30 19:49:46.189311 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:49:46.189081    4892 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/dc4774a0-3da8-4c65-a0b3-887a476ab465-credential-socket" pod "dc4774a0-3da8-4c65-a0b3-887a476ab465" (UID: "dc4774a0-3da8-4c65-a0b3-887a476ab465"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:49:46.189311 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:49:46.189091    4892 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/dc4774a0-3da8-4c65-a0b3-887a476ab465-istiod-ca-cert" pod "dc4774a0-3da8-4c65-a0b3-887a476ab465" (UID: "dc4774a0-3da8-4c65-a0b3-887a476ab465"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 19:49:46.189311 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:49:46.189298    4892 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6befc94b-40e5-42a4-adf5-392b5ecde123-credential-socket" pod "6befc94b-40e5-42a4-adf5-392b5ecde123" (UID: "6befc94b-40e5-42a4-adf5-392b5ecde123"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:49:46.190746 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:49:46.189854    4892 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6befc94b-40e5-42a4-adf5-392b5ecde123-workload-socket\") on node \"ip-10-0-132-82.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 19:49:46.190746 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:49:46.189882    4892 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/dc4774a0-3da8-4c65-a0b3-887a476ab465-istiod-ca-cert\") on node \"ip-10-0-132-82.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 19:49:46.190746 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:49:46.189896    4892 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6befc94b-40e5-42a4-adf5-392b5ecde123-istiod-ca-cert\") on node \"ip-10-0-132-82.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 19:49:46.190746 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:49:46.189910    4892 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/dc4774a0-3da8-4c65-a0b3-887a476ab465-workload-certs\") on node \"ip-10-0-132-82.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 19:49:46.190746 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:49:46.189925    4892 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6befc94b-40e5-42a4-adf5-392b5ecde123-credential-socket\") on node \"ip-10-0-132-82.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 19:49:46.190746 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:49:46.189940    4892 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/dc4774a0-3da8-4c65-a0b3-887a476ab465-credential-socket\") on node \"ip-10-0-132-82.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 19:49:46.190746 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:49:46.190185    4892 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6befc94b-40e5-42a4-adf5-392b5ecde123-workload-certs" pod "6befc94b-40e5-42a4-adf5-392b5ecde123" (UID: "6befc94b-40e5-42a4-adf5-392b5ecde123"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:49:46.190746 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:49:46.190390    4892 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/dc4774a0-3da8-4c65-a0b3-887a476ab465-workload-socket" pod "dc4774a0-3da8-4c65-a0b3-887a476ab465" (UID: "dc4774a0-3da8-4c65-a0b3-887a476ab465"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:49:46.190746 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:49:46.190492    4892 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6befc94b-40e5-42a4-adf5-392b5ecde123-istio-data" pod "6befc94b-40e5-42a4-adf5-392b5ecde123" (UID: "6befc94b-40e5-42a4-adf5-392b5ecde123"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:49:46.190746 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:49:46.190569    4892 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/dc4774a0-3da8-4c65-a0b3-887a476ab465-istio-data" pod "dc4774a0-3da8-4c65-a0b3-887a476ab465" (UID: "dc4774a0-3da8-4c65-a0b3-887a476ab465"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:49:46.191668 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:49:46.191640    4892 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/dc4774a0-3da8-4c65-a0b3-887a476ab465-istio-token" pod "dc4774a0-3da8-4c65-a0b3-887a476ab465" (UID: "dc4774a0-3da8-4c65-a0b3-887a476ab465"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:49:46.191883 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:49:46.191821    4892 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/dc4774a0-3da8-4c65-a0b3-887a476ab465-istio-envoy" pod "dc4774a0-3da8-4c65-a0b3-887a476ab465" (UID: "dc4774a0-3da8-4c65-a0b3-887a476ab465"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:49:46.192092 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:49:46.192065    4892 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/6befc94b-40e5-42a4-adf5-392b5ecde123-istio-podinfo" pod "6befc94b-40e5-42a4-adf5-392b5ecde123" (UID: "6befc94b-40e5-42a4-adf5-392b5ecde123"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 19:49:46.192313 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:49:46.192285    4892 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/dc4774a0-3da8-4c65-a0b3-887a476ab465-istio-podinfo" pod "dc4774a0-3da8-4c65-a0b3-887a476ab465" (UID: "dc4774a0-3da8-4c65-a0b3-887a476ab465"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 19:49:46.192423 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:49:46.192309    4892 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6befc94b-40e5-42a4-adf5-392b5ecde123-kube-api-access-z7f72" pod "6befc94b-40e5-42a4-adf5-392b5ecde123" (UID: "6befc94b-40e5-42a4-adf5-392b5ecde123"). InnerVolumeSpecName "kube-api-access-z7f72". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:49:46.192622 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:49:46.192598    4892 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/dc4774a0-3da8-4c65-a0b3-887a476ab465-kube-api-access-72lrd" pod "dc4774a0-3da8-4c65-a0b3-887a476ab465" (UID: "dc4774a0-3da8-4c65-a0b3-887a476ab465"). InnerVolumeSpecName "kube-api-access-72lrd". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:49:46.193229 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:49:46.193206    4892 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6befc94b-40e5-42a4-adf5-392b5ecde123-istio-envoy" pod "6befc94b-40e5-42a4-adf5-392b5ecde123" (UID: "6befc94b-40e5-42a4-adf5-392b5ecde123"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:49:46.193376 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:49:46.193361    4892 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6befc94b-40e5-42a4-adf5-392b5ecde123-istio-token" pod "6befc94b-40e5-42a4-adf5-392b5ecde123" (UID: "6befc94b-40e5-42a4-adf5-392b5ecde123"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:49:46.306990 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:49:46.294753    4892 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/dc4774a0-3da8-4c65-a0b3-887a476ab465-workload-socket\") on node \"ip-10-0-132-82.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 19:49:46.306990 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:49:46.294786    4892 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6befc94b-40e5-42a4-adf5-392b5ecde123-istio-data\") on node \"ip-10-0-132-82.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 19:49:46.306990 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:49:46.294801    4892 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/dc4774a0-3da8-4c65-a0b3-887a476ab465-istio-podinfo\") on node \"ip-10-0-132-82.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 19:49:46.306990 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:49:46.294814    4892 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/dc4774a0-3da8-4c65-a0b3-887a476ab465-istio-token\") on node \"ip-10-0-132-82.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 19:49:46.306990 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:49:46.294827    4892 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/dc4774a0-3da8-4c65-a0b3-887a476ab465-istio-envoy\") on node \"ip-10-0-132-82.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 19:49:46.306990 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:49:46.294839    4892 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-72lrd\" (UniqueName: \"kubernetes.io/projected/dc4774a0-3da8-4c65-a0b3-887a476ab465-kube-api-access-72lrd\") on node \"ip-10-0-132-82.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 19:49:46.306990 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:49:46.294854    4892 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6befc94b-40e5-42a4-adf5-392b5ecde123-istio-podinfo\") on node \"ip-10-0-132-82.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 19:49:46.306990 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:49:46.294868    4892 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6befc94b-40e5-42a4-adf5-392b5ecde123-istio-token\") on node \"ip-10-0-132-82.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 19:49:46.306990 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:49:46.294881    4892 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-z7f72\" (UniqueName: \"kubernetes.io/projected/6befc94b-40e5-42a4-adf5-392b5ecde123-kube-api-access-z7f72\") on node \"ip-10-0-132-82.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 19:49:46.306990 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:49:46.294893    4892 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/dc4774a0-3da8-4c65-a0b3-887a476ab465-istio-data\") on node \"ip-10-0-132-82.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 19:49:46.306990 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:49:46.294905    4892 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6befc94b-40e5-42a4-adf5-392b5ecde123-istio-envoy\") on node \"ip-10-0-132-82.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 19:49:46.306990 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:49:46.294917    4892 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6befc94b-40e5-42a4-adf5-392b5ecde123-workload-certs\") on node \"ip-10-0-132-82.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 19:49:46.708679 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:49:46.708647    4892 generic.go:356] "Generic (PLEG): container finished" podID="dc4774a0-3da8-4c65-a0b3-887a476ab465" containerID="440d694500bb30cdd0530699bd2d413223f8c1d1f41cccf42775b61856b664e4" exitCode=0
Oct 30 19:49:46.708884 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:49:46.708713    4892 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-d52df0-openshift-default-6d5f887b45-lxbtl"
Oct 30 19:49:46.708884 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:49:46.708741    4892 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-d52df0-openshift-default-6d5f887b45-lxbtl" event={"ID":"dc4774a0-3da8-4c65-a0b3-887a476ab465","Type":"ContainerDied","Data":"440d694500bb30cdd0530699bd2d413223f8c1d1f41cccf42775b61856b664e4"}
Oct 30 19:49:46.708884 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:49:46.708797    4892 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-d52df0-openshift-default-6d5f887b45-lxbtl" event={"ID":"dc4774a0-3da8-4c65-a0b3-887a476ab465","Type":"ContainerDied","Data":"ff7273f384e00244298e2b9e18ca867407e0f695aca8cf48d9209ef628100391"}
Oct 30 19:49:46.708884 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:49:46.708829    4892 scope.go:122] "RemoveContainer" containerID="440d694500bb30cdd0530699bd2d413223f8c1d1f41cccf42775b61856b664e4"
Oct 30 19:49:46.710979 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:49:46.710892    4892 generic.go:356] "Generic (PLEG): container finished" podID="6befc94b-40e5-42a4-adf5-392b5ecde123" containerID="46e079b04871b67635b433adadb16a2bbd7b3117558a7b1d0bb392e0be15de8c" exitCode=0
Oct 30 19:49:46.710979 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:49:46.710952    4892 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-d52df0-openshift-default-6d5f887b45-2f4xc"
Oct 30 19:49:46.711155 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:49:46.711007    4892 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-d52df0-openshift-default-6d5f887b45-2f4xc" event={"ID":"6befc94b-40e5-42a4-adf5-392b5ecde123","Type":"ContainerDied","Data":"46e079b04871b67635b433adadb16a2bbd7b3117558a7b1d0bb392e0be15de8c"}
Oct 30 19:49:46.711155 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:49:46.711064    4892 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-d52df0-openshift-default-6d5f887b45-2f4xc" event={"ID":"6befc94b-40e5-42a4-adf5-392b5ecde123","Type":"ContainerDied","Data":"5d2d7cb823c7697e722c326abf9879dc0a4c1c47aa81a1b2e99322bbff75354a"}
Oct 30 19:49:46.721440 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:49:46.721421    4892 scope.go:122] "RemoveContainer" containerID="440d694500bb30cdd0530699bd2d413223f8c1d1f41cccf42775b61856b664e4"
Oct 30 19:49:46.721714 ip-10-0-132-82 kubenswrapper[4892]: E1030 19:49:46.721696    4892 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"440d694500bb30cdd0530699bd2d413223f8c1d1f41cccf42775b61856b664e4\": container with ID starting with 440d694500bb30cdd0530699bd2d413223f8c1d1f41cccf42775b61856b664e4 not found: ID does not exist" containerID="440d694500bb30cdd0530699bd2d413223f8c1d1f41cccf42775b61856b664e4"
Oct 30 19:49:46.721804 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:49:46.721724    4892 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"440d694500bb30cdd0530699bd2d413223f8c1d1f41cccf42775b61856b664e4"} err="failed to get container status \"440d694500bb30cdd0530699bd2d413223f8c1d1f41cccf42775b61856b664e4\": rpc error: code = NotFound desc = could not find container \"440d694500bb30cdd0530699bd2d413223f8c1d1f41cccf42775b61856b664e4\": container with ID starting with 440d694500bb30cdd0530699bd2d413223f8c1d1f41cccf42775b61856b664e4 not found: ID does not exist"
Oct 30 19:49:46.721804 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:49:46.721750    4892 scope.go:122] "RemoveContainer" containerID="46e079b04871b67635b433adadb16a2bbd7b3117558a7b1d0bb392e0be15de8c"
Oct 30 19:49:46.732689 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:49:46.732662    4892 scope.go:122] "RemoveContainer" containerID="46e079b04871b67635b433adadb16a2bbd7b3117558a7b1d0bb392e0be15de8c"
Oct 30 19:49:46.732958 ip-10-0-132-82 kubenswrapper[4892]: E1030 19:49:46.732933    4892 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"46e079b04871b67635b433adadb16a2bbd7b3117558a7b1d0bb392e0be15de8c\": container with ID starting with 46e079b04871b67635b433adadb16a2bbd7b3117558a7b1d0bb392e0be15de8c not found: ID does not exist" containerID="46e079b04871b67635b433adadb16a2bbd7b3117558a7b1d0bb392e0be15de8c"
Oct 30 19:49:46.733038 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:49:46.732958    4892 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"46e079b04871b67635b433adadb16a2bbd7b3117558a7b1d0bb392e0be15de8c"} err="failed to get container status \"46e079b04871b67635b433adadb16a2bbd7b3117558a7b1d0bb392e0be15de8c\": rpc error: code = NotFound desc = could not find container \"46e079b04871b67635b433adadb16a2bbd7b3117558a7b1d0bb392e0be15de8c\": container with ID starting with 46e079b04871b67635b433adadb16a2bbd7b3117558a7b1d0bb392e0be15de8c not found: ID does not exist"
Oct 30 19:49:46.881203 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:49:46.881108    4892 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-d52df0-openshift-default-6d5f887b45-lxbtl"]
Oct 30 19:49:46.888129 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:49:46.888100    4892 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w1-d52df0-openshift-default-6d5f887b45-lxbtl"]
Oct 30 19:49:46.990484 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:49:46.990442    4892 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-d52df0-openshift-default-6d5f887b45-2f4xc"]
Oct 30 19:49:46.994638 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:49:46.994612    4892 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w1-d52df0-openshift-default-6d5f887b45-2f4xc"]
Oct 30 19:49:47.727678 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:49:47.727644    4892 generic.go:356] "Generic (PLEG): container finished" podID="0bdf7217-4cc3-4710-b13e-ffcee12febf0" containerID="3dd4fa0d82cb56258cb9c53b4db22f21fb21a22a306f2079be7486ee58300540" exitCode=0
Oct 30 19:49:47.728111 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:49:47.727709    4892 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-b5178f-openshift-default-c8fbdff8c-wgm46" event={"ID":"0bdf7217-4cc3-4710-b13e-ffcee12febf0","Type":"ContainerDied","Data":"3dd4fa0d82cb56258cb9c53b4db22f21fb21a22a306f2079be7486ee58300540"}
Oct 30 19:49:47.783933 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:49:47.783872    4892 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-b5178f-openshift-default-c8fbdff8c-wgm46"
Oct 30 19:49:47.846125 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:49:47.844416    4892 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0bdf7217-4cc3-4710-b13e-ffcee12febf0-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0bdf7217-4cc3-4710-b13e-ffcee12febf0-workload-certs\") pod \"0bdf7217-4cc3-4710-b13e-ffcee12febf0\" (UID: \"0bdf7217-4cc3-4710-b13e-ffcee12febf0\") "
Oct 30 19:49:47.846125 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:49:47.844482    4892 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0bdf7217-4cc3-4710-b13e-ffcee12febf0-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0bdf7217-4cc3-4710-b13e-ffcee12febf0-workload-socket\") pod \"0bdf7217-4cc3-4710-b13e-ffcee12febf0\" (UID: \"0bdf7217-4cc3-4710-b13e-ffcee12febf0\") "
Oct 30 19:49:47.846125 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:49:47.844594    4892 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0bdf7217-4cc3-4710-b13e-ffcee12febf0-kube-api-access-4fwdg\" (UniqueName: \"kubernetes.io/projected/0bdf7217-4cc3-4710-b13e-ffcee12febf0-kube-api-access-4fwdg\") pod \"0bdf7217-4cc3-4710-b13e-ffcee12febf0\" (UID: \"0bdf7217-4cc3-4710-b13e-ffcee12febf0\") "
Oct 30 19:49:47.846125 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:49:47.844670    4892 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/0bdf7217-4cc3-4710-b13e-ffcee12febf0-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0bdf7217-4cc3-4710-b13e-ffcee12febf0-istio-podinfo\") pod \"0bdf7217-4cc3-4710-b13e-ffcee12febf0\" (UID: \"0bdf7217-4cc3-4710-b13e-ffcee12febf0\") "
Oct 30 19:49:47.846125 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:49:47.844726    4892 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0bdf7217-4cc3-4710-b13e-ffcee12febf0-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0bdf7217-4cc3-4710-b13e-ffcee12febf0-credential-socket\") pod \"0bdf7217-4cc3-4710-b13e-ffcee12febf0\" (UID: \"0bdf7217-4cc3-4710-b13e-ffcee12febf0\") "
Oct 30 19:49:47.846125 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:49:47.844834    4892 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0bdf7217-4cc3-4710-b13e-ffcee12febf0-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0bdf7217-4cc3-4710-b13e-ffcee12febf0-istio-data\") pod \"0bdf7217-4cc3-4710-b13e-ffcee12febf0\" (UID: \"0bdf7217-4cc3-4710-b13e-ffcee12febf0\") "
Oct 30 19:49:47.846125 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:49:47.844860    4892 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/0bdf7217-4cc3-4710-b13e-ffcee12febf0-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0bdf7217-4cc3-4710-b13e-ffcee12febf0-istiod-ca-cert\") pod \"0bdf7217-4cc3-4710-b13e-ffcee12febf0\" (UID: \"0bdf7217-4cc3-4710-b13e-ffcee12febf0\") "
Oct 30 19:49:47.846125 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:49:47.844893    4892 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0bdf7217-4cc3-4710-b13e-ffcee12febf0-istio-token\" (UniqueName: \"kubernetes.io/projected/0bdf7217-4cc3-4710-b13e-ffcee12febf0-istio-token\") pod \"0bdf7217-4cc3-4710-b13e-ffcee12febf0\" (UID: \"0bdf7217-4cc3-4710-b13e-ffcee12febf0\") "
Oct 30 19:49:47.846125 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:49:47.845118    4892 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0bdf7217-4cc3-4710-b13e-ffcee12febf0-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0bdf7217-4cc3-4710-b13e-ffcee12febf0-istio-envoy\") pod \"0bdf7217-4cc3-4710-b13e-ffcee12febf0\" (UID: \"0bdf7217-4cc3-4710-b13e-ffcee12febf0\") "
Oct 30 19:49:47.848489 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:49:47.848322    4892 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0bdf7217-4cc3-4710-b13e-ffcee12febf0-istio-data" pod "0bdf7217-4cc3-4710-b13e-ffcee12febf0" (UID: "0bdf7217-4cc3-4710-b13e-ffcee12febf0"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:49:47.848815 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:49:47.848641    4892 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0bdf7217-4cc3-4710-b13e-ffcee12febf0-credential-socket" pod "0bdf7217-4cc3-4710-b13e-ffcee12febf0" (UID: "0bdf7217-4cc3-4710-b13e-ffcee12febf0"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:49:47.849316 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:49:47.849060    4892 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0bdf7217-4cc3-4710-b13e-ffcee12febf0-istiod-ca-cert" pod "0bdf7217-4cc3-4710-b13e-ffcee12febf0" (UID: "0bdf7217-4cc3-4710-b13e-ffcee12febf0"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 19:49:47.858110 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:49:47.858075    4892 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0bdf7217-4cc3-4710-b13e-ffcee12febf0-workload-socket" pod "0bdf7217-4cc3-4710-b13e-ffcee12febf0" (UID: "0bdf7217-4cc3-4710-b13e-ffcee12febf0"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:49:47.859368 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:49:47.859332    4892 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0bdf7217-4cc3-4710-b13e-ffcee12febf0-workload-certs" pod "0bdf7217-4cc3-4710-b13e-ffcee12febf0" (UID: "0bdf7217-4cc3-4710-b13e-ffcee12febf0"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:49:47.873849 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:49:47.873436    4892 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0bdf7217-4cc3-4710-b13e-ffcee12febf0-istio-envoy" pod "0bdf7217-4cc3-4710-b13e-ffcee12febf0" (UID: "0bdf7217-4cc3-4710-b13e-ffcee12febf0"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:49:47.890669 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:49:47.889986    4892 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/0bdf7217-4cc3-4710-b13e-ffcee12febf0-istio-podinfo" pod "0bdf7217-4cc3-4710-b13e-ffcee12febf0" (UID: "0bdf7217-4cc3-4710-b13e-ffcee12febf0"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 19:49:47.891029 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:49:47.890989    4892 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0bdf7217-4cc3-4710-b13e-ffcee12febf0-istio-token" pod "0bdf7217-4cc3-4710-b13e-ffcee12febf0" (UID: "0bdf7217-4cc3-4710-b13e-ffcee12febf0"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:49:47.892132 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:49:47.892094    4892 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0bdf7217-4cc3-4710-b13e-ffcee12febf0-kube-api-access-4fwdg" pod "0bdf7217-4cc3-4710-b13e-ffcee12febf0" (UID: "0bdf7217-4cc3-4710-b13e-ffcee12febf0"). InnerVolumeSpecName "kube-api-access-4fwdg". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:49:47.949649 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:49:47.949608    4892 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0bdf7217-4cc3-4710-b13e-ffcee12febf0-istio-data\") on node \"ip-10-0-132-82.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 19:49:47.949925 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:49:47.949866    4892 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0bdf7217-4cc3-4710-b13e-ffcee12febf0-istiod-ca-cert\") on node \"ip-10-0-132-82.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 19:49:47.950085 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:49:47.950070    4892 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0bdf7217-4cc3-4710-b13e-ffcee12febf0-istio-token\") on node \"ip-10-0-132-82.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 19:49:47.950175 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:49:47.950164    4892 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0bdf7217-4cc3-4710-b13e-ffcee12febf0-istio-envoy\") on node \"ip-10-0-132-82.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 19:49:47.950254 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:49:47.950244    4892 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0bdf7217-4cc3-4710-b13e-ffcee12febf0-workload-certs\") on node \"ip-10-0-132-82.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 19:49:47.950341 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:49:47.950331    4892 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0bdf7217-4cc3-4710-b13e-ffcee12febf0-workload-socket\") on node \"ip-10-0-132-82.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 19:49:47.950421 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:49:47.950407    4892 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4fwdg\" (UniqueName: \"kubernetes.io/projected/0bdf7217-4cc3-4710-b13e-ffcee12febf0-kube-api-access-4fwdg\") on node \"ip-10-0-132-82.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 19:49:47.950526 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:49:47.950514    4892 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0bdf7217-4cc3-4710-b13e-ffcee12febf0-istio-podinfo\") on node \"ip-10-0-132-82.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 19:49:47.950607 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:49:47.950597    4892 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0bdf7217-4cc3-4710-b13e-ffcee12febf0-credential-socket\") on node \"ip-10-0-132-82.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 19:49:48.572337 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:49:48.572294    4892 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="6befc94b-40e5-42a4-adf5-392b5ecde123" path="/var/lib/kubelet/pods/6befc94b-40e5-42a4-adf5-392b5ecde123/volumes"
Oct 30 19:49:48.572836 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:49:48.572818    4892 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="dc4774a0-3da8-4c65-a0b3-887a476ab465" path="/var/lib/kubelet/pods/dc4774a0-3da8-4c65-a0b3-887a476ab465/volumes"
Oct 30 19:49:48.791988 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:49:48.790738    4892 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-b5178f-openshift-default-c8fbdff8c-wgm46" event={"ID":"0bdf7217-4cc3-4710-b13e-ffcee12febf0","Type":"ContainerDied","Data":"0962651f2bd8b1d3e0e38fc296559bb9edaa48e70811a27a9457bf15b133f212"}
Oct 30 19:49:48.791988 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:49:48.790813    4892 scope.go:122] "RemoveContainer" containerID="3dd4fa0d82cb56258cb9c53b4db22f21fb21a22a306f2079be7486ee58300540"
Oct 30 19:49:48.791988 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:49:48.791005    4892 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-b5178f-openshift-default-c8fbdff8c-wgm46"
Oct 30 19:49:49.053991 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:49:49.053010    4892 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-b5178f-openshift-default-c8fbdff8c-wgm46"]
Oct 30 19:49:49.075985 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:49:49.072082    4892 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w4-b5178f-openshift-default-c8fbdff8c-wgm46"]
Oct 30 19:49:50.043012 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:49:50.040081    4892 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-d52df0/distinct-2-w1-d52df0-kserve-7846558464-xrkdc" podUID="9c614ab6-30e2-4fe1-8609-d5e2dadc198a" containerName="main" probeResult="failure" output="Get \"https://10.132.0.83:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:49:50.065989 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:49:50.065006    4892 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ae4eb22f-mxz6s"]
Oct 30 19:49:50.189992 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:49:50.188981    4892 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ae4eb22f-mxz6s"]
Oct 30 19:49:50.332521 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:49:50.332425    4892 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-2d9861-openshift-default-58ddf574b9-w2r9m"]
Oct 30 19:49:50.333515 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:49:50.333478    4892 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w0-2d9861-openshift-default-58ddf574b9-w2r9m" podUID="0f4fa4be-451c-43b5-bc35-cdf507a2dfb3" containerName="istio-proxy" containerID="cri-o://772e8c1633aca86a1b725cee1d4db72ab0e0927004a64e78f0ea6d7cb13f9bf6" gracePeriod=30
Oct 30 19:49:50.632045 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:49:50.626309    4892 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w1-d52df0/simulator-w1-d52df0-kserve-67f74bc577-q5bjz" podUID="81b7bb84-1003-4452-b7e0-b4aaba6fefa4" containerName="main" probeResult="failure" output="Get \"https://10.132.0.57:8000/health\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:49:50.807992 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:49:50.807444    4892 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-5f55cd/premium-w6-5f55cd-kserve-7f785c544b-pcvhb" podUID="191593a2-e3a9-4ce4-a1c6-c089723b0237" containerName="main" probeResult="failure" output="Get \"https://10.132.0.60:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:49:50.818820 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:49:50.818780    4892 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0bdf7217-4cc3-4710-b13e-ffcee12febf0" path="/var/lib/kubelet/pods/0bdf7217-4cc3-4710-b13e-ffcee12febf0/volumes"
Oct 30 19:49:50.838220 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:49:50.820416    4892 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d299acaf-6665-4ab0-8333-cca9953d8f47" path="/var/lib/kubelet/pods/d299acaf-6665-4ab0-8333-cca9953d8f47/volumes"
Oct 30 19:49:50.841944 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:49:50.838519    4892 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-ada8d7/simulator-w3-ada8d7-kserve-7d575cfcb5-5txgn" podUID="10a5c7c8-6146-47b0-9643-bb137cf10bf5" containerName="main" probeResult="failure" output="Get \"https://10.132.0.55:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:49:50.890285 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:49:50.890240    4892 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-686689/distinct-2-w5-686689-kserve-5ff578b4cd-rkc6s" podUID="a8829732-ac6f-4238-b8ce-fc781084668b" containerName="main" probeResult="failure" output="Get \"https://10.132.0.87:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:49:50.982994 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:49:50.960569    4892 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-a3f9b7/simulator-w2-a3f9b7-kserve-79bbc4cb5f-8m488" podUID="50fb2283-bf3f-41da-835b-f407ee1a04f3" containerName="main" probeResult="failure" output="Get \"https://10.132.0.59:8000/ready\": context deadline exceeded"
Oct 30 19:49:51.009986 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:49:50.993425    4892 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-d52df0/simulator-w1-d52df0-kserve-67f74bc577-q5bjz" podUID="81b7bb84-1003-4452-b7e0-b4aaba6fefa4" containerName="main" probeResult="failure" output="Get \"https://10.132.0.57:8000/ready\": context deadline exceeded"
Oct 30 19:49:51.050304 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:49:51.050207    4892 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-d52df0/distinct-2-w1-d52df0-kserve-7846558464-xrkdc" podUID="9c614ab6-30e2-4fe1-8609-d5e2dadc198a" containerName="main" probeResult="failure" output="Get \"https://10.132.0.83:8000/ready\": context deadline exceeded"
Oct 30 19:49:51.052328 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:49:51.052304    4892 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-d52df0/distinct-2-w1-d52df0-kserve-7846558464-xrkdc"
Oct 30 19:49:51.393991 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:49:51.388061    4892 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-b5178f/distinct-2-w4-b5178f-kserve-69ddf8b665-9pdth" podUID="fb71d961-8500-4367-b48c-88d385962bd5" containerName="main" probeResult="failure" output="Get \"https://10.132.0.90:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:49:51.396051 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:49:51.395001    4892 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-d52df0/distinct-2-w1-d52df0-kserve-7846558464-xrkdc"
Oct 30 19:49:51.614991 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:49:51.613450    4892 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-5f55cd/simulator-w6-5f55cd-kserve-5ff86b7447-h9tm7" podUID="3cb47a99-8be1-4756-b31a-43b58cc6b8dc" containerName="main" probeResult="failure" output="Get \"https://10.132.0.58:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:49:51.684034 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:49:51.673558    4892 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w0-2d9861/premium-w0-2d9861-kserve-67945c6848-4n6kz" podUID="798988b2-c769-4f73-a7c8-ca15de0beef1" containerName="main" probeResult="failure" output="Get \"https://10.132.0.62:8000/health\": context deadline exceeded"
Oct 30 19:49:51.793985 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:49:51.793040    4892 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-8a3c7851-z798v"]
Oct 30 19:49:51.844194 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:49:51.844140    4892 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-ada8d7/simulator-w3-ada8d7-kserve-7d575cfcb5-5txgn" podUID="10a5c7c8-6146-47b0-9643-bb137cf10bf5" containerName="main" probeResult="failure" output="Get \"https://10.132.0.55:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:49:51.845177 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:49:51.845153    4892 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-ada8d7/simulator-w3-ada8d7-kserve-7d575cfcb5-5txgn"
Oct 30 19:49:51.896228 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:49:51.892333    4892 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-686689/distinct-2-w5-686689-kserve-5ff578b4cd-rkc6s" podUID="a8829732-ac6f-4238-b8ce-fc781084668b" containerName="main" probeResult="failure" output="Get \"https://10.132.0.87:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:49:51.896890 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:49:51.896867    4892 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w5-686689/distinct-2-w5-686689-kserve-5ff578b4cd-rkc6s"
Oct 30 19:49:51.913045 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:49:51.913014    4892 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-8a3c7851-z798v"]
Oct 30 19:49:51.926292 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:49:51.926265    4892 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-derive-8a3c7851-openshift-default-7845b86bf7-xx8vl"]
Oct 30 19:49:51.926806 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:49:51.926776    4892 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-derive-8a3c7851-openshift-default-7845b86bf7-xx8vl" podUID="9c98da0e-489b-4333-b450-704beaf2f325" containerName="istio-proxy" containerID="cri-o://9d7be60ad01352d352c6af18b909548df3c56c93ca1db09aa276ee8eedea5ec1" gracePeriod=30
Oct 30 19:49:52.040659 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:49:52.035199    4892 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-d52df0/simulator-w1-d52df0-kserve-67f74bc577-q5bjz" podUID="81b7bb84-1003-4452-b7e0-b4aaba6fefa4" containerName="main" probeResult="failure" output="Get \"https://10.132.0.57:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:49:52.040659 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:49:52.035583    4892 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-d52df0/simulator-w1-d52df0-kserve-67f74bc577-q5bjz"
Oct 30 19:49:52.040659 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:49:52.035097    4892 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-a3f9b7/simulator-w2-a3f9b7-kserve-79bbc4cb5f-8m488" podUID="50fb2283-bf3f-41da-835b-f407ee1a04f3" containerName="main" probeResult="failure" output="Get \"https://10.132.0.59:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:49:52.040659 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:49:52.036979    4892 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-a3f9b7/simulator-w2-a3f9b7-kserve-79bbc4cb5f-8m488"
Oct 30 19:49:52.099007 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:49:52.085144    4892 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-2d9861/premium-w0-2d9861-kserve-67945c6848-4n6kz" podUID="798988b2-c769-4f73-a7c8-ca15de0beef1" containerName="main" probeResult="failure" output="Get \"https://10.132.0.62:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:49:52.301169 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:49:52.301075    4892 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w4-b5178f/distinct-2-w4-b5178f-kserve-69ddf8b665-9pdth" podUID="fb71d961-8500-4367-b48c-88d385962bd5" containerName="main" probeResult="failure" output="Get \"https://10.132.0.90:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:49:52.383991 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:49:52.383281    4892 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w3-ada8d7/simulator-w3-ada8d7-kserve-7d575cfcb5-5txgn" podUID="10a5c7c8-6146-47b0-9643-bb137cf10bf5" containerName="main" probeResult="failure" output="Get \"https://10.132.0.55:8000/health\": context deadline exceeded"
Oct 30 19:49:52.409107 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:49:52.409066    4892 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-b5178f/distinct-2-w4-b5178f-kserve-69ddf8b665-9pdth" podUID="fb71d961-8500-4367-b48c-88d385962bd5" containerName="main" probeResult="failure" output="Get \"https://10.132.0.90:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:49:52.413442 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:49:52.409362    4892 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-d52df0/distinct-2-w1-d52df0-kserve-7846558464-xrkdc" podUID="9c614ab6-30e2-4fe1-8609-d5e2dadc198a" containerName="main" probeResult="failure" output="Get \"https://10.132.0.83:8000/ready\": context deadline exceeded"
Oct 30 19:49:52.414610 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:49:52.413710    4892 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-b5178f/distinct-2-w4-b5178f-kserve-69ddf8b665-9pdth"
Oct 30 19:49:52.615294 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:49:52.615187    4892 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-5f55cd/simulator-w6-5f55cd-kserve-5ff86b7447-h9tm7" podUID="3cb47a99-8be1-4756-b31a-43b58cc6b8dc" containerName="main" probeResult="failure" output="Get \"https://10.132.0.58:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:49:52.833269 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:49:52.833229    4892 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-5f55cd/premium-w6-5f55cd-kserve-7f785c544b-pcvhb" podUID="191593a2-e3a9-4ce4-a1c6-c089723b0237" containerName="main" probeResult="failure" output="Get \"https://10.132.0.60:8000/ready\": context deadline exceeded"
Oct 30 19:49:52.854077 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:49:52.850402    4892 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-ada8d7/simulator-w3-ada8d7-kserve-7d575cfcb5-5txgn" podUID="10a5c7c8-6146-47b0-9643-bb137cf10bf5" containerName="main" probeResult="failure" output="Get \"https://10.132.0.55:8000/ready\": context deadline exceeded"
Oct 30 19:49:52.875070 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:49:52.874105    4892 patch_prober.go:29] interesting pod/e2e-derive-8a3c7851-openshift-default-7845b86bf7-xx8vl container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Oct 30 19:49:52.875070 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:49:52.874160    4892 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-derive-8a3c7851-openshift-default-7845b86bf7-xx8vl" podUID="9c98da0e-489b-4333-b450-704beaf2f325" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Oct 30 19:49:52.896268 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:49:52.896233    4892 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="5b7dcbb7-54cd-4d78-af9b-f1d8ccf00d3f" path="/var/lib/kubelet/pods/5b7dcbb7-54cd-4d78-af9b-f1d8ccf00d3f/volumes"
Oct 30 19:49:52.897056 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:49:52.897031    4892 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w6-5f55cd/simulator-w6-5f55cd-kserve-5ff86b7447-h9tm7"
Oct 30 19:49:52.897255 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:49:52.897239    4892 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w5-686689/distinct-2-w5-686689-kserve-5ff578b4cd-rkc6s"
Oct 30 19:49:53.037127 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:49:53.037082    4892 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-a3f9b7/simulator-w2-a3f9b7-kserve-79bbc4cb5f-8m488" podUID="50fb2283-bf3f-41da-835b-f407ee1a04f3" containerName="main" probeResult="failure" output="Get \"https://10.132.0.59:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:49:53.049995 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:49:53.037679    4892 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-d52df0/simulator-w1-d52df0-kserve-67f74bc577-q5bjz" podUID="81b7bb84-1003-4452-b7e0-b4aaba6fefa4" containerName="main" probeResult="failure" output="Get \"https://10.132.0.57:8000/ready\": context deadline exceeded"
Oct 30 19:49:53.087994 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:49:53.086180    4892 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-2d9861/premium-w0-2d9861-kserve-67945c6848-4n6kz" podUID="798988b2-c769-4f73-a7c8-ca15de0beef1" containerName="main" probeResult="failure" output="Get \"https://10.132.0.62:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:49:53.087994 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:49:53.087021    4892 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-2d9861/premium-w0-2d9861-kserve-67945c6848-4n6kz"
Oct 30 19:49:53.440991 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:49:53.422232    4892 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-b5178f/distinct-2-w4-b5178f-kserve-69ddf8b665-9pdth" podUID="fb71d961-8500-4367-b48c-88d385962bd5" containerName="main" probeResult="failure" output="Get \"https://10.132.0.90:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:49:53.440991 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:49:53.422617    4892 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-d52df0/distinct-2-w1-d52df0-kserve-7846558464-xrkdc" podUID="9c614ab6-30e2-4fe1-8609-d5e2dadc198a" containerName="main" probeResult="failure" output="Get \"https://10.132.0.83:8000/ready\": context deadline exceeded"
Oct 30 19:49:53.440991 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:49:53.423405    4892 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-d52df0/distinct-2-w1-d52df0-kserve-7846558464-xrkdc"
Oct 30 19:49:53.498992 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:49:53.484924    4892 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-a3f9b7/simulator-w2-a3f9b7-kserve-79bbc4cb5f-8m488"
Oct 30 19:49:53.572918 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:49:53.572886    4892 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-2d9861/premium-w0-2d9861-kserve-67945c6848-4n6kz"
Oct 30 19:49:53.642627 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:49:53.642599    4892 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-d52df0/simulator-w1-d52df0-kserve-67f74bc577-q5bjz"
Oct 30 19:49:53.668514 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:49:53.668484    4892 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-d52df0/distinct-2-w1-d52df0-kserve-7846558464-xrkdc"
Oct 30 19:49:53.698172 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:49:53.679572    4892 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-5f55cd/simulator-w6-5f55cd-kserve-5ff86b7447-h9tm7" podUID="3cb47a99-8be1-4756-b31a-43b58cc6b8dc" containerName="main" probeResult="failure" output="Get \"https://10.132.0.58:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:49:53.834593 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:49:53.833068    4892 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-686689/distinct-2-w5-686689-kserve-5ff578b4cd-rkc6s" podUID="a8829732-ac6f-4238-b8ce-fc781084668b" containerName="main" probeResult="failure" output="Get \"https://10.132.0.87:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:49:53.858233 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:49:53.858193    4892 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-ada8d7/simulator-w3-ada8d7-kserve-7d575cfcb5-5txgn" podUID="10a5c7c8-6146-47b0-9643-bb137cf10bf5" containerName="main" probeResult="failure" output="Get \"https://10.132.0.55:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:49:54.014053 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:49:54.012612    4892 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w6-5f55cd/simulator-w6-5f55cd-kserve-5ff86b7447-h9tm7"
Oct 30 19:49:54.101605 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:49:54.101583    4892 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-b5178f/distinct-2-w4-b5178f-kserve-69ddf8b665-9pdth"
Oct 30 19:49:54.272546 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:49:54.272459    4892 scope.go:122] "RemoveContainer" containerID="533908949b3e37aae1637ec976559e6519109a433712a90acd9c5909c5f8dfc7"
Oct 30 19:49:54.710979 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:49:54.708850    4892 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-d52df0/distinct-2-w1-d52df0-kserve-7846558464-xrkdc" podUID="9c614ab6-30e2-4fe1-8609-d5e2dadc198a" containerName="main" probeResult="failure" output="Get \"https://10.132.0.83:8000/ready\": context deadline exceeded"
Oct 30 19:49:54.750995 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:49:54.741871    4892 scope.go:122] "RemoveContainer" containerID="100400e8af473dfd0c5ee8c32ba2e4d297751b0570f398347d1851d22ccae74b"
Oct 30 19:49:54.802155 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:49:54.798041    4892 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-5f55cd/premium-w6-5f55cd-kserve-7f785c544b-pcvhb" podUID="191593a2-e3a9-4ce4-a1c6-c089723b0237" containerName="main" probeResult="failure" output="Get \"https://10.132.0.60:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:49:54.909713 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:49:54.896170    4892 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-ada8d7/simulator-w3-ada8d7-kserve-7d575cfcb5-5txgn" podUID="10a5c7c8-6146-47b0-9643-bb137cf10bf5" containerName="main" probeResult="failure" output="Get \"https://10.132.0.55:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:49:54.909713 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:49:54.907839    4892 scope.go:122] "RemoveContainer" containerID="1b500499c8fafcb5e4fcde940952876bf77ddfc4d09e1fb4d2a9009c7f3579c0"
Oct 30 19:49:54.942210 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:49:54.942142    4892 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-686689/distinct-2-w5-686689-kserve-5ff578b4cd-rkc6s" podUID="a8829732-ac6f-4238-b8ce-fc781084668b" containerName="main" probeResult="failure" output="Get \"https://10.132.0.87:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:49:54.943019 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:49:54.942998    4892 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w5-686689/distinct-2-w5-686689-kserve-5ff578b4cd-rkc6s"
Oct 30 19:49:54.967531 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:49:54.966116    4892 scope.go:122] "RemoveContainer" containerID="c8ec16c869191b30e60c22a2527e52f0af8397c65126fd4b67a07aebd051e74c"
Oct 30 19:49:55.042990 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:49:55.038063    4892 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-a3f9b7/simulator-w2-a3f9b7-kserve-79bbc4cb5f-8m488" podUID="50fb2283-bf3f-41da-835b-f407ee1a04f3" containerName="main" probeResult="failure" output="Get \"https://10.132.0.59:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:49:55.206197 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:49:55.199146    4892 scope.go:122] "RemoveContainer" containerID="5abf0c86a1d954512e2f7f1807817cd053d2062b9c7efc8ed2964e380e8d8c11"
Oct 30 19:49:55.391523 ip-10-0-132-82 kubenswrapper[4892]: E1030 19:49:55.391487    4892 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-jrm87" containerName="extractor"
Oct 30 19:49:55.534518 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:49:55.534195    4892 scope.go:122] "RemoveContainer" containerID="1e4bbc8e4a1be1435698519557f20d55c89fe7141636220fec506120abd765c9"
Oct 30 19:49:55.623985 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:49:55.621133    4892 generic.go:356] "Generic (PLEG): container finished" podID="0f4fa4be-451c-43b5-bc35-cdf507a2dfb3" containerID="772e8c1633aca86a1b725cee1d4db72ab0e0927004a64e78f0ea6d7cb13f9bf6" exitCode=0
Oct 30 19:49:55.623985 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:49:55.621271    4892 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-2d9861-openshift-default-58ddf574b9-w2r9m" event={"ID":"0f4fa4be-451c-43b5-bc35-cdf507a2dfb3","Type":"ContainerDied","Data":"772e8c1633aca86a1b725cee1d4db72ab0e0927004a64e78f0ea6d7cb13f9bf6"}
Oct 30 19:49:55.673275 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:49:55.671853    4892 scope.go:122] "RemoveContainer" containerID="a9cf9716c9ffd9533e0d5be12ef76b0ae3a9f1d8cfcfadbf908c24401af1867c"
Oct 30 19:49:55.779138 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:49:55.773928    4892 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w5-686689/distinct-2-w5-686689-kserve-5ff578b4cd-rkc6s"
Oct 30 19:49:55.815990 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:49:55.814550    4892 scope.go:122] "RemoveContainer" containerID="52bce220e3c1d35a73bed99b72853ee002edf1bcb14ee03dbc00b8f4f34a24b0"
Oct 30 19:49:55.911390 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:49:55.910266    4892 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-ada8d7/simulator-w3-ada8d7-kserve-7d575cfcb5-5txgn" podUID="10a5c7c8-6146-47b0-9643-bb137cf10bf5" containerName="main" probeResult="failure" output="Get \"https://10.132.0.55:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:49:55.935235 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:49:55.914876    4892 scope.go:122] "RemoveContainer" containerID="0cad2fcd9c5118cef7cb15d31b0b6314826d4bf1d98894a490c4464b77e8a715"
Oct 30 19:49:56.077072 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:49:56.076064    4892 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-5f55cd/simulator-w6-5f55cd-kserve-5ff86b7447-h9tm7" podUID="3cb47a99-8be1-4756-b31a-43b58cc6b8dc" containerName="main" probeResult="failure" output="Get \"https://10.132.0.58:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:49:56.441008 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:49:56.436381    4892 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-ada8d7/simulator-w3-ada8d7-kserve-7d575cfcb5-5txgn"
Oct 30 19:49:56.488996 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:49:56.485087    4892 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-b5178f/distinct-2-w4-b5178f-kserve-69ddf8b665-9pdth" podUID="fb71d961-8500-4367-b48c-88d385962bd5" containerName="main" probeResult="failure" output="Get \"https://10.132.0.90:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:49:56.794071 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:49:56.792254    4892 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-2d9861-openshift-default-58ddf574b9-w2r9m" event={"ID":"0f4fa4be-451c-43b5-bc35-cdf507a2dfb3","Type":"ContainerDied","Data":"19c39f69dacc7b2dbe4c49b40289e50a3c265c3da952b59487f4042ad43ae610"}
Oct 30 19:49:56.794071 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:49:56.792352    4892 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="19c39f69dacc7b2dbe4c49b40289e50a3c265c3da952b59487f4042ad43ae610"
Oct 30 19:49:56.794631 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:49:56.794398    4892 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-5f55cd/premium-w6-5f55cd-kserve-7f785c544b-pcvhb" podUID="191593a2-e3a9-4ce4-a1c6-c089723b0237" containerName="main" probeResult="failure" output="Get \"https://10.132.0.60:8000/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:49:56.796164 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:49:56.796132    4892 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-686689/distinct-2-w5-686689-kserve-5ff578b4cd-rkc6s" podUID="a8829732-ac6f-4238-b8ce-fc781084668b" containerName="main" probeResult="failure" output="Get \"https://10.132.0.87:8000/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:49:56.807786 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:49:56.807295    4892 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-2d9861-openshift-default-58ddf574b9-w2r9m"
Oct 30 19:49:56.884286 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:49:56.869175    4892 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0f4fa4be-451c-43b5-bc35-cdf507a2dfb3-kube-api-access-tbvsg\" (UniqueName: \"kubernetes.io/projected/0f4fa4be-451c-43b5-bc35-cdf507a2dfb3-kube-api-access-tbvsg\") pod \"0f4fa4be-451c-43b5-bc35-cdf507a2dfb3\" (UID: \"0f4fa4be-451c-43b5-bc35-cdf507a2dfb3\") "
Oct 30 19:49:56.884286 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:49:56.869243    4892 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0f4fa4be-451c-43b5-bc35-cdf507a2dfb3-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0f4fa4be-451c-43b5-bc35-cdf507a2dfb3-credential-socket\") pod \"0f4fa4be-451c-43b5-bc35-cdf507a2dfb3\" (UID: \"0f4fa4be-451c-43b5-bc35-cdf507a2dfb3\") "
Oct 30 19:49:56.884286 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:49:56.869290    4892 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0f4fa4be-451c-43b5-bc35-cdf507a2dfb3-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0f4fa4be-451c-43b5-bc35-cdf507a2dfb3-workload-socket\") pod \"0f4fa4be-451c-43b5-bc35-cdf507a2dfb3\" (UID: \"0f4fa4be-451c-43b5-bc35-cdf507a2dfb3\") "
Oct 30 19:49:56.884286 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:49:56.869344    4892 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0f4fa4be-451c-43b5-bc35-cdf507a2dfb3-istio-token\" (UniqueName: \"kubernetes.io/projected/0f4fa4be-451c-43b5-bc35-cdf507a2dfb3-istio-token\") pod \"0f4fa4be-451c-43b5-bc35-cdf507a2dfb3\" (UID: \"0f4fa4be-451c-43b5-bc35-cdf507a2dfb3\") "
Oct 30 19:49:56.884286 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:49:56.869444    4892 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/0f4fa4be-451c-43b5-bc35-cdf507a2dfb3-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0f4fa4be-451c-43b5-bc35-cdf507a2dfb3-istiod-ca-cert\") pod \"0f4fa4be-451c-43b5-bc35-cdf507a2dfb3\" (UID: \"0f4fa4be-451c-43b5-bc35-cdf507a2dfb3\") "
Oct 30 19:49:56.884286 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:49:56.869549    4892 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/0f4fa4be-451c-43b5-bc35-cdf507a2dfb3-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0f4fa4be-451c-43b5-bc35-cdf507a2dfb3-istio-podinfo\") pod \"0f4fa4be-451c-43b5-bc35-cdf507a2dfb3\" (UID: \"0f4fa4be-451c-43b5-bc35-cdf507a2dfb3\") "
Oct 30 19:49:56.884286 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:49:56.869684    4892 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0f4fa4be-451c-43b5-bc35-cdf507a2dfb3-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0f4fa4be-451c-43b5-bc35-cdf507a2dfb3-istio-envoy\") pod \"0f4fa4be-451c-43b5-bc35-cdf507a2dfb3\" (UID: \"0f4fa4be-451c-43b5-bc35-cdf507a2dfb3\") "
Oct 30 19:49:56.884286 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:49:56.869770    4892 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0f4fa4be-451c-43b5-bc35-cdf507a2dfb3-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0f4fa4be-451c-43b5-bc35-cdf507a2dfb3-istio-data\") pod \"0f4fa4be-451c-43b5-bc35-cdf507a2dfb3\" (UID: \"0f4fa4be-451c-43b5-bc35-cdf507a2dfb3\") "
Oct 30 19:49:56.884286 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:49:56.869807    4892 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0f4fa4be-451c-43b5-bc35-cdf507a2dfb3-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0f4fa4be-451c-43b5-bc35-cdf507a2dfb3-workload-certs\") pod \"0f4fa4be-451c-43b5-bc35-cdf507a2dfb3\" (UID: \"0f4fa4be-451c-43b5-bc35-cdf507a2dfb3\") "
Oct 30 19:49:56.884286 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:49:56.871569    4892 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0f4fa4be-451c-43b5-bc35-cdf507a2dfb3-workload-certs" pod "0f4fa4be-451c-43b5-bc35-cdf507a2dfb3" (UID: "0f4fa4be-451c-43b5-bc35-cdf507a2dfb3"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:49:56.885092 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:49:56.884324    4892 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0f4fa4be-451c-43b5-bc35-cdf507a2dfb3-credential-socket" pod "0f4fa4be-451c-43b5-bc35-cdf507a2dfb3" (UID: "0f4fa4be-451c-43b5-bc35-cdf507a2dfb3"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:49:56.885092 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:49:56.884536    4892 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0f4fa4be-451c-43b5-bc35-cdf507a2dfb3-workload-socket" pod "0f4fa4be-451c-43b5-bc35-cdf507a2dfb3" (UID: "0f4fa4be-451c-43b5-bc35-cdf507a2dfb3"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:49:56.907034 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:49:56.889138    4892 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0f4fa4be-451c-43b5-bc35-cdf507a2dfb3-kube-api-access-tbvsg" pod "0f4fa4be-451c-43b5-bc35-cdf507a2dfb3" (UID: "0f4fa4be-451c-43b5-bc35-cdf507a2dfb3"). InnerVolumeSpecName "kube-api-access-tbvsg". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:49:56.907034 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:49:56.889371    4892 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0f4fa4be-451c-43b5-bc35-cdf507a2dfb3-istio-data" pod "0f4fa4be-451c-43b5-bc35-cdf507a2dfb3" (UID: "0f4fa4be-451c-43b5-bc35-cdf507a2dfb3"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:49:56.907034 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:49:56.889794    4892 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0f4fa4be-451c-43b5-bc35-cdf507a2dfb3-istiod-ca-cert" pod "0f4fa4be-451c-43b5-bc35-cdf507a2dfb3" (UID: "0f4fa4be-451c-43b5-bc35-cdf507a2dfb3"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 19:49:56.907034 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:49:56.893143    4892 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0f4fa4be-451c-43b5-bc35-cdf507a2dfb3-istio-envoy" pod "0f4fa4be-451c-43b5-bc35-cdf507a2dfb3" (UID: "0f4fa4be-451c-43b5-bc35-cdf507a2dfb3"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:49:56.907034 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:49:56.898289    4892 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0f4fa4be-451c-43b5-bc35-cdf507a2dfb3-istio-token" pod "0f4fa4be-451c-43b5-bc35-cdf507a2dfb3" (UID: "0f4fa4be-451c-43b5-bc35-cdf507a2dfb3"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:49:56.907034 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:49:56.904636    4892 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/0f4fa4be-451c-43b5-bc35-cdf507a2dfb3-istio-podinfo" pod "0f4fa4be-451c-43b5-bc35-cdf507a2dfb3" (UID: "0f4fa4be-451c-43b5-bc35-cdf507a2dfb3"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 19:49:56.974153 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:49:56.972306    4892 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0f4fa4be-451c-43b5-bc35-cdf507a2dfb3-istio-envoy\") on node \"ip-10-0-132-82.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 19:49:56.974153 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:49:56.972344    4892 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0f4fa4be-451c-43b5-bc35-cdf507a2dfb3-istio-data\") on node \"ip-10-0-132-82.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 19:49:56.974153 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:49:56.972360    4892 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0f4fa4be-451c-43b5-bc35-cdf507a2dfb3-workload-certs\") on node \"ip-10-0-132-82.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 19:49:56.974153 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:49:56.972375    4892 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tbvsg\" (UniqueName: \"kubernetes.io/projected/0f4fa4be-451c-43b5-bc35-cdf507a2dfb3-kube-api-access-tbvsg\") on node \"ip-10-0-132-82.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 19:49:56.974153 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:49:56.972389    4892 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0f4fa4be-451c-43b5-bc35-cdf507a2dfb3-credential-socket\") on node \"ip-10-0-132-82.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 19:49:56.974153 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:49:56.972404    4892 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0f4fa4be-451c-43b5-bc35-cdf507a2dfb3-workload-socket\") on node \"ip-10-0-132-82.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 19:49:56.974153 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:49:56.972417    4892 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0f4fa4be-451c-43b5-bc35-cdf507a2dfb3-istio-token\") on node \"ip-10-0-132-82.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 19:49:56.974153 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:49:56.972430    4892 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0f4fa4be-451c-43b5-bc35-cdf507a2dfb3-istiod-ca-cert\") on node \"ip-10-0-132-82.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 19:49:56.974153 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:49:56.972443    4892 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0f4fa4be-451c-43b5-bc35-cdf507a2dfb3-istio-podinfo\") on node \"ip-10-0-132-82.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 19:49:57.293217 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:49:57.293193    4892 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-derive-8a3c7851-openshift-default-7845b86bf7-xx8vl"
Oct 30 19:49:57.379168 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:49:57.379137    4892 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9c98da0e-489b-4333-b450-704beaf2f325-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9c98da0e-489b-4333-b450-704beaf2f325-istio-envoy\") pod \"9c98da0e-489b-4333-b450-704beaf2f325\" (UID: \"9c98da0e-489b-4333-b450-704beaf2f325\") "
Oct 30 19:49:57.379325 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:49:57.379229    4892 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9c98da0e-489b-4333-b450-704beaf2f325-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9c98da0e-489b-4333-b450-704beaf2f325-istio-data\") pod \"9c98da0e-489b-4333-b450-704beaf2f325\" (UID: \"9c98da0e-489b-4333-b450-704beaf2f325\") "
Oct 30 19:49:57.379392 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:49:57.379340    4892 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9c98da0e-489b-4333-b450-704beaf2f325-kube-api-access-h5gnr\" (UniqueName: \"kubernetes.io/projected/9c98da0e-489b-4333-b450-704beaf2f325-kube-api-access-h5gnr\") pod \"9c98da0e-489b-4333-b450-704beaf2f325\" (UID: \"9c98da0e-489b-4333-b450-704beaf2f325\") "
Oct 30 19:49:57.379549 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:49:57.379512    4892 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9c98da0e-489b-4333-b450-704beaf2f325-istio-data" pod "9c98da0e-489b-4333-b450-704beaf2f325" (UID: "9c98da0e-489b-4333-b450-704beaf2f325"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:49:57.379767 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:49:57.379749    4892 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9c98da0e-489b-4333-b450-704beaf2f325-istio-token\" (UniqueName: \"kubernetes.io/projected/9c98da0e-489b-4333-b450-704beaf2f325-istio-token\") pod \"9c98da0e-489b-4333-b450-704beaf2f325\" (UID: \"9c98da0e-489b-4333-b450-704beaf2f325\") "
Oct 30 19:49:57.380193 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:49:57.380173    4892 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9c98da0e-489b-4333-b450-704beaf2f325-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9c98da0e-489b-4333-b450-704beaf2f325-workload-certs\") pod \"9c98da0e-489b-4333-b450-704beaf2f325\" (UID: \"9c98da0e-489b-4333-b450-704beaf2f325\") "
Oct 30 19:49:57.380326 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:49:57.380232    4892 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/9c98da0e-489b-4333-b450-704beaf2f325-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9c98da0e-489b-4333-b450-704beaf2f325-istiod-ca-cert\") pod \"9c98da0e-489b-4333-b450-704beaf2f325\" (UID: \"9c98da0e-489b-4333-b450-704beaf2f325\") "
Oct 30 19:49:57.380326 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:49:57.380306    4892 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9c98da0e-489b-4333-b450-704beaf2f325-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9c98da0e-489b-4333-b450-704beaf2f325-workload-socket\") pod \"9c98da0e-489b-4333-b450-704beaf2f325\" (UID: \"9c98da0e-489b-4333-b450-704beaf2f325\") "
Oct 30 19:49:57.380468 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:49:57.380351    4892 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9c98da0e-489b-4333-b450-704beaf2f325-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9c98da0e-489b-4333-b450-704beaf2f325-credential-socket\") pod \"9c98da0e-489b-4333-b450-704beaf2f325\" (UID: \"9c98da0e-489b-4333-b450-704beaf2f325\") "
Oct 30 19:49:57.380533 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:49:57.380479    4892 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9c98da0e-489b-4333-b450-704beaf2f325-workload-certs" pod "9c98da0e-489b-4333-b450-704beaf2f325" (UID: "9c98da0e-489b-4333-b450-704beaf2f325"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:49:57.380533 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:49:57.380517    4892 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/9c98da0e-489b-4333-b450-704beaf2f325-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9c98da0e-489b-4333-b450-704beaf2f325-istio-podinfo\") pod \"9c98da0e-489b-4333-b450-704beaf2f325\" (UID: \"9c98da0e-489b-4333-b450-704beaf2f325\") "
Oct 30 19:49:57.380950 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:49:57.380761    4892 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9c98da0e-489b-4333-b450-704beaf2f325-istiod-ca-cert" pod "9c98da0e-489b-4333-b450-704beaf2f325" (UID: "9c98da0e-489b-4333-b450-704beaf2f325"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 19:49:57.380950 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:49:57.380917    4892 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9c98da0e-489b-4333-b450-704beaf2f325-workload-socket" pod "9c98da0e-489b-4333-b450-704beaf2f325" (UID: "9c98da0e-489b-4333-b450-704beaf2f325"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:49:57.380950 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:49:57.380937    4892 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9c98da0e-489b-4333-b450-704beaf2f325-credential-socket" pod "9c98da0e-489b-4333-b450-704beaf2f325" (UID: "9c98da0e-489b-4333-b450-704beaf2f325"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:49:57.381335 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:49:57.381315    4892 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9c98da0e-489b-4333-b450-704beaf2f325-istio-data\") on node \"ip-10-0-132-82.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 19:49:57.381387 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:49:57.381341    4892 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9c98da0e-489b-4333-b450-704beaf2f325-workload-certs\") on node \"ip-10-0-132-82.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 19:49:57.381445 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:49:57.381359    4892 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9c98da0e-489b-4333-b450-704beaf2f325-istiod-ca-cert\") on node \"ip-10-0-132-82.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 19:49:57.381484 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:49:57.381452    4892 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9c98da0e-489b-4333-b450-704beaf2f325-workload-socket\") on node \"ip-10-0-132-82.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 19:49:57.381484 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:49:57.381470    4892 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9c98da0e-489b-4333-b450-704beaf2f325-credential-socket\") on node \"ip-10-0-132-82.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 19:49:57.382369 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:49:57.382342    4892 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9c98da0e-489b-4333-b450-704beaf2f325-istio-envoy" pod "9c98da0e-489b-4333-b450-704beaf2f325" (UID: "9c98da0e-489b-4333-b450-704beaf2f325"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:49:57.382369 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:49:57.382353    4892 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9c98da0e-489b-4333-b450-704beaf2f325-kube-api-access-h5gnr" pod "9c98da0e-489b-4333-b450-704beaf2f325" (UID: "9c98da0e-489b-4333-b450-704beaf2f325"). InnerVolumeSpecName "kube-api-access-h5gnr". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:49:57.383381 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:49:57.383358    4892 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9c98da0e-489b-4333-b450-704beaf2f325-istio-token" pod "9c98da0e-489b-4333-b450-704beaf2f325" (UID: "9c98da0e-489b-4333-b450-704beaf2f325"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:49:57.384060 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:49:57.384036    4892 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/9c98da0e-489b-4333-b450-704beaf2f325-istio-podinfo" pod "9c98da0e-489b-4333-b450-704beaf2f325" (UID: "9c98da0e-489b-4333-b450-704beaf2f325"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 19:49:57.483451 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:49:57.483417    4892 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9c98da0e-489b-4333-b450-704beaf2f325-istio-envoy\") on node \"ip-10-0-132-82.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 19:49:57.483715 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:49:57.483661    4892 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-h5gnr\" (UniqueName: \"kubernetes.io/projected/9c98da0e-489b-4333-b450-704beaf2f325-kube-api-access-h5gnr\") on node \"ip-10-0-132-82.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 19:49:57.483830 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:49:57.483816    4892 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9c98da0e-489b-4333-b450-704beaf2f325-istio-token\") on node \"ip-10-0-132-82.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 19:49:57.483927 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:49:57.483915    4892 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9c98da0e-489b-4333-b450-704beaf2f325-istio-podinfo\") on node \"ip-10-0-132-82.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 19:49:57.815759 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:49:57.815712    4892 generic.go:356] "Generic (PLEG): container finished" podID="9c98da0e-489b-4333-b450-704beaf2f325" containerID="9d7be60ad01352d352c6af18b909548df3c56c93ca1db09aa276ee8eedea5ec1" exitCode=0
Oct 30 19:49:57.816280 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:49:57.815793    4892 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-derive-8a3c7851-openshift-default-7845b86bf7-xx8vl"
Oct 30 19:49:57.816280 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:49:57.815955    4892 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-derive-8a3c7851-openshift-default-7845b86bf7-xx8vl" event={"ID":"9c98da0e-489b-4333-b450-704beaf2f325","Type":"ContainerDied","Data":"9d7be60ad01352d352c6af18b909548df3c56c93ca1db09aa276ee8eedea5ec1"}
Oct 30 19:49:57.816280 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:49:57.816091    4892 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-derive-8a3c7851-openshift-default-7845b86bf7-xx8vl" event={"ID":"9c98da0e-489b-4333-b450-704beaf2f325","Type":"ContainerDied","Data":"ca2c1dbf03b0c840fb882c9c8ef4807899d5e4d89d547e6029662cd2d9a57d7f"}
Oct 30 19:49:57.816280 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:49:57.816134    4892 scope.go:122] "RemoveContainer" containerID="9d7be60ad01352d352c6af18b909548df3c56c93ca1db09aa276ee8eedea5ec1"
Oct 30 19:49:57.816804 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:49:57.816773    4892 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-2d9861-openshift-default-58ddf574b9-w2r9m"
Oct 30 19:49:57.839504 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:49:57.839478    4892 scope.go:122] "RemoveContainer" containerID="9b069f96d6daf9fc90a38ed7626db9a2a20aa656f12734127a85f4d468ed67cb"
Oct 30 19:49:57.857763 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:49:57.857737    4892 scope.go:122] "RemoveContainer" containerID="9d7be60ad01352d352c6af18b909548df3c56c93ca1db09aa276ee8eedea5ec1"
Oct 30 19:49:57.858125 ip-10-0-132-82 kubenswrapper[4892]: E1030 19:49:57.858104    4892 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9d7be60ad01352d352c6af18b909548df3c56c93ca1db09aa276ee8eedea5ec1\": container with ID starting with 9d7be60ad01352d352c6af18b909548df3c56c93ca1db09aa276ee8eedea5ec1 not found: ID does not exist" containerID="9d7be60ad01352d352c6af18b909548df3c56c93ca1db09aa276ee8eedea5ec1"
Oct 30 19:49:57.858188 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:49:57.858134    4892 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9d7be60ad01352d352c6af18b909548df3c56c93ca1db09aa276ee8eedea5ec1"} err="failed to get container status \"9d7be60ad01352d352c6af18b909548df3c56c93ca1db09aa276ee8eedea5ec1\": rpc error: code = NotFound desc = could not find container \"9d7be60ad01352d352c6af18b909548df3c56c93ca1db09aa276ee8eedea5ec1\": container with ID starting with 9d7be60ad01352d352c6af18b909548df3c56c93ca1db09aa276ee8eedea5ec1 not found: ID does not exist"
Oct 30 19:49:57.858188 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:49:57.858152    4892 scope.go:122] "RemoveContainer" containerID="9b069f96d6daf9fc90a38ed7626db9a2a20aa656f12734127a85f4d468ed67cb"
Oct 30 19:49:57.858489 ip-10-0-132-82 kubenswrapper[4892]: E1030 19:49:57.858423    4892 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9b069f96d6daf9fc90a38ed7626db9a2a20aa656f12734127a85f4d468ed67cb\": container with ID starting with 9b069f96d6daf9fc90a38ed7626db9a2a20aa656f12734127a85f4d468ed67cb not found: ID does not exist" containerID="9b069f96d6daf9fc90a38ed7626db9a2a20aa656f12734127a85f4d468ed67cb"
Oct 30 19:49:57.858489 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:49:57.858465    4892 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9b069f96d6daf9fc90a38ed7626db9a2a20aa656f12734127a85f4d468ed67cb"} err="failed to get container status \"9b069f96d6daf9fc90a38ed7626db9a2a20aa656f12734127a85f4d468ed67cb\": rpc error: code = NotFound desc = could not find container \"9b069f96d6daf9fc90a38ed7626db9a2a20aa656f12734127a85f4d468ed67cb\": container with ID starting with 9b069f96d6daf9fc90a38ed7626db9a2a20aa656f12734127a85f4d468ed67cb not found: ID does not exist"
Oct 30 19:49:58.053072 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:49:58.053040    4892 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-2d9861-openshift-default-58ddf574b9-w2r9m"]
Oct 30 19:49:58.103242 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:49:58.103161    4892 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w0-2d9861-openshift-default-58ddf574b9-w2r9m"]
Oct 30 19:49:58.180732 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:49:58.180702    4892 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-derive-8a3c7851-openshift-default-7845b86bf7-xx8vl"]
Oct 30 19:49:58.185145 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:49:58.185122    4892 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-derive-8a3c7851-openshift-default-7845b86bf7-xx8vl"]
Oct 30 19:49:58.575941 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:49:58.575911    4892 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0f4fa4be-451c-43b5-bc35-cdf507a2dfb3" path="/var/lib/kubelet/pods/0f4fa4be-451c-43b5-bc35-cdf507a2dfb3/volumes"
Oct 30 19:49:58.576504 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:49:58.576486    4892 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="9c98da0e-489b-4333-b450-704beaf2f325" path="/var/lib/kubelet/pods/9c98da0e-489b-4333-b450-704beaf2f325/volumes"
Oct 30 19:50:02.039033 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:50:02.038998    4892 patch_prober.go:29] interesting pod/e2e-mt-5bf13ff3-openshift-default-7558bbbc65-9j7w6 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.171:15021/healthz/ready\": dial tcp 10.132.0.171:15021: connect: connection refused" start-of-body=
Oct 30 19:50:02.039425 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:50:02.039080    4892 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-mt-5bf13ff3-openshift-default-7558bbbc65-9j7w6" podUID="474b0b88-a062-4ee2-9994-022ae1253f28" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.171:15021/healthz/ready\": dial tcp 10.132.0.171:15021: connect: connection refused"
Oct 30 19:50:02.928107 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:50:02.927952    4892 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-86e4a7-85cc57c56c-vhlcj"]
Oct 30 19:50:02.947021 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:50:02.935994    4892 cpu_manager.go:414] "containerMap: removing container" podUID="dc4774a0-3da8-4c65-a0b3-887a476ab465" containerName="istio-proxy"
Oct 30 19:50:02.947021 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:50:02.936027    4892 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="dc4774a0-3da8-4c65-a0b3-887a476ab465" containerName="istio-proxy"
Oct 30 19:50:02.947021 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:50:02.936053    4892 cpu_manager.go:414] "containerMap: removing container" podUID="fcc8f06d-659a-42e9-9ee8-4390a2591f5b" containerName="istio-proxy"
Oct 30 19:50:02.947021 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:50:02.936061    4892 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="fcc8f06d-659a-42e9-9ee8-4390a2591f5b" containerName="istio-proxy"
Oct 30 19:50:02.947021 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:50:02.936085    4892 cpu_manager.go:414] "containerMap: removing container" podUID="1419be25-72fa-4db5-a92b-44972189a2cb" containerName="revoke-keys"
Oct 30 19:50:02.947021 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:50:02.936093    4892 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1419be25-72fa-4db5-a92b-44972189a2cb" containerName="revoke-keys"
Oct 30 19:50:02.947021 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:50:02.936143    4892 cpu_manager.go:414] "containerMap: removing container" podUID="0bdf7217-4cc3-4710-b13e-ffcee12febf0" containerName="istio-proxy"
Oct 30 19:50:02.947021 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:50:02.936152    4892 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0bdf7217-4cc3-4710-b13e-ffcee12febf0" containerName="istio-proxy"
Oct 30 19:50:02.947021 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:50:02.936167    4892 cpu_manager.go:414] "containerMap: removing container" podUID="6befc94b-40e5-42a4-adf5-392b5ecde123" containerName="istio-proxy"
Oct 30 19:50:02.947021 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:50:02.936174    4892 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6befc94b-40e5-42a4-adf5-392b5ecde123" containerName="istio-proxy"
Oct 30 19:50:02.947021 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:50:02.936199    4892 cpu_manager.go:414] "containerMap: removing container" podUID="0f4fa4be-451c-43b5-bc35-cdf507a2dfb3" containerName="istio-proxy"
Oct 30 19:50:02.947021 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:50:02.936206    4892 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0f4fa4be-451c-43b5-bc35-cdf507a2dfb3" containerName="istio-proxy"
Oct 30 19:50:02.947021 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:50:02.936216    4892 cpu_manager.go:414] "containerMap: removing container" podUID="9c98da0e-489b-4333-b450-704beaf2f325" containerName="istio-proxy"
Oct 30 19:50:02.947021 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:50:02.936223    4892 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9c98da0e-489b-4333-b450-704beaf2f325" containerName="istio-proxy"
Oct 30 19:50:02.947021 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:50:02.936237    4892 cpu_manager.go:414] "containerMap: removing container" podUID="9c98da0e-489b-4333-b450-704beaf2f325" containerName="istio-proxy"
Oct 30 19:50:02.947021 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:50:02.936244    4892 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9c98da0e-489b-4333-b450-704beaf2f325" containerName="istio-proxy"
Oct 30 19:50:02.947021 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:50:02.936413    4892 memory_manager.go:365] "RemoveStaleState removing state" podUID="9c98da0e-489b-4333-b450-704beaf2f325" containerName="istio-proxy"
Oct 30 19:50:02.947021 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:50:02.936431    4892 memory_manager.go:365] "RemoveStaleState removing state" podUID="dc4774a0-3da8-4c65-a0b3-887a476ab465" containerName="istio-proxy"
Oct 30 19:50:02.947021 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:50:02.936447    4892 memory_manager.go:365] "RemoveStaleState removing state" podUID="fcc8f06d-659a-42e9-9ee8-4390a2591f5b" containerName="istio-proxy"
Oct 30 19:50:02.947021 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:50:02.936461    4892 memory_manager.go:365] "RemoveStaleState removing state" podUID="1419be25-72fa-4db5-a92b-44972189a2cb" containerName="revoke-keys"
Oct 30 19:50:02.947021 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:50:02.936479    4892 memory_manager.go:365] "RemoveStaleState removing state" podUID="0bdf7217-4cc3-4710-b13e-ffcee12febf0" containerName="istio-proxy"
Oct 30 19:50:02.947021 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:50:02.936490    4892 memory_manager.go:365] "RemoveStaleState removing state" podUID="6befc94b-40e5-42a4-adf5-392b5ecde123" containerName="istio-proxy"
Oct 30 19:50:02.947021 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:50:02.936502    4892 memory_manager.go:365] "RemoveStaleState removing state" podUID="0f4fa4be-451c-43b5-bc35-cdf507a2dfb3" containerName="istio-proxy"
Oct 30 19:50:02.952062 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:50:02.952000    4892 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-a-86e4a7-85cc57c56c-vhlcj"
Oct 30 19:50:02.953671 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:50:02.953589    4892 generic.go:356] "Generic (PLEG): container finished" podID="474b0b88-a062-4ee2-9994-022ae1253f28" containerID="191522a582678a8a642c14a3c468d7a6d308a6de40f6275658f2e3176f86eeec" exitCode=0
Oct 30 19:50:02.953827 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:50:02.953698    4892 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-5bf13ff3-openshift-default-7558bbbc65-9j7w6" event={"ID":"474b0b88-a062-4ee2-9994-022ae1253f28","Type":"ContainerDied","Data":"191522a582678a8a642c14a3c468d7a6d308a6de40f6275658f2e3176f86eeec"}
Oct 30 19:50:02.954132 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:50:02.954102    4892 scope.go:122] "RemoveContainer" containerID="191522a582678a8a642c14a3c468d7a6d308a6de40f6275658f2e3176f86eeec"
Oct 30 19:50:02.956227 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:50:02.956193    4892 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892000Ki","pods":"250"}
Oct 30 19:50:02.956553 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:50:02.956379    4892 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892000Ki","pods":"250"}
Oct 30 19:50:02.956691 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:50:02.956676    4892 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892000Ki","pods":"250"}
Oct 30 19:50:02.974495 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:50:02.974470    4892 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-86e4a7-85cc57c56c-vhlcj"]
Oct 30 19:50:03.048911 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:50:03.029415    4892 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-api-a-86e4a7\""
Oct 30 19:50:03.048911 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:50:03.029604    4892 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-api-a-86e4a7-dockercfg-cmtzb\""
Oct 30 19:50:03.098790 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:50:03.076199    4892 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8e13a394-77f5-46a4-b58a-7f00a1483e9e-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-a-86e4a7-85cc57c56c-vhlcj\" (UID: \"8e13a394-77f5-46a4-b58a-7f00a1483e9e\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-86e4a7-85cc57c56c-vhlcj"
Oct 30 19:50:03.098790 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:50:03.076454    4892 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jnwb8\" (UniqueName: \"kubernetes.io/projected/8e13a394-77f5-46a4-b58a-7f00a1483e9e-kube-api-access-jnwb8\") pod \"payload-pre-processing-e2e-api-a-86e4a7-85cc57c56c-vhlcj\" (UID: \"8e13a394-77f5-46a4-b58a-7f00a1483e9e\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-86e4a7-85cc57c56c-vhlcj"
Oct 30 19:50:03.185179 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:50:03.125668    4892 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-86e4a7-5968698996-xkbd8"]
Oct 30 19:50:03.185179 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:50:03.145342    4892 memory_manager.go:365] "RemoveStaleState removing state" podUID="9c98da0e-489b-4333-b450-704beaf2f325" containerName="istio-proxy"
Oct 30 19:50:03.185179 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:50:03.163413    4892 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-a-86e4a7-5968698996-xkbd8"
Oct 30 19:50:03.236818 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:50:03.186522    4892 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8e13a394-77f5-46a4-b58a-7f00a1483e9e-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-a-86e4a7-85cc57c56c-vhlcj\" (UID: \"8e13a394-77f5-46a4-b58a-7f00a1483e9e\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-86e4a7-85cc57c56c-vhlcj"
Oct 30 19:50:03.236818 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:50:03.186757    4892 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jnwb8\" (UniqueName: \"kubernetes.io/projected/8e13a394-77f5-46a4-b58a-7f00a1483e9e-kube-api-access-jnwb8\") pod \"payload-pre-processing-e2e-api-a-86e4a7-85cc57c56c-vhlcj\" (UID: \"8e13a394-77f5-46a4-b58a-7f00a1483e9e\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-86e4a7-85cc57c56c-vhlcj"
Oct 30 19:50:03.236818 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:50:03.188605    4892 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8e13a394-77f5-46a4-b58a-7f00a1483e9e-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-a-86e4a7-85cc57c56c-vhlcj\" (UID: \"8e13a394-77f5-46a4-b58a-7f00a1483e9e\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-86e4a7-85cc57c56c-vhlcj"
Oct 30 19:50:03.341205 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:50:03.333579    4892 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/61795905-fd72-424f-bf9d-07f97f3f252c-plugins-config-volume\") pod \"payload-processing-e2e-api-a-86e4a7-5968698996-xkbd8\" (UID: \"61795905-fd72-424f-bf9d-07f97f3f252c\") " pod="openshift-ingress/payload-processing-e2e-api-a-86e4a7-5968698996-xkbd8"
Oct 30 19:50:03.341205 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:50:03.333660    4892 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-p8mtx\" (UniqueName: \"kubernetes.io/projected/61795905-fd72-424f-bf9d-07f97f3f252c-kube-api-access-p8mtx\") pod \"payload-processing-e2e-api-a-86e4a7-5968698996-xkbd8\" (UID: \"61795905-fd72-424f-bf9d-07f97f3f252c\") " pod="openshift-ingress/payload-processing-e2e-api-a-86e4a7-5968698996-xkbd8"
Oct 30 19:50:03.341205 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:50:03.335131    4892 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-86e4a7-5968698996-xkbd8"]
Oct 30 19:50:03.341205 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:50:03.337223    4892 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jnwb8\" (UniqueName: \"kubernetes.io/projected/8e13a394-77f5-46a4-b58a-7f00a1483e9e-kube-api-access-jnwb8\") pod \"payload-pre-processing-e2e-api-a-86e4a7-85cc57c56c-vhlcj\" (UID: \"8e13a394-77f5-46a4-b58a-7f00a1483e9e\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-86e4a7-85cc57c56c-vhlcj"
Oct 30 19:50:03.349521 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:50:03.348593    4892 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-df789477-gkkmz"]
Oct 30 19:50:03.357267 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:50:03.357133    4892 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-df789477-gkkmz"]
Oct 30 19:50:03.442070 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:50:03.441300    4892 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/61795905-fd72-424f-bf9d-07f97f3f252c-plugins-config-volume\") pod \"payload-processing-e2e-api-a-86e4a7-5968698996-xkbd8\" (UID: \"61795905-fd72-424f-bf9d-07f97f3f252c\") " pod="openshift-ingress/payload-processing-e2e-api-a-86e4a7-5968698996-xkbd8"
Oct 30 19:50:03.442070 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:50:03.441374    4892 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-p8mtx\" (UniqueName: \"kubernetes.io/projected/61795905-fd72-424f-bf9d-07f97f3f252c-kube-api-access-p8mtx\") pod \"payload-processing-e2e-api-a-86e4a7-5968698996-xkbd8\" (UID: \"61795905-fd72-424f-bf9d-07f97f3f252c\") " pod="openshift-ingress/payload-processing-e2e-api-a-86e4a7-5968698996-xkbd8"
Oct 30 19:50:03.442561 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:50:03.442538    4892 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/61795905-fd72-424f-bf9d-07f97f3f252c-plugins-config-volume\") pod \"payload-processing-e2e-api-a-86e4a7-5968698996-xkbd8\" (UID: \"61795905-fd72-424f-bf9d-07f97f3f252c\") " pod="openshift-ingress/payload-processing-e2e-api-a-86e4a7-5968698996-xkbd8"
Oct 30 19:50:03.497825 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:50:03.497792    4892 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-p8mtx\" (UniqueName: \"kubernetes.io/projected/61795905-fd72-424f-bf9d-07f97f3f252c-kube-api-access-p8mtx\") pod \"payload-processing-e2e-api-a-86e4a7-5968698996-xkbd8\" (UID: \"61795905-fd72-424f-bf9d-07f97f3f252c\") " pod="openshift-ingress/payload-processing-e2e-api-a-86e4a7-5968698996-xkbd8"
Oct 30 19:50:03.569137 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:50:03.569097    4892 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-a-86e4a7-85cc57c56c-vhlcj"
Oct 30 19:50:03.645322 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:50:03.645295    4892 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-a-86e4a7-5968698996-xkbd8"
Oct 30 19:50:03.965064 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:50:03.965036    4892 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-86e4a7-85cc57c56c-vhlcj"]
Oct 30 19:50:03.966151 ip-10-0-132-82 kubenswrapper[4892]: W1030 19:50:03.966127    4892 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8e13a394_77f5_46a4_b58a_7f00a1483e9e.slice/crio-f46ae4e478f8fbfc7f7061f9d3c519fc51e23ee611cfd82043fae35c797b208a WatchSource:0}: Error finding container f46ae4e478f8fbfc7f7061f9d3c519fc51e23ee611cfd82043fae35c797b208a: Status 404 returned error can't find the container with id f46ae4e478f8fbfc7f7061f9d3c519fc51e23ee611cfd82043fae35c797b208a
Oct 30 19:50:03.969707 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:50:03.969648    4892 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-5bf13ff3-openshift-default-7558bbbc65-9j7w6" event={"ID":"474b0b88-a062-4ee2-9994-022ae1253f28","Type":"ContainerStarted","Data":"7a0c5c0b438a6855ad7cf45b76d06609bb2d8d93068cde2d1f3c4db2227e00c6"}
Oct 30 19:50:04.083118 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:50:04.083087    4892 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-86e4a7-5968698996-xkbd8"]
Oct 30 19:50:04.085185 ip-10-0-132-82 kubenswrapper[4892]: W1030 19:50:04.085145    4892 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod61795905_fd72_424f_bf9d_07f97f3f252c.slice/crio-0c594382b7c57fb1256adc5a4b32ab34bd49eeabfa8bcf88d707ae29df9c8ba6 WatchSource:0}: Error finding container 0c594382b7c57fb1256adc5a4b32ab34bd49eeabfa8bcf88d707ae29df9c8ba6: Status 404 returned error can't find the container with id 0c594382b7c57fb1256adc5a4b32ab34bd49eeabfa8bcf88d707ae29df9c8ba6
Oct 30 19:50:04.322342 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:50:04.322248    4892 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-mt-5bf13ff3-openshift-default-7558bbbc65-9j7w6"
Oct 30 19:50:04.322521 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:50:04.322498    4892 patch_prober.go:29] interesting pod/e2e-mt-5bf13ff3-openshift-default-7558bbbc65-9j7w6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.171:15021/healthz/ready\": dial tcp 10.132.0.171:15021: connect: connection refused" start-of-body=
Oct 30 19:50:04.322578 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:50:04.322545    4892 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-5bf13ff3-openshift-default-7558bbbc65-9j7w6" podUID="474b0b88-a062-4ee2-9994-022ae1253f28" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.171:15021/healthz/ready\": dial tcp 10.132.0.171:15021: connect: connection refused"
Oct 30 19:50:04.573237 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:50:04.573157    4892 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="1419be25-72fa-4db5-a92b-44972189a2cb" path="/var/lib/kubelet/pods/1419be25-72fa-4db5-a92b-44972189a2cb/volumes"
Oct 30 19:50:04.986992 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:50:04.981142    4892 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-a-86e4a7-5968698996-xkbd8" event={"ID":"61795905-fd72-424f-bf9d-07f97f3f252c","Type":"ContainerStarted","Data":"6121afbd6daf498d128007ce3e4415f1bbca4bdffc28081b8cc8df57a4c08f8b"}
Oct 30 19:50:04.986992 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:50:04.981226    4892 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-a-86e4a7-5968698996-xkbd8" event={"ID":"61795905-fd72-424f-bf9d-07f97f3f252c","Type":"ContainerStarted","Data":"0c594382b7c57fb1256adc5a4b32ab34bd49eeabfa8bcf88d707ae29df9c8ba6"}
Oct 30 19:50:04.986992 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:50:04.981481    4892 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-api-a-86e4a7-5968698996-xkbd8"
Oct 30 19:50:04.996005 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:50:04.993105    4892 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-a-86e4a7-85cc57c56c-vhlcj" event={"ID":"8e13a394-77f5-46a4-b58a-7f00a1483e9e","Type":"ContainerStarted","Data":"16c42645a660c63f8bfe07abde4a06668c19107b4fdd20daa431eb56c141b549"}
Oct 30 19:50:04.996005 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:50:04.993171    4892 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-api-a-86e4a7-85cc57c56c-vhlcj"
Oct 30 19:50:04.996005 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:50:04.993193    4892 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-a-86e4a7-85cc57c56c-vhlcj" event={"ID":"8e13a394-77f5-46a4-b58a-7f00a1483e9e","Type":"ContainerStarted","Data":"f46ae4e478f8fbfc7f7061f9d3c519fc51e23ee611cfd82043fae35c797b208a"}
Oct 30 19:50:05.240038 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:50:05.231749    4892 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-api-a-86e4a7-5968698996-xkbd8" podStartSLOduration=2.231729437 podStartE2EDuration="2.231729437s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:50:03 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:50:05.114272886 +0000 UTC m=+2052.684738520" watchObservedRunningTime="2026-10-30 19:50:05.231729437 +0000 UTC m=+2052.802195037"
Oct 30 19:50:05.242000 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:50:05.241780    4892 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-api-a-86e4a7-85cc57c56c-vhlcj" podStartSLOduration=3.241766347 podStartE2EDuration="3.241766347s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:50:02 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:50:05.241487881 +0000 UTC m=+2052.811953481" watchObservedRunningTime="2026-10-30 19:50:05.241766347 +0000 UTC m=+2052.812231947"
Oct 30 19:50:05.322991 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:50:05.322892    4892 patch_prober.go:29] interesting pod/e2e-mt-5bf13ff3-openshift-default-7558bbbc65-9j7w6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.171:15021/healthz/ready\": dial tcp 10.132.0.171:15021: connect: connection refused" start-of-body=
Oct 30 19:50:05.322991 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:50:05.322953    4892 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-5bf13ff3-openshift-default-7558bbbc65-9j7w6" podUID="474b0b88-a062-4ee2-9994-022ae1253f28" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.171:15021/healthz/ready\": dial tcp 10.132.0.171:15021: connect: connection refused"
Oct 30 19:50:06.322132 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:50:06.322097    4892 patch_prober.go:29] interesting pod/e2e-mt-5bf13ff3-openshift-default-7558bbbc65-9j7w6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.171:15021/healthz/ready\": dial tcp 10.132.0.171:15021: connect: connection refused" start-of-body=
Oct 30 19:50:06.322520 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:50:06.322150    4892 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-5bf13ff3-openshift-default-7558bbbc65-9j7w6" podUID="474b0b88-a062-4ee2-9994-022ae1253f28" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.171:15021/healthz/ready\": dial tcp 10.132.0.171:15021: connect: connection refused"
Oct 30 19:50:06.912931 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:50:06.912895    4892 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-8958-c57c0a"]
Oct 30 19:50:06.917221 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:50:06.917203    4892 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-8958-c57c0a"
Oct 30 19:50:06.920307 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:50:06.920282    4892 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wv588\" (UniqueName: \"kubernetes.io/projected/f8095838-79c0-4aa3-b670-3ee86243e0b6-kube-api-access-wv588\") pod \"test-curl-8958-c57c0a\" (UID: \"f8095838-79c0-4aa3-b670-3ee86243e0b6\") " pod="opendatahub/test-curl-8958-c57c0a"
Oct 30 19:50:06.976485 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:50:06.973278    4892 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-8958-c57c0a"]
Oct 30 19:50:07.017331 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:50:07.017298    4892 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"default-dockercfg-q2f25\""
Oct 30 19:50:07.021545 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:50:07.021514    4892 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wv588\" (UniqueName: \"kubernetes.io/projected/f8095838-79c0-4aa3-b670-3ee86243e0b6-kube-api-access-wv588\") pod \"test-curl-8958-c57c0a\" (UID: \"f8095838-79c0-4aa3-b670-3ee86243e0b6\") " pod="opendatahub/test-curl-8958-c57c0a"
Oct 30 19:50:07.079672 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:50:07.079639    4892 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wv588\" (UniqueName: \"kubernetes.io/projected/f8095838-79c0-4aa3-b670-3ee86243e0b6-kube-api-access-wv588\") pod \"test-curl-8958-c57c0a\" (UID: \"f8095838-79c0-4aa3-b670-3ee86243e0b6\") " pod="opendatahub/test-curl-8958-c57c0a"
Oct 30 19:50:07.258121 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:50:07.251516    4892 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-8958-c57c0a"
Oct 30 19:50:07.322211 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:50:07.322100    4892 patch_prober.go:29] interesting pod/e2e-mt-5bf13ff3-openshift-default-7558bbbc65-9j7w6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.171:15021/healthz/ready\": dial tcp 10.132.0.171:15021: connect: connection refused" start-of-body=
Oct 30 19:50:07.322211 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:50:07.322180    4892 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-5bf13ff3-openshift-default-7558bbbc65-9j7w6" podUID="474b0b88-a062-4ee2-9994-022ae1253f28" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.171:15021/healthz/ready\": dial tcp 10.132.0.171:15021: connect: connection refused"
Oct 30 19:50:07.654989 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:50:07.650022    4892 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-8958-c57c0a"]
Oct 30 19:50:07.664899 ip-10-0-132-82 kubenswrapper[4892]: W1030 19:50:07.664251    4892 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podf8095838_79c0_4aa3_b670_3ee86243e0b6.slice/crio-9a5d0c89e5f1a099c9f5cc21e96ebbeb27736a750317f08116e9695d6ddaa3f1 WatchSource:0}: Error finding container 9a5d0c89e5f1a099c9f5cc21e96ebbeb27736a750317f08116e9695d6ddaa3f1: Status 404 returned error can't find the container with id 9a5d0c89e5f1a099c9f5cc21e96ebbeb27736a750317f08116e9695d6ddaa3f1
Oct 30 19:50:08.087058 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:50:08.085882    4892 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-8958-c57c0a" event={"ID":"f8095838-79c0-4aa3-b670-3ee86243e0b6","Type":"ContainerStarted","Data":"597bc4245a345ce360ad71fe571edd8169240810f362c0993e712a06629e3c46"}
Oct 30 19:50:08.087058 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:50:08.085936    4892 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-8958-c57c0a" event={"ID":"f8095838-79c0-4aa3-b670-3ee86243e0b6","Type":"ContainerStarted","Data":"9a5d0c89e5f1a099c9f5cc21e96ebbeb27736a750317f08116e9695d6ddaa3f1"}
Oct 30 19:50:08.210312 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:50:08.210252    4892 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/test-curl-8958-c57c0a" podStartSLOduration=2.210238281 podStartE2EDuration="2.210238281s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:50:06 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:50:08.208237946 +0000 UTC m=+2055.778703546" watchObservedRunningTime="2026-10-30 19:50:08.210238281 +0000 UTC m=+2055.780703880"
Oct 30 19:50:08.322074 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:50:08.322036    4892 patch_prober.go:29] interesting pod/e2e-mt-5bf13ff3-openshift-default-7558bbbc65-9j7w6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.171:15021/healthz/ready\": dial tcp 10.132.0.171:15021: connect: connection refused" start-of-body=
Oct 30 19:50:08.322249 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:50:08.322091    4892 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-5bf13ff3-openshift-default-7558bbbc65-9j7w6" podUID="474b0b88-a062-4ee2-9994-022ae1253f28" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.171:15021/healthz/ready\": dial tcp 10.132.0.171:15021: connect: connection refused"
Oct 30 19:50:08.963991 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:50:08.963296    4892 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-mt-dd9b8d79/e2e-lifecycle-model-dd9b8d79-kserve-7659b4b7d9-r2flr"]
Oct 30 19:50:08.971016 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:50:08.970587    4892 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-mt-dd9b8d79/e2e-lifecycle-model-dd9b8d79-kserve-7659b4b7d9-r2flr"
Oct 30 19:50:09.003087 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:50:09.003060    4892 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-mt-dd9b8d79/e2e-lifecycle-model-dd9b8d79-kserve-7659b4b7d9-r2flr"]
Oct 30 19:50:09.058985 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:50:09.058473    4892 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-mt-dd9b8d79\"/\"openshift-service-ca.crt\""
Oct 30 19:50:09.059201 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:50:09.059112    4892 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-mt-dd9b8d79\"/\"default-dockercfg-6bbhk\""
Oct 30 19:50:09.073693 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:50:09.073520    4892 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-mt-dd9b8d79\"/\"kube-root-ca.crt\""
Oct 30 19:50:09.073693 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:50:09.073532    4892 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-mt-dd9b8d79\"/\"e2e-lifecycle-model-dd9b8d79-kserve-self-signed-certs\""
Oct 30 19:50:09.073881 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:50:09.073862    4892 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e469e8c3-e158-4b45-9135-6da275fc897f-home\") pod \"e2e-lifecycle-model-dd9b8d79-kserve-7659b4b7d9-r2flr\" (UID: \"e469e8c3-e158-4b45-9135-6da275fc897f\") " pod="ai-tenant-e2e-mt-dd9b8d79/e2e-lifecycle-model-dd9b8d79-kserve-7659b4b7d9-r2flr"
Oct 30 19:50:09.073929 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:50:09.073895    4892 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e469e8c3-e158-4b45-9135-6da275fc897f-tls-certs\") pod \"e2e-lifecycle-model-dd9b8d79-kserve-7659b4b7d9-r2flr\" (UID: \"e469e8c3-e158-4b45-9135-6da275fc897f\") " pod="ai-tenant-e2e-mt-dd9b8d79/e2e-lifecycle-model-dd9b8d79-kserve-7659b4b7d9-r2flr"
Oct 30 19:50:09.074031 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:50:09.074016    4892 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e469e8c3-e158-4b45-9135-6da275fc897f-model-cache\") pod \"e2e-lifecycle-model-dd9b8d79-kserve-7659b4b7d9-r2flr\" (UID: \"e469e8c3-e158-4b45-9135-6da275fc897f\") " pod="ai-tenant-e2e-mt-dd9b8d79/e2e-lifecycle-model-dd9b8d79-kserve-7659b4b7d9-r2flr"
Oct 30 19:50:09.074219 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:50:09.074203    4892 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e469e8c3-e158-4b45-9135-6da275fc897f-tmp-dir\") pod \"e2e-lifecycle-model-dd9b8d79-kserve-7659b4b7d9-r2flr\" (UID: \"e469e8c3-e158-4b45-9135-6da275fc897f\") " pod="ai-tenant-e2e-mt-dd9b8d79/e2e-lifecycle-model-dd9b8d79-kserve-7659b4b7d9-r2flr"
Oct 30 19:50:09.074297 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:50:09.074272    4892 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e469e8c3-e158-4b45-9135-6da275fc897f-dshm\") pod \"e2e-lifecycle-model-dd9b8d79-kserve-7659b4b7d9-r2flr\" (UID: \"e469e8c3-e158-4b45-9135-6da275fc897f\") " pod="ai-tenant-e2e-mt-dd9b8d79/e2e-lifecycle-model-dd9b8d79-kserve-7659b4b7d9-r2flr"
Oct 30 19:50:09.074334 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:50:09.074294    4892 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-m4ps7\" (UniqueName: \"kubernetes.io/projected/e469e8c3-e158-4b45-9135-6da275fc897f-kube-api-access-m4ps7\") pod \"e2e-lifecycle-model-dd9b8d79-kserve-7659b4b7d9-r2flr\" (UID: \"e469e8c3-e158-4b45-9135-6da275fc897f\") " pod="ai-tenant-e2e-mt-dd9b8d79/e2e-lifecycle-model-dd9b8d79-kserve-7659b4b7d9-r2flr"
Oct 30 19:50:09.205998 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:50:09.195591    4892 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e469e8c3-e158-4b45-9135-6da275fc897f-tmp-dir\") pod \"e2e-lifecycle-model-dd9b8d79-kserve-7659b4b7d9-r2flr\" (UID: \"e469e8c3-e158-4b45-9135-6da275fc897f\") " pod="ai-tenant-e2e-mt-dd9b8d79/e2e-lifecycle-model-dd9b8d79-kserve-7659b4b7d9-r2flr"
Oct 30 19:50:09.205998 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:50:09.201170    4892 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e469e8c3-e158-4b45-9135-6da275fc897f-dshm\") pod \"e2e-lifecycle-model-dd9b8d79-kserve-7659b4b7d9-r2flr\" (UID: \"e469e8c3-e158-4b45-9135-6da275fc897f\") " pod="ai-tenant-e2e-mt-dd9b8d79/e2e-lifecycle-model-dd9b8d79-kserve-7659b4b7d9-r2flr"
Oct 30 19:50:09.205998 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:50:09.201218    4892 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-m4ps7\" (UniqueName: \"kubernetes.io/projected/e469e8c3-e158-4b45-9135-6da275fc897f-kube-api-access-m4ps7\") pod \"e2e-lifecycle-model-dd9b8d79-kserve-7659b4b7d9-r2flr\" (UID: \"e469e8c3-e158-4b45-9135-6da275fc897f\") " pod="ai-tenant-e2e-mt-dd9b8d79/e2e-lifecycle-model-dd9b8d79-kserve-7659b4b7d9-r2flr"
Oct 30 19:50:09.205998 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:50:09.201321    4892 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e469e8c3-e158-4b45-9135-6da275fc897f-home\") pod \"e2e-lifecycle-model-dd9b8d79-kserve-7659b4b7d9-r2flr\" (UID: \"e469e8c3-e158-4b45-9135-6da275fc897f\") " pod="ai-tenant-e2e-mt-dd9b8d79/e2e-lifecycle-model-dd9b8d79-kserve-7659b4b7d9-r2flr"
Oct 30 19:50:09.205998 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:50:09.201349    4892 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e469e8c3-e158-4b45-9135-6da275fc897f-tls-certs\") pod \"e2e-lifecycle-model-dd9b8d79-kserve-7659b4b7d9-r2flr\" (UID: \"e469e8c3-e158-4b45-9135-6da275fc897f\") " pod="ai-tenant-e2e-mt-dd9b8d79/e2e-lifecycle-model-dd9b8d79-kserve-7659b4b7d9-r2flr"
Oct 30 19:50:09.205998 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:50:09.201406    4892 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e469e8c3-e158-4b45-9135-6da275fc897f-model-cache\") pod \"e2e-lifecycle-model-dd9b8d79-kserve-7659b4b7d9-r2flr\" (UID: \"e469e8c3-e158-4b45-9135-6da275fc897f\") " pod="ai-tenant-e2e-mt-dd9b8d79/e2e-lifecycle-model-dd9b8d79-kserve-7659b4b7d9-r2flr"
Oct 30 19:50:09.205998 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:50:09.201953    4892 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e469e8c3-e158-4b45-9135-6da275fc897f-model-cache\") pod \"e2e-lifecycle-model-dd9b8d79-kserve-7659b4b7d9-r2flr\" (UID: \"e469e8c3-e158-4b45-9135-6da275fc897f\") " pod="ai-tenant-e2e-mt-dd9b8d79/e2e-lifecycle-model-dd9b8d79-kserve-7659b4b7d9-r2flr"
Oct 30 19:50:09.205998 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:50:09.196089    4892 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e469e8c3-e158-4b45-9135-6da275fc897f-tmp-dir\") pod \"e2e-lifecycle-model-dd9b8d79-kserve-7659b4b7d9-r2flr\" (UID: \"e469e8c3-e158-4b45-9135-6da275fc897f\") " pod="ai-tenant-e2e-mt-dd9b8d79/e2e-lifecycle-model-dd9b8d79-kserve-7659b4b7d9-r2flr"
Oct 30 19:50:09.206545 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:50:09.206075    4892 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e469e8c3-e158-4b45-9135-6da275fc897f-home\") pod \"e2e-lifecycle-model-dd9b8d79-kserve-7659b4b7d9-r2flr\" (UID: \"e469e8c3-e158-4b45-9135-6da275fc897f\") " pod="ai-tenant-e2e-mt-dd9b8d79/e2e-lifecycle-model-dd9b8d79-kserve-7659b4b7d9-r2flr"
Oct 30 19:50:09.215999 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:50:09.215526    4892 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e469e8c3-e158-4b45-9135-6da275fc897f-dshm\") pod \"e2e-lifecycle-model-dd9b8d79-kserve-7659b4b7d9-r2flr\" (UID: \"e469e8c3-e158-4b45-9135-6da275fc897f\") " pod="ai-tenant-e2e-mt-dd9b8d79/e2e-lifecycle-model-dd9b8d79-kserve-7659b4b7d9-r2flr"
Oct 30 19:50:09.217991 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:50:09.217935    4892 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e469e8c3-e158-4b45-9135-6da275fc897f-tls-certs\") pod \"e2e-lifecycle-model-dd9b8d79-kserve-7659b4b7d9-r2flr\" (UID: \"e469e8c3-e158-4b45-9135-6da275fc897f\") " pod="ai-tenant-e2e-mt-dd9b8d79/e2e-lifecycle-model-dd9b8d79-kserve-7659b4b7d9-r2flr"
Oct 30 19:50:09.273039 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:50:09.273003    4892 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-m4ps7\" (UniqueName: \"kubernetes.io/projected/e469e8c3-e158-4b45-9135-6da275fc897f-kube-api-access-m4ps7\") pod \"e2e-lifecycle-model-dd9b8d79-kserve-7659b4b7d9-r2flr\" (UID: \"e469e8c3-e158-4b45-9135-6da275fc897f\") " pod="ai-tenant-e2e-mt-dd9b8d79/e2e-lifecycle-model-dd9b8d79-kserve-7659b4b7d9-r2flr"
Oct 30 19:50:09.284746 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:50:09.284715    4892 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-mt-dd9b8d79/e2e-lifecycle-model-dd9b8d79-kserve-7659b4b7d9-r2flr"
Oct 30 19:50:09.397991 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:50:09.383106    4892 patch_prober.go:29] interesting pod/e2e-mt-5bf13ff3-openshift-default-7558bbbc65-9j7w6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.171:15021/healthz/ready\": dial tcp 10.132.0.171:15021: connect: connection refused" start-of-body=
Oct 30 19:50:09.397991 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:50:09.383190    4892 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-5bf13ff3-openshift-default-7558bbbc65-9j7w6" podUID="474b0b88-a062-4ee2-9994-022ae1253f28" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.171:15021/healthz/ready\": dial tcp 10.132.0.171:15021: connect: connection refused"
Oct 30 19:50:09.872794 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:50:09.872711    4892 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-mt-dd9b8d79/e2e-lifecycle-model-dd9b8d79-kserve-7659b4b7d9-r2flr"]
Oct 30 19:50:09.877542 ip-10-0-132-82 kubenswrapper[4892]: W1030 19:50:09.877513    4892 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pode469e8c3_e158_4b45_9135_6da275fc897f.slice/crio-ea8d1e5a49bc366dde5a9a98f2088dcaad9c4510e7ba6ec89fe6da1946901a35 WatchSource:0}: Error finding container ea8d1e5a49bc366dde5a9a98f2088dcaad9c4510e7ba6ec89fe6da1946901a35: Status 404 returned error can't find the container with id ea8d1e5a49bc366dde5a9a98f2088dcaad9c4510e7ba6ec89fe6da1946901a35
Oct 30 19:50:09.888376 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:50:09.888336    4892 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-8958-c57c0a"]
Oct 30 19:50:09.893727 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:50:09.893705    4892 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-8958-c57c0a"]
Oct 30 19:50:10.189287 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:50:10.189250    4892 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-mt-dd9b8d79/e2e-lifecycle-model-dd9b8d79-kserve-7659b4b7d9-r2flr" event={"ID":"e469e8c3-e158-4b45-9135-6da275fc897f","Type":"ContainerStarted","Data":"a8265c669ccf9c1b2da8388e1843b62adc4cb67ef19134165d5034efe5354ee0"}
Oct 30 19:50:10.189482 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:50:10.189297    4892 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-mt-dd9b8d79/e2e-lifecycle-model-dd9b8d79-kserve-7659b4b7d9-r2flr" event={"ID":"e469e8c3-e158-4b45-9135-6da275fc897f","Type":"ContainerStarted","Data":"ea8d1e5a49bc366dde5a9a98f2088dcaad9c4510e7ba6ec89fe6da1946901a35"}
Oct 30 19:50:10.189532 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:50:10.189490    4892 kuberuntime_container.go:898] "Killing container with a grace period" pod="opendatahub/test-curl-8958-c57c0a" podUID="f8095838-79c0-4aa3-b670-3ee86243e0b6" containerName="test-curl-8958-c57c0a" containerID="cri-o://597bc4245a345ce360ad71fe571edd8169240810f362c0993e712a06629e3c46" gracePeriod=2
Oct 30 19:50:10.285369 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:50:10.285326    4892 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-mt-dd9b8d79/e2e-lifecycle-model-dd9b8d79-kserve-7659b4b7d9-r2flr"
Oct 30 19:50:10.303764 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:50:10.303712    4892 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-mt-dd9b8d79/e2e-lifecycle-model-dd9b8d79-kserve-7659b4b7d9-r2flr" podStartSLOduration=2.303689031 podStartE2EDuration="2.303689031s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:50:08 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:50:10.300882994 +0000 UTC m=+2057.871348594" watchObservedRunningTime="2026-10-30 19:50:10.303689031 +0000 UTC m=+2057.874154676"
Oct 30 19:50:10.326518 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:50:10.326497    4892 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-mt-5bf13ff3-openshift-default-7558bbbc65-9j7w6"
Oct 30 19:50:10.326872 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:50:10.326844    4892 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-mt-5bf13ff3-openshift-default-7558bbbc65-9j7w6"
Oct 30 19:50:10.327662 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:50:10.327641    4892 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-mt-5bf13ff3-openshift-default-7558bbbc65-9j7w6"
Oct 30 19:50:10.351226 ip-10-0-132-82 kubenswrapper[4892]: E1030 19:50:10.351188    4892 status_manager.go:1045] "Failed to get status for pod" err="pods \"test-curl-8958-c57c0a\" is forbidden: User \"system:node:ip-10-0-132-82.us-west-1.compute.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-132-82.us-west-1.compute.internal' and this object" podUID="f8095838-79c0-4aa3-b670-3ee86243e0b6" pod="opendatahub/test-curl-8958-c57c0a"
Oct 30 19:50:10.516157 ip-10-0-132-82 kubenswrapper[4892]: E1030 19:50:10.516067    4892 status_manager.go:1045] "Failed to get status for pod" err="pods \"test-curl-8958-c57c0a\" is forbidden: User \"system:node:ip-10-0-132-82.us-west-1.compute.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-132-82.us-west-1.compute.internal' and this object" podUID="f8095838-79c0-4aa3-b670-3ee86243e0b6" pod="opendatahub/test-curl-8958-c57c0a"
Oct 30 19:50:12.459034 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:50:12.459010    4892 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-8958-c57c0a"
Oct 30 19:50:12.554238 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:50:12.554206    4892 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f8095838-79c0-4aa3-b670-3ee86243e0b6-kube-api-access-wv588\" (UniqueName: \"kubernetes.io/projected/f8095838-79c0-4aa3-b670-3ee86243e0b6-kube-api-access-wv588\") pod \"f8095838-79c0-4aa3-b670-3ee86243e0b6\" (UID: \"f8095838-79c0-4aa3-b670-3ee86243e0b6\") "
Oct 30 19:50:12.556566 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:50:12.556523    4892 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f8095838-79c0-4aa3-b670-3ee86243e0b6-kube-api-access-wv588" pod "f8095838-79c0-4aa3-b670-3ee86243e0b6" (UID: "f8095838-79c0-4aa3-b670-3ee86243e0b6"). InnerVolumeSpecName "kube-api-access-wv588". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:50:12.573269 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:50:12.573244    4892 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f8095838-79c0-4aa3-b670-3ee86243e0b6" path="/var/lib/kubelet/pods/f8095838-79c0-4aa3-b670-3ee86243e0b6/volumes"
Oct 30 19:50:12.656280 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:50:12.656253    4892 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wv588\" (UniqueName: \"kubernetes.io/projected/f8095838-79c0-4aa3-b670-3ee86243e0b6-kube-api-access-wv588\") on node \"ip-10-0-132-82.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 19:50:13.213828 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:50:13.213790    4892 generic.go:356] "Generic (PLEG): container finished" podID="f8095838-79c0-4aa3-b670-3ee86243e0b6" containerID="597bc4245a345ce360ad71fe571edd8169240810f362c0993e712a06629e3c46" exitCode=137
Oct 30 19:50:13.213828 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:50:13.213828    4892 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-8958-c57c0a"
Oct 30 19:50:13.214099 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:50:13.213883    4892 scope.go:122] "RemoveContainer" containerID="597bc4245a345ce360ad71fe571edd8169240810f362c0993e712a06629e3c46"
Oct 30 19:50:13.224134 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:50:13.224114    4892 scope.go:122] "RemoveContainer" containerID="597bc4245a345ce360ad71fe571edd8169240810f362c0993e712a06629e3c46"
Oct 30 19:50:13.224435 ip-10-0-132-82 kubenswrapper[4892]: E1030 19:50:13.224411    4892 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"597bc4245a345ce360ad71fe571edd8169240810f362c0993e712a06629e3c46\": container with ID starting with 597bc4245a345ce360ad71fe571edd8169240810f362c0993e712a06629e3c46 not found: ID does not exist" containerID="597bc4245a345ce360ad71fe571edd8169240810f362c0993e712a06629e3c46"
Oct 30 19:50:13.224500 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:50:13.224441    4892 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"597bc4245a345ce360ad71fe571edd8169240810f362c0993e712a06629e3c46"} err="failed to get container status \"597bc4245a345ce360ad71fe571edd8169240810f362c0993e712a06629e3c46\": rpc error: code = NotFound desc = could not find container \"597bc4245a345ce360ad71fe571edd8169240810f362c0993e712a06629e3c46\": container with ID starting with 597bc4245a345ce360ad71fe571edd8169240810f362c0993e712a06629e3c46 not found: ID does not exist"
Oct 30 19:50:15.970035 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:50:15.965595    4892 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-cb14ed-c8b7565cf-h6zcq"]
Oct 30 19:50:15.970035 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:50:15.966887    4892 cpu_manager.go:414] "containerMap: removing container" podUID="f8095838-79c0-4aa3-b670-3ee86243e0b6" containerName="test-curl-8958-c57c0a"
Oct 30 19:50:15.970035 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:50:15.966909    4892 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f8095838-79c0-4aa3-b670-3ee86243e0b6" containerName="test-curl-8958-c57c0a"
Oct 30 19:50:15.970035 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:50:15.967391    4892 memory_manager.go:365] "RemoveStaleState removing state" podUID="f8095838-79c0-4aa3-b670-3ee86243e0b6" containerName="test-curl-8958-c57c0a"
Oct 30 19:50:16.010636 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:50:16.010310    4892 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-a-cb14ed-c8b7565cf-h6zcq"
Oct 30 19:50:16.019431 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:50:16.018906    4892 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-api-a-86e4a7-85cc57c56c-vhlcj"
Oct 30 19:50:16.022348 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:50:16.020242    4892 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-api-a-86e4a7-5968698996-xkbd8"
Oct 30 19:50:16.025454 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:50:16.025430    4892 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-cb14ed-c8b7565cf-h6zcq"]
Oct 30 19:50:16.146800 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:50:16.146614    4892 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-api-a-cb14ed\""
Oct 30 19:50:16.281323 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:50:16.244741    4892 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-api-a-cb14ed-dockercfg-9bgz7\""
Oct 30 19:50:16.281591 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:50:16.267165    4892 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-cb14ed-6f8d8d5f84-6pqqb"]
Oct 30 19:50:16.311056 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:50:16.292149    4892 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6e76e92e-5a95-4a67-99a0-6907a8f22c71-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-a-cb14ed-c8b7565cf-h6zcq\" (UID: \"6e76e92e-5a95-4a67-99a0-6907a8f22c71\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-cb14ed-c8b7565cf-h6zcq"
Oct 30 19:50:16.396608 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:50:16.322556    4892 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-a-cb14ed-6f8d8d5f84-6pqqb"
Oct 30 19:50:16.396919 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:50:16.396886    4892 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5z69q\" (UniqueName: \"kubernetes.io/projected/6e76e92e-5a95-4a67-99a0-6907a8f22c71-kube-api-access-5z69q\") pod \"payload-pre-processing-e2e-api-a-cb14ed-c8b7565cf-h6zcq\" (UID: \"6e76e92e-5a95-4a67-99a0-6907a8f22c71\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-cb14ed-c8b7565cf-h6zcq"
Oct 30 19:50:16.405743 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:50:16.405713    4892 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-cb14ed-6f8d8d5f84-6pqqb"]
Oct 30 19:50:16.498368 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:50:16.498334    4892 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0a20e1c6-2c3f-4c1b-ac66-9fb5dfe44bfb-plugins-config-volume\") pod \"payload-processing-e2e-api-a-cb14ed-6f8d8d5f84-6pqqb\" (UID: \"0a20e1c6-2c3f-4c1b-ac66-9fb5dfe44bfb\") " pod="openshift-ingress/payload-processing-e2e-api-a-cb14ed-6f8d8d5f84-6pqqb"
Oct 30 19:50:16.498542 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:50:16.498398    4892 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5z69q\" (UniqueName: \"kubernetes.io/projected/6e76e92e-5a95-4a67-99a0-6907a8f22c71-kube-api-access-5z69q\") pod \"payload-pre-processing-e2e-api-a-cb14ed-c8b7565cf-h6zcq\" (UID: \"6e76e92e-5a95-4a67-99a0-6907a8f22c71\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-cb14ed-c8b7565cf-h6zcq"
Oct 30 19:50:16.498675 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:50:16.498643    4892 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-44svl\" (UniqueName: \"kubernetes.io/projected/0a20e1c6-2c3f-4c1b-ac66-9fb5dfe44bfb-kube-api-access-44svl\") pod \"payload-processing-e2e-api-a-cb14ed-6f8d8d5f84-6pqqb\" (UID: \"0a20e1c6-2c3f-4c1b-ac66-9fb5dfe44bfb\") " pod="openshift-ingress/payload-processing-e2e-api-a-cb14ed-6f8d8d5f84-6pqqb"
Oct 30 19:50:16.498803 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:50:16.498784    4892 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6e76e92e-5a95-4a67-99a0-6907a8f22c71-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-a-cb14ed-c8b7565cf-h6zcq\" (UID: \"6e76e92e-5a95-4a67-99a0-6907a8f22c71\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-cb14ed-c8b7565cf-h6zcq"
Oct 30 19:50:16.499504 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:50:16.499479    4892 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6e76e92e-5a95-4a67-99a0-6907a8f22c71-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-a-cb14ed-c8b7565cf-h6zcq\" (UID: \"6e76e92e-5a95-4a67-99a0-6907a8f22c71\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-cb14ed-c8b7565cf-h6zcq"
Oct 30 19:50:16.582365 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:50:16.580874    4892 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5z69q\" (UniqueName: \"kubernetes.io/projected/6e76e92e-5a95-4a67-99a0-6907a8f22c71-kube-api-access-5z69q\") pod \"payload-pre-processing-e2e-api-a-cb14ed-c8b7565cf-h6zcq\" (UID: \"6e76e92e-5a95-4a67-99a0-6907a8f22c71\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-cb14ed-c8b7565cf-h6zcq"
Oct 30 19:50:16.605990 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:50:16.605463    4892 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0a20e1c6-2c3f-4c1b-ac66-9fb5dfe44bfb-plugins-config-volume\") pod \"payload-processing-e2e-api-a-cb14ed-6f8d8d5f84-6pqqb\" (UID: \"0a20e1c6-2c3f-4c1b-ac66-9fb5dfe44bfb\") " pod="openshift-ingress/payload-processing-e2e-api-a-cb14ed-6f8d8d5f84-6pqqb"
Oct 30 19:50:16.605990 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:50:16.605804    4892 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-44svl\" (UniqueName: \"kubernetes.io/projected/0a20e1c6-2c3f-4c1b-ac66-9fb5dfe44bfb-kube-api-access-44svl\") pod \"payload-processing-e2e-api-a-cb14ed-6f8d8d5f84-6pqqb\" (UID: \"0a20e1c6-2c3f-4c1b-ac66-9fb5dfe44bfb\") " pod="openshift-ingress/payload-processing-e2e-api-a-cb14ed-6f8d8d5f84-6pqqb"
Oct 30 19:50:16.608035 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:50:16.607385    4892 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0a20e1c6-2c3f-4c1b-ac66-9fb5dfe44bfb-plugins-config-volume\") pod \"payload-processing-e2e-api-a-cb14ed-6f8d8d5f84-6pqqb\" (UID: \"0a20e1c6-2c3f-4c1b-ac66-9fb5dfe44bfb\") " pod="openshift-ingress/payload-processing-e2e-api-a-cb14ed-6f8d8d5f84-6pqqb"
Oct 30 19:50:16.628811 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:50:16.628785    4892 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-a-cb14ed-c8b7565cf-h6zcq"
Oct 30 19:50:16.665762 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:50:16.665737    4892 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-44svl\" (UniqueName: \"kubernetes.io/projected/0a20e1c6-2c3f-4c1b-ac66-9fb5dfe44bfb-kube-api-access-44svl\") pod \"payload-processing-e2e-api-a-cb14ed-6f8d8d5f84-6pqqb\" (UID: \"0a20e1c6-2c3f-4c1b-ac66-9fb5dfe44bfb\") " pod="openshift-ingress/payload-processing-e2e-api-a-cb14ed-6f8d8d5f84-6pqqb"
Oct 30 19:50:16.759992 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:50:16.757313    4892 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-a-cb14ed-6f8d8d5f84-6pqqb"
Oct 30 19:50:17.415773 ip-10-0-132-82 kubenswrapper[4892]: W1030 19:50:17.415743    4892 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6e76e92e_5a95_4a67_99a0_6907a8f22c71.slice/crio-6780f905bb9d78366d0bef0670d986d2208ca1c29a0ca76605cf93d48334602e WatchSource:0}: Error finding container 6780f905bb9d78366d0bef0670d986d2208ca1c29a0ca76605cf93d48334602e: Status 404 returned error can't find the container with id 6780f905bb9d78366d0bef0670d986d2208ca1c29a0ca76605cf93d48334602e
Oct 30 19:50:17.416593 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:50:17.416567    4892 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-cb14ed-c8b7565cf-h6zcq"]
Oct 30 19:50:17.515403 ip-10-0-132-82 kubenswrapper[4892]: W1030 19:50:17.515373    4892 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0a20e1c6_2c3f_4c1b_ac66_9fb5dfe44bfb.slice/crio-f9ba13e179f83e78e2ef3f48ddabda5aa6777c57a427c57c511e265c82727196 WatchSource:0}: Error finding container f9ba13e179f83e78e2ef3f48ddabda5aa6777c57a427c57c511e265c82727196: Status 404 returned error can't find the container with id f9ba13e179f83e78e2ef3f48ddabda5aa6777c57a427c57c511e265c82727196
Oct 30 19:50:17.516170 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:50:17.516134    4892 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-cb14ed-6f8d8d5f84-6pqqb"]
Oct 30 19:50:18.381991 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:50:18.379950    4892 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-fsfvk"]
Oct 30 19:50:18.463040 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:50:18.460607    4892 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-a-cb14ed-c8b7565cf-h6zcq" event={"ID":"6e76e92e-5a95-4a67-99a0-6907a8f22c71","Type":"ContainerStarted","Data":"f178ca9b51110b45933a71367152d6a75569134dcf93da7742f47b01e31f48f7"}
Oct 30 19:50:18.463040 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:50:18.460673    4892 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-a-cb14ed-c8b7565cf-h6zcq" event={"ID":"6e76e92e-5a95-4a67-99a0-6907a8f22c71","Type":"ContainerStarted","Data":"6780f905bb9d78366d0bef0670d986d2208ca1c29a0ca76605cf93d48334602e"}
Oct 30 19:50:18.463040 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:50:18.461396    4892 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-api-a-cb14ed-c8b7565cf-h6zcq"
Oct 30 19:50:18.463040 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:50:18.462921    4892 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-a-cb14ed-6f8d8d5f84-6pqqb" event={"ID":"0a20e1c6-2c3f-4c1b-ac66-9fb5dfe44bfb","Type":"ContainerStarted","Data":"ced8287e7025b8a937c88a4ea862cd7804e46bc986d06f4ce6f594219447b426"}
Oct 30 19:50:18.463040 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:50:18.462985    4892 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-a-cb14ed-6f8d8d5f84-6pqqb" event={"ID":"0a20e1c6-2c3f-4c1b-ac66-9fb5dfe44bfb","Type":"ContainerStarted","Data":"f9ba13e179f83e78e2ef3f48ddabda5aa6777c57a427c57c511e265c82727196"}
Oct 30 19:50:18.463841 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:50:18.463456    4892 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-api-a-cb14ed-6f8d8d5f84-6pqqb"
Oct 30 19:50:18.595923 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:50:18.595859    4892 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-api-a-cb14ed-c8b7565cf-h6zcq" podStartSLOduration=3.59584169 podStartE2EDuration="3.59584169s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:50:15 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:50:18.590784839 +0000 UTC m=+2066.161250432" watchObservedRunningTime="2026-10-30 19:50:18.59584169 +0000 UTC m=+2066.166307293"
Oct 30 19:50:18.722857 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:50:18.722790    4892 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-api-a-cb14ed-6f8d8d5f84-6pqqb" podStartSLOduration=2.7227667650000003 podStartE2EDuration="2.722766765s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:50:16 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:50:18.703421224 +0000 UTC m=+2066.273886826" watchObservedRunningTime="2026-10-30 19:50:18.722766765 +0000 UTC m=+2066.293232370"
Oct 30 19:50:19.285233 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:50:19.285197    4892 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-mt-dd9b8d79/e2e-lifecycle-model-dd9b8d79-kserve-7659b4b7d9-r2flr"
Oct 30 19:50:19.298036 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:50:19.298002    4892 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-mt-dd9b8d79/e2e-lifecycle-model-dd9b8d79-kserve-7659b4b7d9-r2flr"
Oct 30 19:50:19.483487 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:50:19.483453    4892 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-mt-dd9b8d79/e2e-lifecycle-model-dd9b8d79-kserve-7659b4b7d9-r2flr"
Oct 30 19:50:20.044757 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:50:20.044728    4892 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-8958-349fce"]
Oct 30 19:50:20.049437 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:50:20.049416    4892 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-8958-349fce"
Oct 30 19:50:20.085997 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:50:20.085956    4892 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kln7h\" (UniqueName: \"kubernetes.io/projected/392bb995-f542-41ac-a271-6b6e89e6a6d5-kube-api-access-kln7h\") pod \"test-curl-8958-349fce\" (UID: \"392bb995-f542-41ac-a271-6b6e89e6a6d5\") " pod="opendatahub/test-curl-8958-349fce"
Oct 30 19:50:20.101501 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:50:20.101477    4892 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-8958-349fce"]
Oct 30 19:50:20.124105 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:50:20.124079    4892 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"default-dockercfg-q2f25\""
Oct 30 19:50:20.187712 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:50:20.187681    4892 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kln7h\" (UniqueName: \"kubernetes.io/projected/392bb995-f542-41ac-a271-6b6e89e6a6d5-kube-api-access-kln7h\") pod \"test-curl-8958-349fce\" (UID: \"392bb995-f542-41ac-a271-6b6e89e6a6d5\") " pod="opendatahub/test-curl-8958-349fce"
Oct 30 19:50:20.244545 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:50:20.244517    4892 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-kln7h\" (UniqueName: \"kubernetes.io/projected/392bb995-f542-41ac-a271-6b6e89e6a6d5-kube-api-access-kln7h\") pod \"test-curl-8958-349fce\" (UID: \"392bb995-f542-41ac-a271-6b6e89e6a6d5\") " pod="opendatahub/test-curl-8958-349fce"
Oct 30 19:50:20.361900 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:50:20.361835    4892 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-8958-349fce"
Oct 30 19:50:20.740200 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:50:20.740167    4892 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-8958-349fce"]
Oct 30 19:50:20.740577 ip-10-0-132-82 kubenswrapper[4892]: W1030 19:50:20.740475    4892 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod392bb995_f542_41ac_a271_6b6e89e6a6d5.slice/crio-70c5479490c71e38a26620d8bb64c5370e5836030c35e4deb15a25e784bc2d96 WatchSource:0}: Error finding container 70c5479490c71e38a26620d8bb64c5370e5836030c35e4deb15a25e784bc2d96: Status 404 returned error can't find the container with id 70c5479490c71e38a26620d8bb64c5370e5836030c35e4deb15a25e784bc2d96
Oct 30 19:50:21.493489 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:50:21.493449    4892 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-8958-349fce" event={"ID":"392bb995-f542-41ac-a271-6b6e89e6a6d5","Type":"ContainerStarted","Data":"e8189294a6287eee7c9b51ee79cd7ef5cb99bd42bb97913c7c8420474041fd52"}
Oct 30 19:50:21.493672 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:50:21.493495    4892 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-8958-349fce" event={"ID":"392bb995-f542-41ac-a271-6b6e89e6a6d5","Type":"ContainerStarted","Data":"70c5479490c71e38a26620d8bb64c5370e5836030c35e4deb15a25e784bc2d96"}
Oct 30 19:50:21.635867 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:50:21.635799    4892 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/test-curl-8958-349fce" podStartSLOduration=1.63578289 podStartE2EDuration="1.63578289s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:50:20 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:50:21.634255566 +0000 UTC m=+2069.204721161" watchObservedRunningTime="2026-10-30 19:50:21.63578289 +0000 UTC m=+2069.206248490"
Oct 30 19:50:23.202197 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:50:23.202162    4892 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-8958-349fce"]
Oct 30 19:50:23.208743 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:50:23.208715    4892 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-8958-349fce"]
Oct 30 19:50:23.359995 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:50:23.359917    4892 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-8958-eeac38"]
Oct 30 19:50:23.360950 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:50:23.360925    4892 cpu_manager.go:414] "containerMap: removing container" podUID="392bb995-f542-41ac-a271-6b6e89e6a6d5" containerName="test-curl-8958-349fce"
Oct 30 19:50:23.360950 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:50:23.360950    4892 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="392bb995-f542-41ac-a271-6b6e89e6a6d5" containerName="test-curl-8958-349fce"
Oct 30 19:50:23.361211 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:50:23.361197    4892 memory_manager.go:365] "RemoveStaleState removing state" podUID="392bb995-f542-41ac-a271-6b6e89e6a6d5" containerName="test-curl-8958-349fce"
Oct 30 19:50:23.365663 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:50:23.365641    4892 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-8958-eeac38"
Oct 30 19:50:23.418441 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:50:23.418416    4892 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-8958-eeac38"]
Oct 30 19:50:23.428276 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:50:23.428244    4892 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tlzsz\" (UniqueName: \"kubernetes.io/projected/3dd9ce95-a4c7-4efa-82ec-ae9a745529fb-kube-api-access-tlzsz\") pod \"test-curl-8958-eeac38\" (UID: \"3dd9ce95-a4c7-4efa-82ec-ae9a745529fb\") " pod="opendatahub/test-curl-8958-eeac38"
Oct 30 19:50:23.511566 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:50:23.511487    4892 kuberuntime_container.go:898] "Killing container with a grace period" pod="opendatahub/test-curl-8958-349fce" podUID="392bb995-f542-41ac-a271-6b6e89e6a6d5" containerName="test-curl-8958-349fce" containerID="cri-o://e8189294a6287eee7c9b51ee79cd7ef5cb99bd42bb97913c7c8420474041fd52" gracePeriod=2
Oct 30 19:50:23.529979 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:50:23.529936    4892 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tlzsz\" (UniqueName: \"kubernetes.io/projected/3dd9ce95-a4c7-4efa-82ec-ae9a745529fb-kube-api-access-tlzsz\") pod \"test-curl-8958-eeac38\" (UID: \"3dd9ce95-a4c7-4efa-82ec-ae9a745529fb\") " pod="opendatahub/test-curl-8958-eeac38"
Oct 30 19:50:23.562122 ip-10-0-132-82 kubenswrapper[4892]: E1030 19:50:23.562077    4892 status_manager.go:1045] "Failed to get status for pod" err="pods \"test-curl-8958-349fce\" is forbidden: User \"system:node:ip-10-0-132-82.us-west-1.compute.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-132-82.us-west-1.compute.internal' and this object" podUID="392bb995-f542-41ac-a271-6b6e89e6a6d5" pod="opendatahub/test-curl-8958-349fce"
Oct 30 19:50:23.586817 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:50:23.586791    4892 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tlzsz\" (UniqueName: \"kubernetes.io/projected/3dd9ce95-a4c7-4efa-82ec-ae9a745529fb-kube-api-access-tlzsz\") pod \"test-curl-8958-eeac38\" (UID: \"3dd9ce95-a4c7-4efa-82ec-ae9a745529fb\") " pod="opendatahub/test-curl-8958-eeac38"
Oct 30 19:50:23.681014 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:50:23.680988    4892 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-8958-eeac38"
Oct 30 19:50:24.066951 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:50:24.065542    4892 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-8958-eeac38"]
Oct 30 19:50:24.066951 ip-10-0-132-82 kubenswrapper[4892]: W1030 19:50:24.066306    4892 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod3dd9ce95_a4c7_4efa_82ec_ae9a745529fb.slice/crio-6cc44a25eadfc0dada97d081189cf4a29de4db3e00efb6fb2e788547a05607d4 WatchSource:0}: Error finding container 6cc44a25eadfc0dada97d081189cf4a29de4db3e00efb6fb2e788547a05607d4: Status 404 returned error can't find the container with id 6cc44a25eadfc0dada97d081189cf4a29de4db3e00efb6fb2e788547a05607d4
Oct 30 19:50:24.688565 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:50:24.688493    4892 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-8958-eeac38" event={"ID":"3dd9ce95-a4c7-4efa-82ec-ae9a745529fb","Type":"ContainerStarted","Data":"e3903cbc9330f9db07febb297bf6b82e08fe68fded4b09c50392f41018927aa6"}
Oct 30 19:50:24.688565 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:50:24.688540    4892 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-8958-eeac38" event={"ID":"3dd9ce95-a4c7-4efa-82ec-ae9a745529fb","Type":"ContainerStarted","Data":"6cc44a25eadfc0dada97d081189cf4a29de4db3e00efb6fb2e788547a05607d4"}
Oct 30 19:50:24.747370 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:50:24.747336    4892 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-daa1ac-7f4d69d955-k5qgg"]
Oct 30 19:50:24.785350 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:50:24.783453    4892 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-daa1ac-7f4d69d955-k5qgg"
Oct 30 19:50:24.815432 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:50:24.809932    4892 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-daa1ac-7f4d69d955-k5qgg"]
Oct 30 19:50:24.815432 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:50:24.813335    4892 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/test-curl-8958-eeac38" podStartSLOduration=1.8133175860000001 podStartE2EDuration="1.813317586s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:50:23 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:50:24.805493636 +0000 UTC m=+2072.375959238" watchObservedRunningTime="2026-10-30 19:50:24.813317586 +0000 UTC m=+2072.383783188"
Oct 30 19:50:24.843989 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:50:24.839328    4892 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-inf-a-daa1ac\""
Oct 30 19:50:24.843989 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:50:24.839495    4892 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-inf-a-daa1ac-dockercfg-2l242\""
Oct 30 19:50:24.887840 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:50:24.887785    4892 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5ce65b2f-c2fb-4c9a-9e3b-783315e390b1-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-a-daa1ac-7f4d69d955-k5qgg\" (UID: \"5ce65b2f-c2fb-4c9a-9e3b-783315e390b1\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-daa1ac-7f4d69d955-k5qgg"
Oct 30 19:50:24.920067 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:50:24.889212    4892 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-h6rh5\" (UniqueName: \"kubernetes.io/projected/5ce65b2f-c2fb-4c9a-9e3b-783315e390b1-kube-api-access-h6rh5\") pod \"payload-pre-processing-e2e-inf-a-daa1ac-7f4d69d955-k5qgg\" (UID: \"5ce65b2f-c2fb-4c9a-9e3b-783315e390b1\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-daa1ac-7f4d69d955-k5qgg"
Oct 30 19:50:25.040312 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:50:25.028331    4892 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5ce65b2f-c2fb-4c9a-9e3b-783315e390b1-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-a-daa1ac-7f4d69d955-k5qgg\" (UID: \"5ce65b2f-c2fb-4c9a-9e3b-783315e390b1\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-daa1ac-7f4d69d955-k5qgg"
Oct 30 19:50:25.053050 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:50:25.040714    4892 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-h6rh5\" (UniqueName: \"kubernetes.io/projected/5ce65b2f-c2fb-4c9a-9e3b-783315e390b1-kube-api-access-h6rh5\") pod \"payload-pre-processing-e2e-inf-a-daa1ac-7f4d69d955-k5qgg\" (UID: \"5ce65b2f-c2fb-4c9a-9e3b-783315e390b1\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-daa1ac-7f4d69d955-k5qgg"
Oct 30 19:50:25.054906 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:50:25.035069    4892 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5ce65b2f-c2fb-4c9a-9e3b-783315e390b1-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-a-daa1ac-7f4d69d955-k5qgg\" (UID: \"5ce65b2f-c2fb-4c9a-9e3b-783315e390b1\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-daa1ac-7f4d69d955-k5qgg"
Oct 30 19:50:25.089066 ip-10-0-132-82 kubenswrapper[4892]: E1030 19:50:25.088938    4892 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-jrm87" containerName="extractor"
Oct 30 19:50:25.098287 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:50:25.098261    4892 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-h6rh5\" (UniqueName: \"kubernetes.io/projected/5ce65b2f-c2fb-4c9a-9e3b-783315e390b1-kube-api-access-h6rh5\") pod \"payload-pre-processing-e2e-inf-a-daa1ac-7f4d69d955-k5qgg\" (UID: \"5ce65b2f-c2fb-4c9a-9e3b-783315e390b1\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-daa1ac-7f4d69d955-k5qgg"
Oct 30 19:50:25.124086 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:50:25.124057    4892 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-daa1ac-7f4d69d955-k5qgg"
Oct 30 19:50:25.512125 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:50:25.512095    4892 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-daa1ac-7f4d69d955-k5qgg"]
Oct 30 19:50:25.512818 ip-10-0-132-82 kubenswrapper[4892]: W1030 19:50:25.512778    4892 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5ce65b2f_c2fb_4c9a_9e3b_783315e390b1.slice/crio-193d288535c50a6d7bac80d4a5c4597f17300df1faa70fab0e9e30531c850bac WatchSource:0}: Error finding container 193d288535c50a6d7bac80d4a5c4597f17300df1faa70fab0e9e30531c850bac: Status 404 returned error can't find the container with id 193d288535c50a6d7bac80d4a5c4597f17300df1faa70fab0e9e30531c850bac
Oct 30 19:50:25.702529 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:50:25.702497    4892 generic.go:356] "Generic (PLEG): container finished" podID="392bb995-f542-41ac-a271-6b6e89e6a6d5" containerID="e8189294a6287eee7c9b51ee79cd7ef5cb99bd42bb97913c7c8420474041fd52" exitCode=137
Oct 30 19:50:25.705506 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:50:25.704805    4892 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-daa1ac-7f4d69d955-k5qgg" event={"ID":"5ce65b2f-c2fb-4c9a-9e3b-783315e390b1","Type":"ContainerStarted","Data":"102578dd6e23bbdb0fabcd53e438b50e14073fcc5d32ebe6cda98f7ab76d2d04"}
Oct 30 19:50:25.705506 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:50:25.704855    4892 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-daa1ac-7f4d69d955-k5qgg" event={"ID":"5ce65b2f-c2fb-4c9a-9e3b-783315e390b1","Type":"ContainerStarted","Data":"193d288535c50a6d7bac80d4a5c4597f17300df1faa70fab0e9e30531c850bac"}
Oct 30 19:50:25.705506 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:50:25.705344    4892 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-daa1ac-7f4d69d955-k5qgg"
Oct 30 19:50:25.786579 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:50:25.786551    4892 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-8958-349fce"
Oct 30 19:50:25.817401 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:50:25.817351    4892 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-daa1ac-7f4d69d955-k5qgg" podStartSLOduration=1.817340073 podStartE2EDuration="1.817340073s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:50:24 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:50:25.816830224 +0000 UTC m=+2073.387295824" watchObservedRunningTime="2026-10-30 19:50:25.817340073 +0000 UTC m=+2073.387805669"
Oct 30 19:50:25.863525 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:50:25.863498    4892 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/392bb995-f542-41ac-a271-6b6e89e6a6d5-kube-api-access-kln7h\" (UniqueName: \"kubernetes.io/projected/392bb995-f542-41ac-a271-6b6e89e6a6d5-kube-api-access-kln7h\") pod \"392bb995-f542-41ac-a271-6b6e89e6a6d5\" (UID: \"392bb995-f542-41ac-a271-6b6e89e6a6d5\") "
Oct 30 19:50:25.866245 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:50:25.866217    4892 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/392bb995-f542-41ac-a271-6b6e89e6a6d5-kube-api-access-kln7h" pod "392bb995-f542-41ac-a271-6b6e89e6a6d5" (UID: "392bb995-f542-41ac-a271-6b6e89e6a6d5"). InnerVolumeSpecName "kube-api-access-kln7h". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:50:25.970179 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:50:25.970134    4892 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kln7h\" (UniqueName: \"kubernetes.io/projected/392bb995-f542-41ac-a271-6b6e89e6a6d5-kube-api-access-kln7h\") on node \"ip-10-0-132-82.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 19:50:26.256689 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:50:26.256653    4892 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-8958-eeac38"]
Oct 30 19:50:26.264836 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:50:26.264805    4892 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-8958-eeac38"]
Oct 30 19:50:26.420732 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:50:26.420691    4892 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-8958-7304f2"]
Oct 30 19:50:26.421574 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:50:26.421549    4892 cpu_manager.go:414] "containerMap: removing container" podUID="3dd9ce95-a4c7-4efa-82ec-ae9a745529fb" containerName="test-curl-8958-eeac38"
Oct 30 19:50:26.421574 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:50:26.421572    4892 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3dd9ce95-a4c7-4efa-82ec-ae9a745529fb" containerName="test-curl-8958-eeac38"
Oct 30 19:50:26.421881 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:50:26.421864    4892 memory_manager.go:365] "RemoveStaleState removing state" podUID="3dd9ce95-a4c7-4efa-82ec-ae9a745529fb" containerName="test-curl-8958-eeac38"
Oct 30 19:50:26.425782 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:50:26.425761    4892 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-8958-7304f2"
Oct 30 19:50:26.479203 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:50:26.479175    4892 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-8958-7304f2"]
Oct 30 19:50:26.571898 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:50:26.571862    4892 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="392bb995-f542-41ac-a271-6b6e89e6a6d5" path="/var/lib/kubelet/pods/392bb995-f542-41ac-a271-6b6e89e6a6d5/volumes"
Oct 30 19:50:26.578713 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:50:26.578683    4892 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-q2twp\" (UniqueName: \"kubernetes.io/projected/0d476e59-3b82-43ed-9b78-25272c4df148-kube-api-access-q2twp\") pod \"test-curl-8958-7304f2\" (UID: \"0d476e59-3b82-43ed-9b78-25272c4df148\") " pod="opendatahub/test-curl-8958-7304f2"
Oct 30 19:50:26.680646 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:50:26.680583    4892 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-q2twp\" (UniqueName: \"kubernetes.io/projected/0d476e59-3b82-43ed-9b78-25272c4df148-kube-api-access-q2twp\") pod \"test-curl-8958-7304f2\" (UID: \"0d476e59-3b82-43ed-9b78-25272c4df148\") " pod="opendatahub/test-curl-8958-7304f2"
Oct 30 19:50:26.714052 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:50:26.714031    4892 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-8958-349fce"
Oct 30 19:50:26.714428 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:50:26.714083    4892 scope.go:122] "RemoveContainer" containerID="e8189294a6287eee7c9b51ee79cd7ef5cb99bd42bb97913c7c8420474041fd52"
Oct 30 19:50:26.714428 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:50:26.714136    4892 kuberuntime_container.go:898] "Killing container with a grace period" pod="opendatahub/test-curl-8958-eeac38" podUID="3dd9ce95-a4c7-4efa-82ec-ae9a745529fb" containerName="test-curl-8958-eeac38" containerID="cri-o://e3903cbc9330f9db07febb297bf6b82e08fe68fded4b09c50392f41018927aa6" gracePeriod=2
Oct 30 19:50:26.736538 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:50:26.736515    4892 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-q2twp\" (UniqueName: \"kubernetes.io/projected/0d476e59-3b82-43ed-9b78-25272c4df148-kube-api-access-q2twp\") pod \"test-curl-8958-7304f2\" (UID: \"0d476e59-3b82-43ed-9b78-25272c4df148\") " pod="opendatahub/test-curl-8958-7304f2"
Oct 30 19:50:26.742343 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:50:26.742320    4892 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-8958-7304f2"
Oct 30 19:50:27.265243 ip-10-0-132-82 kubenswrapper[4892]: W1030 19:50:27.265205    4892 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod0d476e59_3b82_43ed_9b78_25272c4df148.slice/crio-de02e01ca785a9d048c0dd179f68fcbb5b7baecb3609ba8dda99e2e518ed9818 WatchSource:0}: Error finding container de02e01ca785a9d048c0dd179f68fcbb5b7baecb3609ba8dda99e2e518ed9818: Status 404 returned error can't find the container with id de02e01ca785a9d048c0dd179f68fcbb5b7baecb3609ba8dda99e2e518ed9818
Oct 30 19:50:27.266707 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:50:27.266678    4892 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-8958-7304f2"]
Oct 30 19:50:27.728844 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:50:27.728802    4892 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-8958-7304f2" event={"ID":"0d476e59-3b82-43ed-9b78-25272c4df148","Type":"ContainerStarted","Data":"ecc5243e8463ab11cd08f6ba02e5dc92704abc8a51d0096289973e27d7336cb0"}
Oct 30 19:50:27.729390 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:50:27.728852    4892 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-8958-7304f2" event={"ID":"0d476e59-3b82-43ed-9b78-25272c4df148","Type":"ContainerStarted","Data":"de02e01ca785a9d048c0dd179f68fcbb5b7baecb3609ba8dda99e2e518ed9818"}
Oct 30 19:50:27.848171 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:50:27.848107    4892 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/test-curl-8958-7304f2" podStartSLOduration=1.848089766 podStartE2EDuration="1.848089766s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:50:26 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:50:27.839846773 +0000 UTC m=+2075.410312373" watchObservedRunningTime="2026-10-30 19:50:27.848089766 +0000 UTC m=+2075.418555366"
Oct 30 19:50:28.986023 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:50:28.985953    4892 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-8958-eeac38"
Oct 30 19:50:29.133386 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:50:29.133355    4892 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3dd9ce95-a4c7-4efa-82ec-ae9a745529fb-kube-api-access-tlzsz\" (UniqueName: \"kubernetes.io/projected/3dd9ce95-a4c7-4efa-82ec-ae9a745529fb-kube-api-access-tlzsz\") pod \"3dd9ce95-a4c7-4efa-82ec-ae9a745529fb\" (UID: \"3dd9ce95-a4c7-4efa-82ec-ae9a745529fb\") "
Oct 30 19:50:29.135604 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:50:29.135567    4892 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3dd9ce95-a4c7-4efa-82ec-ae9a745529fb-kube-api-access-tlzsz" pod "3dd9ce95-a4c7-4efa-82ec-ae9a745529fb" (UID: "3dd9ce95-a4c7-4efa-82ec-ae9a745529fb"). InnerVolumeSpecName "kube-api-access-tlzsz". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:50:29.235784 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:50:29.235750    4892 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tlzsz\" (UniqueName: \"kubernetes.io/projected/3dd9ce95-a4c7-4efa-82ec-ae9a745529fb-kube-api-access-tlzsz\") on node \"ip-10-0-132-82.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 19:50:29.377797 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:50:29.377725    4892 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-8958-7304f2"]
Oct 30 19:50:29.383720 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:50:29.383696    4892 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-8958-7304f2"]
Oct 30 19:50:29.476068 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:50:29.476039    4892 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-api-a-cb14ed-6f8d8d5f84-6pqqb"
Oct 30 19:50:29.750705 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:50:29.750672    4892 generic.go:356] "Generic (PLEG): container finished" podID="3dd9ce95-a4c7-4efa-82ec-ae9a745529fb" containerID="e3903cbc9330f9db07febb297bf6b82e08fe68fded4b09c50392f41018927aa6" exitCode=137
Oct 30 19:50:29.750905 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:50:29.750731    4892 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-8958-eeac38"
Oct 30 19:50:29.750905 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:50:29.750771    4892 scope.go:122] "RemoveContainer" containerID="e3903cbc9330f9db07febb297bf6b82e08fe68fded4b09c50392f41018927aa6"
Oct 30 19:50:29.751079 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:50:29.750939    4892 kuberuntime_container.go:898] "Killing container with a grace period" pod="opendatahub/test-curl-8958-7304f2" podUID="0d476e59-3b82-43ed-9b78-25272c4df148" containerName="test-curl-8958-7304f2" containerID="cri-o://ecc5243e8463ab11cd08f6ba02e5dc92704abc8a51d0096289973e27d7336cb0" gracePeriod=2
Oct 30 19:50:29.768658 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:50:29.768632    4892 scope.go:122] "RemoveContainer" containerID="e3903cbc9330f9db07febb297bf6b82e08fe68fded4b09c50392f41018927aa6"
Oct 30 19:50:29.769186 ip-10-0-132-82 kubenswrapper[4892]: E1030 19:50:29.769157    4892 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e3903cbc9330f9db07febb297bf6b82e08fe68fded4b09c50392f41018927aa6\": container with ID starting with e3903cbc9330f9db07febb297bf6b82e08fe68fded4b09c50392f41018927aa6 not found: ID does not exist" containerID="e3903cbc9330f9db07febb297bf6b82e08fe68fded4b09c50392f41018927aa6"
Oct 30 19:50:29.769291 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:50:29.769195    4892 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e3903cbc9330f9db07febb297bf6b82e08fe68fded4b09c50392f41018927aa6"} err="failed to get container status \"e3903cbc9330f9db07febb297bf6b82e08fe68fded4b09c50392f41018927aa6\": rpc error: code = NotFound desc = could not find container \"e3903cbc9330f9db07febb297bf6b82e08fe68fded4b09c50392f41018927aa6\": container with ID starting with e3903cbc9330f9db07febb297bf6b82e08fe68fded4b09c50392f41018927aa6 not found: ID does not exist"
Oct 30 19:50:29.801497 ip-10-0-132-82 kubenswrapper[4892]: E1030 19:50:29.801465    4892 status_manager.go:1045] "Failed to get status for pod" err="pods \"test-curl-8958-7304f2\" is forbidden: User \"system:node:ip-10-0-132-82.us-west-1.compute.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-132-82.us-west-1.compute.internal' and this object" podUID="0d476e59-3b82-43ed-9b78-25272c4df148" pod="opendatahub/test-curl-8958-7304f2"
Oct 30 19:50:29.852176 ip-10-0-132-82 kubenswrapper[4892]: E1030 19:50:29.852144    4892 status_manager.go:1045] "Failed to get status for pod" err="pods \"test-curl-8958-eeac38\" is forbidden: User \"system:node:ip-10-0-132-82.us-west-1.compute.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-132-82.us-west-1.compute.internal' and this object" podUID="3dd9ce95-a4c7-4efa-82ec-ae9a745529fb" pod="opendatahub/test-curl-8958-eeac38"
Oct 30 19:50:29.903219 ip-10-0-132-82 kubenswrapper[4892]: E1030 19:50:29.903182    4892 status_manager.go:1045] "Failed to get status for pod" err="pods \"test-curl-8958-7304f2\" is forbidden: User \"system:node:ip-10-0-132-82.us-west-1.compute.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-132-82.us-west-1.compute.internal' and this object" podUID="0d476e59-3b82-43ed-9b78-25272c4df148" pod="opendatahub/test-curl-8958-7304f2"
Oct 30 19:50:30.486305 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:50:30.486277    4892 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-api-a-cb14ed-c8b7565cf-h6zcq"
Oct 30 19:50:30.537033 ip-10-0-132-82 kubenswrapper[4892]: E1030 19:50:30.536959    4892 status_manager.go:1045] "Failed to get status for pod" err="pods \"test-curl-8958-7304f2\" is forbidden: User \"system:node:ip-10-0-132-82.us-west-1.compute.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-132-82.us-west-1.compute.internal' and this object" podUID="0d476e59-3b82-43ed-9b78-25272c4df148" pod="opendatahub/test-curl-8958-7304f2"
Oct 30 19:50:30.575905 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:50:30.575875    4892 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="3dd9ce95-a4c7-4efa-82ec-ae9a745529fb" path="/var/lib/kubelet/pods/3dd9ce95-a4c7-4efa-82ec-ae9a745529fb/volumes"
Oct 30 19:50:30.587325 ip-10-0-132-82 kubenswrapper[4892]: E1030 19:50:30.587297    4892 status_manager.go:1045] "Failed to get status for pod" err="pods \"test-curl-8958-eeac38\" is forbidden: User \"system:node:ip-10-0-132-82.us-west-1.compute.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-132-82.us-west-1.compute.internal' and this object" podUID="3dd9ce95-a4c7-4efa-82ec-ae9a745529fb" pod="opendatahub/test-curl-8958-eeac38"
Oct 30 19:50:32.033135 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:50:32.033091    4892 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-8958-7304f2"
Oct 30 19:50:32.086414 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:50:32.086388    4892 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0d476e59-3b82-43ed-9b78-25272c4df148-kube-api-access-q2twp\" (UniqueName: \"kubernetes.io/projected/0d476e59-3b82-43ed-9b78-25272c4df148-kube-api-access-q2twp\") pod \"0d476e59-3b82-43ed-9b78-25272c4df148\" (UID: \"0d476e59-3b82-43ed-9b78-25272c4df148\") "
Oct 30 19:50:32.088667 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:50:32.088637    4892 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0d476e59-3b82-43ed-9b78-25272c4df148-kube-api-access-q2twp" pod "0d476e59-3b82-43ed-9b78-25272c4df148" (UID: "0d476e59-3b82-43ed-9b78-25272c4df148"). InnerVolumeSpecName "kube-api-access-q2twp". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:50:32.188297 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:50:32.188266    4892 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-q2twp\" (UniqueName: \"kubernetes.io/projected/0d476e59-3b82-43ed-9b78-25272c4df148-kube-api-access-q2twp\") on node \"ip-10-0-132-82.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 19:50:32.575668 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:50:32.575638    4892 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0d476e59-3b82-43ed-9b78-25272c4df148" path="/var/lib/kubelet/pods/0d476e59-3b82-43ed-9b78-25272c4df148/volumes"
Oct 30 19:50:32.829047 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:50:32.828785    4892 generic.go:356] "Generic (PLEG): container finished" podID="0d476e59-3b82-43ed-9b78-25272c4df148" containerID="ecc5243e8463ab11cd08f6ba02e5dc92704abc8a51d0096289973e27d7336cb0" exitCode=137
Oct 30 19:50:32.829047 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:50:32.828932    4892 scope.go:122] "RemoveContainer" containerID="ecc5243e8463ab11cd08f6ba02e5dc92704abc8a51d0096289973e27d7336cb0"
Oct 30 19:50:32.829284 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:50:32.829118    4892 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-8958-7304f2"
Oct 30 19:50:32.922124 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:50:32.922089    4892 scope.go:122] "RemoveContainer" containerID="ecc5243e8463ab11cd08f6ba02e5dc92704abc8a51d0096289973e27d7336cb0"
Oct 30 19:50:32.924293 ip-10-0-132-82 kubenswrapper[4892]: E1030 19:50:32.924256    4892 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ecc5243e8463ab11cd08f6ba02e5dc92704abc8a51d0096289973e27d7336cb0\": container with ID starting with ecc5243e8463ab11cd08f6ba02e5dc92704abc8a51d0096289973e27d7336cb0 not found: ID does not exist" containerID="ecc5243e8463ab11cd08f6ba02e5dc92704abc8a51d0096289973e27d7336cb0"
Oct 30 19:50:32.924436 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:50:32.924303    4892 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ecc5243e8463ab11cd08f6ba02e5dc92704abc8a51d0096289973e27d7336cb0"} err="failed to get container status \"ecc5243e8463ab11cd08f6ba02e5dc92704abc8a51d0096289973e27d7336cb0\": rpc error: code = NotFound desc = could not find container \"ecc5243e8463ab11cd08f6ba02e5dc92704abc8a51d0096289973e27d7336cb0\": container with ID starting with ecc5243e8463ab11cd08f6ba02e5dc92704abc8a51d0096289973e27d7336cb0 not found: ID does not exist"
Oct 30 19:50:36.717853 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:50:36.717827    4892 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-daa1ac-7f4d69d955-k5qgg"
Oct 30 19:50:48.883597 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:50:48.883546    4892 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-6873e0-78bf88f8b7-hhml5"]
Oct 30 19:50:48.885540 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:50:48.885499    4892 cpu_manager.go:414] "containerMap: removing container" podUID="0d476e59-3b82-43ed-9b78-25272c4df148" containerName="test-curl-8958-7304f2"
Oct 30 19:50:48.895799 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:50:48.895773    4892 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0d476e59-3b82-43ed-9b78-25272c4df148" containerName="test-curl-8958-7304f2"
Oct 30 19:50:48.896348 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:50:48.896309    4892 memory_manager.go:365] "RemoveStaleState removing state" podUID="0d476e59-3b82-43ed-9b78-25272c4df148" containerName="test-curl-8958-7304f2"
Oct 30 19:50:48.900709 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:50:48.900687    4892 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-6873e0-78bf88f8b7-hhml5"
Oct 30 19:50:48.940811 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:50:48.940785    4892 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-6873e0-78bf88f8b7-hhml5"]
Oct 30 19:50:48.949712 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:50:48.949683    4892 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9c8d8a28-2b91-4b8f-a582-f8059580f6e0-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-a-6873e0-78bf88f8b7-hhml5\" (UID: \"9c8d8a28-2b91-4b8f-a582-f8059580f6e0\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-6873e0-78bf88f8b7-hhml5"
Oct 30 19:50:48.949858 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:50:48.949776    4892 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9vfbh\" (UniqueName: \"kubernetes.io/projected/9c8d8a28-2b91-4b8f-a582-f8059580f6e0-kube-api-access-9vfbh\") pod \"payload-pre-processing-e2e-inf-a-6873e0-78bf88f8b7-hhml5\" (UID: \"9c8d8a28-2b91-4b8f-a582-f8059580f6e0\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-6873e0-78bf88f8b7-hhml5"
Oct 30 19:50:48.983330 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:50:48.983284    4892 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-inf-a-6873e0\""
Oct 30 19:50:49.072331 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:50:49.059030    4892 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9c8d8a28-2b91-4b8f-a582-f8059580f6e0-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-a-6873e0-78bf88f8b7-hhml5\" (UID: \"9c8d8a28-2b91-4b8f-a582-f8059580f6e0\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-6873e0-78bf88f8b7-hhml5"
Oct 30 19:50:49.076988 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:50:49.072723    4892 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9vfbh\" (UniqueName: \"kubernetes.io/projected/9c8d8a28-2b91-4b8f-a582-f8059580f6e0-kube-api-access-9vfbh\") pod \"payload-pre-processing-e2e-inf-a-6873e0-78bf88f8b7-hhml5\" (UID: \"9c8d8a28-2b91-4b8f-a582-f8059580f6e0\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-6873e0-78bf88f8b7-hhml5"
Oct 30 19:50:49.086063 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:50:49.060074    4892 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9c8d8a28-2b91-4b8f-a582-f8059580f6e0-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-a-6873e0-78bf88f8b7-hhml5\" (UID: \"9c8d8a28-2b91-4b8f-a582-f8059580f6e0\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-6873e0-78bf88f8b7-hhml5"
Oct 30 19:50:49.102599 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:50:49.102563    4892 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-6873e0-5ddd7bfd58-cbrnz"]
Oct 30 19:50:49.147171 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:50:49.147093    4892 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-6873e0-5ddd7bfd58-cbrnz"]
Oct 30 19:50:49.147487 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:50:49.147469    4892 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-a-6873e0-5ddd7bfd58-cbrnz"
Oct 30 19:50:49.206948 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:50:49.206914    4892 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-inf-a-6873e0-dockercfg-tbxfr\""
Oct 30 19:50:49.209120 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:50:49.209081    4892 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9vfbh\" (UniqueName: \"kubernetes.io/projected/9c8d8a28-2b91-4b8f-a582-f8059580f6e0-kube-api-access-9vfbh\") pod \"payload-pre-processing-e2e-inf-a-6873e0-78bf88f8b7-hhml5\" (UID: \"9c8d8a28-2b91-4b8f-a582-f8059580f6e0\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-6873e0-78bf88f8b7-hhml5"
Oct 30 19:50:49.219521 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:50:49.219492    4892 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-6873e0-78bf88f8b7-hhml5"
Oct 30 19:50:49.309439 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:50:49.309260    4892 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/cc52cc8e-3920-45b4-9807-b78a778537a0-plugins-config-volume\") pod \"payload-processing-e2e-inf-a-6873e0-5ddd7bfd58-cbrnz\" (UID: \"cc52cc8e-3920-45b4-9807-b78a778537a0\") " pod="openshift-ingress/payload-processing-e2e-inf-a-6873e0-5ddd7bfd58-cbrnz"
Oct 30 19:50:49.309439 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:50:49.309321    4892 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-phh7k\" (UniqueName: \"kubernetes.io/projected/cc52cc8e-3920-45b4-9807-b78a778537a0-kube-api-access-phh7k\") pod \"payload-processing-e2e-inf-a-6873e0-5ddd7bfd58-cbrnz\" (UID: \"cc52cc8e-3920-45b4-9807-b78a778537a0\") " pod="openshift-ingress/payload-processing-e2e-inf-a-6873e0-5ddd7bfd58-cbrnz"
Oct 30 19:50:49.410641 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:50:49.410556    4892 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/cc52cc8e-3920-45b4-9807-b78a778537a0-plugins-config-volume\") pod \"payload-processing-e2e-inf-a-6873e0-5ddd7bfd58-cbrnz\" (UID: \"cc52cc8e-3920-45b4-9807-b78a778537a0\") " pod="openshift-ingress/payload-processing-e2e-inf-a-6873e0-5ddd7bfd58-cbrnz"
Oct 30 19:50:49.410641 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:50:49.410601    4892 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-phh7k\" (UniqueName: \"kubernetes.io/projected/cc52cc8e-3920-45b4-9807-b78a778537a0-kube-api-access-phh7k\") pod \"payload-processing-e2e-inf-a-6873e0-5ddd7bfd58-cbrnz\" (UID: \"cc52cc8e-3920-45b4-9807-b78a778537a0\") " pod="openshift-ingress/payload-processing-e2e-inf-a-6873e0-5ddd7bfd58-cbrnz"
Oct 30 19:50:49.411349 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:50:49.411322    4892 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/cc52cc8e-3920-45b4-9807-b78a778537a0-plugins-config-volume\") pod \"payload-processing-e2e-inf-a-6873e0-5ddd7bfd58-cbrnz\" (UID: \"cc52cc8e-3920-45b4-9807-b78a778537a0\") " pod="openshift-ingress/payload-processing-e2e-inf-a-6873e0-5ddd7bfd58-cbrnz"
Oct 30 19:50:49.467007 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:50:49.466948    4892 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-phh7k\" (UniqueName: \"kubernetes.io/projected/cc52cc8e-3920-45b4-9807-b78a778537a0-kube-api-access-phh7k\") pod \"payload-processing-e2e-inf-a-6873e0-5ddd7bfd58-cbrnz\" (UID: \"cc52cc8e-3920-45b4-9807-b78a778537a0\") " pod="openshift-ingress/payload-processing-e2e-inf-a-6873e0-5ddd7bfd58-cbrnz"
Oct 30 19:50:49.521893 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:50:49.521867    4892 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-a-6873e0-5ddd7bfd58-cbrnz"
Oct 30 19:50:49.636576 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:50:49.636165    4892 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-6873e0-78bf88f8b7-hhml5"]
Oct 30 19:50:49.638006 ip-10-0-132-82 kubenswrapper[4892]: W1030 19:50:49.637138    4892 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9c8d8a28_2b91_4b8f_a582_f8059580f6e0.slice/crio-1534beaecb6928e04c6696f925f6a169f2ef9d61644b623312879897d0821271 WatchSource:0}: Error finding container 1534beaecb6928e04c6696f925f6a169f2ef9d61644b623312879897d0821271: Status 404 returned error can't find the container with id 1534beaecb6928e04c6696f925f6a169f2ef9d61644b623312879897d0821271
Oct 30 19:50:49.905489 ip-10-0-132-82 kubenswrapper[4892]: W1030 19:50:49.905456    4892 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcc52cc8e_3920_45b4_9807_b78a778537a0.slice/crio-aed3dd0f99b28c0ccec37b5df3fd213322f75009e0caf0edbeb2963deaac7479 WatchSource:0}: Error finding container aed3dd0f99b28c0ccec37b5df3fd213322f75009e0caf0edbeb2963deaac7479: Status 404 returned error can't find the container with id aed3dd0f99b28c0ccec37b5df3fd213322f75009e0caf0edbeb2963deaac7479
Oct 30 19:50:49.906730 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:50:49.906704    4892 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-6873e0-5ddd7bfd58-cbrnz"]
Oct 30 19:50:49.980213 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:50:49.980177    4892 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-a-6873e0-5ddd7bfd58-cbrnz" event={"ID":"cc52cc8e-3920-45b4-9807-b78a778537a0","Type":"ContainerStarted","Data":"aed3dd0f99b28c0ccec37b5df3fd213322f75009e0caf0edbeb2963deaac7479"}
Oct 30 19:50:49.981935 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:50:49.981900    4892 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-6873e0-78bf88f8b7-hhml5" event={"ID":"9c8d8a28-2b91-4b8f-a582-f8059580f6e0","Type":"ContainerStarted","Data":"f1507e6e223fe2e8daf7e8ca514ca76fcee879dae9d0d3cc2226945c440304b8"}
Oct 30 19:50:49.982174 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:50:49.981947    4892 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-6873e0-78bf88f8b7-hhml5" event={"ID":"9c8d8a28-2b91-4b8f-a582-f8059580f6e0","Type":"ContainerStarted","Data":"1534beaecb6928e04c6696f925f6a169f2ef9d61644b623312879897d0821271"}
Oct 30 19:50:49.982174 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:50:49.982039    4892 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-6873e0-78bf88f8b7-hhml5"
Oct 30 19:50:50.098630 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:50:50.098576    4892 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-6873e0-78bf88f8b7-hhml5" podStartSLOduration=2.098562764 podStartE2EDuration="2.098562764s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:50:48 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:50:50.094956323 +0000 UTC m=+2097.665421924" watchObservedRunningTime="2026-10-30 19:50:50.098562764 +0000 UTC m=+2097.669028363"
Oct 30 19:50:50.991275 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:50:50.991227    4892 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-a-6873e0-5ddd7bfd58-cbrnz" event={"ID":"cc52cc8e-3920-45b4-9807-b78a778537a0","Type":"ContainerStarted","Data":"c1c3c5c30831784587be0fb455ed4839366c861e63ee2ae57e5c70f301313221"}
Oct 30 19:50:51.108098 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:50:51.108046    4892 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-inf-a-6873e0-5ddd7bfd58-cbrnz" podStartSLOduration=3.108032168 podStartE2EDuration="3.108032168s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:50:48 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:50:51.103895735 +0000 UTC m=+2098.674361334" watchObservedRunningTime="2026-10-30 19:50:51.108032168 +0000 UTC m=+2098.678497769"
Oct 30 19:50:51.998628 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:50:51.998597    4892 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-inf-a-6873e0-5ddd7bfd58-cbrnz"
Oct 30 19:50:53.230359 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:50:53.230329    4892 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-tg5xj_11db9cec-653f-4aee-8937-0cf5b367efac/console-operator/3.log"
Oct 30 19:50:53.276770 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:50:53.276745    4892 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-tg5xj_11db9cec-653f-4aee-8937-0cf5b367efac/console-operator/3.log"
Oct 30 19:50:54.845885 ip-10-0-132-82 kubenswrapper[4892]: E1030 19:50:54.845847    4892 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-jrm87" containerName="extractor"
Oct 30 19:50:57.054263 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:50:57.054227    4892 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-affa4c07-25tmc"]
Oct 30 19:50:57.101038 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:50:57.101012    4892 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-affa4c07-25tmc"]
Oct 30 19:50:57.110991 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:50:57.110940    4892 scope.go:122] "RemoveContainer" containerID="772e8c1633aca86a1b725cee1d4db72ab0e0927004a64e78f0ea6d7cb13f9bf6"
Oct 30 19:50:57.121884 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:50:57.121867    4892 scope.go:122] "RemoveContainer" containerID="bea3091d950188a93bb395ce0766482893c449df4c5fa38957304cb79b74db1f"
Oct 30 19:50:57.133588 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:50:57.133556    4892 scope.go:122] "RemoveContainer" containerID="41779301ffee7b2cd47f02b4782841173bc03bd11947d1459aa6d7c6189ae9a2"
Oct 30 19:50:58.574150 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:50:58.574106    4892 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="75ab7a24-1dfb-4dd4-a1c7-e0a1c0f9209a" path="/var/lib/kubelet/pods/75ab7a24-1dfb-4dd4-a1c7-e0a1c0f9209a/volumes"
Oct 30 19:50:58.672135 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:50:58.672100    4892 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-fsfvk"]
Oct 30 19:51:00.995257 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:51:00.995227    4892 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-6873e0-78bf88f8b7-hhml5"
Oct 30 19:51:03.011041 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:51:03.011007    4892 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-inf-a-6873e0-5ddd7bfd58-cbrnz"
Oct 30 19:51:24.847763 ip-10-0-132-82 kubenswrapper[4892]: E1030 19:51:24.847723    4892 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-jrm87" containerName="extractor"
Oct 30 19:51:54.636877 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:51:54.636833    4892 prober.go:122] "Probe failed" probeType="Liveness" pod="kube-system/kube-apiserver-proxy-ip-10-0-132-82.us-west-1.compute.internal" podUID="f9af2f5b3d8deb871a93a1e3f0011dc6" containerName="haproxy" probeResult="failure" output="Get \"https://172.20.0.1:6443/version\": context deadline exceeded"
Oct 30 19:51:54.844811 ip-10-0-132-82 kubenswrapper[4892]: E1030 19:51:54.844776    4892 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-jrm87" containerName="extractor"
Oct 30 19:51:57.210166 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:51:57.210138    4892 scope.go:122] "RemoveContainer" containerID="212e7cfe45e7685162afd76de6796d429e249e786dffc0f7d559b385f84766e0"
Oct 30 19:51:57.220993 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:51:57.220959    4892 scope.go:122] "RemoveContainer" containerID="452359c17c5092139de6baf674ddbca722db515a7f9b501432124b253e3bc396"
Oct 30 19:51:57.230555 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:51:57.230534    4892 scope.go:122] "RemoveContainer" containerID="fe08b7b3aac0e21b63ef8e2a571d670ae5dfb48ee4f7c82134b5c20fa0ff6fc7"
Oct 30 19:52:03.783747 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:52:03.783713    4892 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-5bf13ff3-4ckjq"]
Oct 30 19:52:03.789662 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:52:03.789640    4892 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-5bf13ff3-4ckjq"
Oct 30 19:52:03.846355 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:52:03.846324    4892 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-5bf13ff3-4ckjq"]
Oct 30 19:52:03.890119 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:52:03.890089    4892 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 19:52:03.890119 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:52:03.890101    4892 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 30 19:52:03.890341 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:52:03.890328    4892 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-nlxp4\""
Oct 30 19:52:03.890388 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:52:03.890330    4892 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 30 19:52:03.914815 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:52:03.914787    4892 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8fb952a7-38ab-4a93-91e7-9383724d128a-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-mt-5bf13ff3-4ckjq\" (UID: \"8fb952a7-38ab-4a93-91e7-9383724d128a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-5bf13ff3-4ckjq"
Oct 30 19:52:03.915066 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:52:03.915041    4892 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vt2f2\" (UniqueName: \"kubernetes.io/projected/8fb952a7-38ab-4a93-91e7-9383724d128a-kube-api-access-vt2f2\") pod \"maas-api-revoke-keys-e2e-mt-5bf13ff3-4ckjq\" (UID: \"8fb952a7-38ab-4a93-91e7-9383724d128a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-5bf13ff3-4ckjq"
Oct 30 19:52:03.915171 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:52:03.915141    4892 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8fb952a7-38ab-4a93-91e7-9383724d128a-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-mt-5bf13ff3-4ckjq\" (UID: \"8fb952a7-38ab-4a93-91e7-9383724d128a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-5bf13ff3-4ckjq"
Oct 30 19:52:04.016353 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:52:04.016311    4892 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8fb952a7-38ab-4a93-91e7-9383724d128a-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-mt-5bf13ff3-4ckjq\" (UID: \"8fb952a7-38ab-4a93-91e7-9383724d128a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-5bf13ff3-4ckjq"
Oct 30 19:52:04.016616 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:52:04.016594    4892 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vt2f2\" (UniqueName: \"kubernetes.io/projected/8fb952a7-38ab-4a93-91e7-9383724d128a-kube-api-access-vt2f2\") pod \"maas-api-revoke-keys-e2e-mt-5bf13ff3-4ckjq\" (UID: \"8fb952a7-38ab-4a93-91e7-9383724d128a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-5bf13ff3-4ckjq"
Oct 30 19:52:04.016705 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:52:04.016689    4892 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8fb952a7-38ab-4a93-91e7-9383724d128a-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-mt-5bf13ff3-4ckjq\" (UID: \"8fb952a7-38ab-4a93-91e7-9383724d128a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-5bf13ff3-4ckjq"
Oct 30 19:52:04.017727 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:52:04.017700    4892 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8fb952a7-38ab-4a93-91e7-9383724d128a-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-mt-5bf13ff3-4ckjq\" (UID: \"8fb952a7-38ab-4a93-91e7-9383724d128a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-5bf13ff3-4ckjq"
Oct 30 19:52:04.019184 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:52:04.019164    4892 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8fb952a7-38ab-4a93-91e7-9383724d128a-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-mt-5bf13ff3-4ckjq\" (UID: \"8fb952a7-38ab-4a93-91e7-9383724d128a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-5bf13ff3-4ckjq"
Oct 30 19:52:04.073002 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:52:04.072912    4892 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vt2f2\" (UniqueName: \"kubernetes.io/projected/8fb952a7-38ab-4a93-91e7-9383724d128a-kube-api-access-vt2f2\") pod \"maas-api-revoke-keys-e2e-mt-5bf13ff3-4ckjq\" (UID: \"8fb952a7-38ab-4a93-91e7-9383724d128a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-5bf13ff3-4ckjq"
Oct 30 19:52:04.103316 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:52:04.103286    4892 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-5bf13ff3-4ckjq"
Oct 30 19:52:04.491591 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:52:04.491560    4892 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-5bf13ff3-4ckjq"]
Oct 30 19:52:04.492507 ip-10-0-132-82 kubenswrapper[4892]: W1030 19:52:04.492469    4892 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8fb952a7_38ab_4a93_91e7_9383724d128a.slice/crio-dc0197f2eaa755dc65757291085f47670e9d1d217aa73b78c0d830aa07591890 WatchSource:0}: Error finding container dc0197f2eaa755dc65757291085f47670e9d1d217aa73b78c0d830aa07591890: Status 404 returned error can't find the container with id dc0197f2eaa755dc65757291085f47670e9d1d217aa73b78c0d830aa07591890
Oct 30 19:52:04.856278 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:52:04.856241    4892 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-5bf13ff3-4ckjq" event={"ID":"8fb952a7-38ab-4a93-91e7-9383724d128a","Type":"ContainerStarted","Data":"46a4cf6ce878ac3694540a5cb6b2de4b9842f55ae0be01a49cb7669fbccb6364"}
Oct 30 19:52:04.856689 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:52:04.856302    4892 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-5bf13ff3-4ckjq" event={"ID":"8fb952a7-38ab-4a93-91e7-9383724d128a","Type":"ContainerStarted","Data":"dc0197f2eaa755dc65757291085f47670e9d1d217aa73b78c0d830aa07591890"}
Oct 30 19:52:04.972394 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:52:04.972336    4892 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-5bf13ff3-4ckjq" podStartSLOduration=1.972322341 podStartE2EDuration="1.972322341s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:52:03 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:52:04.968654889 +0000 UTC m=+2172.539120490" watchObservedRunningTime="2026-10-30 19:52:04.972322341 +0000 UTC m=+2172.542787940"
Oct 30 19:52:05.867218 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:52:05.867130    4892 generic.go:356] "Generic (PLEG): container finished" podID="8fb952a7-38ab-4a93-91e7-9383724d128a" containerID="46a4cf6ce878ac3694540a5cb6b2de4b9842f55ae0be01a49cb7669fbccb6364" exitCode=0
Oct 30 19:52:05.867218 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:52:05.867169    4892 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-5bf13ff3-4ckjq" event={"ID":"8fb952a7-38ab-4a93-91e7-9383724d128a","Type":"ContainerDied","Data":"46a4cf6ce878ac3694540a5cb6b2de4b9842f55ae0be01a49cb7669fbccb6364"}
Oct 30 19:52:07.050891 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:52:07.050863    4892 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-5bf13ff3-4ckjq"
Oct 30 19:52:07.161313 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:52:07.161288    4892 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/8fb952a7-38ab-4a93-91e7-9383724d128a-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8fb952a7-38ab-4a93-91e7-9383724d128a-maas-api-xks-ca\") pod \"8fb952a7-38ab-4a93-91e7-9383724d128a\" (UID: \"8fb952a7-38ab-4a93-91e7-9383724d128a\") "
Oct 30 19:52:07.161504 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:52:07.161351    4892 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/8fb952a7-38ab-4a93-91e7-9383724d128a-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8fb952a7-38ab-4a93-91e7-9383724d128a-maas-api-openshift-ca\") pod \"8fb952a7-38ab-4a93-91e7-9383724d128a\" (UID: \"8fb952a7-38ab-4a93-91e7-9383724d128a\") "
Oct 30 19:52:07.161504 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:52:07.161389    4892 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8fb952a7-38ab-4a93-91e7-9383724d128a-kube-api-access-vt2f2\" (UniqueName: \"kubernetes.io/projected/8fb952a7-38ab-4a93-91e7-9383724d128a-kube-api-access-vt2f2\") pod \"8fb952a7-38ab-4a93-91e7-9383724d128a\" (UID: \"8fb952a7-38ab-4a93-91e7-9383724d128a\") "
Oct 30 19:52:07.161727 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:52:07.161701    4892 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8fb952a7-38ab-4a93-91e7-9383724d128a-maas-api-openshift-ca" pod "8fb952a7-38ab-4a93-91e7-9383724d128a" (UID: "8fb952a7-38ab-4a93-91e7-9383724d128a"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 19:52:07.162033 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:52:07.162009    4892 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8fb952a7-38ab-4a93-91e7-9383724d128a-maas-api-openshift-ca\") on node \"ip-10-0-132-82.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 19:52:07.163658 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:52:07.163628    4892 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8fb952a7-38ab-4a93-91e7-9383724d128a-kube-api-access-vt2f2" pod "8fb952a7-38ab-4a93-91e7-9383724d128a" (UID: "8fb952a7-38ab-4a93-91e7-9383724d128a"). InnerVolumeSpecName "kube-api-access-vt2f2". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:52:07.163747 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:52:07.163714    4892 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8fb952a7-38ab-4a93-91e7-9383724d128a-maas-api-xks-ca" pod "8fb952a7-38ab-4a93-91e7-9383724d128a" (UID: "8fb952a7-38ab-4a93-91e7-9383724d128a"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 19:52:07.263870 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:52:07.263832    4892 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8fb952a7-38ab-4a93-91e7-9383724d128a-maas-api-xks-ca\") on node \"ip-10-0-132-82.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 19:52:07.263870 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:52:07.263865    4892 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vt2f2\" (UniqueName: \"kubernetes.io/projected/8fb952a7-38ab-4a93-91e7-9383724d128a-kube-api-access-vt2f2\") on node \"ip-10-0-132-82.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 19:52:07.883314 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:52:07.883283    4892 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-5bf13ff3-4ckjq"
Oct 30 19:52:07.883512 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:52:07.883319    4892 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-5bf13ff3-4ckjq" event={"ID":"8fb952a7-38ab-4a93-91e7-9383724d128a","Type":"ContainerDied","Data":"dc0197f2eaa755dc65757291085f47670e9d1d217aa73b78c0d830aa07591890"}
Oct 30 19:52:07.883512 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:52:07.883363    4892 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="dc0197f2eaa755dc65757291085f47670e9d1d217aa73b78c0d830aa07591890"
Oct 30 19:52:12.155990 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:52:12.155793    4892 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-fsfvk"]
Oct 30 19:52:14.969546 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:52:14.969501    4892 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-5bf13ff3-5554f7696-czdq8"]
Oct 30 19:52:15.085724 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:52:15.085689    4892 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-5bf13ff3-7b94c6cdf9-8ll9x"]
Oct 30 19:52:19.969808 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:52:19.969756    4892 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-mt-5bf13ff3-5554f7696-czdq8" podUID="0647888c-d306-43e3-b54c-132157f9da65" containerName="payload-processing" containerID="cri-o://928aba9079baa7ff6d25bc5701642c3afc1bdc35af7ced24ee284b862d11680b" gracePeriod=25
Oct 30 19:52:20.086533 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:52:20.086470    4892 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-mt-5bf13ff3-7b94c6cdf9-8ll9x" podUID="3189293d-967f-4896-8fce-94249776053b" containerName="payload-pre-processing" containerID="cri-o://1999bb58215eba07d37bd2ccb42867d25dde775138fd6bdb4133e8034c38c641" gracePeriod=25
Oct 30 19:52:20.270602 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:52:20.270576    4892 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-5bf13ff3-5554f7696-czdq8"
Oct 30 19:52:20.305268 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:52:20.305235    4892 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0647888c-d306-43e3-b54c-132157f9da65-kube-api-access-fhl6r\" (UniqueName: \"kubernetes.io/projected/0647888c-d306-43e3-b54c-132157f9da65-kube-api-access-fhl6r\") pod \"0647888c-d306-43e3-b54c-132157f9da65\" (UID: \"0647888c-d306-43e3-b54c-132157f9da65\") "
Oct 30 19:52:20.305445 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:52:20.305352    4892 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/0647888c-d306-43e3-b54c-132157f9da65-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0647888c-d306-43e3-b54c-132157f9da65-plugins-config-volume\") pod \"0647888c-d306-43e3-b54c-132157f9da65\" (UID: \"0647888c-d306-43e3-b54c-132157f9da65\") "
Oct 30 19:52:20.305983 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:52:20.305944    4892 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0647888c-d306-43e3-b54c-132157f9da65-plugins-config-volume" pod "0647888c-d306-43e3-b54c-132157f9da65" (UID: "0647888c-d306-43e3-b54c-132157f9da65"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 19:52:20.306098 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:52:20.306082    4892 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0647888c-d306-43e3-b54c-132157f9da65-plugins-config-volume\") on node \"ip-10-0-132-82.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 19:52:20.308277 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:52:20.308251    4892 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0647888c-d306-43e3-b54c-132157f9da65-kube-api-access-fhl6r" pod "0647888c-d306-43e3-b54c-132157f9da65" (UID: "0647888c-d306-43e3-b54c-132157f9da65"). InnerVolumeSpecName "kube-api-access-fhl6r". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:52:20.374751 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:52:20.374728    4892 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-5bf13ff3-7b94c6cdf9-8ll9x"
Oct 30 19:52:20.407390 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:52:20.407360    4892 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/3189293d-967f-4896-8fce-94249776053b-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3189293d-967f-4896-8fce-94249776053b-plugins-config-volume\") pod \"3189293d-967f-4896-8fce-94249776053b\" (UID: \"3189293d-967f-4896-8fce-94249776053b\") "
Oct 30 19:52:20.407672 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:52:20.407650    4892 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3189293d-967f-4896-8fce-94249776053b-kube-api-access-pvdmw\" (UniqueName: \"kubernetes.io/projected/3189293d-967f-4896-8fce-94249776053b-kube-api-access-pvdmw\") pod \"3189293d-967f-4896-8fce-94249776053b\" (UID: \"3189293d-967f-4896-8fce-94249776053b\") "
Oct 30 19:52:20.407757 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:52:20.407717    4892 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3189293d-967f-4896-8fce-94249776053b-plugins-config-volume" pod "3189293d-967f-4896-8fce-94249776053b" (UID: "3189293d-967f-4896-8fce-94249776053b"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 19:52:20.408340 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:52:20.408307    4892 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3189293d-967f-4896-8fce-94249776053b-plugins-config-volume\") on node \"ip-10-0-132-82.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 19:52:20.408340 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:52:20.408337    4892 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fhl6r\" (UniqueName: \"kubernetes.io/projected/0647888c-d306-43e3-b54c-132157f9da65-kube-api-access-fhl6r\") on node \"ip-10-0-132-82.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 19:52:20.409998 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:52:20.409934    4892 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3189293d-967f-4896-8fce-94249776053b-kube-api-access-pvdmw" pod "3189293d-967f-4896-8fce-94249776053b" (UID: "3189293d-967f-4896-8fce-94249776053b"). InnerVolumeSpecName "kube-api-access-pvdmw". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:52:20.509480 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:52:20.509417    4892 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pvdmw\" (UniqueName: \"kubernetes.io/projected/3189293d-967f-4896-8fce-94249776053b-kube-api-access-pvdmw\") on node \"ip-10-0-132-82.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 19:52:21.043520 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:52:21.043490    4892 generic.go:356] "Generic (PLEG): container finished" podID="0647888c-d306-43e3-b54c-132157f9da65" containerID="928aba9079baa7ff6d25bc5701642c3afc1bdc35af7ced24ee284b862d11680b" exitCode=0
Oct 30 19:52:21.044038 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:52:21.043536    4892 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-5bf13ff3-5554f7696-czdq8"
Oct 30 19:52:21.044038 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:52:21.043559    4892 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-5bf13ff3-5554f7696-czdq8" event={"ID":"0647888c-d306-43e3-b54c-132157f9da65","Type":"ContainerDied","Data":"928aba9079baa7ff6d25bc5701642c3afc1bdc35af7ced24ee284b862d11680b"}
Oct 30 19:52:21.044038 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:52:21.043723    4892 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-5bf13ff3-5554f7696-czdq8" event={"ID":"0647888c-d306-43e3-b54c-132157f9da65","Type":"ContainerDied","Data":"3c5108899ea98ed6f3049cdae925895ee46abd470c0f51b80c72ec9e26436edf"}
Oct 30 19:52:21.044038 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:52:21.043734    4892 scope.go:122] "RemoveContainer" containerID="928aba9079baa7ff6d25bc5701642c3afc1bdc35af7ced24ee284b862d11680b"
Oct 30 19:52:21.045364 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:52:21.045340    4892 generic.go:356] "Generic (PLEG): container finished" podID="3189293d-967f-4896-8fce-94249776053b" containerID="1999bb58215eba07d37bd2ccb42867d25dde775138fd6bdb4133e8034c38c641" exitCode=0
Oct 30 19:52:21.045439 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:52:21.045373    4892 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-5bf13ff3-7b94c6cdf9-8ll9x" event={"ID":"3189293d-967f-4896-8fce-94249776053b","Type":"ContainerDied","Data":"1999bb58215eba07d37bd2ccb42867d25dde775138fd6bdb4133e8034c38c641"}
Oct 30 19:52:21.045439 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:52:21.045396    4892 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-5bf13ff3-7b94c6cdf9-8ll9x"
Oct 30 19:52:21.045439 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:52:21.045413    4892 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-5bf13ff3-7b94c6cdf9-8ll9x" event={"ID":"3189293d-967f-4896-8fce-94249776053b","Type":"ContainerDied","Data":"04fb3444dc4125a75ec2e227eb8315d15c8a2aad4db1b6b9e23f3a19b14c5210"}
Oct 30 19:52:21.055233 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:52:21.055214    4892 scope.go:122] "RemoveContainer" containerID="928aba9079baa7ff6d25bc5701642c3afc1bdc35af7ced24ee284b862d11680b"
Oct 30 19:52:21.055478 ip-10-0-132-82 kubenswrapper[4892]: E1030 19:52:21.055459    4892 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"928aba9079baa7ff6d25bc5701642c3afc1bdc35af7ced24ee284b862d11680b\": container with ID starting with 928aba9079baa7ff6d25bc5701642c3afc1bdc35af7ced24ee284b862d11680b not found: ID does not exist" containerID="928aba9079baa7ff6d25bc5701642c3afc1bdc35af7ced24ee284b862d11680b"
Oct 30 19:52:21.055527 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:52:21.055486    4892 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"928aba9079baa7ff6d25bc5701642c3afc1bdc35af7ced24ee284b862d11680b"} err="failed to get container status \"928aba9079baa7ff6d25bc5701642c3afc1bdc35af7ced24ee284b862d11680b\": rpc error: code = NotFound desc = could not find container \"928aba9079baa7ff6d25bc5701642c3afc1bdc35af7ced24ee284b862d11680b\": container with ID starting with 928aba9079baa7ff6d25bc5701642c3afc1bdc35af7ced24ee284b862d11680b not found: ID does not exist"
Oct 30 19:52:21.055527 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:52:21.055504    4892 scope.go:122] "RemoveContainer" containerID="1999bb58215eba07d37bd2ccb42867d25dde775138fd6bdb4133e8034c38c641"
Oct 30 19:52:21.065462 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:52:21.065435    4892 scope.go:122] "RemoveContainer" containerID="1999bb58215eba07d37bd2ccb42867d25dde775138fd6bdb4133e8034c38c641"
Oct 30 19:52:21.065769 ip-10-0-132-82 kubenswrapper[4892]: E1030 19:52:21.065746    4892 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1999bb58215eba07d37bd2ccb42867d25dde775138fd6bdb4133e8034c38c641\": container with ID starting with 1999bb58215eba07d37bd2ccb42867d25dde775138fd6bdb4133e8034c38c641 not found: ID does not exist" containerID="1999bb58215eba07d37bd2ccb42867d25dde775138fd6bdb4133e8034c38c641"
Oct 30 19:52:21.065882 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:52:21.065776    4892 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1999bb58215eba07d37bd2ccb42867d25dde775138fd6bdb4133e8034c38c641"} err="failed to get container status \"1999bb58215eba07d37bd2ccb42867d25dde775138fd6bdb4133e8034c38c641\": rpc error: code = NotFound desc = could not find container \"1999bb58215eba07d37bd2ccb42867d25dde775138fd6bdb4133e8034c38c641\": container with ID starting with 1999bb58215eba07d37bd2ccb42867d25dde775138fd6bdb4133e8034c38c641 not found: ID does not exist"
Oct 30 19:52:21.208948 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:52:21.208918    4892 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-5bf13ff3-7b94c6cdf9-8ll9x"]
Oct 30 19:52:21.214823 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:52:21.214800    4892 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-5bf13ff3-7b94c6cdf9-8ll9x"]
Oct 30 19:52:21.323286 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:52:21.323206    4892 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-5bf13ff3-5554f7696-czdq8"]
Oct 30 19:52:21.327321 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:52:21.327296    4892 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-5bf13ff3-5554f7696-czdq8"]
Oct 30 19:52:21.908248 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:52:21.908215    4892 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-8964-6ec67f"]
Oct 30 19:52:21.908891 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:52:21.908875    4892 cpu_manager.go:414] "containerMap: removing container" podUID="0647888c-d306-43e3-b54c-132157f9da65" containerName="payload-processing"
Oct 30 19:52:21.908937 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:52:21.908894    4892 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0647888c-d306-43e3-b54c-132157f9da65" containerName="payload-processing"
Oct 30 19:52:21.908937 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:52:21.908907    4892 cpu_manager.go:414] "containerMap: removing container" podUID="8fb952a7-38ab-4a93-91e7-9383724d128a" containerName="revoke-keys"
Oct 30 19:52:21.908937 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:52:21.908912    4892 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8fb952a7-38ab-4a93-91e7-9383724d128a" containerName="revoke-keys"
Oct 30 19:52:21.908937 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:52:21.908926    4892 cpu_manager.go:414] "containerMap: removing container" podUID="3189293d-967f-4896-8fce-94249776053b" containerName="payload-pre-processing"
Oct 30 19:52:21.908937 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:52:21.908931    4892 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3189293d-967f-4896-8fce-94249776053b" containerName="payload-pre-processing"
Oct 30 19:52:21.909155 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:52:21.909073    4892 memory_manager.go:365] "RemoveStaleState removing state" podUID="0647888c-d306-43e3-b54c-132157f9da65" containerName="payload-processing"
Oct 30 19:52:21.909155 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:52:21.909084    4892 memory_manager.go:365] "RemoveStaleState removing state" podUID="8fb952a7-38ab-4a93-91e7-9383724d128a" containerName="revoke-keys"
Oct 30 19:52:21.909155 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:52:21.909091    4892 memory_manager.go:365] "RemoveStaleState removing state" podUID="3189293d-967f-4896-8fce-94249776053b" containerName="payload-pre-processing"
Oct 30 19:52:21.914226 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:52:21.914208    4892 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-8964-6ec67f"
Oct 30 19:52:21.965471 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:52:21.965446    4892 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-8964-6ec67f"]
Oct 30 19:52:22.010042 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:52:22.010008    4892 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"default-dockercfg-q2f25\""
Oct 30 19:52:22.028838 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:52:22.028805    4892 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4kdtv\" (UniqueName: \"kubernetes.io/projected/8b4de3f4-3b56-4666-9b24-fa58d5d3eaf8-kube-api-access-4kdtv\") pod \"test-curl-8964-6ec67f\" (UID: \"8b4de3f4-3b56-4666-9b24-fa58d5d3eaf8\") " pod="opendatahub/test-curl-8964-6ec67f"
Oct 30 19:52:22.130304 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:52:22.130270    4892 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4kdtv\" (UniqueName: \"kubernetes.io/projected/8b4de3f4-3b56-4666-9b24-fa58d5d3eaf8-kube-api-access-4kdtv\") pod \"test-curl-8964-6ec67f\" (UID: \"8b4de3f4-3b56-4666-9b24-fa58d5d3eaf8\") " pod="opendatahub/test-curl-8964-6ec67f"
Oct 30 19:52:22.187120 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:52:22.187053    4892 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4kdtv\" (UniqueName: \"kubernetes.io/projected/8b4de3f4-3b56-4666-9b24-fa58d5d3eaf8-kube-api-access-4kdtv\") pod \"test-curl-8964-6ec67f\" (UID: \"8b4de3f4-3b56-4666-9b24-fa58d5d3eaf8\") " pod="opendatahub/test-curl-8964-6ec67f"
Oct 30 19:52:22.226566 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:52:22.226543    4892 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-8964-6ec67f"
Oct 30 19:52:22.574348 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:52:22.574270    4892 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0647888c-d306-43e3-b54c-132157f9da65" path="/var/lib/kubelet/pods/0647888c-d306-43e3-b54c-132157f9da65/volumes"
Oct 30 19:52:22.574778 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:52:22.574760    4892 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="3189293d-967f-4896-8fce-94249776053b" path="/var/lib/kubelet/pods/3189293d-967f-4896-8fce-94249776053b/volumes"
Oct 30 19:52:22.611028 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:52:22.610997    4892 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-8964-6ec67f"]
Oct 30 19:52:23.066467 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:52:23.066429    4892 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-8964-6ec67f" event={"ID":"8b4de3f4-3b56-4666-9b24-fa58d5d3eaf8","Type":"ContainerStarted","Data":"0ecadf667272c1efe037e85645aef2c0c4f4c75bbad905393c2e95f472727f96"}
Oct 30 19:52:23.066680 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:52:23.066485    4892 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-8964-6ec67f" event={"ID":"8b4de3f4-3b56-4666-9b24-fa58d5d3eaf8","Type":"ContainerStarted","Data":"1733d1a8580d6f052fce14e990b827bdde9931ff8133cd7d69b3bb5e55726f74"}
Oct 30 19:52:23.178229 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:52:23.178172    4892 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/test-curl-8964-6ec67f" podStartSLOduration=2.178157335 podStartE2EDuration="2.178157335s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:52:21 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:52:23.176882368 +0000 UTC m=+2190.747347967" watchObservedRunningTime="2026-10-30 19:52:23.178157335 +0000 UTC m=+2190.748622964"
Oct 30 19:52:24.845577 ip-10-0-132-82 kubenswrapper[4892]: E1030 19:52:24.845543    4892 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-jrm87" containerName="extractor"
Oct 30 19:52:34.643991 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:52:34.643875    4892 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-8964-6ec67f"]
Oct 30 19:52:34.644468 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:52:34.644152    4892 kuberuntime_container.go:898] "Killing container with a grace period" pod="opendatahub/test-curl-8964-6ec67f" podUID="8b4de3f4-3b56-4666-9b24-fa58d5d3eaf8" containerName="test-curl-8964-6ec67f" containerID="cri-o://0ecadf667272c1efe037e85645aef2c0c4f4c75bbad905393c2e95f472727f96" gracePeriod=2
Oct 30 19:52:34.649608 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:52:34.649583    4892 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-8964-6ec67f"]
Oct 30 19:52:34.694888 ip-10-0-132-82 kubenswrapper[4892]: E1030 19:52:34.694850    4892 status_manager.go:1045] "Failed to get status for pod" err="pods \"test-curl-8964-6ec67f\" is forbidden: User \"system:node:ip-10-0-132-82.us-west-1.compute.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-132-82.us-west-1.compute.internal' and this object" podUID="8b4de3f4-3b56-4666-9b24-fa58d5d3eaf8" pod="opendatahub/test-curl-8964-6ec67f"
Oct 30 19:52:35.345357 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:52:35.345313    4892 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-dd9b8d79-fdkrb"]
Oct 30 19:52:35.346839 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:52:35.346811    4892 cpu_manager.go:414] "containerMap: removing container" podUID="8b4de3f4-3b56-4666-9b24-fa58d5d3eaf8" containerName="test-curl-8964-6ec67f"
Oct 30 19:52:35.347000 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:52:35.346876    4892 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8b4de3f4-3b56-4666-9b24-fa58d5d3eaf8" containerName="test-curl-8964-6ec67f"
Oct 30 19:52:35.347288 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:52:35.347270    4892 memory_manager.go:365] "RemoveStaleState removing state" podUID="8b4de3f4-3b56-4666-9b24-fa58d5d3eaf8" containerName="test-curl-8964-6ec67f"
Oct 30 19:52:35.352667 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:52:35.352643    4892 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-dd9b8d79-fdkrb"
Oct 30 19:52:35.410563 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:52:35.410536    4892 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-dd9b8d79-fdkrb"]
Oct 30 19:52:35.454110 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:52:35.454074    4892 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-nlxp4\""
Oct 30 19:52:35.454280 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:52:35.454123    4892 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 19:52:35.454280 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:52:35.454226    4892 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 30 19:52:35.454398 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:52:35.454375    4892 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 30 19:52:35.516889 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:52:35.516853    4892 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6539580c-56b5-4501-abe2-116743a7096b-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-mt-dd9b8d79-fdkrb\" (UID: \"6539580c-56b5-4501-abe2-116743a7096b\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-dd9b8d79-fdkrb"
Oct 30 19:52:35.517072 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:52:35.517000    4892 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6539580c-56b5-4501-abe2-116743a7096b-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-mt-dd9b8d79-fdkrb\" (UID: \"6539580c-56b5-4501-abe2-116743a7096b\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-dd9b8d79-fdkrb"
Oct 30 19:52:35.517177 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:52:35.517149    4892 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8hkvf\" (UniqueName: \"kubernetes.io/projected/6539580c-56b5-4501-abe2-116743a7096b-kube-api-access-8hkvf\") pod \"maas-api-revoke-keys-e2e-mt-dd9b8d79-fdkrb\" (UID: \"6539580c-56b5-4501-abe2-116743a7096b\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-dd9b8d79-fdkrb"
Oct 30 19:52:35.553645 ip-10-0-132-82 kubenswrapper[4892]: E1030 19:52:35.553604    4892 status_manager.go:1045] "Failed to get status for pod" err="pods \"test-curl-8964-6ec67f\" is forbidden: User \"system:node:ip-10-0-132-82.us-west-1.compute.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-132-82.us-west-1.compute.internal' and this object" podUID="8b4de3f4-3b56-4666-9b24-fa58d5d3eaf8" pod="opendatahub/test-curl-8964-6ec67f"
Oct 30 19:52:35.618300 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:52:35.618227    4892 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8hkvf\" (UniqueName: \"kubernetes.io/projected/6539580c-56b5-4501-abe2-116743a7096b-kube-api-access-8hkvf\") pod \"maas-api-revoke-keys-e2e-mt-dd9b8d79-fdkrb\" (UID: \"6539580c-56b5-4501-abe2-116743a7096b\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-dd9b8d79-fdkrb"
Oct 30 19:52:35.618433 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:52:35.618349    4892 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6539580c-56b5-4501-abe2-116743a7096b-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-mt-dd9b8d79-fdkrb\" (UID: \"6539580c-56b5-4501-abe2-116743a7096b\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-dd9b8d79-fdkrb"
Oct 30 19:52:35.618433 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:52:35.618414    4892 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6539580c-56b5-4501-abe2-116743a7096b-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-mt-dd9b8d79-fdkrb\" (UID: \"6539580c-56b5-4501-abe2-116743a7096b\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-dd9b8d79-fdkrb"
Oct 30 19:52:35.619106 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:52:35.619087    4892 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6539580c-56b5-4501-abe2-116743a7096b-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-mt-dd9b8d79-fdkrb\" (UID: \"6539580c-56b5-4501-abe2-116743a7096b\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-dd9b8d79-fdkrb"
Oct 30 19:52:35.621002 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:52:35.620980    4892 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6539580c-56b5-4501-abe2-116743a7096b-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-mt-dd9b8d79-fdkrb\" (UID: \"6539580c-56b5-4501-abe2-116743a7096b\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-dd9b8d79-fdkrb"
Oct 30 19:52:35.678368 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:52:35.678341    4892 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8hkvf\" (UniqueName: \"kubernetes.io/projected/6539580c-56b5-4501-abe2-116743a7096b-kube-api-access-8hkvf\") pod \"maas-api-revoke-keys-e2e-mt-dd9b8d79-fdkrb\" (UID: \"6539580c-56b5-4501-abe2-116743a7096b\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-dd9b8d79-fdkrb"
Oct 30 19:52:35.968510 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:52:35.968466    4892 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-dd9b8d79-fdkrb"
Oct 30 19:52:36.348920 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:52:36.348887    4892 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-dd9b8d79-fdkrb"]
Oct 30 19:52:36.349234 ip-10-0-132-82 kubenswrapper[4892]: W1030 19:52:36.349205    4892 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6539580c_56b5_4501_abe2_116743a7096b.slice/crio-010b781e9d7dec0385301c417aa836b77f6df15a9e4ea0460020ae784361595b WatchSource:0}: Error finding container 010b781e9d7dec0385301c417aa836b77f6df15a9e4ea0460020ae784361595b: Status 404 returned error can't find the container with id 010b781e9d7dec0385301c417aa836b77f6df15a9e4ea0460020ae784361595b
Oct 30 19:52:36.871841 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:52:36.871817    4892 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-8964-6ec67f"
Oct 30 19:52:37.064083 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:52:37.062596    4892 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-7c2ecd3d-pft4g"]
Oct 30 19:52:37.064083 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:52:37.063355    4892 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8b4de3f4-3b56-4666-9b24-fa58d5d3eaf8-kube-api-access-4kdtv\" (UniqueName: \"kubernetes.io/projected/8b4de3f4-3b56-4666-9b24-fa58d5d3eaf8-kube-api-access-4kdtv\") pod \"8b4de3f4-3b56-4666-9b24-fa58d5d3eaf8\" (UID: \"8b4de3f4-3b56-4666-9b24-fa58d5d3eaf8\") "
Oct 30 19:52:37.092264 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:52:37.084560    4892 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8b4de3f4-3b56-4666-9b24-fa58d5d3eaf8-kube-api-access-4kdtv" pod "8b4de3f4-3b56-4666-9b24-fa58d5d3eaf8" (UID: "8b4de3f4-3b56-4666-9b24-fa58d5d3eaf8"). InnerVolumeSpecName "kube-api-access-4kdtv". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:52:37.110470 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:52:37.110413    4892 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-7c2ecd3d-pft4g"]
Oct 30 19:52:37.165561 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:52:37.165528    4892 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4kdtv\" (UniqueName: \"kubernetes.io/projected/8b4de3f4-3b56-4666-9b24-fa58d5d3eaf8-kube-api-access-4kdtv\") on node \"ip-10-0-132-82.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 19:52:37.195040 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:52:37.195004    4892 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-dd9b8d79-fdkrb" event={"ID":"6539580c-56b5-4501-abe2-116743a7096b","Type":"ContainerStarted","Data":"168cedc919b24607d50684cb50ed7c4b8fc1fd9016cff3ef8af293b6ccd1df5c"}
Oct 30 19:52:37.195204 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:52:37.195071    4892 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-dd9b8d79-fdkrb" event={"ID":"6539580c-56b5-4501-abe2-116743a7096b","Type":"ContainerStarted","Data":"010b781e9d7dec0385301c417aa836b77f6df15a9e4ea0460020ae784361595b"}
Oct 30 19:52:37.196717 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:52:37.196689    4892 generic.go:356] "Generic (PLEG): container finished" podID="8b4de3f4-3b56-4666-9b24-fa58d5d3eaf8" containerID="0ecadf667272c1efe037e85645aef2c0c4f4c75bbad905393c2e95f472727f96" exitCode=137
Oct 30 19:52:37.196841 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:52:37.196729    4892 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-8964-6ec67f"
Oct 30 19:52:37.196841 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:52:37.196795    4892 scope.go:122] "RemoveContainer" containerID="0ecadf667272c1efe037e85645aef2c0c4f4c75bbad905393c2e95f472727f96"
Oct 30 19:52:37.211083 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:52:37.211063    4892 scope.go:122] "RemoveContainer" containerID="0ecadf667272c1efe037e85645aef2c0c4f4c75bbad905393c2e95f472727f96"
Oct 30 19:52:37.211392 ip-10-0-132-82 kubenswrapper[4892]: E1030 19:52:37.211373    4892 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0ecadf667272c1efe037e85645aef2c0c4f4c75bbad905393c2e95f472727f96\": container with ID starting with 0ecadf667272c1efe037e85645aef2c0c4f4c75bbad905393c2e95f472727f96 not found: ID does not exist" containerID="0ecadf667272c1efe037e85645aef2c0c4f4c75bbad905393c2e95f472727f96"
Oct 30 19:52:37.211440 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:52:37.211399    4892 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0ecadf667272c1efe037e85645aef2c0c4f4c75bbad905393c2e95f472727f96"} err="failed to get container status \"0ecadf667272c1efe037e85645aef2c0c4f4c75bbad905393c2e95f472727f96\": rpc error: code = NotFound desc = could not find container \"0ecadf667272c1efe037e85645aef2c0c4f4c75bbad905393c2e95f472727f96\": container with ID starting with 0ecadf667272c1efe037e85645aef2c0c4f4c75bbad905393c2e95f472727f96 not found: ID does not exist"
Oct 30 19:52:37.307730 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:52:37.307673    4892 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-dd9b8d79-fdkrb" podStartSLOduration=2.307659217 podStartE2EDuration="2.307659217s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:52:35 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:52:37.306564886 +0000 UTC m=+2204.877030486" watchObservedRunningTime="2026-10-30 19:52:37.307659217 +0000 UTC m=+2204.878124816"
Oct 30 19:52:37.357030 ip-10-0-132-82 kubenswrapper[4892]: E1030 19:52:37.356937    4892 status_manager.go:1045] "Failed to get status for pod" err="pods \"test-curl-8964-6ec67f\" is forbidden: User \"system:node:ip-10-0-132-82.us-west-1.compute.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-132-82.us-west-1.compute.internal' and this object" podUID="8b4de3f4-3b56-4666-9b24-fa58d5d3eaf8" pod="opendatahub/test-curl-8964-6ec67f"
Oct 30 19:52:38.207111 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:52:38.207070    4892 generic.go:356] "Generic (PLEG): container finished" podID="6539580c-56b5-4501-abe2-116743a7096b" containerID="168cedc919b24607d50684cb50ed7c4b8fc1fd9016cff3ef8af293b6ccd1df5c" exitCode=0
Oct 30 19:52:38.207533 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:52:38.207154    4892 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-dd9b8d79-fdkrb" event={"ID":"6539580c-56b5-4501-abe2-116743a7096b","Type":"ContainerDied","Data":"168cedc919b24607d50684cb50ed7c4b8fc1fd9016cff3ef8af293b6ccd1df5c"}
Oct 30 19:52:38.369507 ip-10-0-132-82 kubenswrapper[4892]: E1030 19:52:38.369466    4892 status_manager.go:1045] "Failed to get status for pod" err="pods \"test-curl-8964-6ec67f\" is forbidden: User \"system:node:ip-10-0-132-82.us-west-1.compute.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-132-82.us-west-1.compute.internal' and this object" podUID="8b4de3f4-3b56-4666-9b24-fa58d5d3eaf8" pod="opendatahub/test-curl-8964-6ec67f"
Oct 30 19:52:38.570641 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:52:38.570560    4892 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="8b4de3f4-3b56-4666-9b24-fa58d5d3eaf8" path="/var/lib/kubelet/pods/8b4de3f4-3b56-4666-9b24-fa58d5d3eaf8/volumes"
Oct 30 19:52:38.571063 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:52:38.571047    4892 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e21b494a-695e-4dd9-a55c-fa48b8f0957b" path="/var/lib/kubelet/pods/e21b494a-695e-4dd9-a55c-fa48b8f0957b/volumes"
Oct 30 19:52:39.385130 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:52:39.385109    4892 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-dd9b8d79-fdkrb"
Oct 30 19:52:39.412556 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:52:39.412527    4892 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/6539580c-56b5-4501-abe2-116743a7096b-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6539580c-56b5-4501-abe2-116743a7096b-maas-api-openshift-ca\") pod \"6539580c-56b5-4501-abe2-116743a7096b\" (UID: \"6539580c-56b5-4501-abe2-116743a7096b\") "
Oct 30 19:52:39.412693 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:52:39.412648    4892 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6539580c-56b5-4501-abe2-116743a7096b-kube-api-access-8hkvf\" (UniqueName: \"kubernetes.io/projected/6539580c-56b5-4501-abe2-116743a7096b-kube-api-access-8hkvf\") pod \"6539580c-56b5-4501-abe2-116743a7096b\" (UID: \"6539580c-56b5-4501-abe2-116743a7096b\") "
Oct 30 19:52:39.412788 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:52:39.412768    4892 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/6539580c-56b5-4501-abe2-116743a7096b-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6539580c-56b5-4501-abe2-116743a7096b-maas-api-xks-ca\") pod \"6539580c-56b5-4501-abe2-116743a7096b\" (UID: \"6539580c-56b5-4501-abe2-116743a7096b\") "
Oct 30 19:52:39.412904 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:52:39.412879    4892 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6539580c-56b5-4501-abe2-116743a7096b-maas-api-openshift-ca" pod "6539580c-56b5-4501-abe2-116743a7096b" (UID: "6539580c-56b5-4501-abe2-116743a7096b"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 19:52:39.413379 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:52:39.413358    4892 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6539580c-56b5-4501-abe2-116743a7096b-maas-api-openshift-ca\") on node \"ip-10-0-132-82.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 19:52:39.414860 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:52:39.414835    4892 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6539580c-56b5-4501-abe2-116743a7096b-kube-api-access-8hkvf" pod "6539580c-56b5-4501-abe2-116743a7096b" (UID: "6539580c-56b5-4501-abe2-116743a7096b"). InnerVolumeSpecName "kube-api-access-8hkvf". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:52:39.414860 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:52:39.414845    4892 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6539580c-56b5-4501-abe2-116743a7096b-maas-api-xks-ca" pod "6539580c-56b5-4501-abe2-116743a7096b" (UID: "6539580c-56b5-4501-abe2-116743a7096b"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 19:52:39.514561 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:52:39.514476    4892 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8hkvf\" (UniqueName: \"kubernetes.io/projected/6539580c-56b5-4501-abe2-116743a7096b-kube-api-access-8hkvf\") on node \"ip-10-0-132-82.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 19:52:39.514561 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:52:39.514515    4892 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6539580c-56b5-4501-abe2-116743a7096b-maas-api-xks-ca\") on node \"ip-10-0-132-82.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 19:52:40.225058 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:52:40.225030    4892 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-dd9b8d79-fdkrb"
Oct 30 19:52:40.225058 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:52:40.225037    4892 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-dd9b8d79-fdkrb" event={"ID":"6539580c-56b5-4501-abe2-116743a7096b","Type":"ContainerDied","Data":"010b781e9d7dec0385301c417aa836b77f6df15a9e4ea0460020ae784361595b"}
Oct 30 19:52:40.225291 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:52:40.225083    4892 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="010b781e9d7dec0385301c417aa836b77f6df15a9e4ea0460020ae784361595b"
Oct 30 19:52:47.202122 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:52:47.202078    4892 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-dd9b8d79-67c8d6d46-m4wc6"]
Oct 30 19:52:47.300559 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:52:47.300528    4892 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-dd9b8d79-867946f667-plb8n"]
Oct 30 19:52:51.493194 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:52:51.493163    4892 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-a3f9b7-openshift-default-7d6d8b8fb5-sfm8n"]
Oct 30 19:52:51.493787 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:52:51.493396    4892 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w2-a3f9b7-openshift-default-7d6d8b8fb5-sfm8n" podUID="1c3346aa-a989-495e-9d54-66c8ec632d91" containerName="istio-proxy" containerID="cri-o://0c4417cc4bb8af483b3db4a9fd5ae18b438a366697c09861876a6f417453a841" gracePeriod=30
Oct 30 19:52:51.707657 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:52:51.707625    4892 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-2d9861-openshift-default-58ddf574b9-lrwlb"]
Oct 30 19:52:51.707858 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:52:51.707837    4892 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w0-2d9861-openshift-default-58ddf574b9-lrwlb" podUID="fd2378f0-12b6-4a3a-a526-beb05ec13e02" containerName="istio-proxy" containerID="cri-o://760dd2a07edede572e89e2ad80b741b750963f274a497b5857f369d9e1d4377a" gracePeriod=30
Oct 30 19:52:52.121936 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:52:52.111215    4892 patch_prober.go:29] interesting pod/e2e-worker-w2-a3f9b7-openshift-default-7d6d8b8fb5-sfm8n container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Oct 30 19:52:52.121936 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:52:52.111283    4892 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w2-a3f9b7-openshift-default-7d6d8b8fb5-sfm8n" podUID="1c3346aa-a989-495e-9d54-66c8ec632d91" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Oct 30 19:52:52.202991 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:52:52.202672    4892 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-mt-dd9b8d79-67c8d6d46-m4wc6" podUID="3c568859-9035-498d-b347-170ab8d1b2e7" containerName="payload-processing" containerID="cri-o://37908c15ba0247793b89d486084e09149301e2174a8dd2d1818e4daa1fdf60ae" gracePeriod=25
Oct 30 19:52:52.301023 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:52:52.300956    4892 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-mt-dd9b8d79-867946f667-plb8n" podUID="f7e053ee-836a-4b0e-b2d9-88e3af83ca7b" containerName="payload-pre-processing" containerID="cri-o://14decd769a728ad5e92f5c606283d140be79657786e6905ffd6151f105b2f1fd" gracePeriod=25
Oct 30 19:52:52.484527 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:52:52.484487    4892 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-mt-dd9b8d79-867946f667-plb8n" podUID="f7e053ee-836a-4b0e-b2d9-88e3af83ca7b" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 30 19:52:52.484527 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:52:52.484499    4892 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-mt-dd9b8d79-67c8d6d46-m4wc6" podUID="3c568859-9035-498d-b347-170ab8d1b2e7" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 30 19:52:54.856548 ip-10-0-132-82 kubenswrapper[4892]: E1030 19:52:54.856507    4892 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-jrm87" containerName="extractor"
Oct 30 19:52:56.842175 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:52:56.842150    4892 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-a3f9b7-openshift-default-7d6d8b8fb5-sfm8n"
Oct 30 19:52:56.970613 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:52:56.970583    4892 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-2d9861-openshift-default-58ddf574b9-lrwlb"
Oct 30 19:52:57.030304 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:52:57.030260    4892 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/1c3346aa-a989-495e-9d54-66c8ec632d91-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1c3346aa-a989-495e-9d54-66c8ec632d91-istiod-ca-cert\") pod \"1c3346aa-a989-495e-9d54-66c8ec632d91\" (UID: \"1c3346aa-a989-495e-9d54-66c8ec632d91\") "
Oct 30 19:52:57.030481 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:52:57.030365    4892 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1c3346aa-a989-495e-9d54-66c8ec632d91-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1c3346aa-a989-495e-9d54-66c8ec632d91-credential-socket\") pod \"1c3346aa-a989-495e-9d54-66c8ec632d91\" (UID: \"1c3346aa-a989-495e-9d54-66c8ec632d91\") "
Oct 30 19:52:57.030708 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:52:57.030669    4892 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1c3346aa-a989-495e-9d54-66c8ec632d91-credential-socket" pod "1c3346aa-a989-495e-9d54-66c8ec632d91" (UID: "1c3346aa-a989-495e-9d54-66c8ec632d91"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:52:57.030708 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:52:57.030670    4892 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1c3346aa-a989-495e-9d54-66c8ec632d91-istiod-ca-cert" pod "1c3346aa-a989-495e-9d54-66c8ec632d91" (UID: "1c3346aa-a989-495e-9d54-66c8ec632d91"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 19:52:57.030851 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:52:57.030789    4892 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1c3346aa-a989-495e-9d54-66c8ec632d91-kube-api-access-9mxx5\" (UniqueName: \"kubernetes.io/projected/1c3346aa-a989-495e-9d54-66c8ec632d91-kube-api-access-9mxx5\") pod \"1c3346aa-a989-495e-9d54-66c8ec632d91\" (UID: \"1c3346aa-a989-495e-9d54-66c8ec632d91\") "
Oct 30 19:52:57.030919 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:52:57.030874    4892 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/1c3346aa-a989-495e-9d54-66c8ec632d91-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1c3346aa-a989-495e-9d54-66c8ec632d91-istio-podinfo\") pod \"1c3346aa-a989-495e-9d54-66c8ec632d91\" (UID: \"1c3346aa-a989-495e-9d54-66c8ec632d91\") "
Oct 30 19:52:57.031013 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:52:57.030945    4892 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1c3346aa-a989-495e-9d54-66c8ec632d91-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1c3346aa-a989-495e-9d54-66c8ec632d91-istio-data\") pod \"1c3346aa-a989-495e-9d54-66c8ec632d91\" (UID: \"1c3346aa-a989-495e-9d54-66c8ec632d91\") "
Oct 30 19:52:57.031079 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:52:57.031010    4892 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1c3346aa-a989-495e-9d54-66c8ec632d91-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1c3346aa-a989-495e-9d54-66c8ec632d91-workload-socket\") pod \"1c3346aa-a989-495e-9d54-66c8ec632d91\" (UID: \"1c3346aa-a989-495e-9d54-66c8ec632d91\") "
Oct 30 19:52:57.031079 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:52:57.031066    4892 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1c3346aa-a989-495e-9d54-66c8ec632d91-istio-token\" (UniqueName: \"kubernetes.io/projected/1c3346aa-a989-495e-9d54-66c8ec632d91-istio-token\") pod \"1c3346aa-a989-495e-9d54-66c8ec632d91\" (UID: \"1c3346aa-a989-495e-9d54-66c8ec632d91\") "
Oct 30 19:52:57.031202 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:52:57.031104    4892 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1c3346aa-a989-495e-9d54-66c8ec632d91-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1c3346aa-a989-495e-9d54-66c8ec632d91-workload-certs\") pod \"1c3346aa-a989-495e-9d54-66c8ec632d91\" (UID: \"1c3346aa-a989-495e-9d54-66c8ec632d91\") "
Oct 30 19:52:57.031426 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:52:57.031395    4892 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1c3346aa-a989-495e-9d54-66c8ec632d91-istio-data" pod "1c3346aa-a989-495e-9d54-66c8ec632d91" (UID: "1c3346aa-a989-495e-9d54-66c8ec632d91"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:52:57.031782 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:52:57.031719    4892 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1c3346aa-a989-495e-9d54-66c8ec632d91-workload-certs" pod "1c3346aa-a989-495e-9d54-66c8ec632d91" (UID: "1c3346aa-a989-495e-9d54-66c8ec632d91"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:52:57.031903 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:52:57.031776    4892 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1c3346aa-a989-495e-9d54-66c8ec632d91-workload-socket" pod "1c3346aa-a989-495e-9d54-66c8ec632d91" (UID: "1c3346aa-a989-495e-9d54-66c8ec632d91"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:52:57.031903 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:52:57.031832    4892 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1c3346aa-a989-495e-9d54-66c8ec632d91-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1c3346aa-a989-495e-9d54-66c8ec632d91-istio-envoy\") pod \"1c3346aa-a989-495e-9d54-66c8ec632d91\" (UID: \"1c3346aa-a989-495e-9d54-66c8ec632d91\") "
Oct 30 19:52:57.032723 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:52:57.032460    4892 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1c3346aa-a989-495e-9d54-66c8ec632d91-istiod-ca-cert\") on node \"ip-10-0-132-82.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 19:52:57.032723 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:52:57.032486    4892 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1c3346aa-a989-495e-9d54-66c8ec632d91-credential-socket\") on node \"ip-10-0-132-82.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 19:52:57.032723 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:52:57.032501    4892 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1c3346aa-a989-495e-9d54-66c8ec632d91-istio-data\") on node \"ip-10-0-132-82.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 19:52:57.032723 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:52:57.032514    4892 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1c3346aa-a989-495e-9d54-66c8ec632d91-workload-socket\") on node \"ip-10-0-132-82.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 19:52:57.032723 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:52:57.032527    4892 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1c3346aa-a989-495e-9d54-66c8ec632d91-workload-certs\") on node \"ip-10-0-132-82.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 19:52:57.034353 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:52:57.034321    4892 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1c3346aa-a989-495e-9d54-66c8ec632d91-istio-token" pod "1c3346aa-a989-495e-9d54-66c8ec632d91" (UID: "1c3346aa-a989-495e-9d54-66c8ec632d91"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:52:57.039729 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:52:57.039701    4892 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/1c3346aa-a989-495e-9d54-66c8ec632d91-istio-podinfo" pod "1c3346aa-a989-495e-9d54-66c8ec632d91" (UID: "1c3346aa-a989-495e-9d54-66c8ec632d91"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 19:52:57.039830 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:52:57.039764    4892 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1c3346aa-a989-495e-9d54-66c8ec632d91-kube-api-access-9mxx5" pod "1c3346aa-a989-495e-9d54-66c8ec632d91" (UID: "1c3346aa-a989-495e-9d54-66c8ec632d91"). InnerVolumeSpecName "kube-api-access-9mxx5". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:52:57.041283 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:52:57.041257    4892 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1c3346aa-a989-495e-9d54-66c8ec632d91-istio-envoy" pod "1c3346aa-a989-495e-9d54-66c8ec632d91" (UID: "1c3346aa-a989-495e-9d54-66c8ec632d91"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:52:57.133947 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:52:57.133916    4892 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/fd2378f0-12b6-4a3a-a526-beb05ec13e02-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/fd2378f0-12b6-4a3a-a526-beb05ec13e02-istiod-ca-cert\") pod \"fd2378f0-12b6-4a3a-a526-beb05ec13e02\" (UID: \"fd2378f0-12b6-4a3a-a526-beb05ec13e02\") "
Oct 30 19:52:57.134162 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:52:57.134032    4892 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/fd2378f0-12b6-4a3a-a526-beb05ec13e02-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/fd2378f0-12b6-4a3a-a526-beb05ec13e02-workload-certs\") pod \"fd2378f0-12b6-4a3a-a526-beb05ec13e02\" (UID: \"fd2378f0-12b6-4a3a-a526-beb05ec13e02\") "
Oct 30 19:52:57.134162 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:52:57.134127    4892 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/fd2378f0-12b6-4a3a-a526-beb05ec13e02-istio-token\" (UniqueName: \"kubernetes.io/projected/fd2378f0-12b6-4a3a-a526-beb05ec13e02-istio-token\") pod \"fd2378f0-12b6-4a3a-a526-beb05ec13e02\" (UID: \"fd2378f0-12b6-4a3a-a526-beb05ec13e02\") "
Oct 30 19:52:57.134278 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:52:57.134210    4892 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/fd2378f0-12b6-4a3a-a526-beb05ec13e02-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/fd2378f0-12b6-4a3a-a526-beb05ec13e02-istio-data\") pod \"fd2378f0-12b6-4a3a-a526-beb05ec13e02\" (UID: \"fd2378f0-12b6-4a3a-a526-beb05ec13e02\") "
Oct 30 19:52:57.134278 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:52:57.134261    4892 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/fd2378f0-12b6-4a3a-a526-beb05ec13e02-kube-api-access-54k6d\" (UniqueName: \"kubernetes.io/projected/fd2378f0-12b6-4a3a-a526-beb05ec13e02-kube-api-access-54k6d\") pod \"fd2378f0-12b6-4a3a-a526-beb05ec13e02\" (UID: \"fd2378f0-12b6-4a3a-a526-beb05ec13e02\") "
Oct 30 19:52:57.134382 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:52:57.134304    4892 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/fd2378f0-12b6-4a3a-a526-beb05ec13e02-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/fd2378f0-12b6-4a3a-a526-beb05ec13e02-istio-envoy\") pod \"fd2378f0-12b6-4a3a-a526-beb05ec13e02\" (UID: \"fd2378f0-12b6-4a3a-a526-beb05ec13e02\") "
Oct 30 19:52:57.134382 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:52:57.134338    4892 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/fd2378f0-12b6-4a3a-a526-beb05ec13e02-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/fd2378f0-12b6-4a3a-a526-beb05ec13e02-credential-socket\") pod \"fd2378f0-12b6-4a3a-a526-beb05ec13e02\" (UID: \"fd2378f0-12b6-4a3a-a526-beb05ec13e02\") "
Oct 30 19:52:57.134492 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:52:57.134378    4892 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/fd2378f0-12b6-4a3a-a526-beb05ec13e02-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/fd2378f0-12b6-4a3a-a526-beb05ec13e02-workload-socket\") pod \"fd2378f0-12b6-4a3a-a526-beb05ec13e02\" (UID: \"fd2378f0-12b6-4a3a-a526-beb05ec13e02\") "
Oct 30 19:52:57.134492 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:52:57.134383    4892 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fd2378f0-12b6-4a3a-a526-beb05ec13e02-workload-certs" pod "fd2378f0-12b6-4a3a-a526-beb05ec13e02" (UID: "fd2378f0-12b6-4a3a-a526-beb05ec13e02"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:52:57.134492 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:52:57.134399    4892 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/fd2378f0-12b6-4a3a-a526-beb05ec13e02-istiod-ca-cert" pod "fd2378f0-12b6-4a3a-a526-beb05ec13e02" (UID: "fd2378f0-12b6-4a3a-a526-beb05ec13e02"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 19:52:57.134492 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:52:57.134467    4892 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/fd2378f0-12b6-4a3a-a526-beb05ec13e02-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/fd2378f0-12b6-4a3a-a526-beb05ec13e02-istio-podinfo\") pod \"fd2378f0-12b6-4a3a-a526-beb05ec13e02\" (UID: \"fd2378f0-12b6-4a3a-a526-beb05ec13e02\") "
Oct 30 19:52:57.134934 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:52:57.134911    4892 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fd2378f0-12b6-4a3a-a526-beb05ec13e02-credential-socket" pod "fd2378f0-12b6-4a3a-a526-beb05ec13e02" (UID: "fd2378f0-12b6-4a3a-a526-beb05ec13e02"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:52:57.135179 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:52:57.135158    4892 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/fd2378f0-12b6-4a3a-a526-beb05ec13e02-workload-certs\") on node \"ip-10-0-132-82.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 19:52:57.135261 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:52:57.135185    4892 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1c3346aa-a989-495e-9d54-66c8ec632d91-istio-token\") on node \"ip-10-0-132-82.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 19:52:57.135261 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:52:57.135200    4892 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1c3346aa-a989-495e-9d54-66c8ec632d91-istio-envoy\") on node \"ip-10-0-132-82.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 19:52:57.135261 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:52:57.135225    4892 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/fd2378f0-12b6-4a3a-a526-beb05ec13e02-credential-socket\") on node \"ip-10-0-132-82.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 19:52:57.135261 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:52:57.135240    4892 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9mxx5\" (UniqueName: \"kubernetes.io/projected/1c3346aa-a989-495e-9d54-66c8ec632d91-kube-api-access-9mxx5\") on node \"ip-10-0-132-82.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 19:52:57.135261 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:52:57.135254    4892 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1c3346aa-a989-495e-9d54-66c8ec632d91-istio-podinfo\") on node \"ip-10-0-132-82.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 19:52:57.135500 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:52:57.135268    4892 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/fd2378f0-12b6-4a3a-a526-beb05ec13e02-istiod-ca-cert\") on node \"ip-10-0-132-82.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 19:52:57.135500 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:52:57.135462    4892 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fd2378f0-12b6-4a3a-a526-beb05ec13e02-workload-socket" pod "fd2378f0-12b6-4a3a-a526-beb05ec13e02" (UID: "fd2378f0-12b6-4a3a-a526-beb05ec13e02"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:52:57.135753 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:52:57.135730    4892 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fd2378f0-12b6-4a3a-a526-beb05ec13e02-istio-data" pod "fd2378f0-12b6-4a3a-a526-beb05ec13e02" (UID: "fd2378f0-12b6-4a3a-a526-beb05ec13e02"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:52:57.137739 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:52:57.137700    4892 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/fd2378f0-12b6-4a3a-a526-beb05ec13e02-istio-podinfo" pod "fd2378f0-12b6-4a3a-a526-beb05ec13e02" (UID: "fd2378f0-12b6-4a3a-a526-beb05ec13e02"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 19:52:57.137739 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:52:57.137697    4892 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fd2378f0-12b6-4a3a-a526-beb05ec13e02-istio-envoy" pod "fd2378f0-12b6-4a3a-a526-beb05ec13e02" (UID: "fd2378f0-12b6-4a3a-a526-beb05ec13e02"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:52:57.138109 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:52:57.138091    4892 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fd2378f0-12b6-4a3a-a526-beb05ec13e02-istio-token" pod "fd2378f0-12b6-4a3a-a526-beb05ec13e02" (UID: "fd2378f0-12b6-4a3a-a526-beb05ec13e02"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:52:57.138343 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:52:57.138328    4892 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fd2378f0-12b6-4a3a-a526-beb05ec13e02-kube-api-access-54k6d" pod "fd2378f0-12b6-4a3a-a526-beb05ec13e02" (UID: "fd2378f0-12b6-4a3a-a526-beb05ec13e02"). InnerVolumeSpecName "kube-api-access-54k6d". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:52:57.236544 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:52:57.236512    4892 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/fd2378f0-12b6-4a3a-a526-beb05ec13e02-istio-token\") on node \"ip-10-0-132-82.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 19:52:57.236544 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:52:57.236541    4892 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/fd2378f0-12b6-4a3a-a526-beb05ec13e02-istio-data\") on node \"ip-10-0-132-82.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 19:52:57.236544 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:52:57.236552    4892 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-54k6d\" (UniqueName: \"kubernetes.io/projected/fd2378f0-12b6-4a3a-a526-beb05ec13e02-kube-api-access-54k6d\") on node \"ip-10-0-132-82.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 19:52:57.236860 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:52:57.236567    4892 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/fd2378f0-12b6-4a3a-a526-beb05ec13e02-istio-envoy\") on node \"ip-10-0-132-82.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 19:52:57.236860 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:52:57.236580    4892 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/fd2378f0-12b6-4a3a-a526-beb05ec13e02-workload-socket\") on node \"ip-10-0-132-82.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 19:52:57.236860 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:52:57.236591    4892 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/fd2378f0-12b6-4a3a-a526-beb05ec13e02-istio-podinfo\") on node \"ip-10-0-132-82.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 19:52:57.273179 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:52:57.273149    4892 scope.go:122] "RemoveContainer" containerID="0c4417cc4bb8af483b3db4a9fd5ae18b438a366697c09861876a6f417453a841"
Oct 30 19:52:57.284439 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:52:57.284421    4892 scope.go:122] "RemoveContainer" containerID="760dd2a07edede572e89e2ad80b741b750963f274a497b5857f369d9e1d4377a"
Oct 30 19:52:57.383693 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:52:57.383658    4892 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-a3f9b7-openshift-default-7d6d8b8fb5-sfm8n" event={"ID":"1c3346aa-a989-495e-9d54-66c8ec632d91","Type":"ContainerDied","Data":"0c4417cc4bb8af483b3db4a9fd5ae18b438a366697c09861876a6f417453a841"}
Oct 30 19:52:57.383913 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:52:57.383716    4892 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-a3f9b7-openshift-default-7d6d8b8fb5-sfm8n" event={"ID":"1c3346aa-a989-495e-9d54-66c8ec632d91","Type":"ContainerDied","Data":"2113779abae42b583b9336b113c943eeb62b8e81401d928b1da7c9abf7cdc5ed"}
Oct 30 19:52:57.383913 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:52:57.383768    4892 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-2d9861-openshift-default-58ddf574b9-lrwlb" event={"ID":"fd2378f0-12b6-4a3a-a526-beb05ec13e02","Type":"ContainerDied","Data":"760dd2a07edede572e89e2ad80b741b750963f274a497b5857f369d9e1d4377a"}
Oct 30 19:52:57.383913 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:52:57.383801    4892 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-a3f9b7-openshift-default-7d6d8b8fb5-sfm8n"
Oct 30 19:52:57.383913 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:52:57.383828    4892 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-2d9861-openshift-default-58ddf574b9-lrwlb" event={"ID":"fd2378f0-12b6-4a3a-a526-beb05ec13e02","Type":"ContainerDied","Data":"823ece61e9500ad7b075eb2e2f78eb18567cd129bb68a3041207eedbf339f464"}
Oct 30 19:52:57.384200 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:52:57.384085    4892 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-2d9861-openshift-default-58ddf574b9-lrwlb"
Oct 30 19:52:57.667891 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:52:57.667815    4892 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-a3f9b7-openshift-default-7d6d8b8fb5-sfm8n"]
Oct 30 19:52:57.673357 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:52:57.673319    4892 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w2-a3f9b7-openshift-default-7d6d8b8fb5-sfm8n"]
Oct 30 19:52:57.847356 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:52:57.847323    4892 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-2d9861-openshift-default-58ddf574b9-lrwlb"]
Oct 30 19:52:57.853046 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:52:57.853018    4892 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w0-2d9861-openshift-default-58ddf574b9-lrwlb"]
Oct 30 19:52:58.573861 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:52:58.573828    4892 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="1c3346aa-a989-495e-9d54-66c8ec632d91" path="/var/lib/kubelet/pods/1c3346aa-a989-495e-9d54-66c8ec632d91/volumes"
Oct 30 19:52:58.574388 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:52:58.574368    4892 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="fd2378f0-12b6-4a3a-a526-beb05ec13e02" path="/var/lib/kubelet/pods/fd2378f0-12b6-4a3a-a526-beb05ec13e02/volumes"
Oct 30 19:52:59.803721 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:52:59.803686    4892 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-inf-a-898968-openshift-default-b55dfc9c9-gmck5"]
Oct 30 19:52:59.804783 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:52:59.804759    4892 cpu_manager.go:414] "containerMap: removing container" podUID="1c3346aa-a989-495e-9d54-66c8ec632d91" containerName="istio-proxy"
Oct 30 19:52:59.804862 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:52:59.804787    4892 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1c3346aa-a989-495e-9d54-66c8ec632d91" containerName="istio-proxy"
Oct 30 19:52:59.804862 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:52:59.804820    4892 cpu_manager.go:414] "containerMap: removing container" podUID="fd2378f0-12b6-4a3a-a526-beb05ec13e02" containerName="istio-proxy"
Oct 30 19:52:59.804862 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:52:59.804828    4892 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="fd2378f0-12b6-4a3a-a526-beb05ec13e02" containerName="istio-proxy"
Oct 30 19:52:59.804862 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:52:59.804849    4892 cpu_manager.go:414] "containerMap: removing container" podUID="6539580c-56b5-4501-abe2-116743a7096b" containerName="revoke-keys"
Oct 30 19:52:59.804862 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:52:59.804857    4892 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6539580c-56b5-4501-abe2-116743a7096b" containerName="revoke-keys"
Oct 30 19:52:59.805072 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:52:59.805037    4892 memory_manager.go:365] "RemoveStaleState removing state" podUID="1c3346aa-a989-495e-9d54-66c8ec632d91" containerName="istio-proxy"
Oct 30 19:52:59.805072 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:52:59.805056    4892 memory_manager.go:365] "RemoveStaleState removing state" podUID="fd2378f0-12b6-4a3a-a526-beb05ec13e02" containerName="istio-proxy"
Oct 30 19:52:59.805072 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:52:59.805069    4892 memory_manager.go:365] "RemoveStaleState removing state" podUID="6539580c-56b5-4501-abe2-116743a7096b" containerName="revoke-keys"
Oct 30 19:52:59.810913 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:52:59.810889    4892 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-898968-openshift-default-b55dfc9c9-gmck5"
Oct 30 19:52:59.864449 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:52:59.864400    4892 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-inf-a-898968-openshift-default-dockercfg-8k7l6\""
Oct 30 19:52:59.897833 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:52:59.897791    4892 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/95cba503-314a-4b92-a90b-c23b551ea329-istio-podinfo\") pod \"e2e-inf-a-898968-openshift-default-b55dfc9c9-gmck5\" (UID: \"95cba503-314a-4b92-a90b-c23b551ea329\") " pod="openshift-ingress/e2e-inf-a-898968-openshift-default-b55dfc9c9-gmck5"
Oct 30 19:52:59.897833 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:52:59.897843    4892 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/95cba503-314a-4b92-a90b-c23b551ea329-workload-certs\") pod \"e2e-inf-a-898968-openshift-default-b55dfc9c9-gmck5\" (UID: \"95cba503-314a-4b92-a90b-c23b551ea329\") " pod="openshift-ingress/e2e-inf-a-898968-openshift-default-b55dfc9c9-gmck5"
Oct 30 19:52:59.898071 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:52:59.898042    4892 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/95cba503-314a-4b92-a90b-c23b551ea329-istiod-ca-cert\") pod \"e2e-inf-a-898968-openshift-default-b55dfc9c9-gmck5\" (UID: \"95cba503-314a-4b92-a90b-c23b551ea329\") " pod="openshift-ingress/e2e-inf-a-898968-openshift-default-b55dfc9c9-gmck5"
Oct 30 19:52:59.898183 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:52:59.898166    4892 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/95cba503-314a-4b92-a90b-c23b551ea329-istio-token\") pod \"e2e-inf-a-898968-openshift-default-b55dfc9c9-gmck5\" (UID: \"95cba503-314a-4b92-a90b-c23b551ea329\") " pod="openshift-ingress/e2e-inf-a-898968-openshift-default-b55dfc9c9-gmck5"
Oct 30 19:52:59.898247 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:52:59.898233    4892 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dcqqd\" (UniqueName: \"kubernetes.io/projected/95cba503-314a-4b92-a90b-c23b551ea329-kube-api-access-dcqqd\") pod \"e2e-inf-a-898968-openshift-default-b55dfc9c9-gmck5\" (UID: \"95cba503-314a-4b92-a90b-c23b551ea329\") " pod="openshift-ingress/e2e-inf-a-898968-openshift-default-b55dfc9c9-gmck5"
Oct 30 19:52:59.898302 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:52:59.898263    4892 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/95cba503-314a-4b92-a90b-c23b551ea329-workload-socket\") pod \"e2e-inf-a-898968-openshift-default-b55dfc9c9-gmck5\" (UID: \"95cba503-314a-4b92-a90b-c23b551ea329\") " pod="openshift-ingress/e2e-inf-a-898968-openshift-default-b55dfc9c9-gmck5"
Oct 30 19:52:59.898380 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:52:59.898362    4892 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/95cba503-314a-4b92-a90b-c23b551ea329-credential-socket\") pod \"e2e-inf-a-898968-openshift-default-b55dfc9c9-gmck5\" (UID: \"95cba503-314a-4b92-a90b-c23b551ea329\") " pod="openshift-ingress/e2e-inf-a-898968-openshift-default-b55dfc9c9-gmck5"
Oct 30 19:52:59.898441 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:52:59.898423    4892 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/95cba503-314a-4b92-a90b-c23b551ea329-istio-data\") pod \"e2e-inf-a-898968-openshift-default-b55dfc9c9-gmck5\" (UID: \"95cba503-314a-4b92-a90b-c23b551ea329\") " pod="openshift-ingress/e2e-inf-a-898968-openshift-default-b55dfc9c9-gmck5"
Oct 30 19:52:59.898603 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:52:59.898580    4892 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/95cba503-314a-4b92-a90b-c23b551ea329-istio-envoy\") pod \"e2e-inf-a-898968-openshift-default-b55dfc9c9-gmck5\" (UID: \"95cba503-314a-4b92-a90b-c23b551ea329\") " pod="openshift-ingress/e2e-inf-a-898968-openshift-default-b55dfc9c9-gmck5"
Oct 30 19:52:59.917750 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:52:59.917721    4892 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-a-898968-openshift-default-b55dfc9c9-gmck5"]
Oct 30 19:52:59.999646 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:52:59.999619    4892 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/95cba503-314a-4b92-a90b-c23b551ea329-credential-socket\") pod \"e2e-inf-a-898968-openshift-default-b55dfc9c9-gmck5\" (UID: \"95cba503-314a-4b92-a90b-c23b551ea329\") " pod="openshift-ingress/e2e-inf-a-898968-openshift-default-b55dfc9c9-gmck5"
Oct 30 19:52:59.999836 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:52:59.999665    4892 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/95cba503-314a-4b92-a90b-c23b551ea329-istio-data\") pod \"e2e-inf-a-898968-openshift-default-b55dfc9c9-gmck5\" (UID: \"95cba503-314a-4b92-a90b-c23b551ea329\") " pod="openshift-ingress/e2e-inf-a-898968-openshift-default-b55dfc9c9-gmck5"
Oct 30 19:52:59.999836 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:52:59.999737    4892 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/95cba503-314a-4b92-a90b-c23b551ea329-istio-envoy\") pod \"e2e-inf-a-898968-openshift-default-b55dfc9c9-gmck5\" (UID: \"95cba503-314a-4b92-a90b-c23b551ea329\") " pod="openshift-ingress/e2e-inf-a-898968-openshift-default-b55dfc9c9-gmck5"
Oct 30 19:52:59.999836 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:52:59.999774    4892 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/95cba503-314a-4b92-a90b-c23b551ea329-istio-podinfo\") pod \"e2e-inf-a-898968-openshift-default-b55dfc9c9-gmck5\" (UID: \"95cba503-314a-4b92-a90b-c23b551ea329\") " pod="openshift-ingress/e2e-inf-a-898968-openshift-default-b55dfc9c9-gmck5"
Oct 30 19:52:59.999836 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:52:59.999801    4892 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/95cba503-314a-4b92-a90b-c23b551ea329-workload-certs\") pod \"e2e-inf-a-898968-openshift-default-b55dfc9c9-gmck5\" (UID: \"95cba503-314a-4b92-a90b-c23b551ea329\") " pod="openshift-ingress/e2e-inf-a-898968-openshift-default-b55dfc9c9-gmck5"
Oct 30 19:53:00.000112 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:52:59.999850    4892 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/95cba503-314a-4b92-a90b-c23b551ea329-istiod-ca-cert\") pod \"e2e-inf-a-898968-openshift-default-b55dfc9c9-gmck5\" (UID: \"95cba503-314a-4b92-a90b-c23b551ea329\") " pod="openshift-ingress/e2e-inf-a-898968-openshift-default-b55dfc9c9-gmck5"
Oct 30 19:53:00.000112 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:52:59.999896    4892 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/95cba503-314a-4b92-a90b-c23b551ea329-istio-token\") pod \"e2e-inf-a-898968-openshift-default-b55dfc9c9-gmck5\" (UID: \"95cba503-314a-4b92-a90b-c23b551ea329\") " pod="openshift-ingress/e2e-inf-a-898968-openshift-default-b55dfc9c9-gmck5"
Oct 30 19:53:00.000112 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:52:59.999944    4892 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dcqqd\" (UniqueName: \"kubernetes.io/projected/95cba503-314a-4b92-a90b-c23b551ea329-kube-api-access-dcqqd\") pod \"e2e-inf-a-898968-openshift-default-b55dfc9c9-gmck5\" (UID: \"95cba503-314a-4b92-a90b-c23b551ea329\") " pod="openshift-ingress/e2e-inf-a-898968-openshift-default-b55dfc9c9-gmck5"
Oct 30 19:53:00.000112 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:52:59.999998    4892 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/95cba503-314a-4b92-a90b-c23b551ea329-workload-socket\") pod \"e2e-inf-a-898968-openshift-default-b55dfc9c9-gmck5\" (UID: \"95cba503-314a-4b92-a90b-c23b551ea329\") " pod="openshift-ingress/e2e-inf-a-898968-openshift-default-b55dfc9c9-gmck5"
Oct 30 19:53:00.000285 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:53:00.000135    4892 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/95cba503-314a-4b92-a90b-c23b551ea329-credential-socket\") pod \"e2e-inf-a-898968-openshift-default-b55dfc9c9-gmck5\" (UID: \"95cba503-314a-4b92-a90b-c23b551ea329\") " pod="openshift-ingress/e2e-inf-a-898968-openshift-default-b55dfc9c9-gmck5"
Oct 30 19:53:00.000285 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:53:00.000160    4892 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/95cba503-314a-4b92-a90b-c23b551ea329-istio-data\") pod \"e2e-inf-a-898968-openshift-default-b55dfc9c9-gmck5\" (UID: \"95cba503-314a-4b92-a90b-c23b551ea329\") " pod="openshift-ingress/e2e-inf-a-898968-openshift-default-b55dfc9c9-gmck5"
Oct 30 19:53:00.000285 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:53:00.000212    4892 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/95cba503-314a-4b92-a90b-c23b551ea329-workload-certs\") pod \"e2e-inf-a-898968-openshift-default-b55dfc9c9-gmck5\" (UID: \"95cba503-314a-4b92-a90b-c23b551ea329\") " pod="openshift-ingress/e2e-inf-a-898968-openshift-default-b55dfc9c9-gmck5"
Oct 30 19:53:00.000424 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:53:00.000307    4892 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/95cba503-314a-4b92-a90b-c23b551ea329-workload-socket\") pod \"e2e-inf-a-898968-openshift-default-b55dfc9c9-gmck5\" (UID: \"95cba503-314a-4b92-a90b-c23b551ea329\") " pod="openshift-ingress/e2e-inf-a-898968-openshift-default-b55dfc9c9-gmck5"
Oct 30 19:53:00.000785 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:53:00.000762    4892 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/95cba503-314a-4b92-a90b-c23b551ea329-istiod-ca-cert\") pod \"e2e-inf-a-898968-openshift-default-b55dfc9c9-gmck5\" (UID: \"95cba503-314a-4b92-a90b-c23b551ea329\") " pod="openshift-ingress/e2e-inf-a-898968-openshift-default-b55dfc9c9-gmck5"
Oct 30 19:53:00.002456 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:53:00.002429    4892 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/95cba503-314a-4b92-a90b-c23b551ea329-istio-envoy\") pod \"e2e-inf-a-898968-openshift-default-b55dfc9c9-gmck5\" (UID: \"95cba503-314a-4b92-a90b-c23b551ea329\") " pod="openshift-ingress/e2e-inf-a-898968-openshift-default-b55dfc9c9-gmck5"
Oct 30 19:53:00.003007 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:53:00.002953    4892 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/95cba503-314a-4b92-a90b-c23b551ea329-istio-podinfo\") pod \"e2e-inf-a-898968-openshift-default-b55dfc9c9-gmck5\" (UID: \"95cba503-314a-4b92-a90b-c23b551ea329\") " pod="openshift-ingress/e2e-inf-a-898968-openshift-default-b55dfc9c9-gmck5"
Oct 30 19:53:00.111024 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:53:00.110929    4892 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/95cba503-314a-4b92-a90b-c23b551ea329-istio-token\") pod \"e2e-inf-a-898968-openshift-default-b55dfc9c9-gmck5\" (UID: \"95cba503-314a-4b92-a90b-c23b551ea329\") " pod="openshift-ingress/e2e-inf-a-898968-openshift-default-b55dfc9c9-gmck5"
Oct 30 19:53:00.112222 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:53:00.112189    4892 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dcqqd\" (UniqueName: \"kubernetes.io/projected/95cba503-314a-4b92-a90b-c23b551ea329-kube-api-access-dcqqd\") pod \"e2e-inf-a-898968-openshift-default-b55dfc9c9-gmck5\" (UID: \"95cba503-314a-4b92-a90b-c23b551ea329\") " pod="openshift-ingress/e2e-inf-a-898968-openshift-default-b55dfc9c9-gmck5"
Oct 30 19:53:00.127844 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:53:00.127246    4892 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-898968-openshift-default-b55dfc9c9-gmck5"
Oct 30 19:53:00.604614 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:53:00.604583    4892 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-a-898968-openshift-default-b55dfc9c9-gmck5"]
Oct 30 19:53:00.643046 ip-10-0-132-82 kubenswrapper[4892]: W1030 19:53:00.643015    4892 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod95cba503_314a_4b92_a90b_c23b551ea329.slice/crio-e45e1352a94cc464f084d485f363341efd16ccd6e76f9d80586719b3a0960805 WatchSource:0}: Error finding container e45e1352a94cc464f084d485f363341efd16ccd6e76f9d80586719b3a0960805: Status 404 returned error can't find the container with id e45e1352a94cc464f084d485f363341efd16ccd6e76f9d80586719b3a0960805
Oct 30 19:53:00.646782 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:53:00.646736    4892 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892000Ki","pods":"250"}
Oct 30 19:53:00.646913 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:53:00.646836    4892 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892000Ki","pods":"250"}
Oct 30 19:53:00.646913 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:53:00.646890    4892 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892000Ki","pods":"250"}
Oct 30 19:53:00.850211 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:53:00.850175    4892 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-fsfvk"]
Oct 30 19:53:01.441049 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:53:01.441001    4892 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-898968-openshift-default-b55dfc9c9-gmck5" event={"ID":"95cba503-314a-4b92-a90b-c23b551ea329","Type":"ContainerStarted","Data":"30d80d93579e16c3d80d44735723524b0cb4e15e7b0540f54d72969c46e9ec1e"}
Oct 30 19:53:01.441237 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:53:01.441076    4892 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-898968-openshift-default-b55dfc9c9-gmck5" event={"ID":"95cba503-314a-4b92-a90b-c23b551ea329","Type":"ContainerStarted","Data":"e45e1352a94cc464f084d485f363341efd16ccd6e76f9d80586719b3a0960805"}
Oct 30 19:53:01.657405 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:53:01.657356    4892 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-inf-a-898968-openshift-default-b55dfc9c9-gmck5" podStartSLOduration=2.657342452 podStartE2EDuration="2.657342452s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:52:59 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:53:01.605143996 +0000 UTC m=+2229.175609594" watchObservedRunningTime="2026-10-30 19:53:01.657342452 +0000 UTC m=+2229.227808051"
Oct 30 19:53:02.134360 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:53:02.134314    4892 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-a-898968-openshift-default-b55dfc9c9-gmck5"
Oct 30 19:53:02.166051 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:53:02.154413    4892 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-a-898968-openshift-default-b55dfc9c9-gmck5"
Oct 30 19:53:02.266110 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:53:02.266012    4892 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-1c5537-6d5ffd475f-8jtnh"]
Oct 30 19:53:02.273720 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:53:02.272003    4892 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-b-1c5537-6d5ffd475f-8jtnh"
Oct 30 19:53:02.303987 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:53:02.303629    4892 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-1c5537-6d5ffd475f-8jtnh"]
Oct 30 19:53:02.341988 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:53:02.325521    4892 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-api-b-1c5537-dockercfg-hg6w4\""
Oct 30 19:53:02.341988 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:53:02.325608    4892 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-api-b-1c5537\""
Oct 30 19:53:02.490263 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:53:02.476739    4892 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-1c5537-7f86d78844-ljthj"]
Oct 30 19:53:02.539269 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:53:02.539222    4892 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/81a5c6ce-725d-42b3-8bba-1f80fb5edfad-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-b-1c5537-6d5ffd475f-8jtnh\" (UID: \"81a5c6ce-725d-42b3-8bba-1f80fb5edfad\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-1c5537-6d5ffd475f-8jtnh"
Oct 30 19:53:02.539504 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:53:02.539471    4892 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2cb52\" (UniqueName: \"kubernetes.io/projected/81a5c6ce-725d-42b3-8bba-1f80fb5edfad-kube-api-access-2cb52\") pod \"payload-pre-processing-e2e-api-b-1c5537-6d5ffd475f-8jtnh\" (UID: \"81a5c6ce-725d-42b3-8bba-1f80fb5edfad\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-1c5537-6d5ffd475f-8jtnh"
Oct 30 19:53:02.540628 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:53:02.540588    4892 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-mt-dd9b8d79-867946f667-plb8n" podUID="f7e053ee-836a-4b0e-b2d9-88e3af83ca7b" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 30 19:53:02.545433 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:53:02.545380    4892 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-mt-dd9b8d79-67c8d6d46-m4wc6" podUID="3c568859-9035-498d-b347-170ab8d1b2e7" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 30 19:53:02.549431 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:53:02.548009    4892 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-b-1c5537-7f86d78844-ljthj"
Oct 30 19:53:02.561584 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:53:02.561561    4892 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-1c5537-7f86d78844-ljthj"]
Oct 30 19:53:02.576394 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:53:02.574636    4892 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-a-898968-openshift-default-b55dfc9c9-gmck5"
Oct 30 19:53:02.576394 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:53:02.574688    4892 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-a-898968-openshift-default-b55dfc9c9-gmck5"
Oct 30 19:53:02.641389 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:53:02.641360    4892 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/81a5c6ce-725d-42b3-8bba-1f80fb5edfad-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-b-1c5537-6d5ffd475f-8jtnh\" (UID: \"81a5c6ce-725d-42b3-8bba-1f80fb5edfad\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-1c5537-6d5ffd475f-8jtnh"
Oct 30 19:53:02.641552 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:53:02.641438    4892 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2cb52\" (UniqueName: \"kubernetes.io/projected/81a5c6ce-725d-42b3-8bba-1f80fb5edfad-kube-api-access-2cb52\") pod \"payload-pre-processing-e2e-api-b-1c5537-6d5ffd475f-8jtnh\" (UID: \"81a5c6ce-725d-42b3-8bba-1f80fb5edfad\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-1c5537-6d5ffd475f-8jtnh"
Oct 30 19:53:02.642053 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:53:02.642025    4892 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/81a5c6ce-725d-42b3-8bba-1f80fb5edfad-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-b-1c5537-6d5ffd475f-8jtnh\" (UID: \"81a5c6ce-725d-42b3-8bba-1f80fb5edfad\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-1c5537-6d5ffd475f-8jtnh"
Oct 30 19:53:02.698329 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:53:02.698304    4892 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2cb52\" (UniqueName: \"kubernetes.io/projected/81a5c6ce-725d-42b3-8bba-1f80fb5edfad-kube-api-access-2cb52\") pod \"payload-pre-processing-e2e-api-b-1c5537-6d5ffd475f-8jtnh\" (UID: \"81a5c6ce-725d-42b3-8bba-1f80fb5edfad\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-1c5537-6d5ffd475f-8jtnh"
Oct 30 19:53:02.742577 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:53:02.742502    4892 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-brwx6\" (UniqueName: \"kubernetes.io/projected/1ca3885d-5fb3-4c84-b1fc-2e9138885566-kube-api-access-brwx6\") pod \"payload-processing-e2e-api-b-1c5537-7f86d78844-ljthj\" (UID: \"1ca3885d-5fb3-4c84-b1fc-2e9138885566\") " pod="openshift-ingress/payload-processing-e2e-api-b-1c5537-7f86d78844-ljthj"
Oct 30 19:53:02.742832 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:53:02.742804    4892 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1ca3885d-5fb3-4c84-b1fc-2e9138885566-plugins-config-volume\") pod \"payload-processing-e2e-api-b-1c5537-7f86d78844-ljthj\" (UID: \"1ca3885d-5fb3-4c84-b1fc-2e9138885566\") " pod="openshift-ingress/payload-processing-e2e-api-b-1c5537-7f86d78844-ljthj"
Oct 30 19:53:02.844161 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:53:02.844121    4892 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1ca3885d-5fb3-4c84-b1fc-2e9138885566-plugins-config-volume\") pod \"payload-processing-e2e-api-b-1c5537-7f86d78844-ljthj\" (UID: \"1ca3885d-5fb3-4c84-b1fc-2e9138885566\") " pod="openshift-ingress/payload-processing-e2e-api-b-1c5537-7f86d78844-ljthj"
Oct 30 19:53:02.844407 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:53:02.844376    4892 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-brwx6\" (UniqueName: \"kubernetes.io/projected/1ca3885d-5fb3-4c84-b1fc-2e9138885566-kube-api-access-brwx6\") pod \"payload-processing-e2e-api-b-1c5537-7f86d78844-ljthj\" (UID: \"1ca3885d-5fb3-4c84-b1fc-2e9138885566\") " pod="openshift-ingress/payload-processing-e2e-api-b-1c5537-7f86d78844-ljthj"
Oct 30 19:53:02.844993 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:53:02.844944    4892 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1ca3885d-5fb3-4c84-b1fc-2e9138885566-plugins-config-volume\") pod \"payload-processing-e2e-api-b-1c5537-7f86d78844-ljthj\" (UID: \"1ca3885d-5fb3-4c84-b1fc-2e9138885566\") " pod="openshift-ingress/payload-processing-e2e-api-b-1c5537-7f86d78844-ljthj"
Oct 30 19:53:02.901558 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:53:02.901524    4892 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-brwx6\" (UniqueName: \"kubernetes.io/projected/1ca3885d-5fb3-4c84-b1fc-2e9138885566-kube-api-access-brwx6\") pod \"payload-processing-e2e-api-b-1c5537-7f86d78844-ljthj\" (UID: \"1ca3885d-5fb3-4c84-b1fc-2e9138885566\") " pod="openshift-ingress/payload-processing-e2e-api-b-1c5537-7f86d78844-ljthj"
Oct 30 19:53:02.907236 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:53:02.907209    4892 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-b-1c5537-6d5ffd475f-8jtnh"
Oct 30 19:53:03.172080 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:53:03.172049    4892 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-b-1c5537-7f86d78844-ljthj"
Oct 30 19:53:03.312098 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:53:03.312018    4892 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-1c5537-6d5ffd475f-8jtnh"]
Oct 30 19:53:03.320318 ip-10-0-132-82 kubenswrapper[4892]: W1030 19:53:03.320290    4892 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod81a5c6ce_725d_42b3_8bba_1f80fb5edfad.slice/crio-768b4e2d6742c5af526e3fdb2b1941959b9b704942403541f5417dfab6f94690 WatchSource:0}: Error finding container 768b4e2d6742c5af526e3fdb2b1941959b9b704942403541f5417dfab6f94690: Status 404 returned error can't find the container with id 768b4e2d6742c5af526e3fdb2b1941959b9b704942403541f5417dfab6f94690
Oct 30 19:53:03.552155 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:53:03.552132    4892 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-1c5537-7f86d78844-ljthj"]
Oct 30 19:53:03.552364 ip-10-0-132-82 kubenswrapper[4892]: W1030 19:53:03.552336    4892 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1ca3885d_5fb3_4c84_b1fc_2e9138885566.slice/crio-ae9366dd0d9e939c5a6b0ca53fd958bbac0b83a35ca20dd846f757f75a59c631 WatchSource:0}: Error finding container ae9366dd0d9e939c5a6b0ca53fd958bbac0b83a35ca20dd846f757f75a59c631: Status 404 returned error can't find the container with id ae9366dd0d9e939c5a6b0ca53fd958bbac0b83a35ca20dd846f757f75a59c631
Oct 30 19:53:03.575802 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:53:03.575734    4892 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-b-1c5537-7f86d78844-ljthj" event={"ID":"1ca3885d-5fb3-4c84-b1fc-2e9138885566","Type":"ContainerStarted","Data":"ae9366dd0d9e939c5a6b0ca53fd958bbac0b83a35ca20dd846f757f75a59c631"}
Oct 30 19:53:03.577886 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:53:03.577840    4892 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-b-1c5537-6d5ffd475f-8jtnh" event={"ID":"81a5c6ce-725d-42b3-8bba-1f80fb5edfad","Type":"ContainerStarted","Data":"693139186d056a664db68dfaa475f1d09fbb7de7e57bc6ff817b6109ec03f6d4"}
Oct 30 19:53:03.578023 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:53:03.577900    4892 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-b-1c5537-6d5ffd475f-8jtnh" event={"ID":"81a5c6ce-725d-42b3-8bba-1f80fb5edfad","Type":"ContainerStarted","Data":"768b4e2d6742c5af526e3fdb2b1941959b9b704942403541f5417dfab6f94690"}
Oct 30 19:53:03.578347 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:53:03.578326    4892 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-api-b-1c5537-6d5ffd475f-8jtnh"
Oct 30 19:53:03.710777 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:53:03.710725    4892 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-api-b-1c5537-6d5ffd475f-8jtnh" podStartSLOduration=1.71071121 podStartE2EDuration="1.71071121s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:53:02 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:53:03.706916899 +0000 UTC m=+2231.277382499" watchObservedRunningTime="2026-10-30 19:53:03.71071121 +0000 UTC m=+2231.281176808"
Oct 30 19:53:04.586485 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:53:04.586456    4892 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-b-1c5537-7f86d78844-ljthj" event={"ID":"1ca3885d-5fb3-4c84-b1fc-2e9138885566","Type":"ContainerStarted","Data":"1646839ada220254ef1045f78b020e60b97cf9fdea4608a14a738e1817b08f8c"}
Oct 30 19:53:04.586922 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:53:04.586523    4892 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-api-b-1c5537-7f86d78844-ljthj"
Oct 30 19:53:04.703917 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:53:04.703863    4892 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-api-b-1c5537-7f86d78844-ljthj" podStartSLOduration=2.703848117 podStartE2EDuration="2.703848117s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:53:02 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:53:04.703121574 +0000 UTC m=+2232.273587173" watchObservedRunningTime="2026-10-30 19:53:04.703848117 +0000 UTC m=+2232.274313717"
Oct 30 19:53:08.109103 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:53:08.109069    4892 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-898968-868d54644c-j8d79"]
Oct 30 19:53:08.113479 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:53:08.113460    4892 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-898968-868d54644c-j8d79"
Oct 30 19:53:08.152293 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:53:08.152260    4892 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6bc6af34-1b37-4634-a8b0-932b105ef626-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-a-898968-868d54644c-j8d79\" (UID: \"6bc6af34-1b37-4634-a8b0-932b105ef626\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-898968-868d54644c-j8d79"
Oct 30 19:53:08.152422 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:53:08.152338    4892 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-q28w8\" (UniqueName: \"kubernetes.io/projected/6bc6af34-1b37-4634-a8b0-932b105ef626-kube-api-access-q28w8\") pod \"payload-pre-processing-e2e-inf-a-898968-868d54644c-j8d79\" (UID: \"6bc6af34-1b37-4634-a8b0-932b105ef626\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-898968-868d54644c-j8d79"
Oct 30 19:53:08.165232 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:53:08.165211    4892 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-inf-a-898968\""
Oct 30 19:53:08.225556 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:53:08.225524    4892 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-inf-a-898968-dockercfg-599f8\""
Oct 30 19:53:08.257988 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:53:08.255515    4892 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-898968-868d54644c-j8d79"]
Oct 30 19:53:08.257988 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:53:08.256407    4892 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6bc6af34-1b37-4634-a8b0-932b105ef626-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-a-898968-868d54644c-j8d79\" (UID: \"6bc6af34-1b37-4634-a8b0-932b105ef626\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-898968-868d54644c-j8d79"
Oct 30 19:53:08.257988 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:53:08.256495    4892 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-q28w8\" (UniqueName: \"kubernetes.io/projected/6bc6af34-1b37-4634-a8b0-932b105ef626-kube-api-access-q28w8\") pod \"payload-pre-processing-e2e-inf-a-898968-868d54644c-j8d79\" (UID: \"6bc6af34-1b37-4634-a8b0-932b105ef626\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-898968-868d54644c-j8d79"
Oct 30 19:53:08.259294 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:53:08.259267    4892 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6bc6af34-1b37-4634-a8b0-932b105ef626-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-a-898968-868d54644c-j8d79\" (UID: \"6bc6af34-1b37-4634-a8b0-932b105ef626\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-898968-868d54644c-j8d79"
Oct 30 19:53:08.365788 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:53:08.361933    4892 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-q28w8\" (UniqueName: \"kubernetes.io/projected/6bc6af34-1b37-4634-a8b0-932b105ef626-kube-api-access-q28w8\") pod \"payload-pre-processing-e2e-inf-a-898968-868d54644c-j8d79\" (UID: \"6bc6af34-1b37-4634-a8b0-932b105ef626\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-898968-868d54644c-j8d79"
Oct 30 19:53:08.398092 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:53:08.398038    4892 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-898968-84888dc6d8-vkg68"]
Oct 30 19:53:08.403847 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:53:08.403817    4892 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-898968-84888dc6d8-vkg68"]
Oct 30 19:53:08.404002 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:53:08.403942    4892 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-a-898968-84888dc6d8-vkg68"
Oct 30 19:53:08.428655 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:53:08.428628    4892 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-898968-868d54644c-j8d79"
Oct 30 19:53:08.463197 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:53:08.463164    4892 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2c311ac3-0fcf-48a9-887c-f992ab3e287f-plugins-config-volume\") pod \"payload-processing-e2e-inf-a-898968-84888dc6d8-vkg68\" (UID: \"2c311ac3-0fcf-48a9-887c-f992ab3e287f\") " pod="openshift-ingress/payload-processing-e2e-inf-a-898968-84888dc6d8-vkg68"
Oct 30 19:53:08.463392 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:53:08.463372    4892 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qwjrc\" (UniqueName: \"kubernetes.io/projected/2c311ac3-0fcf-48a9-887c-f992ab3e287f-kube-api-access-qwjrc\") pod \"payload-processing-e2e-inf-a-898968-84888dc6d8-vkg68\" (UID: \"2c311ac3-0fcf-48a9-887c-f992ab3e287f\") " pod="openshift-ingress/payload-processing-e2e-inf-a-898968-84888dc6d8-vkg68"
Oct 30 19:53:08.565066 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:53:08.564982    4892 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qwjrc\" (UniqueName: \"kubernetes.io/projected/2c311ac3-0fcf-48a9-887c-f992ab3e287f-kube-api-access-qwjrc\") pod \"payload-processing-e2e-inf-a-898968-84888dc6d8-vkg68\" (UID: \"2c311ac3-0fcf-48a9-887c-f992ab3e287f\") " pod="openshift-ingress/payload-processing-e2e-inf-a-898968-84888dc6d8-vkg68"
Oct 30 19:53:08.565229 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:53:08.565185    4892 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2c311ac3-0fcf-48a9-887c-f992ab3e287f-plugins-config-volume\") pod \"payload-processing-e2e-inf-a-898968-84888dc6d8-vkg68\" (UID: \"2c311ac3-0fcf-48a9-887c-f992ab3e287f\") " pod="openshift-ingress/payload-processing-e2e-inf-a-898968-84888dc6d8-vkg68"
Oct 30 19:53:08.566106 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:53:08.566082    4892 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2c311ac3-0fcf-48a9-887c-f992ab3e287f-plugins-config-volume\") pod \"payload-processing-e2e-inf-a-898968-84888dc6d8-vkg68\" (UID: \"2c311ac3-0fcf-48a9-887c-f992ab3e287f\") " pod="openshift-ingress/payload-processing-e2e-inf-a-898968-84888dc6d8-vkg68"
Oct 30 19:53:08.621538 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:53:08.621460    4892 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qwjrc\" (UniqueName: \"kubernetes.io/projected/2c311ac3-0fcf-48a9-887c-f992ab3e287f-kube-api-access-qwjrc\") pod \"payload-processing-e2e-inf-a-898968-84888dc6d8-vkg68\" (UID: \"2c311ac3-0fcf-48a9-887c-f992ab3e287f\") " pod="openshift-ingress/payload-processing-e2e-inf-a-898968-84888dc6d8-vkg68"
Oct 30 19:53:08.726717 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:53:08.726687    4892 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-a-898968-84888dc6d8-vkg68"
Oct 30 19:53:08.823099 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:53:08.823071    4892 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-898968-868d54644c-j8d79"]
Oct 30 19:53:09.117322 ip-10-0-132-82 kubenswrapper[4892]: W1030 19:53:09.117295    4892 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2c311ac3_0fcf_48a9_887c_f992ab3e287f.slice/crio-abe797bb99debdde1b72161a6681a28a3af73d21404a7ba538ddce81763c0d6c WatchSource:0}: Error finding container abe797bb99debdde1b72161a6681a28a3af73d21404a7ba538ddce81763c0d6c: Status 404 returned error can't find the container with id abe797bb99debdde1b72161a6681a28a3af73d21404a7ba538ddce81763c0d6c
Oct 30 19:53:09.118086 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:53:09.118060    4892 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-898968-84888dc6d8-vkg68"]
Oct 30 19:53:09.625684 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:53:09.625600    4892 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-898968-868d54644c-j8d79" event={"ID":"6bc6af34-1b37-4634-a8b0-932b105ef626","Type":"ContainerStarted","Data":"22f0d01d6bbfe35c4063388902e1bdd45af61a639b533e788c11b745b3ac7766"}
Oct 30 19:53:09.625684 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:53:09.625658    4892 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-898968-868d54644c-j8d79" event={"ID":"6bc6af34-1b37-4634-a8b0-932b105ef626","Type":"ContainerStarted","Data":"41f288f9b1d1a8cee3150e931429dbb40069f56c28582fb093f66b758af71b86"}
Oct 30 19:53:09.625684 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:53:09.625686    4892 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-898968-868d54644c-j8d79"
Oct 30 19:53:09.627554 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:53:09.627517    4892 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-a-898968-84888dc6d8-vkg68" event={"ID":"2c311ac3-0fcf-48a9-887c-f992ab3e287f","Type":"ContainerStarted","Data":"d8c35861464ea146fecd6790dedd729084469bf3a39a7cc5f621d9546489adbd"}
Oct 30 19:53:09.627676 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:53:09.627570    4892 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-a-898968-84888dc6d8-vkg68" event={"ID":"2c311ac3-0fcf-48a9-887c-f992ab3e287f","Type":"ContainerStarted","Data":"abe797bb99debdde1b72161a6681a28a3af73d21404a7ba538ddce81763c0d6c"}
Oct 30 19:53:09.627676 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:53:09.627600    4892 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-inf-a-898968-84888dc6d8-vkg68"
Oct 30 19:53:09.740101 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:53:09.740051    4892 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-898968-868d54644c-j8d79" podStartSLOduration=1.74003685 podStartE2EDuration="1.74003685s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:53:08 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:53:09.738888224 +0000 UTC m=+2237.309353822" watchObservedRunningTime="2026-10-30 19:53:09.74003685 +0000 UTC m=+2237.310502449"
Oct 30 19:53:09.904990 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:53:09.898393    4892 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-inf-a-898968-84888dc6d8-vkg68" podStartSLOduration=1.898373376 podStartE2EDuration="1.898373376s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:53:08 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:53:09.865664912 +0000 UTC m=+2237.436130513" watchObservedRunningTime="2026-10-30 19:53:09.898373376 +0000 UTC m=+2237.468838977"
Oct 30 19:53:12.484077 ip-10-0-132-82 kubenswrapper[4892]: W1030 19:53:12.483959    4892 logging.go:55] [core] [Channel #3275 SubChannel #3276]grpc: addrConn.createTransport failed to connect to {Addr: "10.132.0.167:50052", ServerName: "10.132.0.167:50052", BalancerAttributes: {"<%!p(pickfirstleaf.managedByPickfirstKeyType={})>": "<%!p(bool=true)>" }}. Err: connection error: desc = "transport: Error while dialing: dial tcp 10.132.0.167:50052: connect: connection refused"
Oct 30 19:53:12.502149 ip-10-0-132-82 kubenswrapper[4892]: W1030 19:53:12.502124    4892 logging.go:55] [core] [Channel #3277 SubChannel #3278]grpc: addrConn.createTransport failed to connect to {Addr: "10.132.0.166:50052", ServerName: "10.132.0.166:50052", BalancerAttributes: {"<%!p(pickfirstleaf.managedByPickfirstKeyType={})>": "<%!p(bool=true)>" }}. Err: connection error: desc = "transport: Error while dialing: dial tcp 10.132.0.166:50052: connect: connection refused"
Oct 30 19:53:12.819092 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:53:12.818738    4892 generic.go:356] "Generic (PLEG): container finished" podID="f7e053ee-836a-4b0e-b2d9-88e3af83ca7b" containerID="14decd769a728ad5e92f5c606283d140be79657786e6905ffd6151f105b2f1fd" exitCode=0
Oct 30 19:53:12.819092 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:53:12.818945    4892 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-dd9b8d79-867946f667-plb8n" event={"ID":"f7e053ee-836a-4b0e-b2d9-88e3af83ca7b","Type":"ContainerDied","Data":"14decd769a728ad5e92f5c606283d140be79657786e6905ffd6151f105b2f1fd"}
Oct 30 19:53:12.841153 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:53:12.825250    4892 generic.go:356] "Generic (PLEG): container finished" podID="3c568859-9035-498d-b347-170ab8d1b2e7" containerID="37908c15ba0247793b89d486084e09149301e2174a8dd2d1818e4daa1fdf60ae" exitCode=0
Oct 30 19:53:12.841153 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:53:12.825584    4892 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-dd9b8d79-67c8d6d46-m4wc6" event={"ID":"3c568859-9035-498d-b347-170ab8d1b2e7","Type":"ContainerDied","Data":"37908c15ba0247793b89d486084e09149301e2174a8dd2d1818e4daa1fdf60ae"}
Oct 30 19:53:13.042368 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:53:13.042339    4892 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-dd9b8d79-67c8d6d46-m4wc6"
Oct 30 19:53:13.106786 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:53:13.106764    4892 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-dd9b8d79-867946f667-plb8n"
Oct 30 19:53:13.135432 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:53:13.135399    4892 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/3c568859-9035-498d-b347-170ab8d1b2e7-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3c568859-9035-498d-b347-170ab8d1b2e7-plugins-config-volume\") pod \"3c568859-9035-498d-b347-170ab8d1b2e7\" (UID: \"3c568859-9035-498d-b347-170ab8d1b2e7\") "
Oct 30 19:53:13.135579 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:53:13.135562    4892 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3c568859-9035-498d-b347-170ab8d1b2e7-kube-api-access-f9hcn\" (UniqueName: \"kubernetes.io/projected/3c568859-9035-498d-b347-170ab8d1b2e7-kube-api-access-f9hcn\") pod \"3c568859-9035-498d-b347-170ab8d1b2e7\" (UID: \"3c568859-9035-498d-b347-170ab8d1b2e7\") "
Oct 30 19:53:13.135776 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:53:13.135750    4892 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3c568859-9035-498d-b347-170ab8d1b2e7-plugins-config-volume" pod "3c568859-9035-498d-b347-170ab8d1b2e7" (UID: "3c568859-9035-498d-b347-170ab8d1b2e7"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 19:53:13.136299 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:53:13.136275    4892 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3c568859-9035-498d-b347-170ab8d1b2e7-plugins-config-volume\") on node \"ip-10-0-132-82.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 19:53:13.137884 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:53:13.137859    4892 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3c568859-9035-498d-b347-170ab8d1b2e7-kube-api-access-f9hcn" pod "3c568859-9035-498d-b347-170ab8d1b2e7" (UID: "3c568859-9035-498d-b347-170ab8d1b2e7"). InnerVolumeSpecName "kube-api-access-f9hcn". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:53:13.237646 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:53:13.237616    4892 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f7e053ee-836a-4b0e-b2d9-88e3af83ca7b-kube-api-access-wbdcr\" (UniqueName: \"kubernetes.io/projected/f7e053ee-836a-4b0e-b2d9-88e3af83ca7b-kube-api-access-wbdcr\") pod \"f7e053ee-836a-4b0e-b2d9-88e3af83ca7b\" (UID: \"f7e053ee-836a-4b0e-b2d9-88e3af83ca7b\") "
Oct 30 19:53:13.237783 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:53:13.237656    4892 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f7e053ee-836a-4b0e-b2d9-88e3af83ca7b-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f7e053ee-836a-4b0e-b2d9-88e3af83ca7b-plugins-config-volume\") pod \"f7e053ee-836a-4b0e-b2d9-88e3af83ca7b\" (UID: \"f7e053ee-836a-4b0e-b2d9-88e3af83ca7b\") "
Oct 30 19:53:13.238021 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:53:13.238001    4892 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-f9hcn\" (UniqueName: \"kubernetes.io/projected/3c568859-9035-498d-b347-170ab8d1b2e7-kube-api-access-f9hcn\") on node \"ip-10-0-132-82.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 19:53:13.238098 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:53:13.238047    4892 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f7e053ee-836a-4b0e-b2d9-88e3af83ca7b-plugins-config-volume" pod "f7e053ee-836a-4b0e-b2d9-88e3af83ca7b" (UID: "f7e053ee-836a-4b0e-b2d9-88e3af83ca7b"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 19:53:13.239910 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:53:13.239887    4892 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f7e053ee-836a-4b0e-b2d9-88e3af83ca7b-kube-api-access-wbdcr" pod "f7e053ee-836a-4b0e-b2d9-88e3af83ca7b" (UID: "f7e053ee-836a-4b0e-b2d9-88e3af83ca7b"). InnerVolumeSpecName "kube-api-access-wbdcr". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:53:13.365058 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:53:13.360274    4892 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wbdcr\" (UniqueName: \"kubernetes.io/projected/f7e053ee-836a-4b0e-b2d9-88e3af83ca7b-kube-api-access-wbdcr\") on node \"ip-10-0-132-82.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 19:53:13.365058 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:53:13.360316    4892 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f7e053ee-836a-4b0e-b2d9-88e3af83ca7b-plugins-config-volume\") on node \"ip-10-0-132-82.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 19:53:13.484147 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:53:13.484094    4892 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-mt-dd9b8d79-67c8d6d46-m4wc6" podUID="3c568859-9035-498d-b347-170ab8d1b2e7" containerName="payload-processing" probeResult="failure" output="timeout: failed to connect service \"10.132.0.167:50052\" within 1s: context deadline exceeded"
Oct 30 19:53:13.484548 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:53:13.484202    4892 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-mt-dd9b8d79-67c8d6d46-m4wc6"
Oct 30 19:53:13.502186 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:53:13.502153    4892 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-mt-dd9b8d79-867946f667-plb8n" podUID="f7e053ee-836a-4b0e-b2d9-88e3af83ca7b" containerName="payload-pre-processing" probeResult="failure" output="timeout: failed to connect service \"10.132.0.166:50052\" within 1s: context deadline exceeded"
Oct 30 19:53:13.502301 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:53:13.502224    4892 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-mt-dd9b8d79-867946f667-plb8n"
Oct 30 19:53:13.836594 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:53:13.836564    4892 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-dd9b8d79-67c8d6d46-m4wc6"
Oct 30 19:53:13.836784 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:53:13.836612    4892 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-dd9b8d79-67c8d6d46-m4wc6" event={"ID":"3c568859-9035-498d-b347-170ab8d1b2e7","Type":"ContainerDied","Data":"6f9e3fca6bfd90c98163099996b6566e8093a6c945018767e05b026c2cc89076"}
Oct 30 19:53:13.836784 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:53:13.836681    4892 scope.go:122] "RemoveContainer" containerID="37908c15ba0247793b89d486084e09149301e2174a8dd2d1818e4daa1fdf60ae"
Oct 30 19:53:13.838525 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:53:13.838434    4892 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-dd9b8d79-867946f667-plb8n" event={"ID":"f7e053ee-836a-4b0e-b2d9-88e3af83ca7b","Type":"ContainerDied","Data":"2d9c145d33e835327bdcd0a2ad4b23ee4165cd350b5d8807cf3a90504ffa99f0"}
Oct 30 19:53:13.838525 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:53:13.838461    4892 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-dd9b8d79-867946f667-plb8n"
Oct 30 19:53:13.850291 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:53:13.850275    4892 scope.go:122] "RemoveContainer" containerID="14decd769a728ad5e92f5c606283d140be79657786e6905ffd6151f105b2f1fd"
Oct 30 19:53:14.002756 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:53:14.002727    4892 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-dd9b8d79-867946f667-plb8n"]
Oct 30 19:53:14.007662 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:53:14.007639    4892 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-dd9b8d79-867946f667-plb8n"]
Oct 30 19:53:14.118899 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:53:14.118813    4892 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-dd9b8d79-67c8d6d46-m4wc6"]
Oct 30 19:53:14.129777 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:53:14.129751    4892 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-dd9b8d79-67c8d6d46-m4wc6"]
Oct 30 19:53:14.141341 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:53:14.141314    4892 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-b5178f-openshift-default-c8fbdff8c-fcqwt"]
Oct 30 19:53:14.141543 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:53:14.141511    4892 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w4-b5178f-openshift-default-c8fbdff8c-fcqwt" podUID="0f1f64ea-aef7-48e0-9238-599b572a90db" containerName="istio-proxy" containerID="cri-o://3013e01c0346a4a9de2b315aaacf81b493eabef65b9af5847f687f534f0b1672" gracePeriod=30
Oct 30 19:53:14.571313 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:53:14.571276    4892 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="3c568859-9035-498d-b347-170ab8d1b2e7" path="/var/lib/kubelet/pods/3c568859-9035-498d-b347-170ab8d1b2e7/volumes"
Oct 30 19:53:14.571811 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:53:14.571796    4892 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f7e053ee-836a-4b0e-b2d9-88e3af83ca7b" path="/var/lib/kubelet/pods/f7e053ee-836a-4b0e-b2d9-88e3af83ca7b/volumes"
Oct 30 19:53:14.590850 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:53:14.590829    4892 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-api-b-1c5537-6d5ffd475f-8jtnh"
Oct 30 19:53:14.644579 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:53:14.644560    4892 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-5bf13ff3-4ckjq"]
Oct 30 19:53:14.648370 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:53:14.648351    4892 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-5bf13ff3-4ckjq"]
Oct 30 19:53:15.618220 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:53:15.615046    4892 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-api-b-1c5537-7f86d78844-ljthj"
Oct 30 19:53:16.571669 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:53:16.571634    4892 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="8fb952a7-38ab-4a93-91e7-9383724d128a" path="/var/lib/kubelet/pods/8fb952a7-38ab-4a93-91e7-9383724d128a/volumes"
Oct 30 19:53:19.216884 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:53:19.216846    4892 patch_prober.go:29] interesting pod/e2e-worker-w4-b5178f-openshift-default-c8fbdff8c-fcqwt container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.96:15021/healthz/ready\": dial tcp 10.132.0.96:15021: connect: connection refused" start-of-body=
Oct 30 19:53:19.216884 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:53:19.216911    4892 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w4-b5178f-openshift-default-c8fbdff8c-fcqwt" podUID="0f1f64ea-aef7-48e0-9238-599b572a90db" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.96:15021/healthz/ready\": dial tcp 10.132.0.96:15021: connect: connection refused"
Oct 30 19:53:19.449022 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:53:19.448992    4892 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-b5178f-openshift-default-c8fbdff8c-fcqwt"
Oct 30 19:53:19.489990 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:53:19.488440    4892 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/0f1f64ea-aef7-48e0-9238-599b572a90db-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0f1f64ea-aef7-48e0-9238-599b572a90db-istio-podinfo\") pod \"0f1f64ea-aef7-48e0-9238-599b572a90db\" (UID: \"0f1f64ea-aef7-48e0-9238-599b572a90db\") "
Oct 30 19:53:19.489990 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:53:19.488500    4892 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0f1f64ea-aef7-48e0-9238-599b572a90db-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0f1f64ea-aef7-48e0-9238-599b572a90db-workload-certs\") pod \"0f1f64ea-aef7-48e0-9238-599b572a90db\" (UID: \"0f1f64ea-aef7-48e0-9238-599b572a90db\") "
Oct 30 19:53:19.489990 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:53:19.488614    4892 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0f1f64ea-aef7-48e0-9238-599b572a90db-kube-api-access-8ppjf\" (UniqueName: \"kubernetes.io/projected/0f1f64ea-aef7-48e0-9238-599b572a90db-kube-api-access-8ppjf\") pod \"0f1f64ea-aef7-48e0-9238-599b572a90db\" (UID: \"0f1f64ea-aef7-48e0-9238-599b572a90db\") "
Oct 30 19:53:19.489990 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:53:19.488658    4892 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0f1f64ea-aef7-48e0-9238-599b572a90db-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0f1f64ea-aef7-48e0-9238-599b572a90db-credential-socket\") pod \"0f1f64ea-aef7-48e0-9238-599b572a90db\" (UID: \"0f1f64ea-aef7-48e0-9238-599b572a90db\") "
Oct 30 19:53:19.489990 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:53:19.488758    4892 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0f1f64ea-aef7-48e0-9238-599b572a90db-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0f1f64ea-aef7-48e0-9238-599b572a90db-workload-socket\") pod \"0f1f64ea-aef7-48e0-9238-599b572a90db\" (UID: \"0f1f64ea-aef7-48e0-9238-599b572a90db\") "
Oct 30 19:53:19.489990 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:53:19.489048    4892 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0f1f64ea-aef7-48e0-9238-599b572a90db-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0f1f64ea-aef7-48e0-9238-599b572a90db-istio-envoy\") pod \"0f1f64ea-aef7-48e0-9238-599b572a90db\" (UID: \"0f1f64ea-aef7-48e0-9238-599b572a90db\") "
Oct 30 19:53:19.489990 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:53:19.489094    4892 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0f1f64ea-aef7-48e0-9238-599b572a90db-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0f1f64ea-aef7-48e0-9238-599b572a90db-istio-data\") pod \"0f1f64ea-aef7-48e0-9238-599b572a90db\" (UID: \"0f1f64ea-aef7-48e0-9238-599b572a90db\") "
Oct 30 19:53:19.489990 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:53:19.489211    4892 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0f1f64ea-aef7-48e0-9238-599b572a90db-istio-token\" (UniqueName: \"kubernetes.io/projected/0f1f64ea-aef7-48e0-9238-599b572a90db-istio-token\") pod \"0f1f64ea-aef7-48e0-9238-599b572a90db\" (UID: \"0f1f64ea-aef7-48e0-9238-599b572a90db\") "
Oct 30 19:53:19.489990 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:53:19.489251    4892 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/0f1f64ea-aef7-48e0-9238-599b572a90db-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0f1f64ea-aef7-48e0-9238-599b572a90db-istiod-ca-cert\") pod \"0f1f64ea-aef7-48e0-9238-599b572a90db\" (UID: \"0f1f64ea-aef7-48e0-9238-599b572a90db\") "
Oct 30 19:53:19.508989 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:53:19.506267    4892 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0f1f64ea-aef7-48e0-9238-599b572a90db-istiod-ca-cert" pod "0f1f64ea-aef7-48e0-9238-599b572a90db" (UID: "0f1f64ea-aef7-48e0-9238-599b572a90db"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 19:53:19.518055 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:53:19.514206    4892 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0f1f64ea-aef7-48e0-9238-599b572a90db-workload-certs" pod "0f1f64ea-aef7-48e0-9238-599b572a90db" (UID: "0f1f64ea-aef7-48e0-9238-599b572a90db"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:53:19.525800 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:53:19.519080    4892 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0f1f64ea-aef7-48e0-9238-599b572a90db-workload-socket" pod "0f1f64ea-aef7-48e0-9238-599b572a90db" (UID: "0f1f64ea-aef7-48e0-9238-599b572a90db"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:53:19.530986 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:53:19.527081    4892 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0f1f64ea-aef7-48e0-9238-599b572a90db-credential-socket" pod "0f1f64ea-aef7-48e0-9238-599b572a90db" (UID: "0f1f64ea-aef7-48e0-9238-599b572a90db"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:53:19.530986 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:53:19.527233    4892 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0f1f64ea-aef7-48e0-9238-599b572a90db-kube-api-access-8ppjf" pod "0f1f64ea-aef7-48e0-9238-599b572a90db" (UID: "0f1f64ea-aef7-48e0-9238-599b572a90db"). InnerVolumeSpecName "kube-api-access-8ppjf". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:53:19.530986 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:53:19.527302    4892 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0f1f64ea-aef7-48e0-9238-599b572a90db-istio-envoy" pod "0f1f64ea-aef7-48e0-9238-599b572a90db" (UID: "0f1f64ea-aef7-48e0-9238-599b572a90db"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:53:19.534988 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:53:19.533558    4892 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/0f1f64ea-aef7-48e0-9238-599b572a90db-istio-podinfo" pod "0f1f64ea-aef7-48e0-9238-599b572a90db" (UID: "0f1f64ea-aef7-48e0-9238-599b572a90db"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 19:53:19.535119 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:53:19.535040    4892 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0f1f64ea-aef7-48e0-9238-599b572a90db-istio-data" pod "0f1f64ea-aef7-48e0-9238-599b572a90db" (UID: "0f1f64ea-aef7-48e0-9238-599b572a90db"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:53:19.553175 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:53:19.552129    4892 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0f1f64ea-aef7-48e0-9238-599b572a90db-istio-token" pod "0f1f64ea-aef7-48e0-9238-599b572a90db" (UID: "0f1f64ea-aef7-48e0-9238-599b572a90db"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:53:19.591839 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:53:19.591799    4892 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0f1f64ea-aef7-48e0-9238-599b572a90db-istio-envoy\") on node \"ip-10-0-132-82.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 19:53:19.591839 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:53:19.591832    4892 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0f1f64ea-aef7-48e0-9238-599b572a90db-istio-data\") on node \"ip-10-0-132-82.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 19:53:19.591839 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:53:19.591846    4892 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0f1f64ea-aef7-48e0-9238-599b572a90db-istio-token\") on node \"ip-10-0-132-82.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 19:53:19.592157 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:53:19.591870    4892 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0f1f64ea-aef7-48e0-9238-599b572a90db-istiod-ca-cert\") on node \"ip-10-0-132-82.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 19:53:19.592157 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:53:19.591882    4892 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0f1f64ea-aef7-48e0-9238-599b572a90db-istio-podinfo\") on node \"ip-10-0-132-82.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 19:53:19.592157 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:53:19.591895    4892 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0f1f64ea-aef7-48e0-9238-599b572a90db-workload-certs\") on node \"ip-10-0-132-82.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 19:53:19.592157 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:53:19.591907    4892 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8ppjf\" (UniqueName: \"kubernetes.io/projected/0f1f64ea-aef7-48e0-9238-599b572a90db-kube-api-access-8ppjf\") on node \"ip-10-0-132-82.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 19:53:19.592157 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:53:19.591922    4892 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0f1f64ea-aef7-48e0-9238-599b572a90db-credential-socket\") on node \"ip-10-0-132-82.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 19:53:19.592157 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:53:19.591934    4892 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0f1f64ea-aef7-48e0-9238-599b572a90db-workload-socket\") on node \"ip-10-0-132-82.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 19:53:20.070158 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:53:20.070115    4892 generic.go:356] "Generic (PLEG): container finished" podID="0f1f64ea-aef7-48e0-9238-599b572a90db" containerID="3013e01c0346a4a9de2b315aaacf81b493eabef65b9af5847f687f534f0b1672" exitCode=0
Oct 30 19:53:20.070343 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:53:20.070175    4892 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-b5178f-openshift-default-c8fbdff8c-fcqwt"
Oct 30 19:53:20.070343 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:53:20.070201    4892 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-b5178f-openshift-default-c8fbdff8c-fcqwt" event={"ID":"0f1f64ea-aef7-48e0-9238-599b572a90db","Type":"ContainerDied","Data":"3013e01c0346a4a9de2b315aaacf81b493eabef65b9af5847f687f534f0b1672"}
Oct 30 19:53:20.070343 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:53:20.070260    4892 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-b5178f-openshift-default-c8fbdff8c-fcqwt" event={"ID":"0f1f64ea-aef7-48e0-9238-599b572a90db","Type":"ContainerDied","Data":"5b060b066edce425a8635627f8fb3b8525a5fd0d57951c568c2eea65b226a3aa"}
Oct 30 19:53:20.070343 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:53:20.070282    4892 scope.go:122] "RemoveContainer" containerID="3013e01c0346a4a9de2b315aaacf81b493eabef65b9af5847f687f534f0b1672"
Oct 30 19:53:20.082278 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:53:20.082261    4892 scope.go:122] "RemoveContainer" containerID="3013e01c0346a4a9de2b315aaacf81b493eabef65b9af5847f687f534f0b1672"
Oct 30 19:53:20.082558 ip-10-0-132-82 kubenswrapper[4892]: E1030 19:53:20.082539    4892 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3013e01c0346a4a9de2b315aaacf81b493eabef65b9af5847f687f534f0b1672\": container with ID starting with 3013e01c0346a4a9de2b315aaacf81b493eabef65b9af5847f687f534f0b1672 not found: ID does not exist" containerID="3013e01c0346a4a9de2b315aaacf81b493eabef65b9af5847f687f534f0b1672"
Oct 30 19:53:20.082602 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:53:20.082568    4892 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3013e01c0346a4a9de2b315aaacf81b493eabef65b9af5847f687f534f0b1672"} err="failed to get container status \"3013e01c0346a4a9de2b315aaacf81b493eabef65b9af5847f687f534f0b1672\": rpc error: code = NotFound desc = could not find container \"3013e01c0346a4a9de2b315aaacf81b493eabef65b9af5847f687f534f0b1672\": container with ID starting with 3013e01c0346a4a9de2b315aaacf81b493eabef65b9af5847f687f534f0b1672 not found: ID does not exist"
Oct 30 19:53:20.239042 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:53:20.239004    4892 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-b5178f-openshift-default-c8fbdff8c-fcqwt"]
Oct 30 19:53:20.285909 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:53:20.285880    4892 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w4-b5178f-openshift-default-c8fbdff8c-fcqwt"]
Oct 30 19:53:20.570927 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:53:20.570896    4892 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0f1f64ea-aef7-48e0-9238-599b572a90db" path="/var/lib/kubelet/pods/0f1f64ea-aef7-48e0-9238-599b572a90db/volumes"
Oct 30 19:53:20.641942 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:53:20.641911    4892 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-inf-a-898968-84888dc6d8-vkg68"
Oct 30 19:53:20.642147 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:53:20.642014    4892 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-898968-868d54644c-j8d79"
Oct 30 19:53:21.979233 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:53:21.979198    4892 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-mt-5bf13ff3-openshift-default-7558bbbc65-9j7w6"]
Oct 30 19:53:21.979681 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:53:21.979409    4892 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-mt-5bf13ff3-openshift-default-7558bbbc65-9j7w6" podUID="474b0b88-a062-4ee2-9994-022ae1253f28" containerName="istio-proxy" containerID="cri-o://7a0c5c0b438a6855ad7cf45b76d06609bb2d8d93068cde2d1f3c4db2227e00c6" gracePeriod=30
Oct 30 19:53:23.547536 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:53:23.547504    4892 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-api-a-6caff5-openshift-default-665946589f-hn4qk"]
Oct 30 19:53:23.548477 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:53:23.548458    4892 cpu_manager.go:414] "containerMap: removing container" podUID="0f1f64ea-aef7-48e0-9238-599b572a90db" containerName="istio-proxy"
Oct 30 19:53:23.548477 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:53:23.548479    4892 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0f1f64ea-aef7-48e0-9238-599b572a90db" containerName="istio-proxy"
Oct 30 19:53:23.548656 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:53:23.548520    4892 cpu_manager.go:414] "containerMap: removing container" podUID="f7e053ee-836a-4b0e-b2d9-88e3af83ca7b" containerName="payload-pre-processing"
Oct 30 19:53:23.548656 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:53:23.548528    4892 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f7e053ee-836a-4b0e-b2d9-88e3af83ca7b" containerName="payload-pre-processing"
Oct 30 19:53:23.548656 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:53:23.548582    4892 cpu_manager.go:414] "containerMap: removing container" podUID="3c568859-9035-498d-b347-170ab8d1b2e7" containerName="payload-processing"
Oct 30 19:53:23.548656 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:53:23.548589    4892 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3c568859-9035-498d-b347-170ab8d1b2e7" containerName="payload-processing"
Oct 30 19:53:23.548862 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:53:23.548762    4892 memory_manager.go:365] "RemoveStaleState removing state" podUID="f7e053ee-836a-4b0e-b2d9-88e3af83ca7b" containerName="payload-pre-processing"
Oct 30 19:53:23.548862 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:53:23.548786    4892 memory_manager.go:365] "RemoveStaleState removing state" podUID="3c568859-9035-498d-b347-170ab8d1b2e7" containerName="payload-processing"
Oct 30 19:53:23.548862 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:53:23.548800    4892 memory_manager.go:365] "RemoveStaleState removing state" podUID="0f1f64ea-aef7-48e0-9238-599b572a90db" containerName="istio-proxy"
Oct 30 19:53:23.553833 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:53:23.553814    4892 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-6caff5-openshift-default-665946589f-hn4qk"
Oct 30 19:53:23.605728 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:53:23.605696    4892 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-api-a-6caff5-openshift-default-dockercfg-9zjb7\""
Oct 30 19:53:23.661981 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:53:23.661929    4892 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-a-6caff5-openshift-default-665946589f-hn4qk"]
Oct 30 19:53:23.703035 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:53:23.694131    4892 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/33098b51-3c25-4761-a108-844516b8d2fa-workload-socket\") pod \"e2e-api-a-6caff5-openshift-default-665946589f-hn4qk\" (UID: \"33098b51-3c25-4761-a108-844516b8d2fa\") " pod="openshift-ingress/e2e-api-a-6caff5-openshift-default-665946589f-hn4qk"
Oct 30 19:53:23.727000 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:53:23.703352    4892 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/33098b51-3c25-4761-a108-844516b8d2fa-credential-socket\") pod \"e2e-api-a-6caff5-openshift-default-665946589f-hn4qk\" (UID: \"33098b51-3c25-4761-a108-844516b8d2fa\") " pod="openshift-ingress/e2e-api-a-6caff5-openshift-default-665946589f-hn4qk"
Oct 30 19:53:23.727000 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:53:23.703418    4892 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vlrmm\" (UniqueName: \"kubernetes.io/projected/33098b51-3c25-4761-a108-844516b8d2fa-kube-api-access-vlrmm\") pod \"e2e-api-a-6caff5-openshift-default-665946589f-hn4qk\" (UID: \"33098b51-3c25-4761-a108-844516b8d2fa\") " pod="openshift-ingress/e2e-api-a-6caff5-openshift-default-665946589f-hn4qk"
Oct 30 19:53:23.727000 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:53:23.703457    4892 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/33098b51-3c25-4761-a108-844516b8d2fa-istio-token\") pod \"e2e-api-a-6caff5-openshift-default-665946589f-hn4qk\" (UID: \"33098b51-3c25-4761-a108-844516b8d2fa\") " pod="openshift-ingress/e2e-api-a-6caff5-openshift-default-665946589f-hn4qk"
Oct 30 19:53:23.727000 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:53:23.703507    4892 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/33098b51-3c25-4761-a108-844516b8d2fa-istiod-ca-cert\") pod \"e2e-api-a-6caff5-openshift-default-665946589f-hn4qk\" (UID: \"33098b51-3c25-4761-a108-844516b8d2fa\") " pod="openshift-ingress/e2e-api-a-6caff5-openshift-default-665946589f-hn4qk"
Oct 30 19:53:23.727000 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:53:23.703538    4892 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/33098b51-3c25-4761-a108-844516b8d2fa-istio-data\") pod \"e2e-api-a-6caff5-openshift-default-665946589f-hn4qk\" (UID: \"33098b51-3c25-4761-a108-844516b8d2fa\") " pod="openshift-ingress/e2e-api-a-6caff5-openshift-default-665946589f-hn4qk"
Oct 30 19:53:23.727000 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:53:23.703725    4892 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/33098b51-3c25-4761-a108-844516b8d2fa-istio-envoy\") pod \"e2e-api-a-6caff5-openshift-default-665946589f-hn4qk\" (UID: \"33098b51-3c25-4761-a108-844516b8d2fa\") " pod="openshift-ingress/e2e-api-a-6caff5-openshift-default-665946589f-hn4qk"
Oct 30 19:53:23.727000 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:53:23.703814    4892 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/33098b51-3c25-4761-a108-844516b8d2fa-workload-certs\") pod \"e2e-api-a-6caff5-openshift-default-665946589f-hn4qk\" (UID: \"33098b51-3c25-4761-a108-844516b8d2fa\") " pod="openshift-ingress/e2e-api-a-6caff5-openshift-default-665946589f-hn4qk"
Oct 30 19:53:23.727000 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:53:23.703949    4892 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/33098b51-3c25-4761-a108-844516b8d2fa-istio-podinfo\") pod \"e2e-api-a-6caff5-openshift-default-665946589f-hn4qk\" (UID: \"33098b51-3c25-4761-a108-844516b8d2fa\") " pod="openshift-ingress/e2e-api-a-6caff5-openshift-default-665946589f-hn4qk"
Oct 30 19:53:23.808746 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:53:23.808652    4892 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/33098b51-3c25-4761-a108-844516b8d2fa-istio-envoy\") pod \"e2e-api-a-6caff5-openshift-default-665946589f-hn4qk\" (UID: \"33098b51-3c25-4761-a108-844516b8d2fa\") " pod="openshift-ingress/e2e-api-a-6caff5-openshift-default-665946589f-hn4qk"
Oct 30 19:53:23.809508 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:53:23.809483    4892 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/33098b51-3c25-4761-a108-844516b8d2fa-workload-certs\") pod \"e2e-api-a-6caff5-openshift-default-665946589f-hn4qk\" (UID: \"33098b51-3c25-4761-a108-844516b8d2fa\") " pod="openshift-ingress/e2e-api-a-6caff5-openshift-default-665946589f-hn4qk"
Oct 30 19:53:23.810079 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:53:23.809941    4892 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/33098b51-3c25-4761-a108-844516b8d2fa-workload-certs\") pod \"e2e-api-a-6caff5-openshift-default-665946589f-hn4qk\" (UID: \"33098b51-3c25-4761-a108-844516b8d2fa\") " pod="openshift-ingress/e2e-api-a-6caff5-openshift-default-665946589f-hn4qk"
Oct 30 19:53:23.810255 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:53:23.810094    4892 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/33098b51-3c25-4761-a108-844516b8d2fa-istio-podinfo\") pod \"e2e-api-a-6caff5-openshift-default-665946589f-hn4qk\" (UID: \"33098b51-3c25-4761-a108-844516b8d2fa\") " pod="openshift-ingress/e2e-api-a-6caff5-openshift-default-665946589f-hn4qk"
Oct 30 19:53:23.810466 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:53:23.810445    4892 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/33098b51-3c25-4761-a108-844516b8d2fa-workload-socket\") pod \"e2e-api-a-6caff5-openshift-default-665946589f-hn4qk\" (UID: \"33098b51-3c25-4761-a108-844516b8d2fa\") " pod="openshift-ingress/e2e-api-a-6caff5-openshift-default-665946589f-hn4qk"
Oct 30 19:53:23.810615 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:53:23.810597    4892 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/33098b51-3c25-4761-a108-844516b8d2fa-credential-socket\") pod \"e2e-api-a-6caff5-openshift-default-665946589f-hn4qk\" (UID: \"33098b51-3c25-4761-a108-844516b8d2fa\") " pod="openshift-ingress/e2e-api-a-6caff5-openshift-default-665946589f-hn4qk"
Oct 30 19:53:23.813740 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:53:23.813715    4892 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vlrmm\" (UniqueName: \"kubernetes.io/projected/33098b51-3c25-4761-a108-844516b8d2fa-kube-api-access-vlrmm\") pod \"e2e-api-a-6caff5-openshift-default-665946589f-hn4qk\" (UID: \"33098b51-3c25-4761-a108-844516b8d2fa\") " pod="openshift-ingress/e2e-api-a-6caff5-openshift-default-665946589f-hn4qk"
Oct 30 19:53:23.813940 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:53:23.813919    4892 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/33098b51-3c25-4761-a108-844516b8d2fa-istio-token\") pod \"e2e-api-a-6caff5-openshift-default-665946589f-hn4qk\" (UID: \"33098b51-3c25-4761-a108-844516b8d2fa\") " pod="openshift-ingress/e2e-api-a-6caff5-openshift-default-665946589f-hn4qk"
Oct 30 19:53:23.817317 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:53:23.812879    4892 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/33098b51-3c25-4761-a108-844516b8d2fa-credential-socket\") pod \"e2e-api-a-6caff5-openshift-default-665946589f-hn4qk\" (UID: \"33098b51-3c25-4761-a108-844516b8d2fa\") " pod="openshift-ingress/e2e-api-a-6caff5-openshift-default-665946589f-hn4qk"
Oct 30 19:53:23.817423 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:53:23.812654    4892 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/33098b51-3c25-4761-a108-844516b8d2fa-workload-socket\") pod \"e2e-api-a-6caff5-openshift-default-665946589f-hn4qk\" (UID: \"33098b51-3c25-4761-a108-844516b8d2fa\") " pod="openshift-ingress/e2e-api-a-6caff5-openshift-default-665946589f-hn4qk"
Oct 30 19:53:23.817423 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:53:23.817298    4892 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/33098b51-3c25-4761-a108-844516b8d2fa-istiod-ca-cert\") pod \"e2e-api-a-6caff5-openshift-default-665946589f-hn4qk\" (UID: \"33098b51-3c25-4761-a108-844516b8d2fa\") " pod="openshift-ingress/e2e-api-a-6caff5-openshift-default-665946589f-hn4qk"
Oct 30 19:53:23.817423 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:53:23.817406    4892 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/33098b51-3c25-4761-a108-844516b8d2fa-istio-data\") pod \"e2e-api-a-6caff5-openshift-default-665946589f-hn4qk\" (UID: \"33098b51-3c25-4761-a108-844516b8d2fa\") " pod="openshift-ingress/e2e-api-a-6caff5-openshift-default-665946589f-hn4qk"
Oct 30 19:53:23.817764 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:53:23.817738    4892 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/33098b51-3c25-4761-a108-844516b8d2fa-istio-data\") pod \"e2e-api-a-6caff5-openshift-default-665946589f-hn4qk\" (UID: \"33098b51-3c25-4761-a108-844516b8d2fa\") " pod="openshift-ingress/e2e-api-a-6caff5-openshift-default-665946589f-hn4qk"
Oct 30 19:53:23.818430 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:53:23.818408    4892 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/33098b51-3c25-4761-a108-844516b8d2fa-istiod-ca-cert\") pod \"e2e-api-a-6caff5-openshift-default-665946589f-hn4qk\" (UID: \"33098b51-3c25-4761-a108-844516b8d2fa\") " pod="openshift-ingress/e2e-api-a-6caff5-openshift-default-665946589f-hn4qk"
Oct 30 19:53:23.820862 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:53:23.820838    4892 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/33098b51-3c25-4761-a108-844516b8d2fa-istio-podinfo\") pod \"e2e-api-a-6caff5-openshift-default-665946589f-hn4qk\" (UID: \"33098b51-3c25-4761-a108-844516b8d2fa\") " pod="openshift-ingress/e2e-api-a-6caff5-openshift-default-665946589f-hn4qk"
Oct 30 19:53:23.821151 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:53:23.821128    4892 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/33098b51-3c25-4761-a108-844516b8d2fa-istio-envoy\") pod \"e2e-api-a-6caff5-openshift-default-665946589f-hn4qk\" (UID: \"33098b51-3c25-4761-a108-844516b8d2fa\") " pod="openshift-ingress/e2e-api-a-6caff5-openshift-default-665946589f-hn4qk"
Oct 30 19:53:23.880894 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:53:23.880861    4892 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vlrmm\" (UniqueName: \"kubernetes.io/projected/33098b51-3c25-4761-a108-844516b8d2fa-kube-api-access-vlrmm\") pod \"e2e-api-a-6caff5-openshift-default-665946589f-hn4qk\" (UID: \"33098b51-3c25-4761-a108-844516b8d2fa\") " pod="openshift-ingress/e2e-api-a-6caff5-openshift-default-665946589f-hn4qk"
Oct 30 19:53:23.885030 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:53:23.885004    4892 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/33098b51-3c25-4761-a108-844516b8d2fa-istio-token\") pod \"e2e-api-a-6caff5-openshift-default-665946589f-hn4qk\" (UID: \"33098b51-3c25-4761-a108-844516b8d2fa\") " pod="openshift-ingress/e2e-api-a-6caff5-openshift-default-665946589f-hn4qk"
Oct 30 19:53:24.173606 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:53:24.173573    4892 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-6caff5-openshift-default-665946589f-hn4qk"
Oct 30 19:53:24.686899 ip-10-0-132-82 kubenswrapper[4892]: W1030 19:53:24.686867    4892 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod33098b51_3c25_4761_a108_844516b8d2fa.slice/crio-d1219230a546ac1ffa504118cbe907692f402bceb49d23f56790fe111558ba76 WatchSource:0}: Error finding container d1219230a546ac1ffa504118cbe907692f402bceb49d23f56790fe111558ba76: Status 404 returned error can't find the container with id d1219230a546ac1ffa504118cbe907692f402bceb49d23f56790fe111558ba76
Oct 30 19:53:24.687266 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:53:24.686947    4892 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-a-6caff5-openshift-default-665946589f-hn4qk"]
Oct 30 19:53:24.690022 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:53:24.689986    4892 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892000Ki","pods":"250"}
Oct 30 19:53:24.690150 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:53:24.690066    4892 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892000Ki","pods":"250"}
Oct 30 19:53:24.690150 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:53:24.690115    4892 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892000Ki","pods":"250"}
Oct 30 19:53:24.855420 ip-10-0-132-82 kubenswrapper[4892]: E1030 19:53:24.855384    4892 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-jrm87" containerName="extractor"
Oct 30 19:53:25.211548 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:53:25.211514    4892 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-6caff5-openshift-default-665946589f-hn4qk" event={"ID":"33098b51-3c25-4761-a108-844516b8d2fa","Type":"ContainerStarted","Data":"87a5399b1210c4c285564ff7854067bfad195cfa2c0f40e0bf87815fb1e09ca4"}
Oct 30 19:53:25.211737 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:53:25.211570    4892 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-6caff5-openshift-default-665946589f-hn4qk" event={"ID":"33098b51-3c25-4761-a108-844516b8d2fa","Type":"ContainerStarted","Data":"d1219230a546ac1ffa504118cbe907692f402bceb49d23f56790fe111558ba76"}
Oct 30 19:53:25.329809 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:53:25.329776    4892 patch_prober.go:29] interesting pod/e2e-mt-5bf13ff3-openshift-default-7558bbbc65-9j7w6 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Oct 30 19:53:25.330039 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:53:25.329834    4892 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-mt-5bf13ff3-openshift-default-7558bbbc65-9j7w6" podUID="474b0b88-a062-4ee2-9994-022ae1253f28" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Oct 30 19:53:25.330158 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:53:25.330106    4892 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-api-a-6caff5-openshift-default-665946589f-hn4qk" podStartSLOduration=2.330089298 podStartE2EDuration="2.330089298s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:53:23 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:53:25.32823549 +0000 UTC m=+2252.898701114" watchObservedRunningTime="2026-10-30 19:53:25.330089298 +0000 UTC m=+2252.900554899"
Oct 30 19:53:26.174098 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:53:26.174061    4892 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-a-6caff5-openshift-default-665946589f-hn4qk"
Oct 30 19:53:26.180098 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:53:26.180071    4892 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-a-6caff5-openshift-default-665946589f-hn4qk"
Oct 30 19:53:26.289298 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:53:26.289256    4892 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-a-6caff5-openshift-default-665946589f-hn4qk"
Oct 30 19:53:26.290722 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:53:26.290702    4892 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-a-6caff5-openshift-default-665946589f-hn4qk"
Oct 30 19:53:26.336773 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:53:26.336736    4892 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-da4c62-79568d7967-nmj4w"]
Oct 30 19:53:26.343556 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:53:26.343529    4892 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-b-da4c62-79568d7967-nmj4w"
Oct 30 19:53:26.396362 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:53:26.396334    4892 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-api-b-da4c62\""
Oct 30 19:53:26.396502 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:53:26.396460    4892 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-api-b-da4c62-dockercfg-gb59r\""
Oct 30 19:53:26.413950 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:53:26.413904    4892 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-da4c62-79568d7967-nmj4w"]
Oct 30 19:53:26.484860 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:53:26.484777    4892 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-da4c62-5899dfc94d-2s5tq"]
Oct 30 19:53:26.489195 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:53:26.489176    4892 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-b-da4c62-5899dfc94d-2s5tq"
Oct 30 19:53:26.601995 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:53:26.600550    4892 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xvbdv\" (UniqueName: \"kubernetes.io/projected/abff79c3-d35b-48bf-86ca-6329ee026b07-kube-api-access-xvbdv\") pod \"payload-pre-processing-e2e-api-b-da4c62-79568d7967-nmj4w\" (UID: \"abff79c3-d35b-48bf-86ca-6329ee026b07\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-da4c62-79568d7967-nmj4w"
Oct 30 19:53:26.601995 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:53:26.600768    4892 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/abff79c3-d35b-48bf-86ca-6329ee026b07-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-b-da4c62-79568d7967-nmj4w\" (UID: \"abff79c3-d35b-48bf-86ca-6329ee026b07\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-da4c62-79568d7967-nmj4w"
Oct 30 19:53:26.601995 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:53:26.600804    4892 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c2d43160-e748-4f65-81e1-0e9895b11593-plugins-config-volume\") pod \"payload-processing-e2e-api-b-da4c62-5899dfc94d-2s5tq\" (UID: \"c2d43160-e748-4f65-81e1-0e9895b11593\") " pod="openshift-ingress/payload-processing-e2e-api-b-da4c62-5899dfc94d-2s5tq"
Oct 30 19:53:26.606998 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:53:26.605796    4892 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qz6p5\" (UniqueName: \"kubernetes.io/projected/c2d43160-e748-4f65-81e1-0e9895b11593-kube-api-access-qz6p5\") pod \"payload-processing-e2e-api-b-da4c62-5899dfc94d-2s5tq\" (UID: \"c2d43160-e748-4f65-81e1-0e9895b11593\") " pod="openshift-ingress/payload-processing-e2e-api-b-da4c62-5899dfc94d-2s5tq"
Oct 30 19:53:26.709001 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:53:26.707855    4892 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qz6p5\" (UniqueName: \"kubernetes.io/projected/c2d43160-e748-4f65-81e1-0e9895b11593-kube-api-access-qz6p5\") pod \"payload-processing-e2e-api-b-da4c62-5899dfc94d-2s5tq\" (UID: \"c2d43160-e748-4f65-81e1-0e9895b11593\") " pod="openshift-ingress/payload-processing-e2e-api-b-da4c62-5899dfc94d-2s5tq"
Oct 30 19:53:26.711341 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:53:26.711276    4892 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xvbdv\" (UniqueName: \"kubernetes.io/projected/abff79c3-d35b-48bf-86ca-6329ee026b07-kube-api-access-xvbdv\") pod \"payload-pre-processing-e2e-api-b-da4c62-79568d7967-nmj4w\" (UID: \"abff79c3-d35b-48bf-86ca-6329ee026b07\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-da4c62-79568d7967-nmj4w"
Oct 30 19:53:26.712045 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:53:26.712018    4892 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/abff79c3-d35b-48bf-86ca-6329ee026b07-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-b-da4c62-79568d7967-nmj4w\" (UID: \"abff79c3-d35b-48bf-86ca-6329ee026b07\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-da4c62-79568d7967-nmj4w"
Oct 30 19:53:26.712253 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:53:26.712232    4892 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c2d43160-e748-4f65-81e1-0e9895b11593-plugins-config-volume\") pod \"payload-processing-e2e-api-b-da4c62-5899dfc94d-2s5tq\" (UID: \"c2d43160-e748-4f65-81e1-0e9895b11593\") " pod="openshift-ingress/payload-processing-e2e-api-b-da4c62-5899dfc94d-2s5tq"
Oct 30 19:53:26.713546 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:53:26.713438    4892 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c2d43160-e748-4f65-81e1-0e9895b11593-plugins-config-volume\") pod \"payload-processing-e2e-api-b-da4c62-5899dfc94d-2s5tq\" (UID: \"c2d43160-e748-4f65-81e1-0e9895b11593\") " pod="openshift-ingress/payload-processing-e2e-api-b-da4c62-5899dfc94d-2s5tq"
Oct 30 19:53:26.713693 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:53:26.713560    4892 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/abff79c3-d35b-48bf-86ca-6329ee026b07-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-b-da4c62-79568d7967-nmj4w\" (UID: \"abff79c3-d35b-48bf-86ca-6329ee026b07\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-da4c62-79568d7967-nmj4w"
Oct 30 19:53:26.721582 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:53:26.721541    4892 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-da4c62-5899dfc94d-2s5tq"]
Oct 30 19:53:26.769386 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:53:26.769322    4892 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qz6p5\" (UniqueName: \"kubernetes.io/projected/c2d43160-e748-4f65-81e1-0e9895b11593-kube-api-access-qz6p5\") pod \"payload-processing-e2e-api-b-da4c62-5899dfc94d-2s5tq\" (UID: \"c2d43160-e748-4f65-81e1-0e9895b11593\") " pod="openshift-ingress/payload-processing-e2e-api-b-da4c62-5899dfc94d-2s5tq"
Oct 30 19:53:26.769563 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:53:26.769323    4892 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xvbdv\" (UniqueName: \"kubernetes.io/projected/abff79c3-d35b-48bf-86ca-6329ee026b07-kube-api-access-xvbdv\") pod \"payload-pre-processing-e2e-api-b-da4c62-79568d7967-nmj4w\" (UID: \"abff79c3-d35b-48bf-86ca-6329ee026b07\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-da4c62-79568d7967-nmj4w"
Oct 30 19:53:26.803218 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:53:26.803192    4892 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-b-da4c62-5899dfc94d-2s5tq"
Oct 30 19:53:26.965978 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:53:26.965928    4892 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-b-da4c62-79568d7967-nmj4w"
Oct 30 19:53:27.185172 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:53:27.185136    4892 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-da4c62-5899dfc94d-2s5tq"]
Oct 30 19:53:27.185616 ip-10-0-132-82 kubenswrapper[4892]: W1030 19:53:27.185446    4892 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc2d43160_e748_4f65_81e1_0e9895b11593.slice/crio-937dd8f89c00d77d92a2cbf5d9845d80dd95f34d127574b790e6d413ecd7dca5 WatchSource:0}: Error finding container 937dd8f89c00d77d92a2cbf5d9845d80dd95f34d127574b790e6d413ecd7dca5: Status 404 returned error can't find the container with id 937dd8f89c00d77d92a2cbf5d9845d80dd95f34d127574b790e6d413ecd7dca5
Oct 30 19:53:27.246247 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:53:27.246208    4892 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-5bf13ff3-openshift-default-7558bbbc65-9j7w6"
Oct 30 19:53:27.298321 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:53:27.298256    4892 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-b-da4c62-5899dfc94d-2s5tq" event={"ID":"c2d43160-e748-4f65-81e1-0e9895b11593","Type":"ContainerStarted","Data":"13ce3ff9e65fb78c33156c11db5f3bd4ae6096c17b535a5e9b6e25c43cc850f9"}
Oct 30 19:53:27.298321 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:53:27.298315    4892 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-b-da4c62-5899dfc94d-2s5tq" event={"ID":"c2d43160-e748-4f65-81e1-0e9895b11593","Type":"ContainerStarted","Data":"937dd8f89c00d77d92a2cbf5d9845d80dd95f34d127574b790e6d413ecd7dca5"}
Oct 30 19:53:27.298510 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:53:27.298401    4892 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-api-b-da4c62-5899dfc94d-2s5tq"
Oct 30 19:53:27.299977 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:53:27.299941    4892 generic.go:356] "Generic (PLEG): container finished" podID="474b0b88-a062-4ee2-9994-022ae1253f28" containerID="7a0c5c0b438a6855ad7cf45b76d06609bb2d8d93068cde2d1f3c4db2227e00c6" exitCode=0
Oct 30 19:53:27.300071 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:53:27.299987    4892 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-5bf13ff3-openshift-default-7558bbbc65-9j7w6"
Oct 30 19:53:27.300071 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:53:27.300009    4892 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-5bf13ff3-openshift-default-7558bbbc65-9j7w6" event={"ID":"474b0b88-a062-4ee2-9994-022ae1253f28","Type":"ContainerDied","Data":"7a0c5c0b438a6855ad7cf45b76d06609bb2d8d93068cde2d1f3c4db2227e00c6"}
Oct 30 19:53:27.300071 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:53:27.300052    4892 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-5bf13ff3-openshift-default-7558bbbc65-9j7w6" event={"ID":"474b0b88-a062-4ee2-9994-022ae1253f28","Type":"ContainerDied","Data":"5fa4d52cf29ae2ba35de6aa200d4d3030384b985a2ecb8536516b0d149ef8fdf"}
Oct 30 19:53:27.300193 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:53:27.300075    4892 scope.go:122] "RemoveContainer" containerID="7a0c5c0b438a6855ad7cf45b76d06609bb2d8d93068cde2d1f3c4db2227e00c6"
Oct 30 19:53:27.311105 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:53:27.311084    4892 scope.go:122] "RemoveContainer" containerID="191522a582678a8a642c14a3c468d7a6d308a6de40f6275658f2e3176f86eeec"
Oct 30 19:53:27.320518 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:53:27.320498    4892 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/474b0b88-a062-4ee2-9994-022ae1253f28-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/474b0b88-a062-4ee2-9994-022ae1253f28-workload-socket\") pod \"474b0b88-a062-4ee2-9994-022ae1253f28\" (UID: \"474b0b88-a062-4ee2-9994-022ae1253f28\") "
Oct 30 19:53:27.320611 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:53:27.320533    4892 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/474b0b88-a062-4ee2-9994-022ae1253f28-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/474b0b88-a062-4ee2-9994-022ae1253f28-workload-certs\") pod \"474b0b88-a062-4ee2-9994-022ae1253f28\" (UID: \"474b0b88-a062-4ee2-9994-022ae1253f28\") "
Oct 30 19:53:27.320611 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:53:27.320585    4892 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/474b0b88-a062-4ee2-9994-022ae1253f28-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/474b0b88-a062-4ee2-9994-022ae1253f28-istio-envoy\") pod \"474b0b88-a062-4ee2-9994-022ae1253f28\" (UID: \"474b0b88-a062-4ee2-9994-022ae1253f28\") "
Oct 30 19:53:27.320782 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:53:27.320641    4892 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/474b0b88-a062-4ee2-9994-022ae1253f28-kube-api-access-5vffk\" (UniqueName: \"kubernetes.io/projected/474b0b88-a062-4ee2-9994-022ae1253f28-kube-api-access-5vffk\") pod \"474b0b88-a062-4ee2-9994-022ae1253f28\" (UID: \"474b0b88-a062-4ee2-9994-022ae1253f28\") "
Oct 30 19:53:27.320782 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:53:27.320706    4892 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/474b0b88-a062-4ee2-9994-022ae1253f28-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/474b0b88-a062-4ee2-9994-022ae1253f28-istio-data\") pod \"474b0b88-a062-4ee2-9994-022ae1253f28\" (UID: \"474b0b88-a062-4ee2-9994-022ae1253f28\") "
Oct 30 19:53:27.320782 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:53:27.320734    4892 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/474b0b88-a062-4ee2-9994-022ae1253f28-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/474b0b88-a062-4ee2-9994-022ae1253f28-istiod-ca-cert\") pod \"474b0b88-a062-4ee2-9994-022ae1253f28\" (UID: \"474b0b88-a062-4ee2-9994-022ae1253f28\") "
Oct 30 19:53:27.320782 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:53:27.320747    4892 scope.go:122] "RemoveContainer" containerID="7a0c5c0b438a6855ad7cf45b76d06609bb2d8d93068cde2d1f3c4db2227e00c6"
Oct 30 19:53:27.321036 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:53:27.320781    4892 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/474b0b88-a062-4ee2-9994-022ae1253f28-workload-socket" pod "474b0b88-a062-4ee2-9994-022ae1253f28" (UID: "474b0b88-a062-4ee2-9994-022ae1253f28"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:53:27.321036 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:53:27.320888    4892 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/474b0b88-a062-4ee2-9994-022ae1253f28-workload-certs" pod "474b0b88-a062-4ee2-9994-022ae1253f28" (UID: "474b0b88-a062-4ee2-9994-022ae1253f28"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:53:27.321036 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:53:27.320900    4892 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/474b0b88-a062-4ee2-9994-022ae1253f28-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/474b0b88-a062-4ee2-9994-022ae1253f28-credential-socket\") pod \"474b0b88-a062-4ee2-9994-022ae1253f28\" (UID: \"474b0b88-a062-4ee2-9994-022ae1253f28\") "
Oct 30 19:53:27.321036 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:53:27.320932    4892 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/474b0b88-a062-4ee2-9994-022ae1253f28-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/474b0b88-a062-4ee2-9994-022ae1253f28-istio-podinfo\") pod \"474b0b88-a062-4ee2-9994-022ae1253f28\" (UID: \"474b0b88-a062-4ee2-9994-022ae1253f28\") "
Oct 30 19:53:27.321036 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:53:27.321004    4892 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/474b0b88-a062-4ee2-9994-022ae1253f28-istio-token\" (UniqueName: \"kubernetes.io/projected/474b0b88-a062-4ee2-9994-022ae1253f28-istio-token\") pod \"474b0b88-a062-4ee2-9994-022ae1253f28\" (UID: \"474b0b88-a062-4ee2-9994-022ae1253f28\") "
Oct 30 19:53:27.321036 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:53:27.321021    4892 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/474b0b88-a062-4ee2-9994-022ae1253f28-istio-data" pod "474b0b88-a062-4ee2-9994-022ae1253f28" (UID: "474b0b88-a062-4ee2-9994-022ae1253f28"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:53:27.321357 ip-10-0-132-82 kubenswrapper[4892]: E1030 19:53:27.321129    4892 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7a0c5c0b438a6855ad7cf45b76d06609bb2d8d93068cde2d1f3c4db2227e00c6\": container with ID starting with 7a0c5c0b438a6855ad7cf45b76d06609bb2d8d93068cde2d1f3c4db2227e00c6 not found: ID does not exist" containerID="7a0c5c0b438a6855ad7cf45b76d06609bb2d8d93068cde2d1f3c4db2227e00c6"
Oct 30 19:53:27.321357 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:53:27.321161    4892 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7a0c5c0b438a6855ad7cf45b76d06609bb2d8d93068cde2d1f3c4db2227e00c6"} err="failed to get container status \"7a0c5c0b438a6855ad7cf45b76d06609bb2d8d93068cde2d1f3c4db2227e00c6\": rpc error: code = NotFound desc = could not find container \"7a0c5c0b438a6855ad7cf45b76d06609bb2d8d93068cde2d1f3c4db2227e00c6\": container with ID starting with 7a0c5c0b438a6855ad7cf45b76d06609bb2d8d93068cde2d1f3c4db2227e00c6 not found: ID does not exist"
Oct 30 19:53:27.321357 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:53:27.321187    4892 scope.go:122] "RemoveContainer" containerID="191522a582678a8a642c14a3c468d7a6d308a6de40f6275658f2e3176f86eeec"
Oct 30 19:53:27.321357 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:53:27.321284    4892 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/474b0b88-a062-4ee2-9994-022ae1253f28-credential-socket" pod "474b0b88-a062-4ee2-9994-022ae1253f28" (UID: "474b0b88-a062-4ee2-9994-022ae1253f28"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:53:27.321357 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:53:27.321300    4892 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/474b0b88-a062-4ee2-9994-022ae1253f28-istiod-ca-cert" pod "474b0b88-a062-4ee2-9994-022ae1253f28" (UID: "474b0b88-a062-4ee2-9994-022ae1253f28"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 19:53:27.321640 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:53:27.321485    4892 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/474b0b88-a062-4ee2-9994-022ae1253f28-istio-data\") on node \"ip-10-0-132-82.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 19:53:27.321640 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:53:27.321497    4892 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/474b0b88-a062-4ee2-9994-022ae1253f28-istiod-ca-cert\") on node \"ip-10-0-132-82.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 19:53:27.321640 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:53:27.321508    4892 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/474b0b88-a062-4ee2-9994-022ae1253f28-credential-socket\") on node \"ip-10-0-132-82.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 19:53:27.321640 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:53:27.321520    4892 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/474b0b88-a062-4ee2-9994-022ae1253f28-workload-socket\") on node \"ip-10-0-132-82.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 19:53:27.321640 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:53:27.321531    4892 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/474b0b88-a062-4ee2-9994-022ae1253f28-workload-certs\") on node \"ip-10-0-132-82.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 19:53:27.321847 ip-10-0-132-82 kubenswrapper[4892]: E1030 19:53:27.321687    4892 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"191522a582678a8a642c14a3c468d7a6d308a6de40f6275658f2e3176f86eeec\": container with ID starting with 191522a582678a8a642c14a3c468d7a6d308a6de40f6275658f2e3176f86eeec not found: ID does not exist" containerID="191522a582678a8a642c14a3c468d7a6d308a6de40f6275658f2e3176f86eeec"
Oct 30 19:53:27.321898 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:53:27.321807    4892 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"191522a582678a8a642c14a3c468d7a6d308a6de40f6275658f2e3176f86eeec"} err="failed to get container status \"191522a582678a8a642c14a3c468d7a6d308a6de40f6275658f2e3176f86eeec\": rpc error: code = NotFound desc = could not find container \"191522a582678a8a642c14a3c468d7a6d308a6de40f6275658f2e3176f86eeec\": container with ID starting with 191522a582678a8a642c14a3c468d7a6d308a6de40f6275658f2e3176f86eeec not found: ID does not exist"
Oct 30 19:53:27.323298 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:53:27.323274    4892 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/474b0b88-a062-4ee2-9994-022ae1253f28-istio-podinfo" pod "474b0b88-a062-4ee2-9994-022ae1253f28" (UID: "474b0b88-a062-4ee2-9994-022ae1253f28"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 19:53:27.323405 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:53:27.323389    4892 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/474b0b88-a062-4ee2-9994-022ae1253f28-istio-envoy" pod "474b0b88-a062-4ee2-9994-022ae1253f28" (UID: "474b0b88-a062-4ee2-9994-022ae1253f28"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:53:27.323452 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:53:27.323397    4892 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/474b0b88-a062-4ee2-9994-022ae1253f28-istio-token" pod "474b0b88-a062-4ee2-9994-022ae1253f28" (UID: "474b0b88-a062-4ee2-9994-022ae1253f28"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:53:27.323452 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:53:27.323424    4892 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/474b0b88-a062-4ee2-9994-022ae1253f28-kube-api-access-5vffk" pod "474b0b88-a062-4ee2-9994-022ae1253f28" (UID: "474b0b88-a062-4ee2-9994-022ae1253f28"). InnerVolumeSpecName "kube-api-access-5vffk". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:53:27.350209 ip-10-0-132-82 kubenswrapper[4892]: W1030 19:53:27.350186    4892 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podabff79c3_d35b_48bf_86ca_6329ee026b07.slice/crio-b6efebd352b1c1220b4079ab92a63ba9d4b52a8755d566645cf9c6bf17432a3c WatchSource:0}: Error finding container b6efebd352b1c1220b4079ab92a63ba9d4b52a8755d566645cf9c6bf17432a3c: Status 404 returned error can't find the container with id b6efebd352b1c1220b4079ab92a63ba9d4b52a8755d566645cf9c6bf17432a3c
Oct 30 19:53:27.351977 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:53:27.351939    4892 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-da4c62-79568d7967-nmj4w"]
Oct 30 19:53:27.423461 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:53:27.422890    4892 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/474b0b88-a062-4ee2-9994-022ae1253f28-istio-envoy\") on node \"ip-10-0-132-82.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 19:53:27.423461 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:53:27.422921    4892 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5vffk\" (UniqueName: \"kubernetes.io/projected/474b0b88-a062-4ee2-9994-022ae1253f28-kube-api-access-5vffk\") on node \"ip-10-0-132-82.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 19:53:27.423461 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:53:27.422933    4892 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/474b0b88-a062-4ee2-9994-022ae1253f28-istio-podinfo\") on node \"ip-10-0-132-82.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 19:53:27.423461 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:53:27.422942    4892 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/474b0b88-a062-4ee2-9994-022ae1253f28-istio-token\") on node \"ip-10-0-132-82.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 19:53:27.476142 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:53:27.476071    4892 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-api-b-da4c62-5899dfc94d-2s5tq" podStartSLOduration=1.476051439 podStartE2EDuration="1.476051439s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:53:26 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:53:27.473826842 +0000 UTC m=+2255.044292442" watchObservedRunningTime="2026-10-30 19:53:27.476051439 +0000 UTC m=+2255.046517039"
Oct 30 19:53:27.741366 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:53:27.741330    4892 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-mt-5bf13ff3-openshift-default-7558bbbc65-9j7w6"]
Oct 30 19:53:27.745929 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:53:27.745902    4892 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-mt-5bf13ff3-openshift-default-7558bbbc65-9j7w6"]
Oct 30 19:53:28.310794 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:53:28.310751    4892 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-b-da4c62-79568d7967-nmj4w" event={"ID":"abff79c3-d35b-48bf-86ca-6329ee026b07","Type":"ContainerStarted","Data":"fc25cf446ce8f4bc7c6a1906171dee830df43635139fb6594740cc9c46cd30e4"}
Oct 30 19:53:28.311299 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:53:28.310805    4892 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-b-da4c62-79568d7967-nmj4w" event={"ID":"abff79c3-d35b-48bf-86ca-6329ee026b07","Type":"ContainerStarted","Data":"b6efebd352b1c1220b4079ab92a63ba9d4b52a8755d566645cf9c6bf17432a3c"}
Oct 30 19:53:28.311299 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:53:28.310952    4892 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-api-b-da4c62-79568d7967-nmj4w"
Oct 30 19:53:28.430902 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:53:28.430843    4892 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-api-b-da4c62-79568d7967-nmj4w" podStartSLOduration=2.430826152 podStartE2EDuration="2.430826152s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:53:26 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:53:28.424177959 +0000 UTC m=+2255.994643562" watchObservedRunningTime="2026-10-30 19:53:28.430826152 +0000 UTC m=+2256.001291756"
Oct 30 19:53:28.599046 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:53:28.597622    4892 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="474b0b88-a062-4ee2-9994-022ae1253f28" path="/var/lib/kubelet/pods/474b0b88-a062-4ee2-9994-022ae1253f28/volumes"
Oct 30 19:53:35.244033 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:53:35.243996    4892 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-b0c827-fc5bcbb66-q72h8"]
Oct 30 19:53:35.244736 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:53:35.244721    4892 cpu_manager.go:414] "containerMap: removing container" podUID="474b0b88-a062-4ee2-9994-022ae1253f28" containerName="istio-proxy"
Oct 30 19:53:35.244781 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:53:35.244736    4892 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="474b0b88-a062-4ee2-9994-022ae1253f28" containerName="istio-proxy"
Oct 30 19:53:35.244781 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:53:35.244769    4892 cpu_manager.go:414] "containerMap: removing container" podUID="474b0b88-a062-4ee2-9994-022ae1253f28" containerName="istio-proxy"
Oct 30 19:53:35.244781 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:53:35.244774    4892 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="474b0b88-a062-4ee2-9994-022ae1253f28" containerName="istio-proxy"
Oct 30 19:53:35.244899 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:53:35.244889    4892 memory_manager.go:365] "RemoveStaleState removing state" podUID="474b0b88-a062-4ee2-9994-022ae1253f28" containerName="istio-proxy"
Oct 30 19:53:35.244935 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:53:35.244906    4892 memory_manager.go:365] "RemoveStaleState removing state" podUID="474b0b88-a062-4ee2-9994-022ae1253f28" containerName="istio-proxy"
Oct 30 19:53:35.249894 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:53:35.249874    4892 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-b-b0c827-fc5bcbb66-q72h8"
Oct 30 19:53:35.320993 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:53:35.311501    4892 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-b0c827-fc5bcbb66-q72h8"]
Oct 30 19:53:35.356090 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:53:35.354396    4892 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-inf-b-b0c827-dockercfg-xr8n7\""
Oct 30 19:53:35.356090 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:53:35.354498    4892 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-inf-b-b0c827\""
Oct 30 19:53:35.369437 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:53:35.369396    4892 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mq8bv\" (UniqueName: \"kubernetes.io/projected/e1a27576-9ec0-4d96-9d2f-3e8df58f9852-kube-api-access-mq8bv\") pod \"payload-processing-e2e-inf-b-b0c827-fc5bcbb66-q72h8\" (UID: \"e1a27576-9ec0-4d96-9d2f-3e8df58f9852\") " pod="openshift-ingress/payload-processing-e2e-inf-b-b0c827-fc5bcbb66-q72h8"
Oct 30 19:53:35.389093 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:53:35.380318    4892 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e1a27576-9ec0-4d96-9d2f-3e8df58f9852-plugins-config-volume\") pod \"payload-processing-e2e-inf-b-b0c827-fc5bcbb66-q72h8\" (UID: \"e1a27576-9ec0-4d96-9d2f-3e8df58f9852\") " pod="openshift-ingress/payload-processing-e2e-inf-b-b0c827-fc5bcbb66-q72h8"
Oct 30 19:53:35.493032 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:53:35.489922    4892 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e1a27576-9ec0-4d96-9d2f-3e8df58f9852-plugins-config-volume\") pod \"payload-processing-e2e-inf-b-b0c827-fc5bcbb66-q72h8\" (UID: \"e1a27576-9ec0-4d96-9d2f-3e8df58f9852\") " pod="openshift-ingress/payload-processing-e2e-inf-b-b0c827-fc5bcbb66-q72h8"
Oct 30 19:53:35.493032 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:53:35.490068    4892 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mq8bv\" (UniqueName: \"kubernetes.io/projected/e1a27576-9ec0-4d96-9d2f-3e8df58f9852-kube-api-access-mq8bv\") pod \"payload-processing-e2e-inf-b-b0c827-fc5bcbb66-q72h8\" (UID: \"e1a27576-9ec0-4d96-9d2f-3e8df58f9852\") " pod="openshift-ingress/payload-processing-e2e-inf-b-b0c827-fc5bcbb66-q72h8"
Oct 30 19:53:35.493032 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:53:35.491390    4892 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e1a27576-9ec0-4d96-9d2f-3e8df58f9852-plugins-config-volume\") pod \"payload-processing-e2e-inf-b-b0c827-fc5bcbb66-q72h8\" (UID: \"e1a27576-9ec0-4d96-9d2f-3e8df58f9852\") " pod="openshift-ingress/payload-processing-e2e-inf-b-b0c827-fc5bcbb66-q72h8"
Oct 30 19:53:35.517286 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:53:35.512217    4892 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-b0c827-7b674bdfbc-vrpw8"]
Oct 30 19:53:35.530025 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:53:35.529732    4892 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-b0c827-7b674bdfbc-vrpw8"]
Oct 30 19:53:35.530025 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:53:35.529847    4892 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-b0c827-7b674bdfbc-vrpw8"
Oct 30 19:53:35.548531 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:53:35.548435    4892 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mq8bv\" (UniqueName: \"kubernetes.io/projected/e1a27576-9ec0-4d96-9d2f-3e8df58f9852-kube-api-access-mq8bv\") pod \"payload-processing-e2e-inf-b-b0c827-fc5bcbb66-q72h8\" (UID: \"e1a27576-9ec0-4d96-9d2f-3e8df58f9852\") " pod="openshift-ingress/payload-processing-e2e-inf-b-b0c827-fc5bcbb66-q72h8"
Oct 30 19:53:35.565084 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:53:35.565063    4892 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-b-b0c827-fc5bcbb66-q72h8"
Oct 30 19:53:35.693661 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:53:35.693627    4892 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dhqrt\" (UniqueName: \"kubernetes.io/projected/d31c06b2-df7e-4d8a-a386-7afc365e6f8a-kube-api-access-dhqrt\") pod \"payload-pre-processing-e2e-inf-b-b0c827-7b674bdfbc-vrpw8\" (UID: \"d31c06b2-df7e-4d8a-a386-7afc365e6f8a\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-b0c827-7b674bdfbc-vrpw8"
Oct 30 19:53:35.693851 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:53:35.693816    4892 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d31c06b2-df7e-4d8a-a386-7afc365e6f8a-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-b-b0c827-7b674bdfbc-vrpw8\" (UID: \"d31c06b2-df7e-4d8a-a386-7afc365e6f8a\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-b0c827-7b674bdfbc-vrpw8"
Oct 30 19:53:35.795210 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:53:35.795143    4892 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d31c06b2-df7e-4d8a-a386-7afc365e6f8a-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-b-b0c827-7b674bdfbc-vrpw8\" (UID: \"d31c06b2-df7e-4d8a-a386-7afc365e6f8a\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-b0c827-7b674bdfbc-vrpw8"
Oct 30 19:53:35.795338 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:53:35.795297    4892 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dhqrt\" (UniqueName: \"kubernetes.io/projected/d31c06b2-df7e-4d8a-a386-7afc365e6f8a-kube-api-access-dhqrt\") pod \"payload-pre-processing-e2e-inf-b-b0c827-7b674bdfbc-vrpw8\" (UID: \"d31c06b2-df7e-4d8a-a386-7afc365e6f8a\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-b0c827-7b674bdfbc-vrpw8"
Oct 30 19:53:35.796045 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:53:35.796021    4892 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d31c06b2-df7e-4d8a-a386-7afc365e6f8a-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-b-b0c827-7b674bdfbc-vrpw8\" (UID: \"d31c06b2-df7e-4d8a-a386-7afc365e6f8a\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-b0c827-7b674bdfbc-vrpw8"
Oct 30 19:53:35.853747 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:53:35.853718    4892 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dhqrt\" (UniqueName: \"kubernetes.io/projected/d31c06b2-df7e-4d8a-a386-7afc365e6f8a-kube-api-access-dhqrt\") pod \"payload-pre-processing-e2e-inf-b-b0c827-7b674bdfbc-vrpw8\" (UID: \"d31c06b2-df7e-4d8a-a386-7afc365e6f8a\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-b0c827-7b674bdfbc-vrpw8"
Oct 30 19:53:35.951165 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:53:35.951133    4892 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-b0c827-fc5bcbb66-q72h8"]
Oct 30 19:53:35.951683 ip-10-0-132-82 kubenswrapper[4892]: W1030 19:53:35.951658    4892 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode1a27576_9ec0_4d96_9d2f_3e8df58f9852.slice/crio-b5b2e1a3a2693af72b5deb767e024d7b6c607efab8730df4dbb3195972cad287 WatchSource:0}: Error finding container b5b2e1a3a2693af72b5deb767e024d7b6c607efab8730df4dbb3195972cad287: Status 404 returned error can't find the container with id b5b2e1a3a2693af72b5deb767e024d7b6c607efab8730df4dbb3195972cad287
Oct 30 19:53:36.148004 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:53:36.147957    4892 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-b0c827-7b674bdfbc-vrpw8"
Oct 30 19:53:36.453581 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:53:36.453489    4892 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-b-b0c827-fc5bcbb66-q72h8" event={"ID":"e1a27576-9ec0-4d96-9d2f-3e8df58f9852","Type":"ContainerStarted","Data":"c49b7da76d0d4843579b5ea620190e267168a4a845eedfd6472eb277027bdddb"}
Oct 30 19:53:36.453581 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:53:36.453552    4892 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-b-b0c827-fc5bcbb66-q72h8" event={"ID":"e1a27576-9ec0-4d96-9d2f-3e8df58f9852","Type":"ContainerStarted","Data":"b5b2e1a3a2693af72b5deb767e024d7b6c607efab8730df4dbb3195972cad287"}
Oct 30 19:53:36.453581 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:53:36.453577    4892 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-inf-b-b0c827-fc5bcbb66-q72h8"
Oct 30 19:53:36.535577 ip-10-0-132-82 kubenswrapper[4892]: W1030 19:53:36.535538    4892 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd31c06b2_df7e_4d8a_a386_7afc365e6f8a.slice/crio-415f78bee68aae355bf1e87b04aa60818dc758d8762205de7c5487b487635b84 WatchSource:0}: Error finding container 415f78bee68aae355bf1e87b04aa60818dc758d8762205de7c5487b487635b84: Status 404 returned error can't find the container with id 415f78bee68aae355bf1e87b04aa60818dc758d8762205de7c5487b487635b84
Oct 30 19:53:36.536999 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:53:36.536947    4892 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-b0c827-7b674bdfbc-vrpw8"]
Oct 30 19:53:36.575565 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:53:36.575516    4892 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-inf-b-b0c827-fc5bcbb66-q72h8" podStartSLOduration=1.575502033 podStartE2EDuration="1.575502033s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:53:35 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:53:36.572162999 +0000 UTC m=+2264.142628600" watchObservedRunningTime="2026-10-30 19:53:36.575502033 +0000 UTC m=+2264.145967667"
Oct 30 19:53:37.466659 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:53:37.466619    4892 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-b0c827-7b674bdfbc-vrpw8" event={"ID":"d31c06b2-df7e-4d8a-a386-7afc365e6f8a","Type":"ContainerStarted","Data":"03dd2e98a14f38631f9f32f5814418a232586ecd97afd10aa43cd9a56927511a"}
Oct 30 19:53:37.467171 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:53:37.466684    4892 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-b0c827-7b674bdfbc-vrpw8" event={"ID":"d31c06b2-df7e-4d8a-a386-7afc365e6f8a","Type":"ContainerStarted","Data":"415f78bee68aae355bf1e87b04aa60818dc758d8762205de7c5487b487635b84"}
Oct 30 19:53:37.467171 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:53:37.466869    4892 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-b0c827-7b674bdfbc-vrpw8"
Oct 30 19:53:37.582420 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:53:37.582367    4892 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-b0c827-7b674bdfbc-vrpw8" podStartSLOduration=2.582354608 podStartE2EDuration="2.582354608s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:53:35 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:53:37.579832791 +0000 UTC m=+2265.150298391" watchObservedRunningTime="2026-10-30 19:53:37.582354608 +0000 UTC m=+2265.152820254"
Oct 30 19:53:38.313811 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:53:38.313785    4892 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-api-b-da4c62-5899dfc94d-2s5tq"
Oct 30 19:53:39.322491 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:53:39.322463    4892 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-api-b-da4c62-79568d7967-nmj4w"
Oct 30 19:53:40.625989 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:53:40.624017    4892 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-6caff5-fcddd97f9-vjnf6"]
Oct 30 19:53:40.649991 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:53:40.642380    4892 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-a-6caff5-fcddd97f9-vjnf6"
Oct 30 19:53:40.693398 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:53:40.681899    4892 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/203e53c4-cf13-4f11-a90f-59319c124125-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-a-6caff5-fcddd97f9-vjnf6\" (UID: \"203e53c4-cf13-4f11-a90f-59319c124125\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-6caff5-fcddd97f9-vjnf6"
Oct 30 19:53:40.693398 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:53:40.682098    4892 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bz4r2\" (UniqueName: \"kubernetes.io/projected/203e53c4-cf13-4f11-a90f-59319c124125-kube-api-access-bz4r2\") pod \"payload-pre-processing-e2e-api-a-6caff5-fcddd97f9-vjnf6\" (UID: \"203e53c4-cf13-4f11-a90f-59319c124125\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-6caff5-fcddd97f9-vjnf6"
Oct 30 19:53:40.710000 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:53:40.707947    4892 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-6caff5-fcddd97f9-vjnf6"]
Oct 30 19:53:40.737048 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:53:40.736221    4892 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-api-a-6caff5-dockercfg-vm4pr\""
Oct 30 19:53:40.737048 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:53:40.736347    4892 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-api-a-6caff5\""
Oct 30 19:53:40.801944 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:53:40.801898    4892 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/203e53c4-cf13-4f11-a90f-59319c124125-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-a-6caff5-fcddd97f9-vjnf6\" (UID: \"203e53c4-cf13-4f11-a90f-59319c124125\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-6caff5-fcddd97f9-vjnf6"
Oct 30 19:53:40.802256 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:53:40.802161    4892 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bz4r2\" (UniqueName: \"kubernetes.io/projected/203e53c4-cf13-4f11-a90f-59319c124125-kube-api-access-bz4r2\") pod \"payload-pre-processing-e2e-api-a-6caff5-fcddd97f9-vjnf6\" (UID: \"203e53c4-cf13-4f11-a90f-59319c124125\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-6caff5-fcddd97f9-vjnf6"
Oct 30 19:53:40.803510 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:53:40.803477    4892 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/203e53c4-cf13-4f11-a90f-59319c124125-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-a-6caff5-fcddd97f9-vjnf6\" (UID: \"203e53c4-cf13-4f11-a90f-59319c124125\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-6caff5-fcddd97f9-vjnf6"
Oct 30 19:53:40.810768 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:53:40.810150    4892 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-6caff5-86dcd5d66d-9r2cn"]
Oct 30 19:53:40.822037 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:53:40.816245    4892 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-a-6caff5-86dcd5d66d-9r2cn"
Oct 30 19:53:40.844995 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:53:40.844684    4892 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-6caff5-86dcd5d66d-9r2cn"]
Oct 30 19:53:40.871987 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:53:40.870212    4892 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bz4r2\" (UniqueName: \"kubernetes.io/projected/203e53c4-cf13-4f11-a90f-59319c124125-kube-api-access-bz4r2\") pod \"payload-pre-processing-e2e-api-a-6caff5-fcddd97f9-vjnf6\" (UID: \"203e53c4-cf13-4f11-a90f-59319c124125\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-6caff5-fcddd97f9-vjnf6"
Oct 30 19:53:40.903744 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:53:40.903712    4892 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/61ff434e-6a69-4316-aacb-478b4b12274e-plugins-config-volume\") pod \"payload-processing-e2e-api-a-6caff5-86dcd5d66d-9r2cn\" (UID: \"61ff434e-6a69-4316-aacb-478b4b12274e\") " pod="openshift-ingress/payload-processing-e2e-api-a-6caff5-86dcd5d66d-9r2cn"
Oct 30 19:53:40.903942 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:53:40.903920    4892 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9z8nl\" (UniqueName: \"kubernetes.io/projected/61ff434e-6a69-4316-aacb-478b4b12274e-kube-api-access-9z8nl\") pod \"payload-processing-e2e-api-a-6caff5-86dcd5d66d-9r2cn\" (UID: \"61ff434e-6a69-4316-aacb-478b4b12274e\") " pod="openshift-ingress/payload-processing-e2e-api-a-6caff5-86dcd5d66d-9r2cn"
Oct 30 19:53:41.005037 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:53:41.005004    4892 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/61ff434e-6a69-4316-aacb-478b4b12274e-plugins-config-volume\") pod \"payload-processing-e2e-api-a-6caff5-86dcd5d66d-9r2cn\" (UID: \"61ff434e-6a69-4316-aacb-478b4b12274e\") " pod="openshift-ingress/payload-processing-e2e-api-a-6caff5-86dcd5d66d-9r2cn"
Oct 30 19:53:41.005208 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:53:41.005163    4892 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9z8nl\" (UniqueName: \"kubernetes.io/projected/61ff434e-6a69-4316-aacb-478b4b12274e-kube-api-access-9z8nl\") pod \"payload-processing-e2e-api-a-6caff5-86dcd5d66d-9r2cn\" (UID: \"61ff434e-6a69-4316-aacb-478b4b12274e\") " pod="openshift-ingress/payload-processing-e2e-api-a-6caff5-86dcd5d66d-9r2cn"
Oct 30 19:53:41.005813 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:53:41.005790    4892 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/61ff434e-6a69-4316-aacb-478b4b12274e-plugins-config-volume\") pod \"payload-processing-e2e-api-a-6caff5-86dcd5d66d-9r2cn\" (UID: \"61ff434e-6a69-4316-aacb-478b4b12274e\") " pod="openshift-ingress/payload-processing-e2e-api-a-6caff5-86dcd5d66d-9r2cn"
Oct 30 19:53:41.031333 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:53:41.031308    4892 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-a-6caff5-fcddd97f9-vjnf6"
Oct 30 19:53:41.063328 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:53:41.063302    4892 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9z8nl\" (UniqueName: \"kubernetes.io/projected/61ff434e-6a69-4316-aacb-478b4b12274e-kube-api-access-9z8nl\") pod \"payload-processing-e2e-api-a-6caff5-86dcd5d66d-9r2cn\" (UID: \"61ff434e-6a69-4316-aacb-478b4b12274e\") " pod="openshift-ingress/payload-processing-e2e-api-a-6caff5-86dcd5d66d-9r2cn"
Oct 30 19:53:41.162180 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:53:41.162080    4892 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-a-6caff5-86dcd5d66d-9r2cn"
Oct 30 19:53:41.415469 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:53:41.415444    4892 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-6caff5-fcddd97f9-vjnf6"]
Oct 30 19:53:41.415700 ip-10-0-132-82 kubenswrapper[4892]: W1030 19:53:41.415678    4892 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod203e53c4_cf13_4f11_a90f_59319c124125.slice/crio-6e794fb4f892eea2bfc0972ea523a0c431f03e048d8ef14329c27e9a3717fc39 WatchSource:0}: Error finding container 6e794fb4f892eea2bfc0972ea523a0c431f03e048d8ef14329c27e9a3717fc39: Status 404 returned error can't find the container with id 6e794fb4f892eea2bfc0972ea523a0c431f03e048d8ef14329c27e9a3717fc39
Oct 30 19:53:41.546803 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:53:41.546775    4892 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-6caff5-86dcd5d66d-9r2cn"]
Oct 30 19:53:41.547424 ip-10-0-132-82 kubenswrapper[4892]: W1030 19:53:41.547395    4892 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod61ff434e_6a69_4316_aacb_478b4b12274e.slice/crio-19edfd7fd0b3f171260aee2449dc4ba7ed6a7b47ccac6ed6a512c1ad8622e86a WatchSource:0}: Error finding container 19edfd7fd0b3f171260aee2449dc4ba7ed6a7b47ccac6ed6a512c1ad8622e86a: Status 404 returned error can't find the container with id 19edfd7fd0b3f171260aee2449dc4ba7ed6a7b47ccac6ed6a512c1ad8622e86a
Oct 30 19:53:41.596374 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:53:41.596345    4892 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-a-6caff5-86dcd5d66d-9r2cn" event={"ID":"61ff434e-6a69-4316-aacb-478b4b12274e","Type":"ContainerStarted","Data":"19edfd7fd0b3f171260aee2449dc4ba7ed6a7b47ccac6ed6a512c1ad8622e86a"}
Oct 30 19:53:41.597894 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:53:41.597871    4892 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-a-6caff5-fcddd97f9-vjnf6" event={"ID":"203e53c4-cf13-4f11-a90f-59319c124125","Type":"ContainerStarted","Data":"32d922a2955c677ec98a2d3ec8a66fa2200b5d7a03918f2bce2d9a45bf54c976"}
Oct 30 19:53:41.597997 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:53:41.597910    4892 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-a-6caff5-fcddd97f9-vjnf6" event={"ID":"203e53c4-cf13-4f11-a90f-59319c124125","Type":"ContainerStarted","Data":"6e794fb4f892eea2bfc0972ea523a0c431f03e048d8ef14329c27e9a3717fc39"}
Oct 30 19:53:41.598050 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:53:41.598017    4892 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-api-a-6caff5-fcddd97f9-vjnf6"
Oct 30 19:53:41.714610 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:53:41.714525    4892 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-api-a-6caff5-fcddd97f9-vjnf6" podStartSLOduration=1.714510999 podStartE2EDuration="1.714510999s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:53:40 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:53:41.711011256 +0000 UTC m=+2269.281476855" watchObservedRunningTime="2026-10-30 19:53:41.714510999 +0000 UTC m=+2269.284976598"
Oct 30 19:53:42.607996 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:53:42.607919    4892 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-a-6caff5-86dcd5d66d-9r2cn" event={"ID":"61ff434e-6a69-4316-aacb-478b4b12274e","Type":"ContainerStarted","Data":"98e19108a5e162853b604d61ee8264b0969acc3416663fd48ff0728f838e7def"}
Oct 30 19:53:42.608190 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:53:42.608024    4892 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-api-a-6caff5-86dcd5d66d-9r2cn"
Oct 30 19:53:42.723579 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:53:42.723519    4892 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-api-a-6caff5-86dcd5d66d-9r2cn" podStartSLOduration=2.723504392 podStartE2EDuration="2.723504392s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:53:40 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:53:42.721068628 +0000 UTC m=+2270.291534229" watchObservedRunningTime="2026-10-30 19:53:42.723504392 +0000 UTC m=+2270.293969991"
Oct 30 19:53:47.470399 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:53:47.470370    4892 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-inf-b-b0c827-fc5bcbb66-q72h8"
Oct 30 19:53:48.485765 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:53:48.485723    4892 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-b0c827-7b674bdfbc-vrpw8"
Oct 30 19:53:49.471500 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:53:49.471468    4892 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-80518402-mssm4"]
Oct 30 19:53:49.476013 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:53:49.475992    4892 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-80518402-mssm4"
Oct 30 19:53:49.533086 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:53:49.533061    4892 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-80518402-mssm4"]
Oct 30 19:53:49.576417 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:53:49.576389    4892 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 19:53:49.576578 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:53:49.576483    4892 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 30 19:53:49.577341 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:53:49.577316    4892 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-nlxp4\""
Oct 30 19:53:49.577341 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:53:49.577324    4892 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 30 19:53:49.617353 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:53:49.617328    4892 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4dcwm\" (UniqueName: \"kubernetes.io/projected/af9f361e-cebe-4309-93cb-c609ef835889-kube-api-access-4dcwm\") pod \"maas-api-revoke-sub-80518402-mssm4\" (UID: \"af9f361e-cebe-4309-93cb-c609ef835889\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-80518402-mssm4"
Oct 30 19:53:49.617528 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:53:49.617508    4892 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/af9f361e-cebe-4309-93cb-c609ef835889-maas-api-xks-ca\") pod \"maas-api-revoke-sub-80518402-mssm4\" (UID: \"af9f361e-cebe-4309-93cb-c609ef835889\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-80518402-mssm4"
Oct 30 19:53:49.617843 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:53:49.617819    4892 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/af9f361e-cebe-4309-93cb-c609ef835889-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-80518402-mssm4\" (UID: \"af9f361e-cebe-4309-93cb-c609ef835889\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-80518402-mssm4"
Oct 30 19:53:49.719465 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:53:49.719434    4892 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/af9f361e-cebe-4309-93cb-c609ef835889-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-80518402-mssm4\" (UID: \"af9f361e-cebe-4309-93cb-c609ef835889\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-80518402-mssm4"
Oct 30 19:53:49.719623 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:53:49.719535    4892 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4dcwm\" (UniqueName: \"kubernetes.io/projected/af9f361e-cebe-4309-93cb-c609ef835889-kube-api-access-4dcwm\") pod \"maas-api-revoke-sub-80518402-mssm4\" (UID: \"af9f361e-cebe-4309-93cb-c609ef835889\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-80518402-mssm4"
Oct 30 19:53:49.719623 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:53:49.719602    4892 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/af9f361e-cebe-4309-93cb-c609ef835889-maas-api-xks-ca\") pod \"maas-api-revoke-sub-80518402-mssm4\" (UID: \"af9f361e-cebe-4309-93cb-c609ef835889\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-80518402-mssm4"
Oct 30 19:53:49.720176 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:53:49.720156    4892 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/af9f361e-cebe-4309-93cb-c609ef835889-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-80518402-mssm4\" (UID: \"af9f361e-cebe-4309-93cb-c609ef835889\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-80518402-mssm4"
Oct 30 19:53:49.722145 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:53:49.722096    4892 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/af9f361e-cebe-4309-93cb-c609ef835889-maas-api-xks-ca\") pod \"maas-api-revoke-sub-80518402-mssm4\" (UID: \"af9f361e-cebe-4309-93cb-c609ef835889\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-80518402-mssm4"
Oct 30 19:53:49.776167 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:53:49.776143    4892 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4dcwm\" (UniqueName: \"kubernetes.io/projected/af9f361e-cebe-4309-93cb-c609ef835889-kube-api-access-4dcwm\") pod \"maas-api-revoke-sub-80518402-mssm4\" (UID: \"af9f361e-cebe-4309-93cb-c609ef835889\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-80518402-mssm4"
Oct 30 19:53:49.791881 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:53:49.791856    4892 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-80518402-mssm4"
Oct 30 19:53:50.172558 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:53:50.172524    4892 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-80518402-mssm4"]
Oct 30 19:53:50.172702 ip-10-0-132-82 kubenswrapper[4892]: W1030 19:53:50.172578    4892 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podaf9f361e_cebe_4309_93cb_c609ef835889.slice/crio-b92b04e3eb2dcea2c8d38657e4e32992ec63acb25729eb44170b586dd80e6094 WatchSource:0}: Error finding container b92b04e3eb2dcea2c8d38657e4e32992ec63acb25729eb44170b586dd80e6094: Status 404 returned error can't find the container with id b92b04e3eb2dcea2c8d38657e4e32992ec63acb25729eb44170b586dd80e6094
Oct 30 19:53:50.688937 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:53:50.688899    4892 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-80518402-mssm4" event={"ID":"af9f361e-cebe-4309-93cb-c609ef835889","Type":"ContainerStarted","Data":"5531a1f3c92b4c7ca4bd40413771c96948f586cd8ca2a54db0009146b7562663"}
Oct 30 19:53:50.689308 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:53:50.688974    4892 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-80518402-mssm4" event={"ID":"af9f361e-cebe-4309-93cb-c609ef835889","Type":"ContainerStarted","Data":"b92b04e3eb2dcea2c8d38657e4e32992ec63acb25729eb44170b586dd80e6094"}
Oct 30 19:53:50.805902 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:53:50.805803    4892 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-80518402-mssm4" podStartSLOduration=1.8057895990000001 podStartE2EDuration="1.805789599s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:53:49 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:53:50.803030041 +0000 UTC m=+2278.373495641" watchObservedRunningTime="2026-10-30 19:53:50.805789599 +0000 UTC m=+2278.376255198"
Oct 30 19:53:51.698202 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:53:51.698166    4892 generic.go:356] "Generic (PLEG): container finished" podID="af9f361e-cebe-4309-93cb-c609ef835889" containerID="5531a1f3c92b4c7ca4bd40413771c96948f586cd8ca2a54db0009146b7562663" exitCode=0
Oct 30 19:53:51.698651 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:53:51.698249    4892 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-80518402-mssm4" event={"ID":"af9f361e-cebe-4309-93cb-c609ef835889","Type":"ContainerDied","Data":"5531a1f3c92b4c7ca4bd40413771c96948f586cd8ca2a54db0009146b7562663"}
Oct 30 19:53:52.610615 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:53:52.610590    4892 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-api-a-6caff5-fcddd97f9-vjnf6"
Oct 30 19:53:52.904034 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:53:52.904007    4892 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-80518402-mssm4"
Oct 30 19:53:53.075518 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:53:53.075493    4892 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/af9f361e-cebe-4309-93cb-c609ef835889-kube-api-access-4dcwm\" (UniqueName: \"kubernetes.io/projected/af9f361e-cebe-4309-93cb-c609ef835889-kube-api-access-4dcwm\") pod \"af9f361e-cebe-4309-93cb-c609ef835889\" (UID: \"af9f361e-cebe-4309-93cb-c609ef835889\") "
Oct 30 19:53:53.075696 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:53:53.075544    4892 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/af9f361e-cebe-4309-93cb-c609ef835889-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/af9f361e-cebe-4309-93cb-c609ef835889-maas-api-openshift-ca\") pod \"af9f361e-cebe-4309-93cb-c609ef835889\" (UID: \"af9f361e-cebe-4309-93cb-c609ef835889\") "
Oct 30 19:53:53.075772 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:53:53.075754    4892 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/af9f361e-cebe-4309-93cb-c609ef835889-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/af9f361e-cebe-4309-93cb-c609ef835889-maas-api-xks-ca\") pod \"af9f361e-cebe-4309-93cb-c609ef835889\" (UID: \"af9f361e-cebe-4309-93cb-c609ef835889\") "
Oct 30 19:53:53.075943 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:53:53.075917    4892 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/af9f361e-cebe-4309-93cb-c609ef835889-maas-api-openshift-ca" pod "af9f361e-cebe-4309-93cb-c609ef835889" (UID: "af9f361e-cebe-4309-93cb-c609ef835889"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 19:53:53.076276 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:53:53.076254    4892 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/af9f361e-cebe-4309-93cb-c609ef835889-maas-api-openshift-ca\") on node \"ip-10-0-132-82.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 19:53:53.077956 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:53:53.077932    4892 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/af9f361e-cebe-4309-93cb-c609ef835889-maas-api-xks-ca" pod "af9f361e-cebe-4309-93cb-c609ef835889" (UID: "af9f361e-cebe-4309-93cb-c609ef835889"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 19:53:53.078152 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:53:53.078136    4892 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/af9f361e-cebe-4309-93cb-c609ef835889-kube-api-access-4dcwm" pod "af9f361e-cebe-4309-93cb-c609ef835889" (UID: "af9f361e-cebe-4309-93cb-c609ef835889"). InnerVolumeSpecName "kube-api-access-4dcwm". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:53:53.177742 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:53:53.177711    4892 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4dcwm\" (UniqueName: \"kubernetes.io/projected/af9f361e-cebe-4309-93cb-c609ef835889-kube-api-access-4dcwm\") on node \"ip-10-0-132-82.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 19:53:53.177742 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:53:53.177742    4892 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/af9f361e-cebe-4309-93cb-c609ef835889-maas-api-xks-ca\") on node \"ip-10-0-132-82.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 19:53:53.619399 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:53:53.619324    4892 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-api-a-6caff5-86dcd5d66d-9r2cn"
Oct 30 19:53:53.716655 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:53:53.716631    4892 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-80518402-mssm4"
Oct 30 19:53:53.716796 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:53:53.716636    4892 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-80518402-mssm4" event={"ID":"af9f361e-cebe-4309-93cb-c609ef835889","Type":"ContainerDied","Data":"b92b04e3eb2dcea2c8d38657e4e32992ec63acb25729eb44170b586dd80e6094"}
Oct 30 19:53:53.716796 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:53:53.716739    4892 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="b92b04e3eb2dcea2c8d38657e4e32992ec63acb25729eb44170b586dd80e6094"
Oct 30 19:53:54.921023 ip-10-0-132-82 kubenswrapper[4892]: E1030 19:53:54.920491    4892 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-jrm87" containerName="extractor"
Oct 30 19:53:57.332744 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:53:57.332712    4892 scope.go:122] "RemoveContainer" containerID="3828be61c54deca37236da277e803386906529cc06b5aafd70d7f10a0f4a9e2f"
Oct 30 19:54:13.470808 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:54:13.470716    4892 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-75b38d-9b874d7b7-z56js"]
Oct 30 19:54:13.471903 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:54:13.471879    4892 cpu_manager.go:414] "containerMap: removing container" podUID="af9f361e-cebe-4309-93cb-c609ef835889" containerName="revoke-keys"
Oct 30 19:54:13.471903 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:54:13.471901    4892 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="af9f361e-cebe-4309-93cb-c609ef835889" containerName="revoke-keys"
Oct 30 19:54:13.472130 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:54:13.472081    4892 memory_manager.go:365] "RemoveStaleState removing state" podUID="af9f361e-cebe-4309-93cb-c609ef835889" containerName="revoke-keys"
Oct 30 19:54:13.478318 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:54:13.478284    4892 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-75b38d-9b874d7b7-z56js"
Oct 30 19:54:13.498464 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:54:13.498440    4892 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/558323d9-9532-440c-a95b-f367d8d0df78-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-b-75b38d-9b874d7b7-z56js\" (UID: \"558323d9-9532-440c-a95b-f367d8d0df78\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-75b38d-9b874d7b7-z56js"
Oct 30 19:54:13.498637 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:54:13.498621    4892 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sgrvk\" (UniqueName: \"kubernetes.io/projected/558323d9-9532-440c-a95b-f367d8d0df78-kube-api-access-sgrvk\") pod \"payload-pre-processing-e2e-inf-b-75b38d-9b874d7b7-z56js\" (UID: \"558323d9-9532-440c-a95b-f367d8d0df78\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-75b38d-9b874d7b7-z56js"
Oct 30 19:54:13.530739 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:54:13.530718    4892 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-75b38d-9b874d7b7-z56js"]
Oct 30 19:54:13.574410 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:54:13.574378    4892 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-inf-b-75b38d\""
Oct 30 19:54:13.574553 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:54:13.574438    4892 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-inf-b-75b38d-dockercfg-6877w\""
Oct 30 19:54:13.599332 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:54:13.599303    4892 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sgrvk\" (UniqueName: \"kubernetes.io/projected/558323d9-9532-440c-a95b-f367d8d0df78-kube-api-access-sgrvk\") pod \"payload-pre-processing-e2e-inf-b-75b38d-9b874d7b7-z56js\" (UID: \"558323d9-9532-440c-a95b-f367d8d0df78\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-75b38d-9b874d7b7-z56js"
Oct 30 19:54:13.599471 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:54:13.599449    4892 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/558323d9-9532-440c-a95b-f367d8d0df78-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-b-75b38d-9b874d7b7-z56js\" (UID: \"558323d9-9532-440c-a95b-f367d8d0df78\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-75b38d-9b874d7b7-z56js"
Oct 30 19:54:13.600263 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:54:13.600238    4892 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/558323d9-9532-440c-a95b-f367d8d0df78-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-b-75b38d-9b874d7b7-z56js\" (UID: \"558323d9-9532-440c-a95b-f367d8d0df78\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-75b38d-9b874d7b7-z56js"
Oct 30 19:54:13.660351 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:54:13.660315    4892 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-sgrvk\" (UniqueName: \"kubernetes.io/projected/558323d9-9532-440c-a95b-f367d8d0df78-kube-api-access-sgrvk\") pod \"payload-pre-processing-e2e-inf-b-75b38d-9b874d7b7-z56js\" (UID: \"558323d9-9532-440c-a95b-f367d8d0df78\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-75b38d-9b874d7b7-z56js"
Oct 30 19:54:13.802351 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:54:13.802264    4892 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-75b38d-9b874d7b7-z56js"
Oct 30 19:54:14.095647 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:54:14.095574    4892 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-fsfvk"]
Oct 30 19:54:14.206749 ip-10-0-132-82 kubenswrapper[4892]: W1030 19:54:14.206717    4892 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod558323d9_9532_440c_a95b_f367d8d0df78.slice/crio-e0cd12b1ae632dc97f71f3f2ff0b534dae0cb778511c219adc0fdc69f4677c8d WatchSource:0}: Error finding container e0cd12b1ae632dc97f71f3f2ff0b534dae0cb778511c219adc0fdc69f4677c8d: Status 404 returned error can't find the container with id e0cd12b1ae632dc97f71f3f2ff0b534dae0cb778511c219adc0fdc69f4677c8d
Oct 30 19:54:14.209728 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:54:14.209703    4892 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-75b38d-9b874d7b7-z56js"]
Oct 30 19:54:14.950727 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:54:14.950635    4892 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-75b38d-9b874d7b7-z56js" event={"ID":"558323d9-9532-440c-a95b-f367d8d0df78","Type":"ContainerStarted","Data":"ba6aa44eddee4783a33fa700389847bca36090d8a680c12dc0d7f1f80e06c634"}
Oct 30 19:54:14.951162 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:54:14.950754    4892 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-75b38d-9b874d7b7-z56js" event={"ID":"558323d9-9532-440c-a95b-f367d8d0df78","Type":"ContainerStarted","Data":"e0cd12b1ae632dc97f71f3f2ff0b534dae0cb778511c219adc0fdc69f4677c8d"}
Oct 30 19:54:14.951162 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:54:14.951035    4892 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-75b38d-9b874d7b7-z56js"
Oct 30 19:54:15.069924 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:54:15.069862    4892 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-75b38d-9b874d7b7-z56js" podStartSLOduration=2.069843829 podStartE2EDuration="2.069843829s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:54:13 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:54:15.064535878 +0000 UTC m=+2302.635001477" watchObservedRunningTime="2026-10-30 19:54:15.069843829 +0000 UTC m=+2302.640309430"
Oct 30 19:54:19.048227 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:54:19.048194    4892 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2e1c40f5-mh9t2"]
Oct 30 19:54:19.097073 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:54:19.097044    4892 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2e1c40f5-mh9t2"]
Oct 30 19:54:20.573642 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:54:20.573602    4892 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="257ea0f4-ca30-42ea-ae62-c4975741b48b" path="/var/lib/kubelet/pods/257ea0f4-ca30-42ea-ae62-c4975741b48b/volumes"
Oct 30 19:54:24.846029 ip-10-0-132-82 kubenswrapper[4892]: E1030 19:54:24.845943    4892 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-jrm87" containerName="extractor"
Oct 30 19:54:25.962259 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:54:25.962227    4892 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-75b38d-9b874d7b7-z56js"
Oct 30 19:54:54.845252 ip-10-0-132-82 kubenswrapper[4892]: E1030 19:54:54.845214    4892 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-jrm87" containerName="extractor"
Oct 30 19:54:57.392749 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:54:57.392710    4892 scope.go:122] "RemoveContainer" containerID="fd9edaa409d311d255f364d9a1c76b454663df1dc342150af3006b900660fca2"
Oct 30 19:55:03.608135 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:55:03.608099    4892 patch_prober.go:29] interesting pod/e2e-inf-a-898968-openshift-default-b55dfc9c9-gmck5 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.192:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 30 19:55:03.610433 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:55:03.608149    4892 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-inf-a-898968-openshift-default-b55dfc9c9-gmck5" podUID="95cba503-314a-4b92-a90b-c23b551ea329" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.192:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:55:05.441795 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:55:05.441761    4892 generic.go:356] "Generic (PLEG): container finished" podID="95cba503-314a-4b92-a90b-c23b551ea329" containerID="30d80d93579e16c3d80d44735723524b0cb4e15e7b0540f54d72969c46e9ec1e" exitCode=0
Oct 30 19:55:05.442321 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:55:05.441831    4892 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-898968-openshift-default-b55dfc9c9-gmck5" event={"ID":"95cba503-314a-4b92-a90b-c23b551ea329","Type":"ContainerDied","Data":"30d80d93579e16c3d80d44735723524b0cb4e15e7b0540f54d72969c46e9ec1e"}
Oct 30 19:55:05.442321 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:55:05.442302    4892 scope.go:122] "RemoveContainer" containerID="30d80d93579e16c3d80d44735723524b0cb4e15e7b0540f54d72969c46e9ec1e"
Oct 30 19:55:05.444231 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:55:05.444198    4892 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892000Ki","pods":"250"}
Oct 30 19:55:05.444316 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:55:05.444272    4892 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892000Ki","pods":"250"}
Oct 30 19:55:05.444316 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:55:05.444310    4892 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892000Ki","pods":"250"}
Oct 30 19:55:06.462089 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:55:06.462056    4892 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-898968-openshift-default-b55dfc9c9-gmck5" event={"ID":"95cba503-314a-4b92-a90b-c23b551ea329","Type":"ContainerStarted","Data":"c406d824e6f76efb5e69812a9ad3ee202c4adb8ee77210c44ab23930d4f19069"}
Oct 30 19:55:07.141994 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:55:07.136291    4892 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-a-898968-openshift-default-b55dfc9c9-gmck5"
Oct 30 19:55:07.141994 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:55:07.136444    4892 patch_prober.go:29] interesting pod/e2e-inf-a-898968-openshift-default-b55dfc9c9-gmck5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.192:15021/healthz/ready\": dial tcp 10.132.0.192:15021: connect: connection refused" start-of-body=
Oct 30 19:55:07.141994 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:55:07.136488    4892 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-898968-openshift-default-b55dfc9c9-gmck5" podUID="95cba503-314a-4b92-a90b-c23b551ea329" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.192:15021/healthz/ready\": dial tcp 10.132.0.192:15021: connect: connection refused"
Oct 30 19:55:08.131370 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:55:08.131330    4892 patch_prober.go:29] interesting pod/e2e-inf-a-898968-openshift-default-b55dfc9c9-gmck5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.192:15021/healthz/ready\": dial tcp 10.132.0.192:15021: connect: connection refused" start-of-body=
Oct 30 19:55:08.132038 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:55:08.132002    4892 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-898968-openshift-default-b55dfc9c9-gmck5" podUID="95cba503-314a-4b92-a90b-c23b551ea329" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.192:15021/healthz/ready\": dial tcp 10.132.0.192:15021: connect: connection refused"
Oct 30 19:55:09.132387 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:55:09.132351    4892 patch_prober.go:29] interesting pod/e2e-inf-a-898968-openshift-default-b55dfc9c9-gmck5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.192:15021/healthz/ready\": dial tcp 10.132.0.192:15021: connect: connection refused" start-of-body=
Oct 30 19:55:09.133021 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:55:09.132986    4892 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-898968-openshift-default-b55dfc9c9-gmck5" podUID="95cba503-314a-4b92-a90b-c23b551ea329" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.192:15021/healthz/ready\": dial tcp 10.132.0.192:15021: connect: connection refused"
Oct 30 19:55:10.131363 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:55:10.131321    4892 patch_prober.go:29] interesting pod/e2e-inf-a-898968-openshift-default-b55dfc9c9-gmck5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.192:15021/healthz/ready\": dial tcp 10.132.0.192:15021: connect: connection refused" start-of-body=
Oct 30 19:55:10.151990 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:55:10.131647    4892 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-898968-openshift-default-b55dfc9c9-gmck5" podUID="95cba503-314a-4b92-a90b-c23b551ea329" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.192:15021/healthz/ready\": dial tcp 10.132.0.192:15021: connect: connection refused"
Oct 30 19:55:11.136327 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:55:11.136287    4892 patch_prober.go:29] interesting pod/e2e-inf-a-898968-openshift-default-b55dfc9c9-gmck5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.192:15021/healthz/ready\": dial tcp 10.132.0.192:15021: connect: connection refused" start-of-body=
Oct 30 19:55:11.136628 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:55:11.136599    4892 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-898968-openshift-default-b55dfc9c9-gmck5" podUID="95cba503-314a-4b92-a90b-c23b551ea329" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.192:15021/healthz/ready\": dial tcp 10.132.0.192:15021: connect: connection refused"
Oct 30 19:55:12.134823 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:55:12.134790    4892 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-a-898968-openshift-default-b55dfc9c9-gmck5"
Oct 30 19:55:12.135395 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:55:12.135350    4892 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-a-898968-openshift-default-b55dfc9c9-gmck5"
Oct 30 19:55:12.136027 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:55:12.136007    4892 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-a-898968-openshift-default-b55dfc9c9-gmck5"
Oct 30 19:55:22.738869 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:55:22.738832    4892 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-lgwz5"]
Oct 30 19:55:22.739424 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:55:22.739087    4892 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-lgwz5" podUID="c63df543-fcb1-4bee-9d52-c50286ac6f91" containerName="istio-proxy" containerID="cri-o://deff73fe4ec3bff9c4fc74f415b493a91a87921ea03ab30b4b7ccc64fab8a84c" gracePeriod=30
Oct 30 19:55:22.746566 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:55:22.746535    4892 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-lqrjf"]
Oct 30 19:55:22.746790 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:55:22.746763    4892 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-lqrjf" podUID="c5510c1c-0da3-4b39-95ee-a9195c32e990" containerName="istio-proxy" containerID="cri-o://cef76da3f34b3aaeb3e66a0565011103bbfd83112c3a11cc808885abcec85f90" gracePeriod=30
Oct 30 19:55:22.751009 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:55:22.750981    4892 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-2pqkg"]
Oct 30 19:55:22.751171 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:55:22.751152    4892 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-2pqkg" podUID="efd3e1ce-d3c6-419d-9b5f-dee2907b2cb6" containerName="istio-proxy" containerID="cri-o://2e8b20742702850f9d506aa439396f9c00dba206ad60eef9d015213953bd4fbd" gracePeriod=30
Oct 30 19:55:22.787719 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:55:22.787686    4892 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-sj8lf"]
Oct 30 19:55:22.787882 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:55:22.787859    4892 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-sj8lf" podUID="aaea6d2c-ad01-47ba-9aa4-8f90ffff6901" containerName="istio-proxy" containerID="cri-o://f7d551895cf5eb0329c0857dfbb29fe3bab5f3901961469267f575fad9f849e4" gracePeriod=30
Oct 30 19:55:23.415855 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:55:23.415817    4892 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-fsfvk"]
Oct 30 19:55:24.846457 ip-10-0-132-82 kubenswrapper[4892]: E1030 19:55:24.846417    4892 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-jrm87" containerName="extractor"
Oct 30 19:55:28.102186 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:55:28.102161    4892 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-2pqkg"
Oct 30 19:55:28.150002 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:55:28.142320    4892 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-lqrjf"
Oct 30 19:55:28.211986 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:55:28.208522    4892 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c5510c1c-0da3-4b39-95ee-a9195c32e990-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c5510c1c-0da3-4b39-95ee-a9195c32e990-workload-certs\") pod \"c5510c1c-0da3-4b39-95ee-a9195c32e990\" (UID: \"c5510c1c-0da3-4b39-95ee-a9195c32e990\") "
Oct 30 19:55:28.211986 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:55:28.208631    4892 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c5510c1c-0da3-4b39-95ee-a9195c32e990-istio-token\" (UniqueName: \"kubernetes.io/projected/c5510c1c-0da3-4b39-95ee-a9195c32e990-istio-token\") pod \"c5510c1c-0da3-4b39-95ee-a9195c32e990\" (UID: \"c5510c1c-0da3-4b39-95ee-a9195c32e990\") "
Oct 30 19:55:28.211986 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:55:28.208666    4892 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/efd3e1ce-d3c6-419d-9b5f-dee2907b2cb6-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/efd3e1ce-d3c6-419d-9b5f-dee2907b2cb6-credential-socket\") pod \"efd3e1ce-d3c6-419d-9b5f-dee2907b2cb6\" (UID: \"efd3e1ce-d3c6-419d-9b5f-dee2907b2cb6\") "
Oct 30 19:55:28.211986 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:55:28.208698    4892 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/efd3e1ce-d3c6-419d-9b5f-dee2907b2cb6-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/efd3e1ce-d3c6-419d-9b5f-dee2907b2cb6-workload-certs\") pod \"efd3e1ce-d3c6-419d-9b5f-dee2907b2cb6\" (UID: \"efd3e1ce-d3c6-419d-9b5f-dee2907b2cb6\") "
Oct 30 19:55:28.211986 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:55:28.208735    4892 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c5510c1c-0da3-4b39-95ee-a9195c32e990-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c5510c1c-0da3-4b39-95ee-a9195c32e990-workload-socket\") pod \"c5510c1c-0da3-4b39-95ee-a9195c32e990\" (UID: \"c5510c1c-0da3-4b39-95ee-a9195c32e990\") "
Oct 30 19:55:28.211986 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:55:28.208775    4892 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c5510c1c-0da3-4b39-95ee-a9195c32e990-kube-api-access-6zkft\" (UniqueName: \"kubernetes.io/projected/c5510c1c-0da3-4b39-95ee-a9195c32e990-kube-api-access-6zkft\") pod \"c5510c1c-0da3-4b39-95ee-a9195c32e990\" (UID: \"c5510c1c-0da3-4b39-95ee-a9195c32e990\") "
Oct 30 19:55:28.211986 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:55:28.208911    4892 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/efd3e1ce-d3c6-419d-9b5f-dee2907b2cb6-kube-api-access-w95lm\" (UniqueName: \"kubernetes.io/projected/efd3e1ce-d3c6-419d-9b5f-dee2907b2cb6-kube-api-access-w95lm\") pod \"efd3e1ce-d3c6-419d-9b5f-dee2907b2cb6\" (UID: \"efd3e1ce-d3c6-419d-9b5f-dee2907b2cb6\") "
Oct 30 19:55:28.211986 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:55:28.208952    4892 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/efd3e1ce-d3c6-419d-9b5f-dee2907b2cb6-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/efd3e1ce-d3c6-419d-9b5f-dee2907b2cb6-istio-data\") pod \"efd3e1ce-d3c6-419d-9b5f-dee2907b2cb6\" (UID: \"efd3e1ce-d3c6-419d-9b5f-dee2907b2cb6\") "
Oct 30 19:55:28.211986 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:55:28.209008    4892 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c5510c1c-0da3-4b39-95ee-a9195c32e990-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c5510c1c-0da3-4b39-95ee-a9195c32e990-istiod-ca-cert\") pod \"c5510c1c-0da3-4b39-95ee-a9195c32e990\" (UID: \"c5510c1c-0da3-4b39-95ee-a9195c32e990\") "
Oct 30 19:55:28.211986 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:55:28.209042    4892 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/efd3e1ce-d3c6-419d-9b5f-dee2907b2cb6-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/efd3e1ce-d3c6-419d-9b5f-dee2907b2cb6-istio-podinfo\") pod \"efd3e1ce-d3c6-419d-9b5f-dee2907b2cb6\" (UID: \"efd3e1ce-d3c6-419d-9b5f-dee2907b2cb6\") "
Oct 30 19:55:28.211986 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:55:28.209085    4892 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/efd3e1ce-d3c6-419d-9b5f-dee2907b2cb6-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/efd3e1ce-d3c6-419d-9b5f-dee2907b2cb6-workload-socket\") pod \"efd3e1ce-d3c6-419d-9b5f-dee2907b2cb6\" (UID: \"efd3e1ce-d3c6-419d-9b5f-dee2907b2cb6\") "
Oct 30 19:55:28.211986 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:55:28.209126    4892 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c5510c1c-0da3-4b39-95ee-a9195c32e990-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c5510c1c-0da3-4b39-95ee-a9195c32e990-credential-socket\") pod \"c5510c1c-0da3-4b39-95ee-a9195c32e990\" (UID: \"c5510c1c-0da3-4b39-95ee-a9195c32e990\") "
Oct 30 19:55:28.211986 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:55:28.209156    4892 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/c5510c1c-0da3-4b39-95ee-a9195c32e990-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c5510c1c-0da3-4b39-95ee-a9195c32e990-istio-podinfo\") pod \"c5510c1c-0da3-4b39-95ee-a9195c32e990\" (UID: \"c5510c1c-0da3-4b39-95ee-a9195c32e990\") "
Oct 30 19:55:28.211986 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:55:28.209183    4892 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/efd3e1ce-d3c6-419d-9b5f-dee2907b2cb6-istio-token\" (UniqueName: \"kubernetes.io/projected/efd3e1ce-d3c6-419d-9b5f-dee2907b2cb6-istio-token\") pod \"efd3e1ce-d3c6-419d-9b5f-dee2907b2cb6\" (UID: \"efd3e1ce-d3c6-419d-9b5f-dee2907b2cb6\") "
Oct 30 19:55:28.211986 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:55:28.209230    4892 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/efd3e1ce-d3c6-419d-9b5f-dee2907b2cb6-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/efd3e1ce-d3c6-419d-9b5f-dee2907b2cb6-istio-envoy\") pod \"efd3e1ce-d3c6-419d-9b5f-dee2907b2cb6\" (UID: \"efd3e1ce-d3c6-419d-9b5f-dee2907b2cb6\") "
Oct 30 19:55:28.211986 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:55:28.209329    4892 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c5510c1c-0da3-4b39-95ee-a9195c32e990-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c5510c1c-0da3-4b39-95ee-a9195c32e990-istio-envoy\") pod \"c5510c1c-0da3-4b39-95ee-a9195c32e990\" (UID: \"c5510c1c-0da3-4b39-95ee-a9195c32e990\") "
Oct 30 19:55:28.211986 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:55:28.209374    4892 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/efd3e1ce-d3c6-419d-9b5f-dee2907b2cb6-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/efd3e1ce-d3c6-419d-9b5f-dee2907b2cb6-istiod-ca-cert\") pod \"efd3e1ce-d3c6-419d-9b5f-dee2907b2cb6\" (UID: \"efd3e1ce-d3c6-419d-9b5f-dee2907b2cb6\") "
Oct 30 19:55:28.211986 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:55:28.209431    4892 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c5510c1c-0da3-4b39-95ee-a9195c32e990-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c5510c1c-0da3-4b39-95ee-a9195c32e990-istio-data\") pod \"c5510c1c-0da3-4b39-95ee-a9195c32e990\" (UID: \"c5510c1c-0da3-4b39-95ee-a9195c32e990\") "
Oct 30 19:55:28.211986 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:55:28.210345    4892 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c5510c1c-0da3-4b39-95ee-a9195c32e990-istio-data" pod "c5510c1c-0da3-4b39-95ee-a9195c32e990" (UID: "c5510c1c-0da3-4b39-95ee-a9195c32e990"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:55:28.211986 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:55:28.210467    4892 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c5510c1c-0da3-4b39-95ee-a9195c32e990-istiod-ca-cert" pod "c5510c1c-0da3-4b39-95ee-a9195c32e990" (UID: "c5510c1c-0da3-4b39-95ee-a9195c32e990"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 19:55:28.211986 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:55:28.210813    4892 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/efd3e1ce-d3c6-419d-9b5f-dee2907b2cb6-istiod-ca-cert" pod "efd3e1ce-d3c6-419d-9b5f-dee2907b2cb6" (UID: "efd3e1ce-d3c6-419d-9b5f-dee2907b2cb6"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 19:55:28.211986 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:55:28.210886    4892 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/efd3e1ce-d3c6-419d-9b5f-dee2907b2cb6-workload-socket" pod "efd3e1ce-d3c6-419d-9b5f-dee2907b2cb6" (UID: "efd3e1ce-d3c6-419d-9b5f-dee2907b2cb6"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:55:28.211986 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:55:28.211014    4892 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c5510c1c-0da3-4b39-95ee-a9195c32e990-credential-socket" pod "c5510c1c-0da3-4b39-95ee-a9195c32e990" (UID: "c5510c1c-0da3-4b39-95ee-a9195c32e990"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:55:28.211986 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:55:28.211021    4892 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/efd3e1ce-d3c6-419d-9b5f-dee2907b2cb6-workload-certs" pod "efd3e1ce-d3c6-419d-9b5f-dee2907b2cb6" (UID: "efd3e1ce-d3c6-419d-9b5f-dee2907b2cb6"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:55:28.211986 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:55:28.211200    4892 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/efd3e1ce-d3c6-419d-9b5f-dee2907b2cb6-credential-socket" pod "efd3e1ce-d3c6-419d-9b5f-dee2907b2cb6" (UID: "efd3e1ce-d3c6-419d-9b5f-dee2907b2cb6"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:55:28.211986 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:55:28.211368    4892 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c5510c1c-0da3-4b39-95ee-a9195c32e990-workload-certs" pod "c5510c1c-0da3-4b39-95ee-a9195c32e990" (UID: "c5510c1c-0da3-4b39-95ee-a9195c32e990"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:55:28.211986 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:55:28.211547    4892 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c5510c1c-0da3-4b39-95ee-a9195c32e990-workload-socket" pod "c5510c1c-0da3-4b39-95ee-a9195c32e990" (UID: "c5510c1c-0da3-4b39-95ee-a9195c32e990"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:55:28.213681 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:55:28.212389    4892 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/efd3e1ce-d3c6-419d-9b5f-dee2907b2cb6-istio-data" pod "efd3e1ce-d3c6-419d-9b5f-dee2907b2cb6" (UID: "efd3e1ce-d3c6-419d-9b5f-dee2907b2cb6"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:55:28.213681 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:55:28.213412    4892 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/c5510c1c-0da3-4b39-95ee-a9195c32e990-istio-podinfo" pod "c5510c1c-0da3-4b39-95ee-a9195c32e990" (UID: "c5510c1c-0da3-4b39-95ee-a9195c32e990"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 19:55:28.213879 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:55:28.213844    4892 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/efd3e1ce-d3c6-419d-9b5f-dee2907b2cb6-kube-api-access-w95lm" pod "efd3e1ce-d3c6-419d-9b5f-dee2907b2cb6" (UID: "efd3e1ce-d3c6-419d-9b5f-dee2907b2cb6"). InnerVolumeSpecName "kube-api-access-w95lm". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:55:28.215437 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:55:28.215417    4892 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c5510c1c-0da3-4b39-95ee-a9195c32e990-istio-token" pod "c5510c1c-0da3-4b39-95ee-a9195c32e990" (UID: "c5510c1c-0da3-4b39-95ee-a9195c32e990"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:55:28.218331 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:55:28.218307    4892 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/efd3e1ce-d3c6-419d-9b5f-dee2907b2cb6-istio-envoy" pod "efd3e1ce-d3c6-419d-9b5f-dee2907b2cb6" (UID: "efd3e1ce-d3c6-419d-9b5f-dee2907b2cb6"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:55:28.218331 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:55:28.218313    4892 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c5510c1c-0da3-4b39-95ee-a9195c32e990-istio-envoy" pod "c5510c1c-0da3-4b39-95ee-a9195c32e990" (UID: "c5510c1c-0da3-4b39-95ee-a9195c32e990"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:55:28.218331 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:55:28.218321    4892 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/efd3e1ce-d3c6-419d-9b5f-dee2907b2cb6-istio-podinfo" pod "efd3e1ce-d3c6-419d-9b5f-dee2907b2cb6" (UID: "efd3e1ce-d3c6-419d-9b5f-dee2907b2cb6"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 19:55:28.218742 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:55:28.218434    4892 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/efd3e1ce-d3c6-419d-9b5f-dee2907b2cb6-istio-token" pod "efd3e1ce-d3c6-419d-9b5f-dee2907b2cb6" (UID: "efd3e1ce-d3c6-419d-9b5f-dee2907b2cb6"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:55:28.219842 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:55:28.219819    4892 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c5510c1c-0da3-4b39-95ee-a9195c32e990-kube-api-access-6zkft" pod "c5510c1c-0da3-4b39-95ee-a9195c32e990" (UID: "c5510c1c-0da3-4b39-95ee-a9195c32e990"). InnerVolumeSpecName "kube-api-access-6zkft". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:55:28.241691 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:55:28.241662    4892 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-sj8lf"
Oct 30 19:55:28.243600 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:55:28.243577    4892 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-lgwz5"
Oct 30 19:55:28.311192 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:55:28.311156    4892 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/aaea6d2c-ad01-47ba-9aa4-8f90ffff6901-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/aaea6d2c-ad01-47ba-9aa4-8f90ffff6901-istio-podinfo\") pod \"aaea6d2c-ad01-47ba-9aa4-8f90ffff6901\" (UID: \"aaea6d2c-ad01-47ba-9aa4-8f90ffff6901\") "
Oct 30 19:55:28.312245 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:55:28.311206    4892 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c63df543-fcb1-4bee-9d52-c50286ac6f91-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c63df543-fcb1-4bee-9d52-c50286ac6f91-istiod-ca-cert\") pod \"c63df543-fcb1-4bee-9d52-c50286ac6f91\" (UID: \"c63df543-fcb1-4bee-9d52-c50286ac6f91\") "
Oct 30 19:55:28.312245 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:55:28.311250    4892 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c63df543-fcb1-4bee-9d52-c50286ac6f91-istio-token\" (UniqueName: \"kubernetes.io/projected/c63df543-fcb1-4bee-9d52-c50286ac6f91-istio-token\") pod \"c63df543-fcb1-4bee-9d52-c50286ac6f91\" (UID: \"c63df543-fcb1-4bee-9d52-c50286ac6f91\") "
Oct 30 19:55:28.312245 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:55:28.311300    4892 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c63df543-fcb1-4bee-9d52-c50286ac6f91-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c63df543-fcb1-4bee-9d52-c50286ac6f91-credential-socket\") pod \"c63df543-fcb1-4bee-9d52-c50286ac6f91\" (UID: \"c63df543-fcb1-4bee-9d52-c50286ac6f91\") "
Oct 30 19:55:28.312245 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:55:28.311347    4892 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c63df543-fcb1-4bee-9d52-c50286ac6f91-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c63df543-fcb1-4bee-9d52-c50286ac6f91-istio-data\") pod \"c63df543-fcb1-4bee-9d52-c50286ac6f91\" (UID: \"c63df543-fcb1-4bee-9d52-c50286ac6f91\") "
Oct 30 19:55:28.312245 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:55:28.311409    4892 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c63df543-fcb1-4bee-9d52-c50286ac6f91-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c63df543-fcb1-4bee-9d52-c50286ac6f91-workload-socket\") pod \"c63df543-fcb1-4bee-9d52-c50286ac6f91\" (UID: \"c63df543-fcb1-4bee-9d52-c50286ac6f91\") "
Oct 30 19:55:28.312245 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:55:28.311436    4892 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/aaea6d2c-ad01-47ba-9aa4-8f90ffff6901-istio-token\" (UniqueName: \"kubernetes.io/projected/aaea6d2c-ad01-47ba-9aa4-8f90ffff6901-istio-token\") pod \"aaea6d2c-ad01-47ba-9aa4-8f90ffff6901\" (UID: \"aaea6d2c-ad01-47ba-9aa4-8f90ffff6901\") "
Oct 30 19:55:28.312245 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:55:28.311492    4892 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/aaea6d2c-ad01-47ba-9aa4-8f90ffff6901-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/aaea6d2c-ad01-47ba-9aa4-8f90ffff6901-istio-envoy\") pod \"aaea6d2c-ad01-47ba-9aa4-8f90ffff6901\" (UID: \"aaea6d2c-ad01-47ba-9aa4-8f90ffff6901\") "
Oct 30 19:55:28.312245 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:55:28.311550    4892 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/aaea6d2c-ad01-47ba-9aa4-8f90ffff6901-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/aaea6d2c-ad01-47ba-9aa4-8f90ffff6901-credential-socket\") pod \"aaea6d2c-ad01-47ba-9aa4-8f90ffff6901\" (UID: \"aaea6d2c-ad01-47ba-9aa4-8f90ffff6901\") "
Oct 30 19:55:28.312245 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:55:28.311601    4892 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c63df543-fcb1-4bee-9d52-c50286ac6f91-kube-api-access-pqlj4\" (UniqueName: \"kubernetes.io/projected/c63df543-fcb1-4bee-9d52-c50286ac6f91-kube-api-access-pqlj4\") pod \"c63df543-fcb1-4bee-9d52-c50286ac6f91\" (UID: \"c63df543-fcb1-4bee-9d52-c50286ac6f91\") "
Oct 30 19:55:28.312245 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:55:28.311655    4892 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/aaea6d2c-ad01-47ba-9aa4-8f90ffff6901-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/aaea6d2c-ad01-47ba-9aa4-8f90ffff6901-workload-socket\") pod \"aaea6d2c-ad01-47ba-9aa4-8f90ffff6901\" (UID: \"aaea6d2c-ad01-47ba-9aa4-8f90ffff6901\") "
Oct 30 19:55:28.312245 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:55:28.311711    4892 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/aaea6d2c-ad01-47ba-9aa4-8f90ffff6901-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/aaea6d2c-ad01-47ba-9aa4-8f90ffff6901-workload-certs\") pod \"aaea6d2c-ad01-47ba-9aa4-8f90ffff6901\" (UID: \"aaea6d2c-ad01-47ba-9aa4-8f90ffff6901\") "
Oct 30 19:55:28.312245 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:55:28.311826    4892 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/aaea6d2c-ad01-47ba-9aa4-8f90ffff6901-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/aaea6d2c-ad01-47ba-9aa4-8f90ffff6901-istio-data\") pod \"aaea6d2c-ad01-47ba-9aa4-8f90ffff6901\" (UID: \"aaea6d2c-ad01-47ba-9aa4-8f90ffff6901\") "
Oct 30 19:55:28.312245 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:55:28.311875    4892 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c63df543-fcb1-4bee-9d52-c50286ac6f91-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c63df543-fcb1-4bee-9d52-c50286ac6f91-istio-envoy\") pod \"c63df543-fcb1-4bee-9d52-c50286ac6f91\" (UID: \"c63df543-fcb1-4bee-9d52-c50286ac6f91\") "
Oct 30 19:55:28.312245 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:55:28.311911    4892 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/aaea6d2c-ad01-47ba-9aa4-8f90ffff6901-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/aaea6d2c-ad01-47ba-9aa4-8f90ffff6901-istiod-ca-cert\") pod \"aaea6d2c-ad01-47ba-9aa4-8f90ffff6901\" (UID: \"aaea6d2c-ad01-47ba-9aa4-8f90ffff6901\") "
Oct 30 19:55:28.312245 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:55:28.311999    4892 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/aaea6d2c-ad01-47ba-9aa4-8f90ffff6901-kube-api-access-6gbvt\" (UniqueName: \"kubernetes.io/projected/aaea6d2c-ad01-47ba-9aa4-8f90ffff6901-kube-api-access-6gbvt\") pod \"aaea6d2c-ad01-47ba-9aa4-8f90ffff6901\" (UID: \"aaea6d2c-ad01-47ba-9aa4-8f90ffff6901\") "
Oct 30 19:55:28.312245 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:55:28.312057    4892 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c63df543-fcb1-4bee-9d52-c50286ac6f91-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c63df543-fcb1-4bee-9d52-c50286ac6f91-workload-certs\") pod \"c63df543-fcb1-4bee-9d52-c50286ac6f91\" (UID: \"c63df543-fcb1-4bee-9d52-c50286ac6f91\") "
Oct 30 19:55:28.312245 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:55:28.312128    4892 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/c63df543-fcb1-4bee-9d52-c50286ac6f91-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c63df543-fcb1-4bee-9d52-c50286ac6f91-istio-podinfo\") pod \"c63df543-fcb1-4bee-9d52-c50286ac6f91\" (UID: \"c63df543-fcb1-4bee-9d52-c50286ac6f91\") "
Oct 30 19:55:28.312245 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:55:28.311867    4892 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c63df543-fcb1-4bee-9d52-c50286ac6f91-istio-data" pod "c63df543-fcb1-4bee-9d52-c50286ac6f91" (UID: "c63df543-fcb1-4bee-9d52-c50286ac6f91"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:55:28.312245 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:55:28.312224    4892 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/aaea6d2c-ad01-47ba-9aa4-8f90ffff6901-workload-socket" pod "aaea6d2c-ad01-47ba-9aa4-8f90ffff6901" (UID: "aaea6d2c-ad01-47ba-9aa4-8f90ffff6901"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:55:28.313325 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:55:28.312302    4892 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c63df543-fcb1-4bee-9d52-c50286ac6f91-workload-socket" pod "c63df543-fcb1-4bee-9d52-c50286ac6f91" (UID: "c63df543-fcb1-4bee-9d52-c50286ac6f91"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:55:28.313325 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:55:28.312451    4892 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/aaea6d2c-ad01-47ba-9aa4-8f90ffff6901-workload-certs" pod "aaea6d2c-ad01-47ba-9aa4-8f90ffff6901" (UID: "aaea6d2c-ad01-47ba-9aa4-8f90ffff6901"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:55:28.313325 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:55:28.312533    4892 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c63df543-fcb1-4bee-9d52-c50286ac6f91-istiod-ca-cert" pod "c63df543-fcb1-4bee-9d52-c50286ac6f91" (UID: "c63df543-fcb1-4bee-9d52-c50286ac6f91"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 19:55:28.313325 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:55:28.312731    4892 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/aaea6d2c-ad01-47ba-9aa4-8f90ffff6901-istio-data" pod "aaea6d2c-ad01-47ba-9aa4-8f90ffff6901" (UID: "aaea6d2c-ad01-47ba-9aa4-8f90ffff6901"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:55:28.313325 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:55:28.312743    4892 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c63df543-fcb1-4bee-9d52-c50286ac6f91-workload-certs" pod "c63df543-fcb1-4bee-9d52-c50286ac6f91" (UID: "c63df543-fcb1-4bee-9d52-c50286ac6f91"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:55:28.313325 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:55:28.313145    4892 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/aaea6d2c-ad01-47ba-9aa4-8f90ffff6901-workload-certs\") on node \"ip-10-0-132-82.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 19:55:28.313325 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:55:28.313166    4892 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c5510c1c-0da3-4b39-95ee-a9195c32e990-istio-token\") on node \"ip-10-0-132-82.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 19:55:28.313325 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:55:28.313181    4892 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/efd3e1ce-d3c6-419d-9b5f-dee2907b2cb6-credential-socket\") on node \"ip-10-0-132-82.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 19:55:28.313325 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:55:28.313199    4892 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/efd3e1ce-d3c6-419d-9b5f-dee2907b2cb6-workload-certs\") on node \"ip-10-0-132-82.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 19:55:28.313325 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:55:28.313213    4892 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c5510c1c-0da3-4b39-95ee-a9195c32e990-workload-socket\") on node \"ip-10-0-132-82.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 19:55:28.313325 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:55:28.313228    4892 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6zkft\" (UniqueName: \"kubernetes.io/projected/c5510c1c-0da3-4b39-95ee-a9195c32e990-kube-api-access-6zkft\") on node \"ip-10-0-132-82.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 19:55:28.313325 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:55:28.313244    4892 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/aaea6d2c-ad01-47ba-9aa4-8f90ffff6901-istio-data\") on node \"ip-10-0-132-82.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 19:55:28.313325 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:55:28.313257    4892 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c63df543-fcb1-4bee-9d52-c50286ac6f91-workload-certs\") on node \"ip-10-0-132-82.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 19:55:28.313325 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:55:28.313272    4892 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c63df543-fcb1-4bee-9d52-c50286ac6f91-istiod-ca-cert\") on node \"ip-10-0-132-82.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 19:55:28.313325 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:55:28.313286    4892 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-w95lm\" (UniqueName: \"kubernetes.io/projected/efd3e1ce-d3c6-419d-9b5f-dee2907b2cb6-kube-api-access-w95lm\") on node \"ip-10-0-132-82.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 19:55:28.313325 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:55:28.313303    4892 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/efd3e1ce-d3c6-419d-9b5f-dee2907b2cb6-istio-data\") on node \"ip-10-0-132-82.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 19:55:28.313325 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:55:28.313305    4892 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/aaea6d2c-ad01-47ba-9aa4-8f90ffff6901-istiod-ca-cert" pod "aaea6d2c-ad01-47ba-9aa4-8f90ffff6901" (UID: "aaea6d2c-ad01-47ba-9aa4-8f90ffff6901"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 19:55:28.313325 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:55:28.313320    4892 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c5510c1c-0da3-4b39-95ee-a9195c32e990-istiod-ca-cert\") on node \"ip-10-0-132-82.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 19:55:28.313325 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:55:28.313335    4892 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/efd3e1ce-d3c6-419d-9b5f-dee2907b2cb6-istio-podinfo\") on node \"ip-10-0-132-82.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 19:55:28.314339 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:55:28.313352    4892 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/efd3e1ce-d3c6-419d-9b5f-dee2907b2cb6-workload-socket\") on node \"ip-10-0-132-82.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 19:55:28.314339 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:55:28.313365    4892 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c5510c1c-0da3-4b39-95ee-a9195c32e990-credential-socket\") on node \"ip-10-0-132-82.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 19:55:28.314339 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:55:28.313380    4892 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c5510c1c-0da3-4b39-95ee-a9195c32e990-istio-podinfo\") on node \"ip-10-0-132-82.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 19:55:28.314339 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:55:28.313394    4892 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/efd3e1ce-d3c6-419d-9b5f-dee2907b2cb6-istio-token\") on node \"ip-10-0-132-82.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 19:55:28.314339 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:55:28.313408    4892 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c63df543-fcb1-4bee-9d52-c50286ac6f91-istio-data\") on node \"ip-10-0-132-82.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 19:55:28.314339 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:55:28.313422    4892 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/efd3e1ce-d3c6-419d-9b5f-dee2907b2cb6-istio-envoy\") on node \"ip-10-0-132-82.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 19:55:28.314339 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:55:28.313436    4892 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c63df543-fcb1-4bee-9d52-c50286ac6f91-workload-socket\") on node \"ip-10-0-132-82.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 19:55:28.314339 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:55:28.313450    4892 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c5510c1c-0da3-4b39-95ee-a9195c32e990-istio-envoy\") on node \"ip-10-0-132-82.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 19:55:28.314339 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:55:28.313464    4892 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/efd3e1ce-d3c6-419d-9b5f-dee2907b2cb6-istiod-ca-cert\") on node \"ip-10-0-132-82.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 19:55:28.314339 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:55:28.313476    4892 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c5510c1c-0da3-4b39-95ee-a9195c32e990-istio-data\") on node \"ip-10-0-132-82.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 19:55:28.314339 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:55:28.313488    4892 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/aaea6d2c-ad01-47ba-9aa4-8f90ffff6901-workload-socket\") on node \"ip-10-0-132-82.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 19:55:28.314339 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:55:28.313504    4892 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c5510c1c-0da3-4b39-95ee-a9195c32e990-workload-certs\") on node \"ip-10-0-132-82.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 19:55:28.314339 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:55:28.314293    4892 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c63df543-fcb1-4bee-9d52-c50286ac6f91-credential-socket" pod "c63df543-fcb1-4bee-9d52-c50286ac6f91" (UID: "c63df543-fcb1-4bee-9d52-c50286ac6f91"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:55:28.315025 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:55:28.314345    4892 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/aaea6d2c-ad01-47ba-9aa4-8f90ffff6901-credential-socket" pod "aaea6d2c-ad01-47ba-9aa4-8f90ffff6901" (UID: "aaea6d2c-ad01-47ba-9aa4-8f90ffff6901"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:55:28.316647 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:55:28.316618    4892 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/c63df543-fcb1-4bee-9d52-c50286ac6f91-istio-podinfo" pod "c63df543-fcb1-4bee-9d52-c50286ac6f91" (UID: "c63df543-fcb1-4bee-9d52-c50286ac6f91"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 19:55:28.321567 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:55:28.321506    4892 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/aaea6d2c-ad01-47ba-9aa4-8f90ffff6901-istio-token" pod "aaea6d2c-ad01-47ba-9aa4-8f90ffff6901" (UID: "aaea6d2c-ad01-47ba-9aa4-8f90ffff6901"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:55:28.322151 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:55:28.322126    4892 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/aaea6d2c-ad01-47ba-9aa4-8f90ffff6901-kube-api-access-6gbvt" pod "aaea6d2c-ad01-47ba-9aa4-8f90ffff6901" (UID: "aaea6d2c-ad01-47ba-9aa4-8f90ffff6901"). InnerVolumeSpecName "kube-api-access-6gbvt". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:55:28.330014 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:55:28.327280    4892 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/aaea6d2c-ad01-47ba-9aa4-8f90ffff6901-istio-envoy" pod "aaea6d2c-ad01-47ba-9aa4-8f90ffff6901" (UID: "aaea6d2c-ad01-47ba-9aa4-8f90ffff6901"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:55:28.330014 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:55:28.327342    4892 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c63df543-fcb1-4bee-9d52-c50286ac6f91-istio-token" pod "c63df543-fcb1-4bee-9d52-c50286ac6f91" (UID: "c63df543-fcb1-4bee-9d52-c50286ac6f91"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:55:28.330014 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:55:28.327368    4892 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c63df543-fcb1-4bee-9d52-c50286ac6f91-kube-api-access-pqlj4" pod "c63df543-fcb1-4bee-9d52-c50286ac6f91" (UID: "c63df543-fcb1-4bee-9d52-c50286ac6f91"). InnerVolumeSpecName "kube-api-access-pqlj4". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:55:28.330014 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:55:28.327468    4892 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/aaea6d2c-ad01-47ba-9aa4-8f90ffff6901-istio-podinfo" pod "aaea6d2c-ad01-47ba-9aa4-8f90ffff6901" (UID: "aaea6d2c-ad01-47ba-9aa4-8f90ffff6901"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 19:55:28.330014 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:55:28.329011    4892 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c63df543-fcb1-4bee-9d52-c50286ac6f91-istio-envoy" pod "c63df543-fcb1-4bee-9d52-c50286ac6f91" (UID: "c63df543-fcb1-4bee-9d52-c50286ac6f91"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:55:28.414455 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:55:28.414425    4892 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c63df543-fcb1-4bee-9d52-c50286ac6f91-istio-envoy\") on node \"ip-10-0-132-82.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 19:55:28.414455 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:55:28.414452    4892 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/aaea6d2c-ad01-47ba-9aa4-8f90ffff6901-istiod-ca-cert\") on node \"ip-10-0-132-82.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 19:55:28.414455 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:55:28.414464    4892 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6gbvt\" (UniqueName: \"kubernetes.io/projected/aaea6d2c-ad01-47ba-9aa4-8f90ffff6901-kube-api-access-6gbvt\") on node \"ip-10-0-132-82.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 19:55:28.414721 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:55:28.414475    4892 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c63df543-fcb1-4bee-9d52-c50286ac6f91-istio-podinfo\") on node \"ip-10-0-132-82.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 19:55:28.414721 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:55:28.414488    4892 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/aaea6d2c-ad01-47ba-9aa4-8f90ffff6901-istio-podinfo\") on node \"ip-10-0-132-82.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 19:55:28.414721 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:55:28.414499    4892 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c63df543-fcb1-4bee-9d52-c50286ac6f91-istio-token\") on node \"ip-10-0-132-82.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 19:55:28.414721 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:55:28.414512    4892 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c63df543-fcb1-4bee-9d52-c50286ac6f91-credential-socket\") on node \"ip-10-0-132-82.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 19:55:28.414721 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:55:28.414523    4892 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/aaea6d2c-ad01-47ba-9aa4-8f90ffff6901-istio-token\") on node \"ip-10-0-132-82.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 19:55:28.414721 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:55:28.414534    4892 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/aaea6d2c-ad01-47ba-9aa4-8f90ffff6901-istio-envoy\") on node \"ip-10-0-132-82.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 19:55:28.414721 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:55:28.414548    4892 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/aaea6d2c-ad01-47ba-9aa4-8f90ffff6901-credential-socket\") on node \"ip-10-0-132-82.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 19:55:28.414721 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:55:28.414561    4892 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pqlj4\" (UniqueName: \"kubernetes.io/projected/c63df543-fcb1-4bee-9d52-c50286ac6f91-kube-api-access-pqlj4\") on node \"ip-10-0-132-82.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 19:55:28.758700 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:55:28.758656    4892 generic.go:356] "Generic (PLEG): container finished" podID="aaea6d2c-ad01-47ba-9aa4-8f90ffff6901" containerID="f7d551895cf5eb0329c0857dfbb29fe3bab5f3901961469267f575fad9f849e4" exitCode=0
Oct 30 19:55:28.758904 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:55:28.758743    4892 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-sj8lf"
Oct 30 19:55:28.758904 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:55:28.758743    4892 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-sj8lf" event={"ID":"aaea6d2c-ad01-47ba-9aa4-8f90ffff6901","Type":"ContainerDied","Data":"f7d551895cf5eb0329c0857dfbb29fe3bab5f3901961469267f575fad9f849e4"}
Oct 30 19:55:28.759064 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:55:28.758906    4892 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-sj8lf" event={"ID":"aaea6d2c-ad01-47ba-9aa4-8f90ffff6901","Type":"ContainerDied","Data":"d80b11f24bb9e8334d3e0bea09aeb14f6402f857fca73e10dc635ccc1075e0f4"}
Oct 30 19:55:28.759064 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:55:28.758939    4892 scope.go:122] "RemoveContainer" containerID="f7d551895cf5eb0329c0857dfbb29fe3bab5f3901961469267f575fad9f849e4"
Oct 30 19:55:28.760881 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:55:28.760857    4892 generic.go:356] "Generic (PLEG): container finished" podID="efd3e1ce-d3c6-419d-9b5f-dee2907b2cb6" containerID="2e8b20742702850f9d506aa439396f9c00dba206ad60eef9d015213953bd4fbd" exitCode=0
Oct 30 19:55:28.761035 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:55:28.760997    4892 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-2pqkg" event={"ID":"efd3e1ce-d3c6-419d-9b5f-dee2907b2cb6","Type":"ContainerDied","Data":"2e8b20742702850f9d506aa439396f9c00dba206ad60eef9d015213953bd4fbd"}
Oct 30 19:55:28.761035 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:55:28.761031    4892 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-2pqkg" event={"ID":"efd3e1ce-d3c6-419d-9b5f-dee2907b2cb6","Type":"ContainerDied","Data":"a5d98329863df4e28dcf48117e12471018be2032b86db49fa33d698b7deb4651"}
Oct 30 19:55:28.761173 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:55:28.761089    4892 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-2pqkg"
Oct 30 19:55:28.764664 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:55:28.764638    4892 generic.go:356] "Generic (PLEG): container finished" podID="c63df543-fcb1-4bee-9d52-c50286ac6f91" containerID="deff73fe4ec3bff9c4fc74f415b493a91a87921ea03ab30b4b7ccc64fab8a84c" exitCode=0
Oct 30 19:55:28.764803 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:55:28.764678    4892 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-lgwz5"
Oct 30 19:55:28.764803 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:55:28.764734    4892 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-lgwz5" event={"ID":"c63df543-fcb1-4bee-9d52-c50286ac6f91","Type":"ContainerDied","Data":"deff73fe4ec3bff9c4fc74f415b493a91a87921ea03ab30b4b7ccc64fab8a84c"}
Oct 30 19:55:28.764803 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:55:28.764770    4892 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-lgwz5" event={"ID":"c63df543-fcb1-4bee-9d52-c50286ac6f91","Type":"ContainerDied","Data":"c19f80eaa33b3667ecb85e2df383a166608f3edf890e9032668b820777400f78"}
Oct 30 19:55:28.766834 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:55:28.766811    4892 generic.go:356] "Generic (PLEG): container finished" podID="c5510c1c-0da3-4b39-95ee-a9195c32e990" containerID="cef76da3f34b3aaeb3e66a0565011103bbfd83112c3a11cc808885abcec85f90" exitCode=0
Oct 30 19:55:28.767076 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:55:28.767015    4892 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-lqrjf" event={"ID":"c5510c1c-0da3-4b39-95ee-a9195c32e990","Type":"ContainerDied","Data":"cef76da3f34b3aaeb3e66a0565011103bbfd83112c3a11cc808885abcec85f90"}
Oct 30 19:55:28.767076 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:55:28.767059    4892 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-lqrjf" event={"ID":"c5510c1c-0da3-4b39-95ee-a9195c32e990","Type":"ContainerDied","Data":"9b6ccaf73a864a336a802613dde37395e73b532d240198ea85b05cc63718ac53"}
Oct 30 19:55:28.767263 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:55:28.767139    4892 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-lqrjf"
Oct 30 19:55:28.773953 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:55:28.773934    4892 scope.go:122] "RemoveContainer" containerID="f7d551895cf5eb0329c0857dfbb29fe3bab5f3901961469267f575fad9f849e4"
Oct 30 19:55:28.774271 ip-10-0-132-82 kubenswrapper[4892]: E1030 19:55:28.774244    4892 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f7d551895cf5eb0329c0857dfbb29fe3bab5f3901961469267f575fad9f849e4\": container with ID starting with f7d551895cf5eb0329c0857dfbb29fe3bab5f3901961469267f575fad9f849e4 not found: ID does not exist" containerID="f7d551895cf5eb0329c0857dfbb29fe3bab5f3901961469267f575fad9f849e4"
Oct 30 19:55:28.774369 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:55:28.774282    4892 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f7d551895cf5eb0329c0857dfbb29fe3bab5f3901961469267f575fad9f849e4"} err="failed to get container status \"f7d551895cf5eb0329c0857dfbb29fe3bab5f3901961469267f575fad9f849e4\": rpc error: code = NotFound desc = could not find container \"f7d551895cf5eb0329c0857dfbb29fe3bab5f3901961469267f575fad9f849e4\": container with ID starting with f7d551895cf5eb0329c0857dfbb29fe3bab5f3901961469267f575fad9f849e4 not found: ID does not exist"
Oct 30 19:55:28.774369 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:55:28.774305    4892 scope.go:122] "RemoveContainer" containerID="2e8b20742702850f9d506aa439396f9c00dba206ad60eef9d015213953bd4fbd"
Oct 30 19:55:28.817086 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:55:28.817065    4892 scope.go:122] "RemoveContainer" containerID="2e8b20742702850f9d506aa439396f9c00dba206ad60eef9d015213953bd4fbd"
Oct 30 19:55:28.817463 ip-10-0-132-82 kubenswrapper[4892]: E1030 19:55:28.817438    4892 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2e8b20742702850f9d506aa439396f9c00dba206ad60eef9d015213953bd4fbd\": container with ID starting with 2e8b20742702850f9d506aa439396f9c00dba206ad60eef9d015213953bd4fbd not found: ID does not exist" containerID="2e8b20742702850f9d506aa439396f9c00dba206ad60eef9d015213953bd4fbd"
Oct 30 19:55:28.817558 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:55:28.817477    4892 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2e8b20742702850f9d506aa439396f9c00dba206ad60eef9d015213953bd4fbd"} err="failed to get container status \"2e8b20742702850f9d506aa439396f9c00dba206ad60eef9d015213953bd4fbd\": rpc error: code = NotFound desc = could not find container \"2e8b20742702850f9d506aa439396f9c00dba206ad60eef9d015213953bd4fbd\": container with ID starting with 2e8b20742702850f9d506aa439396f9c00dba206ad60eef9d015213953bd4fbd not found: ID does not exist"
Oct 30 19:55:28.817558 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:55:28.817500    4892 scope.go:122] "RemoveContainer" containerID="deff73fe4ec3bff9c4fc74f415b493a91a87921ea03ab30b4b7ccc64fab8a84c"
Oct 30 19:55:28.828724 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:55:28.828698    4892 scope.go:122] "RemoveContainer" containerID="deff73fe4ec3bff9c4fc74f415b493a91a87921ea03ab30b4b7ccc64fab8a84c"
Oct 30 19:55:28.829026 ip-10-0-132-82 kubenswrapper[4892]: E1030 19:55:28.829001    4892 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"deff73fe4ec3bff9c4fc74f415b493a91a87921ea03ab30b4b7ccc64fab8a84c\": container with ID starting with deff73fe4ec3bff9c4fc74f415b493a91a87921ea03ab30b4b7ccc64fab8a84c not found: ID does not exist" containerID="deff73fe4ec3bff9c4fc74f415b493a91a87921ea03ab30b4b7ccc64fab8a84c"
Oct 30 19:55:28.829119 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:55:28.829032    4892 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"deff73fe4ec3bff9c4fc74f415b493a91a87921ea03ab30b4b7ccc64fab8a84c"} err="failed to get container status \"deff73fe4ec3bff9c4fc74f415b493a91a87921ea03ab30b4b7ccc64fab8a84c\": rpc error: code = NotFound desc = could not find container \"deff73fe4ec3bff9c4fc74f415b493a91a87921ea03ab30b4b7ccc64fab8a84c\": container with ID starting with deff73fe4ec3bff9c4fc74f415b493a91a87921ea03ab30b4b7ccc64fab8a84c not found: ID does not exist"
Oct 30 19:55:28.829119 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:55:28.829047    4892 scope.go:122] "RemoveContainer" containerID="cef76da3f34b3aaeb3e66a0565011103bbfd83112c3a11cc808885abcec85f90"
Oct 30 19:55:28.838976 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:55:28.838941    4892 scope.go:122] "RemoveContainer" containerID="cef76da3f34b3aaeb3e66a0565011103bbfd83112c3a11cc808885abcec85f90"
Oct 30 19:55:28.839310 ip-10-0-132-82 kubenswrapper[4892]: E1030 19:55:28.839291    4892 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"cef76da3f34b3aaeb3e66a0565011103bbfd83112c3a11cc808885abcec85f90\": container with ID starting with cef76da3f34b3aaeb3e66a0565011103bbfd83112c3a11cc808885abcec85f90 not found: ID does not exist" containerID="cef76da3f34b3aaeb3e66a0565011103bbfd83112c3a11cc808885abcec85f90"
Oct 30 19:55:28.839397 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:55:28.839315    4892 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"cef76da3f34b3aaeb3e66a0565011103bbfd83112c3a11cc808885abcec85f90"} err="failed to get container status \"cef76da3f34b3aaeb3e66a0565011103bbfd83112c3a11cc808885abcec85f90\": rpc error: code = NotFound desc = could not find container \"cef76da3f34b3aaeb3e66a0565011103bbfd83112c3a11cc808885abcec85f90\": container with ID starting with cef76da3f34b3aaeb3e66a0565011103bbfd83112c3a11cc808885abcec85f90 not found: ID does not exist"
Oct 30 19:55:28.927851 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:55:28.927825    4892 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-sj8lf"]
Oct 30 19:55:28.934426 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:55:28.934400    4892 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-sj8lf"]
Oct 30 19:55:29.038942 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:55:29.038913    4892 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-lqrjf"]
Oct 30 19:55:29.045311 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:55:29.045284    4892 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-lqrjf"]
Oct 30 19:55:29.151301 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:55:29.149541    4892 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-lgwz5"]
Oct 30 19:55:29.154493 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:55:29.154466    4892 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-lgwz5"]
Oct 30 19:55:29.261367 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:55:29.261291    4892 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-2pqkg"]
Oct 30 19:55:29.264866 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:55:29.264847    4892 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-2pqkg"]
Oct 30 19:55:30.575953 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:55:30.575911    4892 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="aaea6d2c-ad01-47ba-9aa4-8f90ffff6901" path="/var/lib/kubelet/pods/aaea6d2c-ad01-47ba-9aa4-8f90ffff6901/volumes"
Oct 30 19:55:30.576661 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:55:30.576638    4892 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c5510c1c-0da3-4b39-95ee-a9195c32e990" path="/var/lib/kubelet/pods/c5510c1c-0da3-4b39-95ee-a9195c32e990/volumes"
Oct 30 19:55:30.577319 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:55:30.577298    4892 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c63df543-fcb1-4bee-9d52-c50286ac6f91" path="/var/lib/kubelet/pods/c63df543-fcb1-4bee-9d52-c50286ac6f91/volumes"
Oct 30 19:55:30.578443 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:55:30.578418    4892 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="efd3e1ce-d3c6-419d-9b5f-dee2907b2cb6" path="/var/lib/kubelet/pods/efd3e1ce-d3c6-419d-9b5f-dee2907b2cb6/volumes"
Oct 30 19:55:31.977261 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:55:31.977177    4892 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-dd9b8d79-fdkrb"]
Oct 30 19:55:32.025374 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:55:32.025345    4892 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-dd9b8d79-fdkrb"]
Oct 30 19:55:32.583433 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:55:32.583392    4892 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="6539580c-56b5-4501-abe2-116743a7096b" path="/var/lib/kubelet/pods/6539580c-56b5-4501-abe2-116743a7096b/volumes"
Oct 30 19:55:42.545430 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:55:42.545397    4892 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-dd9b8d79-867946f667-2p8kf"]
Oct 30 19:55:42.546151 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:55:42.546135    4892 cpu_manager.go:414] "containerMap: removing container" podUID="aaea6d2c-ad01-47ba-9aa4-8f90ffff6901" containerName="istio-proxy"
Oct 30 19:55:42.546222 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:55:42.546152    4892 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="aaea6d2c-ad01-47ba-9aa4-8f90ffff6901" containerName="istio-proxy"
Oct 30 19:55:42.546222 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:55:42.546162    4892 cpu_manager.go:414] "containerMap: removing container" podUID="efd3e1ce-d3c6-419d-9b5f-dee2907b2cb6" containerName="istio-proxy"
Oct 30 19:55:42.546222 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:55:42.546167    4892 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="efd3e1ce-d3c6-419d-9b5f-dee2907b2cb6" containerName="istio-proxy"
Oct 30 19:55:42.546222 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:55:42.546195    4892 cpu_manager.go:414] "containerMap: removing container" podUID="c63df543-fcb1-4bee-9d52-c50286ac6f91" containerName="istio-proxy"
Oct 30 19:55:42.546222 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:55:42.546203    4892 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c63df543-fcb1-4bee-9d52-c50286ac6f91" containerName="istio-proxy"
Oct 30 19:55:42.546222 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:55:42.546213    4892 cpu_manager.go:414] "containerMap: removing container" podUID="c5510c1c-0da3-4b39-95ee-a9195c32e990" containerName="istio-proxy"
Oct 30 19:55:42.546222 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:55:42.546217    4892 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c5510c1c-0da3-4b39-95ee-a9195c32e990" containerName="istio-proxy"
Oct 30 19:55:42.546440 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:55:42.546342    4892 memory_manager.go:365] "RemoveStaleState removing state" podUID="c63df543-fcb1-4bee-9d52-c50286ac6f91" containerName="istio-proxy"
Oct 30 19:55:42.546440 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:55:42.546351    4892 memory_manager.go:365] "RemoveStaleState removing state" podUID="c5510c1c-0da3-4b39-95ee-a9195c32e990" containerName="istio-proxy"
Oct 30 19:55:42.546440 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:55:42.546366    4892 memory_manager.go:365] "RemoveStaleState removing state" podUID="aaea6d2c-ad01-47ba-9aa4-8f90ffff6901" containerName="istio-proxy"
Oct 30 19:55:42.546440 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:55:42.546372    4892 memory_manager.go:365] "RemoveStaleState removing state" podUID="efd3e1ce-d3c6-419d-9b5f-dee2907b2cb6" containerName="istio-proxy"
Oct 30 19:55:42.553133 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:55:42.553102    4892 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-dd9b8d79-867946f667-2p8kf"
Oct 30 19:55:42.592939 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:55:42.592913    4892 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xpnvp\" (UniqueName: \"kubernetes.io/projected/27ecc98d-6201-4d1c-b9b2-7df1fabbd48c-kube-api-access-xpnvp\") pod \"payload-pre-processing-e2e-mt-dd9b8d79-867946f667-2p8kf\" (UID: \"27ecc98d-6201-4d1c-b9b2-7df1fabbd48c\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-dd9b8d79-867946f667-2p8kf"
Oct 30 19:55:42.593106 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:55:42.593053    4892 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/27ecc98d-6201-4d1c-b9b2-7df1fabbd48c-plugins-config-volume\") pod \"payload-pre-processing-e2e-mt-dd9b8d79-867946f667-2p8kf\" (UID: \"27ecc98d-6201-4d1c-b9b2-7df1fabbd48c\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-dd9b8d79-867946f667-2p8kf"
Oct 30 19:55:42.609668 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:55:42.609637    4892 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-dd9b8d79-867946f667-2p8kf"]
Oct 30 19:55:42.658189 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:55:42.656297    4892 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-mt-dd9b8d79\""
Oct 30 19:55:42.658189 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:55:42.656474    4892 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-mt-dd9b8d79-dockercfg-mswm7\""
Oct 30 19:55:42.799036 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:55:42.798079    4892 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xpnvp\" (UniqueName: \"kubernetes.io/projected/27ecc98d-6201-4d1c-b9b2-7df1fabbd48c-kube-api-access-xpnvp\") pod \"payload-pre-processing-e2e-mt-dd9b8d79-867946f667-2p8kf\" (UID: \"27ecc98d-6201-4d1c-b9b2-7df1fabbd48c\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-dd9b8d79-867946f667-2p8kf"
Oct 30 19:55:42.799036 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:55:42.798211    4892 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/27ecc98d-6201-4d1c-b9b2-7df1fabbd48c-plugins-config-volume\") pod \"payload-pre-processing-e2e-mt-dd9b8d79-867946f667-2p8kf\" (UID: \"27ecc98d-6201-4d1c-b9b2-7df1fabbd48c\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-dd9b8d79-867946f667-2p8kf"
Oct 30 19:55:42.799274 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:55:42.799249    4892 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/27ecc98d-6201-4d1c-b9b2-7df1fabbd48c-plugins-config-volume\") pod \"payload-pre-processing-e2e-mt-dd9b8d79-867946f667-2p8kf\" (UID: \"27ecc98d-6201-4d1c-b9b2-7df1fabbd48c\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-dd9b8d79-867946f667-2p8kf"
Oct 30 19:55:42.966741 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:55:42.966294    4892 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-dd9b8d79-67c8d6d46-mvgdr"]
Oct 30 19:55:42.974621 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:55:42.974162    4892 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-dd9b8d79-67c8d6d46-mvgdr"]
Oct 30 19:55:42.974621 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:55:42.974286    4892 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-dd9b8d79-67c8d6d46-mvgdr"
Oct 30 19:55:43.018891 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:55:43.018858    4892 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xpnvp\" (UniqueName: \"kubernetes.io/projected/27ecc98d-6201-4d1c-b9b2-7df1fabbd48c-kube-api-access-xpnvp\") pod \"payload-pre-processing-e2e-mt-dd9b8d79-867946f667-2p8kf\" (UID: \"27ecc98d-6201-4d1c-b9b2-7df1fabbd48c\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-dd9b8d79-867946f667-2p8kf"
Oct 30 19:55:43.115257 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:55:43.115176    4892 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-h86bt\" (UniqueName: \"kubernetes.io/projected/4a87b754-ce9f-4ff5-855e-c27f1db74eb9-kube-api-access-h86bt\") pod \"payload-processing-e2e-mt-dd9b8d79-67c8d6d46-mvgdr\" (UID: \"4a87b754-ce9f-4ff5-855e-c27f1db74eb9\") " pod="openshift-ingress/payload-processing-e2e-mt-dd9b8d79-67c8d6d46-mvgdr"
Oct 30 19:55:43.115257 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:55:43.115232    4892 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4a87b754-ce9f-4ff5-855e-c27f1db74eb9-plugins-config-volume\") pod \"payload-processing-e2e-mt-dd9b8d79-67c8d6d46-mvgdr\" (UID: \"4a87b754-ce9f-4ff5-855e-c27f1db74eb9\") " pod="openshift-ingress/payload-processing-e2e-mt-dd9b8d79-67c8d6d46-mvgdr"
Oct 30 19:55:43.174347 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:55:43.174322    4892 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-dd9b8d79-867946f667-2p8kf"
Oct 30 19:55:43.216910 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:55:43.216879    4892 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-h86bt\" (UniqueName: \"kubernetes.io/projected/4a87b754-ce9f-4ff5-855e-c27f1db74eb9-kube-api-access-h86bt\") pod \"payload-processing-e2e-mt-dd9b8d79-67c8d6d46-mvgdr\" (UID: \"4a87b754-ce9f-4ff5-855e-c27f1db74eb9\") " pod="openshift-ingress/payload-processing-e2e-mt-dd9b8d79-67c8d6d46-mvgdr"
Oct 30 19:55:43.217090 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:55:43.216925    4892 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4a87b754-ce9f-4ff5-855e-c27f1db74eb9-plugins-config-volume\") pod \"payload-processing-e2e-mt-dd9b8d79-67c8d6d46-mvgdr\" (UID: \"4a87b754-ce9f-4ff5-855e-c27f1db74eb9\") " pod="openshift-ingress/payload-processing-e2e-mt-dd9b8d79-67c8d6d46-mvgdr"
Oct 30 19:55:43.217654 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:55:43.217629    4892 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4a87b754-ce9f-4ff5-855e-c27f1db74eb9-plugins-config-volume\") pod \"payload-processing-e2e-mt-dd9b8d79-67c8d6d46-mvgdr\" (UID: \"4a87b754-ce9f-4ff5-855e-c27f1db74eb9\") " pod="openshift-ingress/payload-processing-e2e-mt-dd9b8d79-67c8d6d46-mvgdr"
Oct 30 19:55:43.273694 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:55:43.273667    4892 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-h86bt\" (UniqueName: \"kubernetes.io/projected/4a87b754-ce9f-4ff5-855e-c27f1db74eb9-kube-api-access-h86bt\") pod \"payload-processing-e2e-mt-dd9b8d79-67c8d6d46-mvgdr\" (UID: \"4a87b754-ce9f-4ff5-855e-c27f1db74eb9\") " pod="openshift-ingress/payload-processing-e2e-mt-dd9b8d79-67c8d6d46-mvgdr"
Oct 30 19:55:43.294610 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:55:43.291627    4892 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-dd9b8d79-67c8d6d46-mvgdr"
Oct 30 19:55:43.555574 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:55:43.555546    4892 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-dd9b8d79-867946f667-2p8kf"]
Oct 30 19:55:43.556648 ip-10-0-132-82 kubenswrapper[4892]: W1030 19:55:43.556624    4892 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod27ecc98d_6201_4d1c_b9b2_7df1fabbd48c.slice/crio-fea1d9ae0fbbc227a577e5a2e46fa9f23a5eab009aa820f0a5171b8cac33608d WatchSource:0}: Error finding container fea1d9ae0fbbc227a577e5a2e46fa9f23a5eab009aa820f0a5171b8cac33608d: Status 404 returned error can't find the container with id fea1d9ae0fbbc227a577e5a2e46fa9f23a5eab009aa820f0a5171b8cac33608d
Oct 30 19:55:43.678547 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:55:43.678518    4892 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-dd9b8d79-67c8d6d46-mvgdr"]
Oct 30 19:55:43.679388 ip-10-0-132-82 kubenswrapper[4892]: W1030 19:55:43.679360    4892 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4a87b754_ce9f_4ff5_855e_c27f1db74eb9.slice/crio-84df25dfea9df9107ff4ac1eae2c6b9a27c7139f5c016482c28758e4016f0fff WatchSource:0}: Error finding container 84df25dfea9df9107ff4ac1eae2c6b9a27c7139f5c016482c28758e4016f0fff: Status 404 returned error can't find the container with id 84df25dfea9df9107ff4ac1eae2c6b9a27c7139f5c016482c28758e4016f0fff
Oct 30 19:55:43.955217 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:55:43.955181    4892 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-dd9b8d79-867946f667-2p8kf" event={"ID":"27ecc98d-6201-4d1c-b9b2-7df1fabbd48c","Type":"ContainerStarted","Data":"0ae383fca745efdaee5e56af30e55a8eed90e9b01a3afbbc6af0de259ca9dedd"}
Oct 30 19:55:43.955444 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:55:43.955247    4892 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-dd9b8d79-867946f667-2p8kf" event={"ID":"27ecc98d-6201-4d1c-b9b2-7df1fabbd48c","Type":"ContainerStarted","Data":"fea1d9ae0fbbc227a577e5a2e46fa9f23a5eab009aa820f0a5171b8cac33608d"}
Oct 30 19:55:43.955444 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:55:43.955282    4892 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-mt-dd9b8d79-867946f667-2p8kf"
Oct 30 19:55:43.957218 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:55:43.957195    4892 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-dd9b8d79-67c8d6d46-mvgdr" event={"ID":"4a87b754-ce9f-4ff5-855e-c27f1db74eb9","Type":"ContainerStarted","Data":"9b9c0d8c510aec854417af981420eb4c6a3b43e9552f8349707e75d867bb8528"}
Oct 30 19:55:43.957343 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:55:43.957247    4892 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-mt-dd9b8d79-67c8d6d46-mvgdr"
Oct 30 19:55:43.957343 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:55:43.957267    4892 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-dd9b8d79-67c8d6d46-mvgdr" event={"ID":"4a87b754-ce9f-4ff5-855e-c27f1db74eb9","Type":"ContainerStarted","Data":"84df25dfea9df9107ff4ac1eae2c6b9a27c7139f5c016482c28758e4016f0fff"}
Oct 30 19:55:44.070416 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:55:44.070367    4892 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-mt-dd9b8d79-867946f667-2p8kf" podStartSLOduration=2.070352371 podStartE2EDuration="2.070352371s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:55:42 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:55:44.067191144 +0000 UTC m=+2391.637656763" watchObservedRunningTime="2026-10-30 19:55:44.070352371 +0000 UTC m=+2391.640817969"
Oct 30 19:55:44.237505 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:55:44.237390    4892 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-mt-dd9b8d79-67c8d6d46-mvgdr" podStartSLOduration=2.237369452 podStartE2EDuration="2.237369452s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:55:42 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:55:44.195365834 +0000 UTC m=+2391.765831434" watchObservedRunningTime="2026-10-30 19:55:44.237369452 +0000 UTC m=+2391.807835053"
Oct 30 19:55:53.291871 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:55:53.291844    4892 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-tg5xj_11db9cec-653f-4aee-8937-0cf5b367efac/console-operator/3.log"
Oct 30 19:55:53.347784 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:55:53.347759    4892 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-tg5xj_11db9cec-653f-4aee-8937-0cf5b367efac/console-operator/3.log"
Oct 30 19:55:54.845803 ip-10-0-132-82 kubenswrapper[4892]: E1030 19:55:54.845769    4892 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-jrm87" containerName="extractor"
Oct 30 19:55:54.970253 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:55:54.970224    4892 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-mt-dd9b8d79-67c8d6d46-mvgdr"
Oct 30 19:55:54.970429 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:55:54.970290    4892 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-mt-dd9b8d79-867946f667-2p8kf"
Oct 30 19:55:57.424076 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:55:57.424041    4892 scope.go:122] "RemoveContainer" containerID="d32a232be94572370539917d7c91e7532b89869bbc3e1bb63275f2c44283ab6e"
Oct 30 19:55:57.437933 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:55:57.437907    4892 scope.go:122] "RemoveContainer" containerID="a671086f7f53ba5ac31f7400543a8d3182bbdb8df5fb97503ee4c07c04bc7eb8"
Oct 30 19:56:00.932213 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:56:00.932174    4892 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-86e4a7-686g8"]
Oct 30 19:56:00.938823 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:56:00.938790    4892 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-86e4a7-686g8"
Oct 30 19:56:00.990467 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:56:00.990434    4892 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/96460de9-c817-4b3a-b08a-6adb3d836530-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-api-a-86e4a7-686g8\" (UID: \"96460de9-c817-4b3a-b08a-6adb3d836530\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-86e4a7-686g8"
Oct 30 19:56:00.990652 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:56:00.990634    4892 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/96460de9-c817-4b3a-b08a-6adb3d836530-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-api-a-86e4a7-686g8\" (UID: \"96460de9-c817-4b3a-b08a-6adb3d836530\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-86e4a7-686g8"
Oct 30 19:56:00.990711 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:56:00.990682    4892 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-njql2\" (UniqueName: \"kubernetes.io/projected/96460de9-c817-4b3a-b08a-6adb3d836530-kube-api-access-njql2\") pod \"maas-api-revoke-keys-e2e-api-a-86e4a7-686g8\" (UID: \"96460de9-c817-4b3a-b08a-6adb3d836530\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-86e4a7-686g8"
Oct 30 19:56:00.997494 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:56:00.997454    4892 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 19:56:01.046990 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:56:01.044861    4892 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-nlxp4\""
Oct 30 19:56:01.046990 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:56:01.045191    4892 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 30 19:56:01.047245 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:56:01.047134    4892 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 30 19:56:01.052418 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:56:01.052118    4892 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-86e4a7-686g8"]
Oct 30 19:56:01.091503 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:56:01.091472    4892 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/96460de9-c817-4b3a-b08a-6adb3d836530-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-api-a-86e4a7-686g8\" (UID: \"96460de9-c817-4b3a-b08a-6adb3d836530\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-86e4a7-686g8"
Oct 30 19:56:01.091687 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:56:01.091634    4892 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/96460de9-c817-4b3a-b08a-6adb3d836530-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-api-a-86e4a7-686g8\" (UID: \"96460de9-c817-4b3a-b08a-6adb3d836530\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-86e4a7-686g8"
Oct 30 19:56:01.091687 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:56:01.091682    4892 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-njql2\" (UniqueName: \"kubernetes.io/projected/96460de9-c817-4b3a-b08a-6adb3d836530-kube-api-access-njql2\") pod \"maas-api-revoke-keys-e2e-api-a-86e4a7-686g8\" (UID: \"96460de9-c817-4b3a-b08a-6adb3d836530\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-86e4a7-686g8"
Oct 30 19:56:01.092318 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:56:01.092277    4892 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/96460de9-c817-4b3a-b08a-6adb3d836530-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-api-a-86e4a7-686g8\" (UID: \"96460de9-c817-4b3a-b08a-6adb3d836530\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-86e4a7-686g8"
Oct 30 19:56:01.094286 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:56:01.094261    4892 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/96460de9-c817-4b3a-b08a-6adb3d836530-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-api-a-86e4a7-686g8\" (UID: \"96460de9-c817-4b3a-b08a-6adb3d836530\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-86e4a7-686g8"
Oct 30 19:56:01.150995 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:56:01.150953    4892 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-njql2\" (UniqueName: \"kubernetes.io/projected/96460de9-c817-4b3a-b08a-6adb3d836530-kube-api-access-njql2\") pod \"maas-api-revoke-keys-e2e-api-a-86e4a7-686g8\" (UID: \"96460de9-c817-4b3a-b08a-6adb3d836530\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-86e4a7-686g8"
Oct 30 19:56:01.254503 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:56:01.254429    4892 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-86e4a7-686g8"
Oct 30 19:56:01.648824 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:56:01.648780    4892 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-86e4a7-686g8"]
Oct 30 19:56:01.650890 ip-10-0-132-82 kubenswrapper[4892]: W1030 19:56:01.650852    4892 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod96460de9_c817_4b3a_b08a_6adb3d836530.slice/crio-a9008962b0615449a743b2c6a3d1d392e0252ea856bd0de2b16162a71f3db354 WatchSource:0}: Error finding container a9008962b0615449a743b2c6a3d1d392e0252ea856bd0de2b16162a71f3db354: Status 404 returned error can't find the container with id a9008962b0615449a743b2c6a3d1d392e0252ea856bd0de2b16162a71f3db354
Oct 30 19:56:02.115200 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:56:02.115161    4892 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-86e4a7-686g8" event={"ID":"96460de9-c817-4b3a-b08a-6adb3d836530","Type":"ContainerStarted","Data":"27391a1d7e8075a1bfc8293fd2e9cbdce81f08b5796699bb4c9a224239de129f"}
Oct 30 19:56:02.115600 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:56:02.115226    4892 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-86e4a7-686g8" event={"ID":"96460de9-c817-4b3a-b08a-6adb3d836530","Type":"ContainerStarted","Data":"a9008962b0615449a743b2c6a3d1d392e0252ea856bd0de2b16162a71f3db354"}
Oct 30 19:56:02.233186 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:56:02.233075    4892 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-86e4a7-686g8" podStartSLOduration=2.233060475 podStartE2EDuration="2.233060475s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:56:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:56:02.228252679 +0000 UTC m=+2409.798718282" watchObservedRunningTime="2026-10-30 19:56:02.233060475 +0000 UTC m=+2409.803526074"
Oct 30 19:56:03.124111 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:56:03.124081    4892 generic.go:356] "Generic (PLEG): container finished" podID="96460de9-c817-4b3a-b08a-6adb3d836530" containerID="27391a1d7e8075a1bfc8293fd2e9cbdce81f08b5796699bb4c9a224239de129f" exitCode=0
Oct 30 19:56:03.124503 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:56:03.124168    4892 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-86e4a7-686g8" event={"ID":"96460de9-c817-4b3a-b08a-6adb3d836530","Type":"ContainerDied","Data":"27391a1d7e8075a1bfc8293fd2e9cbdce81f08b5796699bb4c9a224239de129f"}
Oct 30 19:56:04.318549 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:56:04.318521    4892 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-86e4a7-686g8"
Oct 30 19:56:04.323664 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:56:04.323646    4892 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/96460de9-c817-4b3a-b08a-6adb3d836530-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/96460de9-c817-4b3a-b08a-6adb3d836530-maas-api-xks-ca\") pod \"96460de9-c817-4b3a-b08a-6adb3d836530\" (UID: \"96460de9-c817-4b3a-b08a-6adb3d836530\") "
Oct 30 19:56:04.323772 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:56:04.323701    4892 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/96460de9-c817-4b3a-b08a-6adb3d836530-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/96460de9-c817-4b3a-b08a-6adb3d836530-maas-api-openshift-ca\") pod \"96460de9-c817-4b3a-b08a-6adb3d836530\" (UID: \"96460de9-c817-4b3a-b08a-6adb3d836530\") "
Oct 30 19:56:04.323860 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:56:04.323824    4892 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/96460de9-c817-4b3a-b08a-6adb3d836530-kube-api-access-njql2\" (UniqueName: \"kubernetes.io/projected/96460de9-c817-4b3a-b08a-6adb3d836530-kube-api-access-njql2\") pod \"96460de9-c817-4b3a-b08a-6adb3d836530\" (UID: \"96460de9-c817-4b3a-b08a-6adb3d836530\") "
Oct 30 19:56:04.324103 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:56:04.324081    4892 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/96460de9-c817-4b3a-b08a-6adb3d836530-maas-api-openshift-ca" pod "96460de9-c817-4b3a-b08a-6adb3d836530" (UID: "96460de9-c817-4b3a-b08a-6adb3d836530"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 19:56:04.324355 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:56:04.324339    4892 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/96460de9-c817-4b3a-b08a-6adb3d836530-maas-api-openshift-ca\") on node \"ip-10-0-132-82.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 19:56:04.325832 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:56:04.325808    4892 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/96460de9-c817-4b3a-b08a-6adb3d836530-maas-api-xks-ca" pod "96460de9-c817-4b3a-b08a-6adb3d836530" (UID: "96460de9-c817-4b3a-b08a-6adb3d836530"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 19:56:04.325987 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:56:04.325953    4892 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/96460de9-c817-4b3a-b08a-6adb3d836530-kube-api-access-njql2" pod "96460de9-c817-4b3a-b08a-6adb3d836530" (UID: "96460de9-c817-4b3a-b08a-6adb3d836530"). InnerVolumeSpecName "kube-api-access-njql2". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:56:04.425135 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:56:04.425105    4892 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-njql2\" (UniqueName: \"kubernetes.io/projected/96460de9-c817-4b3a-b08a-6adb3d836530-kube-api-access-njql2\") on node \"ip-10-0-132-82.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 19:56:04.425135 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:56:04.425134    4892 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/96460de9-c817-4b3a-b08a-6adb3d836530-maas-api-xks-ca\") on node \"ip-10-0-132-82.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 19:56:05.143813 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:56:05.143772    4892 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-86e4a7-686g8"
Oct 30 19:56:05.144041 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:56:05.143807    4892 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-86e4a7-686g8" event={"ID":"96460de9-c817-4b3a-b08a-6adb3d836530","Type":"ContainerDied","Data":"a9008962b0615449a743b2c6a3d1d392e0252ea856bd0de2b16162a71f3db354"}
Oct 30 19:56:05.144041 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:56:05.143867    4892 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="a9008962b0615449a743b2c6a3d1d392e0252ea856bd0de2b16162a71f3db354"
Oct 30 19:56:08.363735 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:56:08.363696    4892 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-898968-kbwn2"]
Oct 30 19:56:08.364462 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:56:08.364446    4892 cpu_manager.go:414] "containerMap: removing container" podUID="96460de9-c817-4b3a-b08a-6adb3d836530" containerName="revoke-keys"
Oct 30 19:56:08.364511 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:56:08.364464    4892 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="96460de9-c817-4b3a-b08a-6adb3d836530" containerName="revoke-keys"
Oct 30 19:56:08.364606 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:56:08.364597    4892 memory_manager.go:365] "RemoveStaleState removing state" podUID="96460de9-c817-4b3a-b08a-6adb3d836530" containerName="revoke-keys"
Oct 30 19:56:08.368532 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:56:08.368514    4892 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-898968-kbwn2"
Oct 30 19:56:08.420464 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:56:08.420426    4892 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 19:56:08.469871 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:56:08.469841    4892 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 30 19:56:08.469871 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:56:08.469874    4892 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-nlxp4\""
Oct 30 19:56:08.470144 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:56:08.469887    4892 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 30 19:56:08.476335 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:56:08.476312    4892 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-898968-kbwn2"]
Oct 30 19:56:08.477866 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:56:08.477843    4892 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/adb0c593-1248-4d0e-bae9-2f6ac76233a9-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-898968-kbwn2\" (UID: \"adb0c593-1248-4d0e-bae9-2f6ac76233a9\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-898968-kbwn2"
Oct 30 19:56:08.477935 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:56:08.477888    4892 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/adb0c593-1248-4d0e-bae9-2f6ac76233a9-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-898968-kbwn2\" (UID: \"adb0c593-1248-4d0e-bae9-2f6ac76233a9\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-898968-kbwn2"
Oct 30 19:56:08.478156 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:56:08.478136    4892 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nlj5t\" (UniqueName: \"kubernetes.io/projected/adb0c593-1248-4d0e-bae9-2f6ac76233a9-kube-api-access-nlj5t\") pod \"maas-api-revoke-keys-e2e-inf-a-898968-kbwn2\" (UID: \"adb0c593-1248-4d0e-bae9-2f6ac76233a9\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-898968-kbwn2"
Oct 30 19:56:08.579683 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:56:08.579658    4892 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nlj5t\" (UniqueName: \"kubernetes.io/projected/adb0c593-1248-4d0e-bae9-2f6ac76233a9-kube-api-access-nlj5t\") pod \"maas-api-revoke-keys-e2e-inf-a-898968-kbwn2\" (UID: \"adb0c593-1248-4d0e-bae9-2f6ac76233a9\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-898968-kbwn2"
Oct 30 19:56:08.579862 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:56:08.579716    4892 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/adb0c593-1248-4d0e-bae9-2f6ac76233a9-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-898968-kbwn2\" (UID: \"adb0c593-1248-4d0e-bae9-2f6ac76233a9\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-898968-kbwn2"
Oct 30 19:56:08.579862 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:56:08.579834    4892 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/adb0c593-1248-4d0e-bae9-2f6ac76233a9-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-898968-kbwn2\" (UID: \"adb0c593-1248-4d0e-bae9-2f6ac76233a9\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-898968-kbwn2"
Oct 30 19:56:08.580361 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:56:08.580341    4892 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/adb0c593-1248-4d0e-bae9-2f6ac76233a9-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-898968-kbwn2\" (UID: \"adb0c593-1248-4d0e-bae9-2f6ac76233a9\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-898968-kbwn2"
Oct 30 19:56:08.582383 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:56:08.582365    4892 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/adb0c593-1248-4d0e-bae9-2f6ac76233a9-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-898968-kbwn2\" (UID: \"adb0c593-1248-4d0e-bae9-2f6ac76233a9\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-898968-kbwn2"
Oct 30 19:56:08.636408 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:56:08.636381    4892 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nlj5t\" (UniqueName: \"kubernetes.io/projected/adb0c593-1248-4d0e-bae9-2f6ac76233a9-kube-api-access-nlj5t\") pod \"maas-api-revoke-keys-e2e-inf-a-898968-kbwn2\" (UID: \"adb0c593-1248-4d0e-bae9-2f6ac76233a9\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-898968-kbwn2"
Oct 30 19:56:08.683157 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:56:08.683132    4892 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-898968-kbwn2"
Oct 30 19:56:09.067859 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:56:09.067829    4892 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-898968-kbwn2"]
Oct 30 19:56:09.068351 ip-10-0-132-82 kubenswrapper[4892]: W1030 19:56:09.068323    4892 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podadb0c593_1248_4d0e_bae9_2f6ac76233a9.slice/crio-2f97aae4bb2a5eb12d7b456d275f61dfb0e1add8071f1be094d4fbf25ff6b559 WatchSource:0}: Error finding container 2f97aae4bb2a5eb12d7b456d275f61dfb0e1add8071f1be094d4fbf25ff6b559: Status 404 returned error can't find the container with id 2f97aae4bb2a5eb12d7b456d275f61dfb0e1add8071f1be094d4fbf25ff6b559
Oct 30 19:56:09.183877 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:56:09.183843    4892 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-898968-kbwn2" event={"ID":"adb0c593-1248-4d0e-bae9-2f6ac76233a9","Type":"ContainerStarted","Data":"2f97aae4bb2a5eb12d7b456d275f61dfb0e1add8071f1be094d4fbf25ff6b559"}
Oct 30 19:56:10.192916 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:56:10.192824    4892 generic.go:356] "Generic (PLEG): container finished" podID="adb0c593-1248-4d0e-bae9-2f6ac76233a9" containerID="97964c9a51be733508b19228fd127ff01a9cba57de9b3c6f5e185401b0b47134" exitCode=0
Oct 30 19:56:10.192916 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:56:10.192904    4892 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-898968-kbwn2" event={"ID":"adb0c593-1248-4d0e-bae9-2f6ac76233a9","Type":"ContainerDied","Data":"97964c9a51be733508b19228fd127ff01a9cba57de9b3c6f5e185401b0b47134"}
Oct 30 19:56:11.401282 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:56:11.401254    4892 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-898968-kbwn2"
Oct 30 19:56:11.424939 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:56:11.424911    4892 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/adb0c593-1248-4d0e-bae9-2f6ac76233a9-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/adb0c593-1248-4d0e-bae9-2f6ac76233a9-maas-api-openshift-ca\") pod \"adb0c593-1248-4d0e-bae9-2f6ac76233a9\" (UID: \"adb0c593-1248-4d0e-bae9-2f6ac76233a9\") "
Oct 30 19:56:11.425093 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:56:11.425042    4892 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/adb0c593-1248-4d0e-bae9-2f6ac76233a9-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/adb0c593-1248-4d0e-bae9-2f6ac76233a9-maas-api-xks-ca\") pod \"adb0c593-1248-4d0e-bae9-2f6ac76233a9\" (UID: \"adb0c593-1248-4d0e-bae9-2f6ac76233a9\") "
Oct 30 19:56:11.425204 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:56:11.425189    4892 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/adb0c593-1248-4d0e-bae9-2f6ac76233a9-kube-api-access-nlj5t\" (UniqueName: \"kubernetes.io/projected/adb0c593-1248-4d0e-bae9-2f6ac76233a9-kube-api-access-nlj5t\") pod \"adb0c593-1248-4d0e-bae9-2f6ac76233a9\" (UID: \"adb0c593-1248-4d0e-bae9-2f6ac76233a9\") "
Oct 30 19:56:11.425301 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:56:11.425275    4892 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/adb0c593-1248-4d0e-bae9-2f6ac76233a9-maas-api-openshift-ca" pod "adb0c593-1248-4d0e-bae9-2f6ac76233a9" (UID: "adb0c593-1248-4d0e-bae9-2f6ac76233a9"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 19:56:11.425662 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:56:11.425643    4892 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/adb0c593-1248-4d0e-bae9-2f6ac76233a9-maas-api-openshift-ca\") on node \"ip-10-0-132-82.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 19:56:11.427272 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:56:11.427248    4892 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/adb0c593-1248-4d0e-bae9-2f6ac76233a9-maas-api-xks-ca" pod "adb0c593-1248-4d0e-bae9-2f6ac76233a9" (UID: "adb0c593-1248-4d0e-bae9-2f6ac76233a9"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 19:56:11.427444 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:56:11.427427    4892 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/adb0c593-1248-4d0e-bae9-2f6ac76233a9-kube-api-access-nlj5t" pod "adb0c593-1248-4d0e-bae9-2f6ac76233a9" (UID: "adb0c593-1248-4d0e-bae9-2f6ac76233a9"). InnerVolumeSpecName "kube-api-access-nlj5t". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:56:11.527643 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:56:11.527561    4892 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nlj5t\" (UniqueName: \"kubernetes.io/projected/adb0c593-1248-4d0e-bae9-2f6ac76233a9-kube-api-access-nlj5t\") on node \"ip-10-0-132-82.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 19:56:11.527643 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:56:11.527602    4892 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/adb0c593-1248-4d0e-bae9-2f6ac76233a9-maas-api-xks-ca\") on node \"ip-10-0-132-82.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 19:56:11.996688 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:56:11.996648    4892 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-fsfvk"]
Oct 30 19:56:12.226187 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:56:12.226160    4892 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-898968-kbwn2"
Oct 30 19:56:12.226187 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:56:12.226165    4892 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-898968-kbwn2" event={"ID":"adb0c593-1248-4d0e-bae9-2f6ac76233a9","Type":"ContainerDied","Data":"2f97aae4bb2a5eb12d7b456d275f61dfb0e1add8071f1be094d4fbf25ff6b559"}
Oct 30 19:56:12.226437 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:56:12.226217    4892 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="2f97aae4bb2a5eb12d7b456d275f61dfb0e1add8071f1be094d4fbf25ff6b559"
Oct 30 19:56:13.034468 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:56:13.034439    4892 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-86e4a7-5968698996-xkbd8"]
Oct 30 19:56:13.147019 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:56:13.146987    4892 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-86e4a7-85cc57c56c-vhlcj"]
Oct 30 19:56:18.036072 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:56:18.036026    4892 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-api-a-86e4a7-5968698996-xkbd8" podUID="61795905-fd72-424f-bf9d-07f97f3f252c" containerName="payload-processing" containerID="cri-o://6121afbd6daf498d128007ce3e4415f1bbca4bdffc28081b8cc8df57a4c08f8b" gracePeriod=25
Oct 30 19:56:18.153994 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:56:18.150111    4892 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-api-a-86e4a7-85cc57c56c-vhlcj" podUID="8e13a394-77f5-46a4-b58a-7f00a1483e9e" containerName="payload-pre-processing" containerID="cri-o://16c42645a660c63f8bfe07abde4a06668c19107b4fdd20daa431eb56c141b549" gracePeriod=25
Oct 30 19:56:18.300042 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:56:18.295035    4892 generic.go:356] "Generic (PLEG): container finished" podID="8e13a394-77f5-46a4-b58a-7f00a1483e9e" containerID="16c42645a660c63f8bfe07abde4a06668c19107b4fdd20daa431eb56c141b549" exitCode=0
Oct 30 19:56:18.300042 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:56:18.295331    4892 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-a-86e4a7-85cc57c56c-vhlcj" event={"ID":"8e13a394-77f5-46a4-b58a-7f00a1483e9e","Type":"ContainerDied","Data":"16c42645a660c63f8bfe07abde4a06668c19107b4fdd20daa431eb56c141b549"}
Oct 30 19:56:18.300042 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:56:18.297320    4892 generic.go:356] "Generic (PLEG): container finished" podID="61795905-fd72-424f-bf9d-07f97f3f252c" containerID="6121afbd6daf498d128007ce3e4415f1bbca4bdffc28081b8cc8df57a4c08f8b" exitCode=0
Oct 30 19:56:18.300042 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:56:18.297396    4892 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-a-86e4a7-5968698996-xkbd8" event={"ID":"61795905-fd72-424f-bf9d-07f97f3f252c","Type":"ContainerDied","Data":"6121afbd6daf498d128007ce3e4415f1bbca4bdffc28081b8cc8df57a4c08f8b"}
Oct 30 19:56:18.418173 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:56:18.418148    4892 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-a-86e4a7-5968698996-xkbd8"
Oct 30 19:56:18.515988 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:56:18.515940    4892 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-a-86e4a7-85cc57c56c-vhlcj"
Oct 30 19:56:18.517252 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:56:18.517236    4892 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/8e13a394-77f5-46a4-b58a-7f00a1483e9e-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8e13a394-77f5-46a4-b58a-7f00a1483e9e-plugins-config-volume\") pod \"8e13a394-77f5-46a4-b58a-7f00a1483e9e\" (UID: \"8e13a394-77f5-46a4-b58a-7f00a1483e9e\") "
Oct 30 19:56:18.517378 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:56:18.517366    4892 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/61795905-fd72-424f-bf9d-07f97f3f252c-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/61795905-fd72-424f-bf9d-07f97f3f252c-plugins-config-volume\") pod \"61795905-fd72-424f-bf9d-07f97f3f252c\" (UID: \"61795905-fd72-424f-bf9d-07f97f3f252c\") "
Oct 30 19:56:18.517417 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:56:18.517397    4892 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8e13a394-77f5-46a4-b58a-7f00a1483e9e-kube-api-access-jnwb8\" (UniqueName: \"kubernetes.io/projected/8e13a394-77f5-46a4-b58a-7f00a1483e9e-kube-api-access-jnwb8\") pod \"8e13a394-77f5-46a4-b58a-7f00a1483e9e\" (UID: \"8e13a394-77f5-46a4-b58a-7f00a1483e9e\") "
Oct 30 19:56:18.517496 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:56:18.517484    4892 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/61795905-fd72-424f-bf9d-07f97f3f252c-kube-api-access-p8mtx\" (UniqueName: \"kubernetes.io/projected/61795905-fd72-424f-bf9d-07f97f3f252c-kube-api-access-p8mtx\") pod \"61795905-fd72-424f-bf9d-07f97f3f252c\" (UID: \"61795905-fd72-424f-bf9d-07f97f3f252c\") "
Oct 30 19:56:18.517678 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:56:18.517646    4892 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8e13a394-77f5-46a4-b58a-7f00a1483e9e-plugins-config-volume" pod "8e13a394-77f5-46a4-b58a-7f00a1483e9e" (UID: "8e13a394-77f5-46a4-b58a-7f00a1483e9e"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 19:56:18.517767 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:56:18.517713    4892 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/61795905-fd72-424f-bf9d-07f97f3f252c-plugins-config-volume" pod "61795905-fd72-424f-bf9d-07f97f3f252c" (UID: "61795905-fd72-424f-bf9d-07f97f3f252c"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 19:56:18.517943 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:56:18.517929    4892 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/61795905-fd72-424f-bf9d-07f97f3f252c-plugins-config-volume\") on node \"ip-10-0-132-82.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 19:56:18.518014 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:56:18.517946    4892 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8e13a394-77f5-46a4-b58a-7f00a1483e9e-plugins-config-volume\") on node \"ip-10-0-132-82.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 19:56:18.519639 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:56:18.519617    4892 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8e13a394-77f5-46a4-b58a-7f00a1483e9e-kube-api-access-jnwb8" pod "8e13a394-77f5-46a4-b58a-7f00a1483e9e" (UID: "8e13a394-77f5-46a4-b58a-7f00a1483e9e"). InnerVolumeSpecName "kube-api-access-jnwb8". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:56:18.519696 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:56:18.519637    4892 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/61795905-fd72-424f-bf9d-07f97f3f252c-kube-api-access-p8mtx" pod "61795905-fd72-424f-bf9d-07f97f3f252c" (UID: "61795905-fd72-424f-bf9d-07f97f3f252c"). InnerVolumeSpecName "kube-api-access-p8mtx". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:56:18.619599 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:56:18.619567    4892 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jnwb8\" (UniqueName: \"kubernetes.io/projected/8e13a394-77f5-46a4-b58a-7f00a1483e9e-kube-api-access-jnwb8\") on node \"ip-10-0-132-82.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 19:56:18.619599 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:56:18.619600    4892 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-p8mtx\" (UniqueName: \"kubernetes.io/projected/61795905-fd72-424f-bf9d-07f97f3f252c-kube-api-access-p8mtx\") on node \"ip-10-0-132-82.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 19:56:19.305866 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:56:19.305836    4892 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-a-86e4a7-5968698996-xkbd8"
Oct 30 19:56:19.305866 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:56:19.305855    4892 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-a-86e4a7-5968698996-xkbd8" event={"ID":"61795905-fd72-424f-bf9d-07f97f3f252c","Type":"ContainerDied","Data":"0c594382b7c57fb1256adc5a4b32ab34bd49eeabfa8bcf88d707ae29df9c8ba6"}
Oct 30 19:56:19.306467 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:56:19.305947    4892 scope.go:122] "RemoveContainer" containerID="6121afbd6daf498d128007ce3e4415f1bbca4bdffc28081b8cc8df57a4c08f8b"
Oct 30 19:56:19.308350 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:56:19.308323    4892 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-a-86e4a7-85cc57c56c-vhlcj"
Oct 30 19:56:19.308495 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:56:19.308333    4892 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-a-86e4a7-85cc57c56c-vhlcj" event={"ID":"8e13a394-77f5-46a4-b58a-7f00a1483e9e","Type":"ContainerDied","Data":"f46ae4e478f8fbfc7f7061f9d3c519fc51e23ee611cfd82043fae35c797b208a"}
Oct 30 19:56:19.317623 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:56:19.317606    4892 scope.go:122] "RemoveContainer" containerID="16c42645a660c63f8bfe07abde4a06668c19107b4fdd20daa431eb56c141b549"
Oct 30 19:56:19.476314 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:56:19.476284    4892 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-86e4a7-85cc57c56c-vhlcj"]
Oct 30 19:56:19.480276 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:56:19.480246    4892 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-86e4a7-85cc57c56c-vhlcj"]
Oct 30 19:56:19.583434 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:56:19.583402    4892 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-86e4a7-5968698996-xkbd8"]
Oct 30 19:56:19.589625 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:56:19.589596    4892 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-86e4a7-5968698996-xkbd8"]
Oct 30 19:56:20.176665 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:56:20.176628    4892 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-898968-84888dc6d8-vkg68"]
Oct 30 19:56:20.282741 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:56:20.282706    4892 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-898968-868d54644c-j8d79"]
Oct 30 19:56:20.575611 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:56:20.575579    4892 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="61795905-fd72-424f-bf9d-07f97f3f252c" path="/var/lib/kubelet/pods/61795905-fd72-424f-bf9d-07f97f3f252c/volumes"
Oct 30 19:56:20.576153 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:56:20.576103    4892 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="8e13a394-77f5-46a4-b58a-7f00a1483e9e" path="/var/lib/kubelet/pods/8e13a394-77f5-46a4-b58a-7f00a1483e9e/volumes"
Oct 30 19:56:24.362848 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:56:24.362814    4892 generic.go:356] "Generic (PLEG): container finished" podID="33098b51-3c25-4761-a108-844516b8d2fa" containerID="87a5399b1210c4c285564ff7854067bfad195cfa2c0f40e0bf87815fb1e09ca4" exitCode=0
Oct 30 19:56:24.363292 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:56:24.362880    4892 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-6caff5-openshift-default-665946589f-hn4qk" event={"ID":"33098b51-3c25-4761-a108-844516b8d2fa","Type":"ContainerDied","Data":"87a5399b1210c4c285564ff7854067bfad195cfa2c0f40e0bf87815fb1e09ca4"}
Oct 30 19:56:24.363418 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:56:24.363401    4892 scope.go:122] "RemoveContainer" containerID="87a5399b1210c4c285564ff7854067bfad195cfa2c0f40e0bf87815fb1e09ca4"
Oct 30 19:56:24.365309 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:56:24.365280    4892 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892000Ki","pods":"250"}
Oct 30 19:56:24.365401 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:56:24.365354    4892 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892000Ki","pods":"250"}
Oct 30 19:56:24.365401 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:56:24.365385    4892 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892000Ki","pods":"250"}
Oct 30 19:56:24.851206 ip-10-0-132-82 kubenswrapper[4892]: E1030 19:56:24.850066    4892 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-jrm87" containerName="extractor"
Oct 30 19:56:25.177576 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:56:25.177531    4892 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-inf-a-898968-84888dc6d8-vkg68" podUID="2c311ac3-0fcf-48a9-887c-f992ab3e287f" containerName="payload-processing" containerID="cri-o://d8c35861464ea146fecd6790dedd729084469bf3a39a7cc5f621d9546489adbd" gracePeriod=25
Oct 30 19:56:25.283467 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:56:25.283417    4892 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-898968-868d54644c-j8d79" podUID="6bc6af34-1b37-4634-a8b0-932b105ef626" containerName="payload-pre-processing" containerID="cri-o://22f0d01d6bbfe35c4063388902e1bdd45af61a639b533e788c11b745b3ac7766" gracePeriod=25
Oct 30 19:56:25.379072 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:56:25.379032    4892 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-6caff5-openshift-default-665946589f-hn4qk" event={"ID":"33098b51-3c25-4761-a108-844516b8d2fa","Type":"ContainerStarted","Data":"4bec4ad58bd687ed8ebc8b6e4853203a4ed0ceb1dbc2377d38f44440e95b626b"}
Oct 30 19:56:25.381761 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:56:25.381731    4892 generic.go:356] "Generic (PLEG): container finished" podID="2c311ac3-0fcf-48a9-887c-f992ab3e287f" containerID="d8c35861464ea146fecd6790dedd729084469bf3a39a7cc5f621d9546489adbd" exitCode=0
Oct 30 19:56:25.381910 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:56:25.381821    4892 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-a-898968-84888dc6d8-vkg68" event={"ID":"2c311ac3-0fcf-48a9-887c-f992ab3e287f","Type":"ContainerDied","Data":"d8c35861464ea146fecd6790dedd729084469bf3a39a7cc5f621d9546489adbd"}
Oct 30 19:56:25.577876 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:56:25.577845    4892 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-a-898968-84888dc6d8-vkg68"
Oct 30 19:56:25.684771 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:56:25.684742    4892 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2c311ac3-0fcf-48a9-887c-f992ab3e287f-kube-api-access-qwjrc\" (UniqueName: \"kubernetes.io/projected/2c311ac3-0fcf-48a9-887c-f992ab3e287f-kube-api-access-qwjrc\") pod \"2c311ac3-0fcf-48a9-887c-f992ab3e287f\" (UID: \"2c311ac3-0fcf-48a9-887c-f992ab3e287f\") "
Oct 30 19:56:25.685028 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:56:25.685011    4892 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/2c311ac3-0fcf-48a9-887c-f992ab3e287f-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2c311ac3-0fcf-48a9-887c-f992ab3e287f-plugins-config-volume\") pod \"2c311ac3-0fcf-48a9-887c-f992ab3e287f\" (UID: \"2c311ac3-0fcf-48a9-887c-f992ab3e287f\") "
Oct 30 19:56:25.685384 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:56:25.685361    4892 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2c311ac3-0fcf-48a9-887c-f992ab3e287f-plugins-config-volume" pod "2c311ac3-0fcf-48a9-887c-f992ab3e287f" (UID: "2c311ac3-0fcf-48a9-887c-f992ab3e287f"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 19:56:25.685614 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:56:25.685594    4892 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2c311ac3-0fcf-48a9-887c-f992ab3e287f-plugins-config-volume\") on node \"ip-10-0-132-82.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 19:56:25.687084 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:56:25.687063    4892 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2c311ac3-0fcf-48a9-887c-f992ab3e287f-kube-api-access-qwjrc" pod "2c311ac3-0fcf-48a9-887c-f992ab3e287f" (UID: "2c311ac3-0fcf-48a9-887c-f992ab3e287f"). InnerVolumeSpecName "kube-api-access-qwjrc". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:56:25.712212 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:56:25.712188    4892 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-898968-868d54644c-j8d79"
Oct 30 19:56:25.787170 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:56:25.787075    4892 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qwjrc\" (UniqueName: \"kubernetes.io/projected/2c311ac3-0fcf-48a9-887c-f992ab3e287f-kube-api-access-qwjrc\") on node \"ip-10-0-132-82.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 19:56:25.924251 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:56:25.924211    4892 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6bc6af34-1b37-4634-a8b0-932b105ef626-kube-api-access-q28w8\" (UniqueName: \"kubernetes.io/projected/6bc6af34-1b37-4634-a8b0-932b105ef626-kube-api-access-q28w8\") pod \"6bc6af34-1b37-4634-a8b0-932b105ef626\" (UID: \"6bc6af34-1b37-4634-a8b0-932b105ef626\") "
Oct 30 19:56:25.924446 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:56:25.924383    4892 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/6bc6af34-1b37-4634-a8b0-932b105ef626-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6bc6af34-1b37-4634-a8b0-932b105ef626-plugins-config-volume\") pod \"6bc6af34-1b37-4634-a8b0-932b105ef626\" (UID: \"6bc6af34-1b37-4634-a8b0-932b105ef626\") "
Oct 30 19:56:25.925091 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:56:25.925056    4892 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6bc6af34-1b37-4634-a8b0-932b105ef626-plugins-config-volume" pod "6bc6af34-1b37-4634-a8b0-932b105ef626" (UID: "6bc6af34-1b37-4634-a8b0-932b105ef626"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 19:56:25.970435 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:56:25.970378    4892 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6bc6af34-1b37-4634-a8b0-932b105ef626-kube-api-access-q28w8" pod "6bc6af34-1b37-4634-a8b0-932b105ef626" (UID: "6bc6af34-1b37-4634-a8b0-932b105ef626"). InnerVolumeSpecName "kube-api-access-q28w8". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:56:26.026099 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:56:26.026068    4892 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-q28w8\" (UniqueName: \"kubernetes.io/projected/6bc6af34-1b37-4634-a8b0-932b105ef626-kube-api-access-q28w8\") on node \"ip-10-0-132-82.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 19:56:26.026099 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:56:26.026095    4892 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6bc6af34-1b37-4634-a8b0-932b105ef626-plugins-config-volume\") on node \"ip-10-0-132-82.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 19:56:26.174301 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:56:26.174254    4892 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-a-6caff5-openshift-default-665946589f-hn4qk"
Oct 30 19:56:26.175732 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:56:26.175704    4892 patch_prober.go:29] interesting pod/e2e-api-a-6caff5-openshift-default-665946589f-hn4qk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.197:15021/healthz/ready\": dial tcp 10.132.0.197:15021: connect: connection refused" start-of-body=
Oct 30 19:56:26.175871 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:56:26.175751    4892 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-6caff5-openshift-default-665946589f-hn4qk" podUID="33098b51-3c25-4761-a108-844516b8d2fa" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.197:15021/healthz/ready\": dial tcp 10.132.0.197:15021: connect: connection refused"
Oct 30 19:56:26.260987 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:56:26.251320    4892 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-cb14ed-2sl4x"]
Oct 30 19:56:26.260987 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:56:26.252413    4892 cpu_manager.go:414] "containerMap: removing container" podUID="adb0c593-1248-4d0e-bae9-2f6ac76233a9" containerName="revoke-keys"
Oct 30 19:56:26.260987 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:56:26.252432    4892 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="adb0c593-1248-4d0e-bae9-2f6ac76233a9" containerName="revoke-keys"
Oct 30 19:56:26.260987 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:56:26.252461    4892 cpu_manager.go:414] "containerMap: removing container" podUID="61795905-fd72-424f-bf9d-07f97f3f252c" containerName="payload-processing"
Oct 30 19:56:26.260987 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:56:26.252469    4892 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="61795905-fd72-424f-bf9d-07f97f3f252c" containerName="payload-processing"
Oct 30 19:56:26.260987 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:56:26.252496    4892 cpu_manager.go:414] "containerMap: removing container" podUID="6bc6af34-1b37-4634-a8b0-932b105ef626" containerName="payload-pre-processing"
Oct 30 19:56:26.260987 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:56:26.252505    4892 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6bc6af34-1b37-4634-a8b0-932b105ef626" containerName="payload-pre-processing"
Oct 30 19:56:26.260987 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:56:26.252557    4892 cpu_manager.go:414] "containerMap: removing container" podUID="2c311ac3-0fcf-48a9-887c-f992ab3e287f" containerName="payload-processing"
Oct 30 19:56:26.260987 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:56:26.252566    4892 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2c311ac3-0fcf-48a9-887c-f992ab3e287f" containerName="payload-processing"
Oct 30 19:56:26.260987 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:56:26.252585    4892 cpu_manager.go:414] "containerMap: removing container" podUID="8e13a394-77f5-46a4-b58a-7f00a1483e9e" containerName="payload-pre-processing"
Oct 30 19:56:26.260987 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:56:26.252592    4892 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8e13a394-77f5-46a4-b58a-7f00a1483e9e" containerName="payload-pre-processing"
Oct 30 19:56:26.260987 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:56:26.252732    4892 memory_manager.go:365] "RemoveStaleState removing state" podUID="2c311ac3-0fcf-48a9-887c-f992ab3e287f" containerName="payload-processing"
Oct 30 19:56:26.260987 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:56:26.252745    4892 memory_manager.go:365] "RemoveStaleState removing state" podUID="8e13a394-77f5-46a4-b58a-7f00a1483e9e" containerName="payload-pre-processing"
Oct 30 19:56:26.260987 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:56:26.252759    4892 memory_manager.go:365] "RemoveStaleState removing state" podUID="adb0c593-1248-4d0e-bae9-2f6ac76233a9" containerName="revoke-keys"
Oct 30 19:56:26.260987 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:56:26.252771    4892 memory_manager.go:365] "RemoveStaleState removing state" podUID="61795905-fd72-424f-bf9d-07f97f3f252c" containerName="payload-processing"
Oct 30 19:56:26.260987 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:56:26.252784    4892 memory_manager.go:365] "RemoveStaleState removing state" podUID="6bc6af34-1b37-4634-a8b0-932b105ef626" containerName="payload-pre-processing"
Oct 30 19:56:26.267330 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:56:26.267303    4892 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-cb14ed-2sl4x"
Oct 30 19:56:26.289087 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:56:26.289043    4892 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-a-6caff5-openshift-default-665946589f-hn4qk"
Oct 30 19:56:26.294609 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:56:26.294585    4892 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-cb14ed-2sl4x"]
Oct 30 19:56:26.318985 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:56:26.318813    4892 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 19:56:26.318985 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:56:26.318869    4892 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-nlxp4\""
Oct 30 19:56:26.318985 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:56:26.318882    4892 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 30 19:56:26.318985 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:56:26.318931    4892 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 30 19:56:26.337132 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:56:26.336714    4892 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/10fe3864-9417-4f6d-84ad-019943adf796-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-api-a-cb14ed-2sl4x\" (UID: \"10fe3864-9417-4f6d-84ad-019943adf796\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-cb14ed-2sl4x"
Oct 30 19:56:26.337132 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:56:26.336768    4892 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xvj7b\" (UniqueName: \"kubernetes.io/projected/10fe3864-9417-4f6d-84ad-019943adf796-kube-api-access-xvj7b\") pod \"maas-api-revoke-keys-e2e-api-a-cb14ed-2sl4x\" (UID: \"10fe3864-9417-4f6d-84ad-019943adf796\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-cb14ed-2sl4x"
Oct 30 19:56:26.337132 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:56:26.337006    4892 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/10fe3864-9417-4f6d-84ad-019943adf796-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-api-a-cb14ed-2sl4x\" (UID: \"10fe3864-9417-4f6d-84ad-019943adf796\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-cb14ed-2sl4x"
Oct 30 19:56:26.391380 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:56:26.391345    4892 generic.go:356] "Generic (PLEG): container finished" podID="6bc6af34-1b37-4634-a8b0-932b105ef626" containerID="22f0d01d6bbfe35c4063388902e1bdd45af61a639b533e788c11b745b3ac7766" exitCode=0
Oct 30 19:56:26.391879 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:56:26.391394    4892 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-898968-868d54644c-j8d79"
Oct 30 19:56:26.391879 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:56:26.391440    4892 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-898968-868d54644c-j8d79" event={"ID":"6bc6af34-1b37-4634-a8b0-932b105ef626","Type":"ContainerDied","Data":"22f0d01d6bbfe35c4063388902e1bdd45af61a639b533e788c11b745b3ac7766"}
Oct 30 19:56:26.391879 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:56:26.391488    4892 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-898968-868d54644c-j8d79" event={"ID":"6bc6af34-1b37-4634-a8b0-932b105ef626","Type":"ContainerDied","Data":"41f288f9b1d1a8cee3150e931429dbb40069f56c28582fb093f66b758af71b86"}
Oct 30 19:56:26.391879 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:56:26.391511    4892 scope.go:122] "RemoveContainer" containerID="22f0d01d6bbfe35c4063388902e1bdd45af61a639b533e788c11b745b3ac7766"
Oct 30 19:56:26.393221 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:56:26.393193    4892 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-a-898968-84888dc6d8-vkg68"
Oct 30 19:56:26.393344 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:56:26.393253    4892 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-a-898968-84888dc6d8-vkg68" event={"ID":"2c311ac3-0fcf-48a9-887c-f992ab3e287f","Type":"ContainerDied","Data":"abe797bb99debdde1b72161a6681a28a3af73d21404a7ba538ddce81763c0d6c"}
Oct 30 19:56:26.403556 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:56:26.403376    4892 scope.go:122] "RemoveContainer" containerID="22f0d01d6bbfe35c4063388902e1bdd45af61a639b533e788c11b745b3ac7766"
Oct 30 19:56:26.403649 ip-10-0-132-82 kubenswrapper[4892]: E1030 19:56:26.403630    4892 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"22f0d01d6bbfe35c4063388902e1bdd45af61a639b533e788c11b745b3ac7766\": container with ID starting with 22f0d01d6bbfe35c4063388902e1bdd45af61a639b533e788c11b745b3ac7766 not found: ID does not exist" containerID="22f0d01d6bbfe35c4063388902e1bdd45af61a639b533e788c11b745b3ac7766"
Oct 30 19:56:26.403716 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:56:26.403660    4892 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"22f0d01d6bbfe35c4063388902e1bdd45af61a639b533e788c11b745b3ac7766"} err="failed to get container status \"22f0d01d6bbfe35c4063388902e1bdd45af61a639b533e788c11b745b3ac7766\": rpc error: code = NotFound desc = could not find container \"22f0d01d6bbfe35c4063388902e1bdd45af61a639b533e788c11b745b3ac7766\": container with ID starting with 22f0d01d6bbfe35c4063388902e1bdd45af61a639b533e788c11b745b3ac7766 not found: ID does not exist"
Oct 30 19:56:26.403716 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:56:26.403698    4892 scope.go:122] "RemoveContainer" containerID="d8c35861464ea146fecd6790dedd729084469bf3a39a7cc5f621d9546489adbd"
Oct 30 19:56:26.452029 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:56:26.442680    4892 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/10fe3864-9417-4f6d-84ad-019943adf796-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-api-a-cb14ed-2sl4x\" (UID: \"10fe3864-9417-4f6d-84ad-019943adf796\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-cb14ed-2sl4x"
Oct 30 19:56:26.452029 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:56:26.442762    4892 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/10fe3864-9417-4f6d-84ad-019943adf796-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-api-a-cb14ed-2sl4x\" (UID: \"10fe3864-9417-4f6d-84ad-019943adf796\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-cb14ed-2sl4x"
Oct 30 19:56:26.452029 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:56:26.442799    4892 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xvj7b\" (UniqueName: \"kubernetes.io/projected/10fe3864-9417-4f6d-84ad-019943adf796-kube-api-access-xvj7b\") pod \"maas-api-revoke-keys-e2e-api-a-cb14ed-2sl4x\" (UID: \"10fe3864-9417-4f6d-84ad-019943adf796\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-cb14ed-2sl4x"
Oct 30 19:56:26.452029 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:56:26.443907    4892 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/10fe3864-9417-4f6d-84ad-019943adf796-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-api-a-cb14ed-2sl4x\" (UID: \"10fe3864-9417-4f6d-84ad-019943adf796\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-cb14ed-2sl4x"
Oct 30 19:56:26.452029 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:56:26.446814    4892 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/10fe3864-9417-4f6d-84ad-019943adf796-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-api-a-cb14ed-2sl4x\" (UID: \"10fe3864-9417-4f6d-84ad-019943adf796\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-cb14ed-2sl4x"
Oct 30 19:56:26.502726 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:56:26.502689    4892 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xvj7b\" (UniqueName: \"kubernetes.io/projected/10fe3864-9417-4f6d-84ad-019943adf796-kube-api-access-xvj7b\") pod \"maas-api-revoke-keys-e2e-api-a-cb14ed-2sl4x\" (UID: \"10fe3864-9417-4f6d-84ad-019943adf796\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-cb14ed-2sl4x"
Oct 30 19:56:26.571912 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:56:26.571880    4892 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-898968-868d54644c-j8d79"]
Oct 30 19:56:26.572089 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:56:26.571921    4892 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-898968-868d54644c-j8d79"]
Oct 30 19:56:26.581979 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:56:26.581935    4892 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-cb14ed-2sl4x"
Oct 30 19:56:26.680015 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:56:26.678553    4892 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-898968-84888dc6d8-vkg68"]
Oct 30 19:56:26.684144 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:56:26.684016    4892 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-898968-84888dc6d8-vkg68"]
Oct 30 19:56:27.004855 ip-10-0-132-82 kubenswrapper[4892]: W1030 19:56:27.004828    4892 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod10fe3864_9417_4f6d_84ad_019943adf796.slice/crio-0dd186a10e5544963486d51f518c1fcc8d9256c4638548f308b0ae2993a606cc WatchSource:0}: Error finding container 0dd186a10e5544963486d51f518c1fcc8d9256c4638548f308b0ae2993a606cc: Status 404 returned error can't find the container with id 0dd186a10e5544963486d51f518c1fcc8d9256c4638548f308b0ae2993a606cc
Oct 30 19:56:27.005893 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:56:27.005855    4892 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-cb14ed-2sl4x"]
Oct 30 19:56:27.173872 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:56:27.173834    4892 patch_prober.go:29] interesting pod/e2e-api-a-6caff5-openshift-default-665946589f-hn4qk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.197:15021/healthz/ready\": dial tcp 10.132.0.197:15021: connect: connection refused" start-of-body=
Oct 30 19:56:27.174053 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:56:27.173891    4892 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-6caff5-openshift-default-665946589f-hn4qk" podUID="33098b51-3c25-4761-a108-844516b8d2fa" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.197:15021/healthz/ready\": dial tcp 10.132.0.197:15021: connect: connection refused"
Oct 30 19:56:27.410295 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:56:27.410254    4892 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-cb14ed-2sl4x" event={"ID":"10fe3864-9417-4f6d-84ad-019943adf796","Type":"ContainerStarted","Data":"0dd186a10e5544963486d51f518c1fcc8d9256c4638548f308b0ae2993a606cc"}
Oct 30 19:56:28.174811 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:56:28.174766    4892 patch_prober.go:29] interesting pod/e2e-api-a-6caff5-openshift-default-665946589f-hn4qk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.197:15021/healthz/ready\": dial tcp 10.132.0.197:15021: connect: connection refused" start-of-body=
Oct 30 19:56:28.175072 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:56:28.174845    4892 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-6caff5-openshift-default-665946589f-hn4qk" podUID="33098b51-3c25-4761-a108-844516b8d2fa" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.197:15021/healthz/ready\": dial tcp 10.132.0.197:15021: connect: connection refused"
Oct 30 19:56:28.421595 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:56:28.421559    4892 generic.go:356] "Generic (PLEG): container finished" podID="10fe3864-9417-4f6d-84ad-019943adf796" containerID="a0e456c551f27194822d5c66f632d8254e7236024a5ec99882aa7e25b893009a" exitCode=0
Oct 30 19:56:28.422061 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:56:28.421609    4892 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-cb14ed-2sl4x" event={"ID":"10fe3864-9417-4f6d-84ad-019943adf796","Type":"ContainerDied","Data":"a0e456c551f27194822d5c66f632d8254e7236024a5ec99882aa7e25b893009a"}
Oct 30 19:56:28.579061 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:56:28.576996    4892 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="2c311ac3-0fcf-48a9-887c-f992ab3e287f" path="/var/lib/kubelet/pods/2c311ac3-0fcf-48a9-887c-f992ab3e287f/volumes"
Oct 30 19:56:28.579061 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:56:28.577480    4892 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="6bc6af34-1b37-4634-a8b0-932b105ef626" path="/var/lib/kubelet/pods/6bc6af34-1b37-4634-a8b0-932b105ef626/volumes"
Oct 30 19:56:29.174593 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:56:29.174546    4892 patch_prober.go:29] interesting pod/e2e-api-a-6caff5-openshift-default-665946589f-hn4qk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.197:15021/healthz/ready\": dial tcp 10.132.0.197:15021: connect: connection refused" start-of-body=
Oct 30 19:56:29.174783 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:56:29.174606    4892 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-6caff5-openshift-default-665946589f-hn4qk" podUID="33098b51-3c25-4761-a108-844516b8d2fa" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.197:15021/healthz/ready\": dial tcp 10.132.0.197:15021: connect: connection refused"
Oct 30 19:56:29.553357 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:56:29.553004    4892 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-daa1ac-swtts"]
Oct 30 19:56:29.574979 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:56:29.573145    4892 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-daa1ac-swtts"
Oct 30 19:56:29.596187 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:56:29.596151    4892 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/39e00956-193c-4dc5-9ccc-4ba8ba94584c-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-daa1ac-swtts\" (UID: \"39e00956-193c-4dc5-9ccc-4ba8ba94584c\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-daa1ac-swtts"
Oct 30 19:56:29.596344 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:56:29.596216    4892 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sg5dw\" (UniqueName: \"kubernetes.io/projected/39e00956-193c-4dc5-9ccc-4ba8ba94584c-kube-api-access-sg5dw\") pod \"maas-api-revoke-keys-e2e-inf-a-daa1ac-swtts\" (UID: \"39e00956-193c-4dc5-9ccc-4ba8ba94584c\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-daa1ac-swtts"
Oct 30 19:56:29.596743 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:56:29.596718    4892 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/39e00956-193c-4dc5-9ccc-4ba8ba94584c-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-daa1ac-swtts\" (UID: \"39e00956-193c-4dc5-9ccc-4ba8ba94584c\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-daa1ac-swtts"
Oct 30 19:56:29.609050 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:56:29.609029    4892 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-daa1ac-swtts"]
Oct 30 19:56:29.699903 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:56:29.699862    4892 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/39e00956-193c-4dc5-9ccc-4ba8ba94584c-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-daa1ac-swtts\" (UID: \"39e00956-193c-4dc5-9ccc-4ba8ba94584c\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-daa1ac-swtts"
Oct 30 19:56:29.700099 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:56:29.700078    4892 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/39e00956-193c-4dc5-9ccc-4ba8ba94584c-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-daa1ac-swtts\" (UID: \"39e00956-193c-4dc5-9ccc-4ba8ba94584c\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-daa1ac-swtts"
Oct 30 19:56:29.700168 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:56:29.700125    4892 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sg5dw\" (UniqueName: \"kubernetes.io/projected/39e00956-193c-4dc5-9ccc-4ba8ba94584c-kube-api-access-sg5dw\") pod \"maas-api-revoke-keys-e2e-inf-a-daa1ac-swtts\" (UID: \"39e00956-193c-4dc5-9ccc-4ba8ba94584c\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-daa1ac-swtts"
Oct 30 19:56:29.700788 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:56:29.700760    4892 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/39e00956-193c-4dc5-9ccc-4ba8ba94584c-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-daa1ac-swtts\" (UID: \"39e00956-193c-4dc5-9ccc-4ba8ba94584c\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-daa1ac-swtts"
Oct 30 19:56:29.703709 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:56:29.703682    4892 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/39e00956-193c-4dc5-9ccc-4ba8ba94584c-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-daa1ac-swtts\" (UID: \"39e00956-193c-4dc5-9ccc-4ba8ba94584c\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-daa1ac-swtts"
Oct 30 19:56:29.721268 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:56:29.721215    4892 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-cb14ed-2sl4x"
Oct 30 19:56:29.760870 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:56:29.760846    4892 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-sg5dw\" (UniqueName: \"kubernetes.io/projected/39e00956-193c-4dc5-9ccc-4ba8ba94584c-kube-api-access-sg5dw\") pod \"maas-api-revoke-keys-e2e-inf-a-daa1ac-swtts\" (UID: \"39e00956-193c-4dc5-9ccc-4ba8ba94584c\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-daa1ac-swtts"
Oct 30 19:56:29.800738 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:56:29.800712    4892 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/10fe3864-9417-4f6d-84ad-019943adf796-kube-api-access-xvj7b\" (UniqueName: \"kubernetes.io/projected/10fe3864-9417-4f6d-84ad-019943adf796-kube-api-access-xvj7b\") pod \"10fe3864-9417-4f6d-84ad-019943adf796\" (UID: \"10fe3864-9417-4f6d-84ad-019943adf796\") "
Oct 30 19:56:29.800862 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:56:29.800805    4892 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/10fe3864-9417-4f6d-84ad-019943adf796-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/10fe3864-9417-4f6d-84ad-019943adf796-maas-api-xks-ca\") pod \"10fe3864-9417-4f6d-84ad-019943adf796\" (UID: \"10fe3864-9417-4f6d-84ad-019943adf796\") "
Oct 30 19:56:29.800862 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:56:29.800834    4892 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/10fe3864-9417-4f6d-84ad-019943adf796-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/10fe3864-9417-4f6d-84ad-019943adf796-maas-api-openshift-ca\") pod \"10fe3864-9417-4f6d-84ad-019943adf796\" (UID: \"10fe3864-9417-4f6d-84ad-019943adf796\") "
Oct 30 19:56:29.801343 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:56:29.801274    4892 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/10fe3864-9417-4f6d-84ad-019943adf796-maas-api-openshift-ca" pod "10fe3864-9417-4f6d-84ad-019943adf796" (UID: "10fe3864-9417-4f6d-84ad-019943adf796"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 19:56:29.801559 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:56:29.801432    4892 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/10fe3864-9417-4f6d-84ad-019943adf796-maas-api-openshift-ca\") on node \"ip-10-0-132-82.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 19:56:29.802863 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:56:29.802831    4892 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/10fe3864-9417-4f6d-84ad-019943adf796-maas-api-xks-ca" pod "10fe3864-9417-4f6d-84ad-019943adf796" (UID: "10fe3864-9417-4f6d-84ad-019943adf796"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 19:56:29.802996 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:56:29.802939    4892 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/10fe3864-9417-4f6d-84ad-019943adf796-kube-api-access-xvj7b" pod "10fe3864-9417-4f6d-84ad-019943adf796" (UID: "10fe3864-9417-4f6d-84ad-019943adf796"). InnerVolumeSpecName "kube-api-access-xvj7b". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:56:29.897979 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:56:29.897932    4892 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-daa1ac-swtts"
Oct 30 19:56:29.902633 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:56:29.902609    4892 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xvj7b\" (UniqueName: \"kubernetes.io/projected/10fe3864-9417-4f6d-84ad-019943adf796-kube-api-access-xvj7b\") on node \"ip-10-0-132-82.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 19:56:29.902741 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:56:29.902635    4892 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/10fe3864-9417-4f6d-84ad-019943adf796-maas-api-xks-ca\") on node \"ip-10-0-132-82.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 19:56:30.174074 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:56:30.174029    4892 patch_prober.go:29] interesting pod/e2e-api-a-6caff5-openshift-default-665946589f-hn4qk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.197:15021/healthz/ready\": dial tcp 10.132.0.197:15021: connect: connection refused" start-of-body=
Oct 30 19:56:30.174253 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:56:30.174099    4892 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-6caff5-openshift-default-665946589f-hn4qk" podUID="33098b51-3c25-4761-a108-844516b8d2fa" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.197:15021/healthz/ready\": dial tcp 10.132.0.197:15021: connect: connection refused"
Oct 30 19:56:30.311917 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:56:30.311886    4892 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-daa1ac-swtts"]
Oct 30 19:56:30.312393 ip-10-0-132-82 kubenswrapper[4892]: W1030 19:56:30.312368    4892 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod39e00956_193c_4dc5_9ccc_4ba8ba94584c.slice/crio-0ba69162fa8fc490af41b37c18640d1489f04a038e3f309d984b7a67ae89b2dc WatchSource:0}: Error finding container 0ba69162fa8fc490af41b37c18640d1489f04a038e3f309d984b7a67ae89b2dc: Status 404 returned error can't find the container with id 0ba69162fa8fc490af41b37c18640d1489f04a038e3f309d984b7a67ae89b2dc
Oct 30 19:56:30.469665 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:56:30.469630    4892 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-daa1ac-swtts" event={"ID":"39e00956-193c-4dc5-9ccc-4ba8ba94584c","Type":"ContainerStarted","Data":"0ba69162fa8fc490af41b37c18640d1489f04a038e3f309d984b7a67ae89b2dc"}
Oct 30 19:56:30.471295 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:56:30.471266    4892 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-cb14ed-2sl4x" event={"ID":"10fe3864-9417-4f6d-84ad-019943adf796","Type":"ContainerDied","Data":"0dd186a10e5544963486d51f518c1fcc8d9256c4638548f308b0ae2993a606cc"}
Oct 30 19:56:30.471433 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:56:30.471294    4892 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-cb14ed-2sl4x"
Oct 30 19:56:30.471433 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:56:30.471312    4892 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="0dd186a10e5544963486d51f518c1fcc8d9256c4638548f308b0ae2993a606cc"
Oct 30 19:56:31.180588 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:56:31.180554    4892 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-a-6caff5-openshift-default-665946589f-hn4qk"
Oct 30 19:56:31.181715 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:56:31.181694    4892 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-a-6caff5-openshift-default-665946589f-hn4qk"
Oct 30 19:56:31.487054 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:56:31.485520    4892 generic.go:356] "Generic (PLEG): container finished" podID="39e00956-193c-4dc5-9ccc-4ba8ba94584c" containerID="556837ed75294bf62881db9353fc7bf97427b9f70c8f0fd3bc03cb435dc0dcfd" exitCode=0
Oct 30 19:56:31.487054 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:56:31.485678    4892 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-daa1ac-swtts" event={"ID":"39e00956-193c-4dc5-9ccc-4ba8ba94584c","Type":"ContainerDied","Data":"556837ed75294bf62881db9353fc7bf97427b9f70c8f0fd3bc03cb435dc0dcfd"}
Oct 30 19:56:32.676334 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:56:32.676304    4892 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-daa1ac-swtts"
Oct 30 19:56:32.739003 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:56:32.738978    4892 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/39e00956-193c-4dc5-9ccc-4ba8ba94584c-kube-api-access-sg5dw\" (UniqueName: \"kubernetes.io/projected/39e00956-193c-4dc5-9ccc-4ba8ba94584c-kube-api-access-sg5dw\") pod \"39e00956-193c-4dc5-9ccc-4ba8ba94584c\" (UID: \"39e00956-193c-4dc5-9ccc-4ba8ba94584c\") "
Oct 30 19:56:32.739166 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:56:32.739069    4892 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/39e00956-193c-4dc5-9ccc-4ba8ba94584c-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/39e00956-193c-4dc5-9ccc-4ba8ba94584c-maas-api-openshift-ca\") pod \"39e00956-193c-4dc5-9ccc-4ba8ba94584c\" (UID: \"39e00956-193c-4dc5-9ccc-4ba8ba94584c\") "
Oct 30 19:56:32.739166 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:56:32.739154    4892 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/39e00956-193c-4dc5-9ccc-4ba8ba94584c-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/39e00956-193c-4dc5-9ccc-4ba8ba94584c-maas-api-xks-ca\") pod \"39e00956-193c-4dc5-9ccc-4ba8ba94584c\" (UID: \"39e00956-193c-4dc5-9ccc-4ba8ba94584c\") "
Oct 30 19:56:32.739429 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:56:32.739406    4892 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/39e00956-193c-4dc5-9ccc-4ba8ba94584c-maas-api-openshift-ca" pod "39e00956-193c-4dc5-9ccc-4ba8ba94584c" (UID: "39e00956-193c-4dc5-9ccc-4ba8ba94584c"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 19:56:32.739766 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:56:32.739742    4892 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/39e00956-193c-4dc5-9ccc-4ba8ba94584c-maas-api-openshift-ca\") on node \"ip-10-0-132-82.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 19:56:32.741417 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:56:32.741395    4892 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/39e00956-193c-4dc5-9ccc-4ba8ba94584c-maas-api-xks-ca" pod "39e00956-193c-4dc5-9ccc-4ba8ba94584c" (UID: "39e00956-193c-4dc5-9ccc-4ba8ba94584c"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 19:56:32.741661 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:56:32.741645    4892 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/39e00956-193c-4dc5-9ccc-4ba8ba94584c-kube-api-access-sg5dw" pod "39e00956-193c-4dc5-9ccc-4ba8ba94584c" (UID: "39e00956-193c-4dc5-9ccc-4ba8ba94584c"). InnerVolumeSpecName "kube-api-access-sg5dw". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:56:32.840750 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:56:32.840711    4892 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-sg5dw\" (UniqueName: \"kubernetes.io/projected/39e00956-193c-4dc5-9ccc-4ba8ba94584c-kube-api-access-sg5dw\") on node \"ip-10-0-132-82.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 19:56:32.840750 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:56:32.840737    4892 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/39e00956-193c-4dc5-9ccc-4ba8ba94584c-maas-api-xks-ca\") on node \"ip-10-0-132-82.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 19:56:33.503057 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:56:33.503029    4892 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-daa1ac-swtts"
Oct 30 19:56:33.503255 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:56:33.503035    4892 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-daa1ac-swtts" event={"ID":"39e00956-193c-4dc5-9ccc-4ba8ba94584c","Type":"ContainerDied","Data":"0ba69162fa8fc490af41b37c18640d1489f04a038e3f309d984b7a67ae89b2dc"}
Oct 30 19:56:33.503255 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:56:33.503143    4892 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="0ba69162fa8fc490af41b37c18640d1489f04a038e3f309d984b7a67ae89b2dc"
Oct 30 19:56:38.012038 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:56:38.012002    4892 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-cb14ed-6f8d8d5f84-6pqqb"]
Oct 30 19:56:38.127926 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:56:38.127895    4892 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-cb14ed-c8b7565cf-h6zcq"]
Oct 30 19:56:39.386522 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:56:39.386489    4892 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-6caff5-wm9m7"]
Oct 30 19:56:39.387171 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:56:39.387156    4892 cpu_manager.go:414] "containerMap: removing container" podUID="10fe3864-9417-4f6d-84ad-019943adf796" containerName="revoke-keys"
Oct 30 19:56:39.387218 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:56:39.387173    4892 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="10fe3864-9417-4f6d-84ad-019943adf796" containerName="revoke-keys"
Oct 30 19:56:39.387218 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:56:39.387193    4892 cpu_manager.go:414] "containerMap: removing container" podUID="39e00956-193c-4dc5-9ccc-4ba8ba94584c" containerName="revoke-keys"
Oct 30 19:56:39.387218 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:56:39.387198    4892 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="39e00956-193c-4dc5-9ccc-4ba8ba94584c" containerName="revoke-keys"
Oct 30 19:56:39.387341 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:56:39.387311    4892 memory_manager.go:365] "RemoveStaleState removing state" podUID="10fe3864-9417-4f6d-84ad-019943adf796" containerName="revoke-keys"
Oct 30 19:56:39.387341 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:56:39.387321    4892 memory_manager.go:365] "RemoveStaleState removing state" podUID="39e00956-193c-4dc5-9ccc-4ba8ba94584c" containerName="revoke-keys"
Oct 30 19:56:39.391126 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:56:39.391110    4892 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-6caff5-wm9m7"
Oct 30 19:56:39.445348 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:56:39.445325    4892 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 19:56:39.445633 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:56:39.445597    4892 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 30 19:56:39.446191 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:56:39.445820    4892 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 30 19:56:39.446191 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:56:39.445855    4892 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-nlxp4\""
Oct 30 19:56:39.451302 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:56:39.451279    4892 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-6caff5-wm9m7"]
Oct 30 19:56:39.648654 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:56:39.648556    4892 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qrd9m\" (UniqueName: \"kubernetes.io/projected/bdc15d2d-feda-4d17-9066-d6f510effbcf-kube-api-access-qrd9m\") pod \"maas-api-revoke-keys-e2e-api-a-6caff5-wm9m7\" (UID: \"bdc15d2d-feda-4d17-9066-d6f510effbcf\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-6caff5-wm9m7"
Oct 30 19:56:39.648946 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:56:39.648919    4892 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/bdc15d2d-feda-4d17-9066-d6f510effbcf-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-api-a-6caff5-wm9m7\" (UID: \"bdc15d2d-feda-4d17-9066-d6f510effbcf\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-6caff5-wm9m7"
Oct 30 19:56:39.649060 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:56:39.649048    4892 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/bdc15d2d-feda-4d17-9066-d6f510effbcf-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-api-a-6caff5-wm9m7\" (UID: \"bdc15d2d-feda-4d17-9066-d6f510effbcf\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-6caff5-wm9m7"
Oct 30 19:56:39.750420 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:56:39.750381    4892 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qrd9m\" (UniqueName: \"kubernetes.io/projected/bdc15d2d-feda-4d17-9066-d6f510effbcf-kube-api-access-qrd9m\") pod \"maas-api-revoke-keys-e2e-api-a-6caff5-wm9m7\" (UID: \"bdc15d2d-feda-4d17-9066-d6f510effbcf\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-6caff5-wm9m7"
Oct 30 19:56:39.750723 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:56:39.750697    4892 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/bdc15d2d-feda-4d17-9066-d6f510effbcf-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-api-a-6caff5-wm9m7\" (UID: \"bdc15d2d-feda-4d17-9066-d6f510effbcf\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-6caff5-wm9m7"
Oct 30 19:56:39.750826 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:56:39.750751    4892 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/bdc15d2d-feda-4d17-9066-d6f510effbcf-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-api-a-6caff5-wm9m7\" (UID: \"bdc15d2d-feda-4d17-9066-d6f510effbcf\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-6caff5-wm9m7"
Oct 30 19:56:39.751884 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:56:39.751856    4892 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/bdc15d2d-feda-4d17-9066-d6f510effbcf-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-api-a-6caff5-wm9m7\" (UID: \"bdc15d2d-feda-4d17-9066-d6f510effbcf\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-6caff5-wm9m7"
Oct 30 19:56:39.754451 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:56:39.754422    4892 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/bdc15d2d-feda-4d17-9066-d6f510effbcf-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-api-a-6caff5-wm9m7\" (UID: \"bdc15d2d-feda-4d17-9066-d6f510effbcf\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-6caff5-wm9m7"
Oct 30 19:56:39.815484 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:56:39.815452    4892 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qrd9m\" (UniqueName: \"kubernetes.io/projected/bdc15d2d-feda-4d17-9066-d6f510effbcf-kube-api-access-qrd9m\") pod \"maas-api-revoke-keys-e2e-api-a-6caff5-wm9m7\" (UID: \"bdc15d2d-feda-4d17-9066-d6f510effbcf\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-6caff5-wm9m7"
Oct 30 19:56:40.005216 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:56:40.005127    4892 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-6caff5-wm9m7"
Oct 30 19:56:40.403130 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:56:40.403093    4892 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-6caff5-wm9m7"]
Oct 30 19:56:40.405504 ip-10-0-132-82 kubenswrapper[4892]: W1030 19:56:40.405475    4892 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbdc15d2d_feda_4d17_9066_d6f510effbcf.slice/crio-31160786579db40152373e494044b41524f28e8e4dc2d574fd93144c058a378b WatchSource:0}: Error finding container 31160786579db40152373e494044b41524f28e8e4dc2d574fd93144c058a378b: Status 404 returned error can't find the container with id 31160786579db40152373e494044b41524f28e8e4dc2d574fd93144c058a378b
Oct 30 19:56:40.579638 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:56:40.579604    4892 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-6caff5-wm9m7" event={"ID":"bdc15d2d-feda-4d17-9066-d6f510effbcf","Type":"ContainerStarted","Data":"31160786579db40152373e494044b41524f28e8e4dc2d574fd93144c058a378b"}
Oct 30 19:56:41.589012 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:56:41.588942    4892 generic.go:356] "Generic (PLEG): container finished" podID="bdc15d2d-feda-4d17-9066-d6f510effbcf" containerID="72f5e3a23d03763ac8f5839a202969d29f16f47a12ff92506f63ea6d30fe414f" exitCode=0
Oct 30 19:56:41.589389 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:56:41.589041    4892 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-6caff5-wm9m7" event={"ID":"bdc15d2d-feda-4d17-9066-d6f510effbcf","Type":"ContainerDied","Data":"72f5e3a23d03763ac8f5839a202969d29f16f47a12ff92506f63ea6d30fe414f"}
Oct 30 19:56:42.181388 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:56:42.181351    4892 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-daa1ac-7f4d69d955-k5qgg"]
Oct 30 19:56:42.799016 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:56:42.798959    4892 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-6caff5-wm9m7"
Oct 30 19:56:42.890902 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:56:42.890827    4892 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/bdc15d2d-feda-4d17-9066-d6f510effbcf-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/bdc15d2d-feda-4d17-9066-d6f510effbcf-maas-api-xks-ca\") pod \"bdc15d2d-feda-4d17-9066-d6f510effbcf\" (UID: \"bdc15d2d-feda-4d17-9066-d6f510effbcf\") "
Oct 30 19:56:42.891049 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:56:42.890925    4892 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/bdc15d2d-feda-4d17-9066-d6f510effbcf-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/bdc15d2d-feda-4d17-9066-d6f510effbcf-maas-api-openshift-ca\") pod \"bdc15d2d-feda-4d17-9066-d6f510effbcf\" (UID: \"bdc15d2d-feda-4d17-9066-d6f510effbcf\") "
Oct 30 19:56:42.891049 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:56:42.890953    4892 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/bdc15d2d-feda-4d17-9066-d6f510effbcf-kube-api-access-qrd9m\" (UniqueName: \"kubernetes.io/projected/bdc15d2d-feda-4d17-9066-d6f510effbcf-kube-api-access-qrd9m\") pod \"bdc15d2d-feda-4d17-9066-d6f510effbcf\" (UID: \"bdc15d2d-feda-4d17-9066-d6f510effbcf\") "
Oct 30 19:56:42.891346 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:56:42.891308    4892 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/bdc15d2d-feda-4d17-9066-d6f510effbcf-maas-api-openshift-ca" pod "bdc15d2d-feda-4d17-9066-d6f510effbcf" (UID: "bdc15d2d-feda-4d17-9066-d6f510effbcf"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 19:56:42.892978 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:56:42.892944    4892 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/bdc15d2d-feda-4d17-9066-d6f510effbcf-maas-api-xks-ca" pod "bdc15d2d-feda-4d17-9066-d6f510effbcf" (UID: "bdc15d2d-feda-4d17-9066-d6f510effbcf"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 19:56:42.893172 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:56:42.893151    4892 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bdc15d2d-feda-4d17-9066-d6f510effbcf-kube-api-access-qrd9m" pod "bdc15d2d-feda-4d17-9066-d6f510effbcf" (UID: "bdc15d2d-feda-4d17-9066-d6f510effbcf"). InnerVolumeSpecName "kube-api-access-qrd9m". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:56:42.992093 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:56:42.992061    4892 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/bdc15d2d-feda-4d17-9066-d6f510effbcf-maas-api-openshift-ca\") on node \"ip-10-0-132-82.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 19:56:42.992093 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:56:42.992087    4892 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qrd9m\" (UniqueName: \"kubernetes.io/projected/bdc15d2d-feda-4d17-9066-d6f510effbcf-kube-api-access-qrd9m\") on node \"ip-10-0-132-82.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 19:56:42.992093 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:56:42.992097    4892 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/bdc15d2d-feda-4d17-9066-d6f510effbcf-maas-api-xks-ca\") on node \"ip-10-0-132-82.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 19:56:43.012793 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:56:43.012755    4892 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-api-a-cb14ed-6f8d8d5f84-6pqqb" podUID="0a20e1c6-2c3f-4c1b-ac66-9fb5dfe44bfb" containerName="payload-processing" containerID="cri-o://ced8287e7025b8a937c88a4ea862cd7804e46bc986d06f4ce6f594219447b426" gracePeriod=25
Oct 30 19:56:43.128584 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:56:43.128541    4892 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-api-a-cb14ed-c8b7565cf-h6zcq" podUID="6e76e92e-5a95-4a67-99a0-6907a8f22c71" containerName="payload-pre-processing" containerID="cri-o://f178ca9b51110b45933a71367152d6a75569134dcf93da7742f47b01e31f48f7" gracePeriod=25
Oct 30 19:56:43.289207 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:56:43.288597    4892 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-a-cb14ed-6f8d8d5f84-6pqqb"
Oct 30 19:56:43.295424 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:56:43.295400    4892 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/0a20e1c6-2c3f-4c1b-ac66-9fb5dfe44bfb-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0a20e1c6-2c3f-4c1b-ac66-9fb5dfe44bfb-plugins-config-volume\") pod \"0a20e1c6-2c3f-4c1b-ac66-9fb5dfe44bfb\" (UID: \"0a20e1c6-2c3f-4c1b-ac66-9fb5dfe44bfb\") "
Oct 30 19:56:43.295530 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:56:43.295477    4892 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0a20e1c6-2c3f-4c1b-ac66-9fb5dfe44bfb-kube-api-access-44svl\" (UniqueName: \"kubernetes.io/projected/0a20e1c6-2c3f-4c1b-ac66-9fb5dfe44bfb-kube-api-access-44svl\") pod \"0a20e1c6-2c3f-4c1b-ac66-9fb5dfe44bfb\" (UID: \"0a20e1c6-2c3f-4c1b-ac66-9fb5dfe44bfb\") "
Oct 30 19:56:43.295816 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:56:43.295783    4892 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0a20e1c6-2c3f-4c1b-ac66-9fb5dfe44bfb-plugins-config-volume" pod "0a20e1c6-2c3f-4c1b-ac66-9fb5dfe44bfb" (UID: "0a20e1c6-2c3f-4c1b-ac66-9fb5dfe44bfb"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 19:56:43.296118 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:56:43.296102    4892 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0a20e1c6-2c3f-4c1b-ac66-9fb5dfe44bfb-plugins-config-volume\") on node \"ip-10-0-132-82.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 19:56:43.298014 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:56:43.297988    4892 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0a20e1c6-2c3f-4c1b-ac66-9fb5dfe44bfb-kube-api-access-44svl" pod "0a20e1c6-2c3f-4c1b-ac66-9fb5dfe44bfb" (UID: "0a20e1c6-2c3f-4c1b-ac66-9fb5dfe44bfb"). InnerVolumeSpecName "kube-api-access-44svl". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:56:43.397450 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:56:43.397418    4892 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-44svl\" (UniqueName: \"kubernetes.io/projected/0a20e1c6-2c3f-4c1b-ac66-9fb5dfe44bfb-kube-api-access-44svl\") on node \"ip-10-0-132-82.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 19:56:43.418615 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:56:43.418586    4892 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-a-cb14ed-c8b7565cf-h6zcq"
Oct 30 19:56:43.528039 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:56:43.508664    4892 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/6e76e92e-5a95-4a67-99a0-6907a8f22c71-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6e76e92e-5a95-4a67-99a0-6907a8f22c71-plugins-config-volume\") pod \"6e76e92e-5a95-4a67-99a0-6907a8f22c71\" (UID: \"6e76e92e-5a95-4a67-99a0-6907a8f22c71\") "
Oct 30 19:56:43.528039 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:56:43.508767    4892 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6e76e92e-5a95-4a67-99a0-6907a8f22c71-kube-api-access-5z69q\" (UniqueName: \"kubernetes.io/projected/6e76e92e-5a95-4a67-99a0-6907a8f22c71-kube-api-access-5z69q\") pod \"6e76e92e-5a95-4a67-99a0-6907a8f22c71\" (UID: \"6e76e92e-5a95-4a67-99a0-6907a8f22c71\") "
Oct 30 19:56:43.541987 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:56:43.536609    4892 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6e76e92e-5a95-4a67-99a0-6907a8f22c71-plugins-config-volume" pod "6e76e92e-5a95-4a67-99a0-6907a8f22c71" (UID: "6e76e92e-5a95-4a67-99a0-6907a8f22c71"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 19:56:43.549850 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:56:43.549356    4892 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6e76e92e-5a95-4a67-99a0-6907a8f22c71-kube-api-access-5z69q" pod "6e76e92e-5a95-4a67-99a0-6907a8f22c71" (UID: "6e76e92e-5a95-4a67-99a0-6907a8f22c71"). InnerVolumeSpecName "kube-api-access-5z69q". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:56:43.626993 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:56:43.624463    4892 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6e76e92e-5a95-4a67-99a0-6907a8f22c71-plugins-config-volume\") on node \"ip-10-0-132-82.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 19:56:43.626993 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:56:43.624497    4892 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5z69q\" (UniqueName: \"kubernetes.io/projected/6e76e92e-5a95-4a67-99a0-6907a8f22c71-kube-api-access-5z69q\") on node \"ip-10-0-132-82.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 19:56:43.654468 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:56:43.654183    4892 generic.go:356] "Generic (PLEG): container finished" podID="6e76e92e-5a95-4a67-99a0-6907a8f22c71" containerID="f178ca9b51110b45933a71367152d6a75569134dcf93da7742f47b01e31f48f7" exitCode=0
Oct 30 19:56:43.654468 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:56:43.654267    4892 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-a-cb14ed-c8b7565cf-h6zcq" event={"ID":"6e76e92e-5a95-4a67-99a0-6907a8f22c71","Type":"ContainerDied","Data":"f178ca9b51110b45933a71367152d6a75569134dcf93da7742f47b01e31f48f7"}
Oct 30 19:56:43.654468 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:56:43.654316    4892 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-a-cb14ed-c8b7565cf-h6zcq" event={"ID":"6e76e92e-5a95-4a67-99a0-6907a8f22c71","Type":"ContainerDied","Data":"6780f905bb9d78366d0bef0670d986d2208ca1c29a0ca76605cf93d48334602e"}
Oct 30 19:56:43.654468 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:56:43.654342    4892 scope.go:122] "RemoveContainer" containerID="f178ca9b51110b45933a71367152d6a75569134dcf93da7742f47b01e31f48f7"
Oct 30 19:56:43.655001 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:56:43.654958    4892 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-a-cb14ed-c8b7565cf-h6zcq"
Oct 30 19:56:43.665516 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:56:43.665493    4892 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-6caff5-wm9m7"
Oct 30 19:56:43.665632 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:56:43.665582    4892 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-6caff5-wm9m7" event={"ID":"bdc15d2d-feda-4d17-9066-d6f510effbcf","Type":"ContainerDied","Data":"31160786579db40152373e494044b41524f28e8e4dc2d574fd93144c058a378b"}
Oct 30 19:56:43.665632 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:56:43.665619    4892 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="31160786579db40152373e494044b41524f28e8e4dc2d574fd93144c058a378b"
Oct 30 19:56:43.667098 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:56:43.667075    4892 generic.go:356] "Generic (PLEG): container finished" podID="0a20e1c6-2c3f-4c1b-ac66-9fb5dfe44bfb" containerID="ced8287e7025b8a937c88a4ea862cd7804e46bc986d06f4ce6f594219447b426" exitCode=0
Oct 30 19:56:43.667187 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:56:43.667115    4892 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-a-cb14ed-6f8d8d5f84-6pqqb"
Oct 30 19:56:43.667187 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:56:43.667137    4892 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-a-cb14ed-6f8d8d5f84-6pqqb" event={"ID":"0a20e1c6-2c3f-4c1b-ac66-9fb5dfe44bfb","Type":"ContainerDied","Data":"ced8287e7025b8a937c88a4ea862cd7804e46bc986d06f4ce6f594219447b426"}
Oct 30 19:56:43.667187 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:56:43.667171    4892 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-a-cb14ed-6f8d8d5f84-6pqqb" event={"ID":"0a20e1c6-2c3f-4c1b-ac66-9fb5dfe44bfb","Type":"ContainerDied","Data":"f9ba13e179f83e78e2ef3f48ddabda5aa6777c57a427c57c511e265c82727196"}
Oct 30 19:56:43.677509 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:56:43.677489    4892 scope.go:122] "RemoveContainer" containerID="f178ca9b51110b45933a71367152d6a75569134dcf93da7742f47b01e31f48f7"
Oct 30 19:56:43.677825 ip-10-0-132-82 kubenswrapper[4892]: E1030 19:56:43.677796    4892 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f178ca9b51110b45933a71367152d6a75569134dcf93da7742f47b01e31f48f7\": container with ID starting with f178ca9b51110b45933a71367152d6a75569134dcf93da7742f47b01e31f48f7 not found: ID does not exist" containerID="f178ca9b51110b45933a71367152d6a75569134dcf93da7742f47b01e31f48f7"
Oct 30 19:56:43.677936 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:56:43.677837    4892 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f178ca9b51110b45933a71367152d6a75569134dcf93da7742f47b01e31f48f7"} err="failed to get container status \"f178ca9b51110b45933a71367152d6a75569134dcf93da7742f47b01e31f48f7\": rpc error: code = NotFound desc = could not find container \"f178ca9b51110b45933a71367152d6a75569134dcf93da7742f47b01e31f48f7\": container with ID starting with f178ca9b51110b45933a71367152d6a75569134dcf93da7742f47b01e31f48f7 not found: ID does not exist"
Oct 30 19:56:43.677936 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:56:43.677860    4892 scope.go:122] "RemoveContainer" containerID="ced8287e7025b8a937c88a4ea862cd7804e46bc986d06f4ce6f594219447b426"
Oct 30 19:56:43.688721 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:56:43.688704    4892 scope.go:122] "RemoveContainer" containerID="ced8287e7025b8a937c88a4ea862cd7804e46bc986d06f4ce6f594219447b426"
Oct 30 19:56:43.688958 ip-10-0-132-82 kubenswrapper[4892]: E1030 19:56:43.688942    4892 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ced8287e7025b8a937c88a4ea862cd7804e46bc986d06f4ce6f594219447b426\": container with ID starting with ced8287e7025b8a937c88a4ea862cd7804e46bc986d06f4ce6f594219447b426 not found: ID does not exist" containerID="ced8287e7025b8a937c88a4ea862cd7804e46bc986d06f4ce6f594219447b426"
Oct 30 19:56:43.689023 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:56:43.688997    4892 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ced8287e7025b8a937c88a4ea862cd7804e46bc986d06f4ce6f594219447b426"} err="failed to get container status \"ced8287e7025b8a937c88a4ea862cd7804e46bc986d06f4ce6f594219447b426\": rpc error: code = NotFound desc = could not find container \"ced8287e7025b8a937c88a4ea862cd7804e46bc986d06f4ce6f594219447b426\": container with ID starting with ced8287e7025b8a937c88a4ea862cd7804e46bc986d06f4ce6f594219447b426 not found: ID does not exist"
Oct 30 19:56:43.902686 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:56:43.902655    4892 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-cb14ed-c8b7565cf-h6zcq"]
Oct 30 19:56:43.906854 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:56:43.906819    4892 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-cb14ed-c8b7565cf-h6zcq"]
Oct 30 19:56:44.009617 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:56:44.009562    4892 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-cb14ed-6f8d8d5f84-6pqqb"]
Oct 30 19:56:44.014553 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:56:44.014527    4892 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-cb14ed-6f8d8d5f84-6pqqb"]
Oct 30 19:56:44.570785 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:56:44.570751    4892 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0a20e1c6-2c3f-4c1b-ac66-9fb5dfe44bfb" path="/var/lib/kubelet/pods/0a20e1c6-2c3f-4c1b-ac66-9fb5dfe44bfb/volumes"
Oct 30 19:56:44.571240 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:56:44.571224    4892 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="6e76e92e-5a95-4a67-99a0-6907a8f22c71" path="/var/lib/kubelet/pods/6e76e92e-5a95-4a67-99a0-6907a8f22c71/volumes"
Oct 30 19:56:47.181648 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:56:47.181604    4892 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-daa1ac-7f4d69d955-k5qgg" podUID="5ce65b2f-c2fb-4c9a-9e3b-783315e390b1" containerName="payload-pre-processing" containerID="cri-o://102578dd6e23bbdb0fabcd53e438b50e14073fcc5d32ebe6cda98f7ab76d2d04" gracePeriod=25
Oct 30 19:56:47.471568 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:56:47.471539    4892 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-daa1ac-7f4d69d955-k5qgg"
Oct 30 19:56:47.586414 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:56:47.585765    4892 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5ce65b2f-c2fb-4c9a-9e3b-783315e390b1-kube-api-access-h6rh5\" (UniqueName: \"kubernetes.io/projected/5ce65b2f-c2fb-4c9a-9e3b-783315e390b1-kube-api-access-h6rh5\") pod \"5ce65b2f-c2fb-4c9a-9e3b-783315e390b1\" (UID: \"5ce65b2f-c2fb-4c9a-9e3b-783315e390b1\") "
Oct 30 19:56:47.586414 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:56:47.585860    4892 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/5ce65b2f-c2fb-4c9a-9e3b-783315e390b1-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5ce65b2f-c2fb-4c9a-9e3b-783315e390b1-plugins-config-volume\") pod \"5ce65b2f-c2fb-4c9a-9e3b-783315e390b1\" (UID: \"5ce65b2f-c2fb-4c9a-9e3b-783315e390b1\") "
Oct 30 19:56:47.586414 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:56:47.586393    4892 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5ce65b2f-c2fb-4c9a-9e3b-783315e390b1-plugins-config-volume" pod "5ce65b2f-c2fb-4c9a-9e3b-783315e390b1" (UID: "5ce65b2f-c2fb-4c9a-9e3b-783315e390b1"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 19:56:47.589145 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:56:47.589114    4892 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5ce65b2f-c2fb-4c9a-9e3b-783315e390b1-kube-api-access-h6rh5" pod "5ce65b2f-c2fb-4c9a-9e3b-783315e390b1" (UID: "5ce65b2f-c2fb-4c9a-9e3b-783315e390b1"). InnerVolumeSpecName "kube-api-access-h6rh5". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:56:47.688105 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:56:47.688078    4892 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-h6rh5\" (UniqueName: \"kubernetes.io/projected/5ce65b2f-c2fb-4c9a-9e3b-783315e390b1-kube-api-access-h6rh5\") on node \"ip-10-0-132-82.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 19:56:47.688105 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:56:47.688107    4892 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5ce65b2f-c2fb-4c9a-9e3b-783315e390b1-plugins-config-volume\") on node \"ip-10-0-132-82.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 19:56:47.703542 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:56:47.703471    4892 generic.go:356] "Generic (PLEG): container finished" podID="5ce65b2f-c2fb-4c9a-9e3b-783315e390b1" containerID="102578dd6e23bbdb0fabcd53e438b50e14073fcc5d32ebe6cda98f7ab76d2d04" exitCode=0
Oct 30 19:56:47.703542 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:56:47.703518    4892 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-daa1ac-7f4d69d955-k5qgg"
Oct 30 19:56:47.703734 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:56:47.703562    4892 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-daa1ac-7f4d69d955-k5qgg" event={"ID":"5ce65b2f-c2fb-4c9a-9e3b-783315e390b1","Type":"ContainerDied","Data":"102578dd6e23bbdb0fabcd53e438b50e14073fcc5d32ebe6cda98f7ab76d2d04"}
Oct 30 19:56:47.703734 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:56:47.703623    4892 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-daa1ac-7f4d69d955-k5qgg" event={"ID":"5ce65b2f-c2fb-4c9a-9e3b-783315e390b1","Type":"ContainerDied","Data":"193d288535c50a6d7bac80d4a5c4597f17300df1faa70fab0e9e30531c850bac"}
Oct 30 19:56:47.703734 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:56:47.703651    4892 scope.go:122] "RemoveContainer" containerID="102578dd6e23bbdb0fabcd53e438b50e14073fcc5d32ebe6cda98f7ab76d2d04"
Oct 30 19:56:47.714802 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:56:47.714784    4892 scope.go:122] "RemoveContainer" containerID="102578dd6e23bbdb0fabcd53e438b50e14073fcc5d32ebe6cda98f7ab76d2d04"
Oct 30 19:56:47.715104 ip-10-0-132-82 kubenswrapper[4892]: E1030 19:56:47.715078    4892 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"102578dd6e23bbdb0fabcd53e438b50e14073fcc5d32ebe6cda98f7ab76d2d04\": container with ID starting with 102578dd6e23bbdb0fabcd53e438b50e14073fcc5d32ebe6cda98f7ab76d2d04 not found: ID does not exist" containerID="102578dd6e23bbdb0fabcd53e438b50e14073fcc5d32ebe6cda98f7ab76d2d04"
Oct 30 19:56:47.715164 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:56:47.715114    4892 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"102578dd6e23bbdb0fabcd53e438b50e14073fcc5d32ebe6cda98f7ab76d2d04"} err="failed to get container status \"102578dd6e23bbdb0fabcd53e438b50e14073fcc5d32ebe6cda98f7ab76d2d04\": rpc error: code = NotFound desc = could not find container \"102578dd6e23bbdb0fabcd53e438b50e14073fcc5d32ebe6cda98f7ab76d2d04\": container with ID starting with 102578dd6e23bbdb0fabcd53e438b50e14073fcc5d32ebe6cda98f7ab76d2d04 not found: ID does not exist"
Oct 30 19:56:47.867779 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:56:47.867753    4892 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-daa1ac-7f4d69d955-k5qgg"]
Oct 30 19:56:47.871543 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:56:47.871523    4892 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-daa1ac-7f4d69d955-k5qgg"]
Oct 30 19:56:48.572897 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:56:48.572860    4892 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="5ce65b2f-c2fb-4c9a-9e3b-783315e390b1" path="/var/lib/kubelet/pods/5ce65b2f-c2fb-4c9a-9e3b-783315e390b1/volumes"
Oct 30 19:56:50.651060 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:56:50.649518    4892 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-6caff5-86dcd5d66d-9r2cn"]
Oct 30 19:56:50.762595 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:56:50.762546    4892 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-6caff5-fcddd97f9-vjnf6"]
Oct 30 19:56:54.849040 ip-10-0-132-82 kubenswrapper[4892]: E1030 19:56:54.848996    4892 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-jrm87" containerName="extractor"
Oct 30 19:56:55.650445 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:56:55.650399    4892 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-api-a-6caff5-86dcd5d66d-9r2cn" podUID="61ff434e-6a69-4316-aacb-478b4b12274e" containerName="payload-processing" containerID="cri-o://98e19108a5e162853b604d61ee8264b0969acc3416663fd48ff0728f838e7def" gracePeriod=25
Oct 30 19:56:55.763834 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:56:55.763756    4892 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-api-a-6caff5-fcddd97f9-vjnf6" podUID="203e53c4-cf13-4f11-a90f-59319c124125" containerName="payload-pre-processing" containerID="cri-o://32d922a2955c677ec98a2d3ec8a66fa2200b5d7a03918f2bce2d9a45bf54c976" gracePeriod=25
Oct 30 19:56:55.775374 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:56:55.775341    4892 generic.go:356] "Generic (PLEG): container finished" podID="61ff434e-6a69-4316-aacb-478b4b12274e" containerID="98e19108a5e162853b604d61ee8264b0969acc3416663fd48ff0728f838e7def" exitCode=0
Oct 30 19:56:55.775498 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:56:55.775400    4892 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-a-6caff5-86dcd5d66d-9r2cn" event={"ID":"61ff434e-6a69-4316-aacb-478b4b12274e","Type":"ContainerDied","Data":"98e19108a5e162853b604d61ee8264b0969acc3416663fd48ff0728f838e7def"}
Oct 30 19:56:55.940208 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:56:55.940177    4892 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-a-6caff5-86dcd5d66d-9r2cn"
Oct 30 19:56:56.054300 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:56:56.054276    4892 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-a-6caff5-fcddd97f9-vjnf6"
Oct 30 19:56:56.098474 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:56:56.098434    4892 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/61ff434e-6a69-4316-aacb-478b4b12274e-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/61ff434e-6a69-4316-aacb-478b4b12274e-plugins-config-volume\") pod \"61ff434e-6a69-4316-aacb-478b4b12274e\" (UID: \"61ff434e-6a69-4316-aacb-478b4b12274e\") "
Oct 30 19:56:56.098639 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:56:56.098493    4892 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/61ff434e-6a69-4316-aacb-478b4b12274e-kube-api-access-9z8nl\" (UniqueName: \"kubernetes.io/projected/61ff434e-6a69-4316-aacb-478b4b12274e-kube-api-access-9z8nl\") pod \"61ff434e-6a69-4316-aacb-478b4b12274e\" (UID: \"61ff434e-6a69-4316-aacb-478b4b12274e\") "
Oct 30 19:56:56.098871 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:56:56.098840    4892 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/61ff434e-6a69-4316-aacb-478b4b12274e-plugins-config-volume" pod "61ff434e-6a69-4316-aacb-478b4b12274e" (UID: "61ff434e-6a69-4316-aacb-478b4b12274e"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 19:56:56.099064 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:56:56.099005    4892 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/61ff434e-6a69-4316-aacb-478b4b12274e-plugins-config-volume\") on node \"ip-10-0-132-82.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 19:56:56.100717 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:56:56.100693    4892 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/61ff434e-6a69-4316-aacb-478b4b12274e-kube-api-access-9z8nl" pod "61ff434e-6a69-4316-aacb-478b4b12274e" (UID: "61ff434e-6a69-4316-aacb-478b4b12274e"). InnerVolumeSpecName "kube-api-access-9z8nl". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:56:56.200190 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:56:56.200099    4892 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/203e53c4-cf13-4f11-a90f-59319c124125-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/203e53c4-cf13-4f11-a90f-59319c124125-plugins-config-volume\") pod \"203e53c4-cf13-4f11-a90f-59319c124125\" (UID: \"203e53c4-cf13-4f11-a90f-59319c124125\") "
Oct 30 19:56:56.200190 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:56:56.200148    4892 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/203e53c4-cf13-4f11-a90f-59319c124125-kube-api-access-bz4r2\" (UniqueName: \"kubernetes.io/projected/203e53c4-cf13-4f11-a90f-59319c124125-kube-api-access-bz4r2\") pod \"203e53c4-cf13-4f11-a90f-59319c124125\" (UID: \"203e53c4-cf13-4f11-a90f-59319c124125\") "
Oct 30 19:56:56.200562 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:56:56.200535    4892 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/203e53c4-cf13-4f11-a90f-59319c124125-plugins-config-volume" pod "203e53c4-cf13-4f11-a90f-59319c124125" (UID: "203e53c4-cf13-4f11-a90f-59319c124125"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 19:56:56.200674 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:56:56.200655    4892 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9z8nl\" (UniqueName: \"kubernetes.io/projected/61ff434e-6a69-4316-aacb-478b4b12274e-kube-api-access-9z8nl\") on node \"ip-10-0-132-82.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 19:56:56.200728 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:56:56.200683    4892 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/203e53c4-cf13-4f11-a90f-59319c124125-plugins-config-volume\") on node \"ip-10-0-132-82.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 19:56:56.202631 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:56:56.202605    4892 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/203e53c4-cf13-4f11-a90f-59319c124125-kube-api-access-bz4r2" pod "203e53c4-cf13-4f11-a90f-59319c124125" (UID: "203e53c4-cf13-4f11-a90f-59319c124125"). InnerVolumeSpecName "kube-api-access-bz4r2". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:56:56.301987 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:56:56.301924    4892 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bz4r2\" (UniqueName: \"kubernetes.io/projected/203e53c4-cf13-4f11-a90f-59319c124125-kube-api-access-bz4r2\") on node \"ip-10-0-132-82.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 19:56:56.784829 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:56:56.784800    4892 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-a-6caff5-86dcd5d66d-9r2cn"
Oct 30 19:56:56.785028 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:56:56.784839    4892 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-a-6caff5-86dcd5d66d-9r2cn" event={"ID":"61ff434e-6a69-4316-aacb-478b4b12274e","Type":"ContainerDied","Data":"19edfd7fd0b3f171260aee2449dc4ba7ed6a7b47ccac6ed6a512c1ad8622e86a"}
Oct 30 19:56:56.785028 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:56:56.784898    4892 scope.go:122] "RemoveContainer" containerID="98e19108a5e162853b604d61ee8264b0969acc3416663fd48ff0728f838e7def"
Oct 30 19:56:56.786698 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:56:56.786672    4892 generic.go:356] "Generic (PLEG): container finished" podID="203e53c4-cf13-4f11-a90f-59319c124125" containerID="32d922a2955c677ec98a2d3ec8a66fa2200b5d7a03918f2bce2d9a45bf54c976" exitCode=0
Oct 30 19:56:56.786819 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:56:56.786724    4892 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-a-6caff5-fcddd97f9-vjnf6"
Oct 30 19:56:56.786819 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:56:56.786749    4892 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-a-6caff5-fcddd97f9-vjnf6" event={"ID":"203e53c4-cf13-4f11-a90f-59319c124125","Type":"ContainerDied","Data":"32d922a2955c677ec98a2d3ec8a66fa2200b5d7a03918f2bce2d9a45bf54c976"}
Oct 30 19:56:56.786819 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:56:56.786786    4892 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-a-6caff5-fcddd97f9-vjnf6" event={"ID":"203e53c4-cf13-4f11-a90f-59319c124125","Type":"ContainerDied","Data":"6e794fb4f892eea2bfc0972ea523a0c431f03e048d8ef14329c27e9a3717fc39"}
Oct 30 19:56:56.796862 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:56:56.796832    4892 scope.go:122] "RemoveContainer" containerID="32d922a2955c677ec98a2d3ec8a66fa2200b5d7a03918f2bce2d9a45bf54c976"
Oct 30 19:56:56.806480 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:56:56.806446    4892 scope.go:122] "RemoveContainer" containerID="32d922a2955c677ec98a2d3ec8a66fa2200b5d7a03918f2bce2d9a45bf54c976"
Oct 30 19:56:56.806725 ip-10-0-132-82 kubenswrapper[4892]: E1030 19:56:56.806705    4892 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"32d922a2955c677ec98a2d3ec8a66fa2200b5d7a03918f2bce2d9a45bf54c976\": container with ID starting with 32d922a2955c677ec98a2d3ec8a66fa2200b5d7a03918f2bce2d9a45bf54c976 not found: ID does not exist" containerID="32d922a2955c677ec98a2d3ec8a66fa2200b5d7a03918f2bce2d9a45bf54c976"
Oct 30 19:56:56.806794 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:56:56.806736    4892 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"32d922a2955c677ec98a2d3ec8a66fa2200b5d7a03918f2bce2d9a45bf54c976"} err="failed to get container status \"32d922a2955c677ec98a2d3ec8a66fa2200b5d7a03918f2bce2d9a45bf54c976\": rpc error: code = NotFound desc = could not find container \"32d922a2955c677ec98a2d3ec8a66fa2200b5d7a03918f2bce2d9a45bf54c976\": container with ID starting with 32d922a2955c677ec98a2d3ec8a66fa2200b5d7a03918f2bce2d9a45bf54c976 not found: ID does not exist"
Oct 30 19:56:56.954555 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:56:56.954528    4892 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-6caff5-fcddd97f9-vjnf6"]
Oct 30 19:56:57.003002 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:56:57.002958    4892 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-6caff5-fcddd97f9-vjnf6"]
Oct 30 19:56:57.110688 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:56:57.110467    4892 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-6caff5-86dcd5d66d-9r2cn"]
Oct 30 19:56:57.114546 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:56:57.114522    4892 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-6caff5-86dcd5d66d-9r2cn"]
Oct 30 19:56:58.570621 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:56:58.570585    4892 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="203e53c4-cf13-4f11-a90f-59319c124125" path="/var/lib/kubelet/pods/203e53c4-cf13-4f11-a90f-59319c124125/volumes"
Oct 30 19:56:58.571221 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:56:58.571201    4892 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="61ff434e-6a69-4316-aacb-478b4b12274e" path="/var/lib/kubelet/pods/61ff434e-6a69-4316-aacb-478b4b12274e/volumes"
Oct 30 19:57:02.349341 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:57:02.349262    4892 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-6caff5-wm9m7"]
Oct 30 19:57:02.398250 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:57:02.398219    4892 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-6caff5-wm9m7"]
Oct 30 19:57:02.571735 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:57:02.571702    4892 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="bdc15d2d-feda-4d17-9066-d6f510effbcf" path="/var/lib/kubelet/pods/bdc15d2d-feda-4d17-9066-d6f510effbcf/volumes"
Oct 30 19:57:09.649579 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:57:09.649543    4892 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-a-6caff5-openshift-default-665946589f-hn4qk"]
Oct 30 19:57:09.650101 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:57:09.649807    4892 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-api-a-6caff5-openshift-default-665946589f-hn4qk" podUID="33098b51-3c25-4761-a108-844516b8d2fa" containerName="istio-proxy" containerID="cri-o://4bec4ad58bd687ed8ebc8b6e4853203a4ed0ceb1dbc2377d38f44440e95b626b" gracePeriod=30
Oct 30 19:57:12.809273 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:57:12.809234    4892 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-6873e0-6f6mc"]
Oct 30 19:57:12.810683 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:57:12.810655    4892 cpu_manager.go:414] "containerMap: removing container" podUID="6e76e92e-5a95-4a67-99a0-6907a8f22c71" containerName="payload-pre-processing"
Oct 30 19:57:12.810683 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:57:12.810681    4892 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6e76e92e-5a95-4a67-99a0-6907a8f22c71" containerName="payload-pre-processing"
Oct 30 19:57:12.810878 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:57:12.810756    4892 cpu_manager.go:414] "containerMap: removing container" podUID="5ce65b2f-c2fb-4c9a-9e3b-783315e390b1" containerName="payload-pre-processing"
Oct 30 19:57:12.810878 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:57:12.810765    4892 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5ce65b2f-c2fb-4c9a-9e3b-783315e390b1" containerName="payload-pre-processing"
Oct 30 19:57:12.810878 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:57:12.810796    4892 cpu_manager.go:414] "containerMap: removing container" podUID="bdc15d2d-feda-4d17-9066-d6f510effbcf" containerName="revoke-keys"
Oct 30 19:57:12.810878 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:57:12.810804    4892 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="bdc15d2d-feda-4d17-9066-d6f510effbcf" containerName="revoke-keys"
Oct 30 19:57:12.810878 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:57:12.810838    4892 cpu_manager.go:414] "containerMap: removing container" podUID="61ff434e-6a69-4316-aacb-478b4b12274e" containerName="payload-processing"
Oct 30 19:57:12.810878 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:57:12.810845    4892 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="61ff434e-6a69-4316-aacb-478b4b12274e" containerName="payload-processing"
Oct 30 19:57:12.810878 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:57:12.810863    4892 cpu_manager.go:414] "containerMap: removing container" podUID="0a20e1c6-2c3f-4c1b-ac66-9fb5dfe44bfb" containerName="payload-processing"
Oct 30 19:57:12.810878 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:57:12.810870    4892 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0a20e1c6-2c3f-4c1b-ac66-9fb5dfe44bfb" containerName="payload-processing"
Oct 30 19:57:12.811284 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:57:12.810895    4892 cpu_manager.go:414] "containerMap: removing container" podUID="203e53c4-cf13-4f11-a90f-59319c124125" containerName="payload-pre-processing"
Oct 30 19:57:12.811284 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:57:12.810903    4892 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="203e53c4-cf13-4f11-a90f-59319c124125" containerName="payload-pre-processing"
Oct 30 19:57:12.811284 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:57:12.811262    4892 memory_manager.go:365] "RemoveStaleState removing state" podUID="5ce65b2f-c2fb-4c9a-9e3b-783315e390b1" containerName="payload-pre-processing"
Oct 30 19:57:12.811419 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:57:12.811290    4892 memory_manager.go:365] "RemoveStaleState removing state" podUID="bdc15d2d-feda-4d17-9066-d6f510effbcf" containerName="revoke-keys"
Oct 30 19:57:12.811419 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:57:12.811310    4892 memory_manager.go:365] "RemoveStaleState removing state" podUID="61ff434e-6a69-4316-aacb-478b4b12274e" containerName="payload-processing"
Oct 30 19:57:12.811419 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:57:12.811325    4892 memory_manager.go:365] "RemoveStaleState removing state" podUID="0a20e1c6-2c3f-4c1b-ac66-9fb5dfe44bfb" containerName="payload-processing"
Oct 30 19:57:12.811419 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:57:12.811343    4892 memory_manager.go:365] "RemoveStaleState removing state" podUID="203e53c4-cf13-4f11-a90f-59319c124125" containerName="payload-pre-processing"
Oct 30 19:57:12.811419 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:57:12.811367    4892 memory_manager.go:365] "RemoveStaleState removing state" podUID="6e76e92e-5a95-4a67-99a0-6907a8f22c71" containerName="payload-pre-processing"
Oct 30 19:57:12.819449 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:57:12.819419    4892 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-6873e0-6f6mc"
Oct 30 19:57:12.864262 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:57:12.864226    4892 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4skbz\" (UniqueName: \"kubernetes.io/projected/561a463b-f457-4bc5-bcdc-aa4b520df908-kube-api-access-4skbz\") pod \"maas-api-revoke-keys-e2e-inf-a-6873e0-6f6mc\" (UID: \"561a463b-f457-4bc5-bcdc-aa4b520df908\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-6873e0-6f6mc"
Oct 30 19:57:12.864545 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:57:12.864518    4892 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/561a463b-f457-4bc5-bcdc-aa4b520df908-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-6873e0-6f6mc\" (UID: \"561a463b-f457-4bc5-bcdc-aa4b520df908\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-6873e0-6f6mc"
Oct 30 19:57:12.864679 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:57:12.864616    4892 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/561a463b-f457-4bc5-bcdc-aa4b520df908-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-6873e0-6f6mc\" (UID: \"561a463b-f457-4bc5-bcdc-aa4b520df908\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-6873e0-6f6mc"
Oct 30 19:57:12.881482 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:57:12.881454    4892 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-6873e0-6f6mc"]
Oct 30 19:57:12.924214 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:57:12.924176    4892 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 19:57:12.924214 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:57:12.924188    4892 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 30 19:57:12.924453 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:57:12.924242    4892 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 30 19:57:12.925676 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:57:12.925659    4892 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-nlxp4\""
Oct 30 19:57:12.965662 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:57:12.965633    4892 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4skbz\" (UniqueName: \"kubernetes.io/projected/561a463b-f457-4bc5-bcdc-aa4b520df908-kube-api-access-4skbz\") pod \"maas-api-revoke-keys-e2e-inf-a-6873e0-6f6mc\" (UID: \"561a463b-f457-4bc5-bcdc-aa4b520df908\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-6873e0-6f6mc"
Oct 30 19:57:12.965837 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:57:12.965699    4892 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/561a463b-f457-4bc5-bcdc-aa4b520df908-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-6873e0-6f6mc\" (UID: \"561a463b-f457-4bc5-bcdc-aa4b520df908\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-6873e0-6f6mc"
Oct 30 19:57:12.965837 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:57:12.965752    4892 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/561a463b-f457-4bc5-bcdc-aa4b520df908-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-6873e0-6f6mc\" (UID: \"561a463b-f457-4bc5-bcdc-aa4b520df908\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-6873e0-6f6mc"
Oct 30 19:57:12.966587 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:57:12.966512    4892 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/561a463b-f457-4bc5-bcdc-aa4b520df908-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-6873e0-6f6mc\" (UID: \"561a463b-f457-4bc5-bcdc-aa4b520df908\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-6873e0-6f6mc"
Oct 30 19:57:12.968397 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:57:12.968370    4892 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/561a463b-f457-4bc5-bcdc-aa4b520df908-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-6873e0-6f6mc\" (UID: \"561a463b-f457-4bc5-bcdc-aa4b520df908\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-6873e0-6f6mc"
Oct 30 19:57:13.023171 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:57:13.023120    4892 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4skbz\" (UniqueName: \"kubernetes.io/projected/561a463b-f457-4bc5-bcdc-aa4b520df908-kube-api-access-4skbz\") pod \"maas-api-revoke-keys-e2e-inf-a-6873e0-6f6mc\" (UID: \"561a463b-f457-4bc5-bcdc-aa4b520df908\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-6873e0-6f6mc"
Oct 30 19:57:13.134441 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:57:13.134417    4892 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-6873e0-6f6mc"
Oct 30 19:57:13.518195 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:57:13.518162    4892 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-6873e0-6f6mc"]
Oct 30 19:57:13.518496 ip-10-0-132-82 kubenswrapper[4892]: W1030 19:57:13.518472    4892 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod561a463b_f457_4bc5_bcdc_aa4b520df908.slice/crio-c3ddad455d996c7c181dc93598c7bba4db3498ce5f2d550f4393eae8c1df757d WatchSource:0}: Error finding container c3ddad455d996c7c181dc93598c7bba4db3498ce5f2d550f4393eae8c1df757d: Status 404 returned error can't find the container with id c3ddad455d996c7c181dc93598c7bba4db3498ce5f2d550f4393eae8c1df757d
Oct 30 19:57:13.927239 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:57:13.927192    4892 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-6873e0-6f6mc" event={"ID":"561a463b-f457-4bc5-bcdc-aa4b520df908","Type":"ContainerStarted","Data":"71c3d80f18f2a473bf806b9f0168ec3acfc61f9cd43fc4a62570438a1e115425"}
Oct 30 19:57:13.927760 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:57:13.927269    4892 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-6873e0-6f6mc" event={"ID":"561a463b-f457-4bc5-bcdc-aa4b520df908","Type":"ContainerStarted","Data":"c3ddad455d996c7c181dc93598c7bba4db3498ce5f2d550f4393eae8c1df757d"}
Oct 30 19:57:14.047421 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:57:14.047363    4892 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-6873e0-6f6mc" podStartSLOduration=2.047344447 podStartE2EDuration="2.047344447s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:57:12 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:57:14.045148686 +0000 UTC m=+2481.615614289" watchObservedRunningTime="2026-10-30 19:57:14.047344447 +0000 UTC m=+2481.617810049"
Oct 30 19:57:14.937800 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:57:14.937722    4892 generic.go:356] "Generic (PLEG): container finished" podID="33098b51-3c25-4761-a108-844516b8d2fa" containerID="4bec4ad58bd687ed8ebc8b6e4853203a4ed0ceb1dbc2377d38f44440e95b626b" exitCode=0
Oct 30 19:57:14.937800 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:57:14.937750    4892 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-6caff5-openshift-default-665946589f-hn4qk" event={"ID":"33098b51-3c25-4761-a108-844516b8d2fa","Type":"ContainerDied","Data":"4bec4ad58bd687ed8ebc8b6e4853203a4ed0ceb1dbc2377d38f44440e95b626b"}
Oct 30 19:57:14.938287 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:57:14.937810    4892 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-6caff5-openshift-default-665946589f-hn4qk" event={"ID":"33098b51-3c25-4761-a108-844516b8d2fa","Type":"ContainerDied","Data":"d1219230a546ac1ffa504118cbe907692f402bceb49d23f56790fe111558ba76"}
Oct 30 19:57:14.938287 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:57:14.937828    4892 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="d1219230a546ac1ffa504118cbe907692f402bceb49d23f56790fe111558ba76"
Oct 30 19:57:14.938287 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:57:14.937841    4892 scope.go:122] "RemoveContainer" containerID="87a5399b1210c4c285564ff7854067bfad195cfa2c0f40e0bf87815fb1e09ca4"
Oct 30 19:57:14.940396 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:57:14.940180    4892 generic.go:356] "Generic (PLEG): container finished" podID="561a463b-f457-4bc5-bcdc-aa4b520df908" containerID="71c3d80f18f2a473bf806b9f0168ec3acfc61f9cd43fc4a62570438a1e115425" exitCode=0
Oct 30 19:57:14.940396 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:57:14.940262    4892 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-6873e0-6f6mc" event={"ID":"561a463b-f457-4bc5-bcdc-aa4b520df908","Type":"ContainerDied","Data":"71c3d80f18f2a473bf806b9f0168ec3acfc61f9cd43fc4a62570438a1e115425"}
Oct 30 19:57:14.954632 ip-10-0-132-82 kubenswrapper[4892]: E1030 19:57:14.954611    4892 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"87a5399b1210c4c285564ff7854067bfad195cfa2c0f40e0bf87815fb1e09ca4\": container with ID starting with 87a5399b1210c4c285564ff7854067bfad195cfa2c0f40e0bf87815fb1e09ca4 not found: ID does not exist" containerID="87a5399b1210c4c285564ff7854067bfad195cfa2c0f40e0bf87815fb1e09ca4"
Oct 30 19:57:14.958245 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:57:14.958223    4892 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-6caff5-openshift-default-665946589f-hn4qk"
Oct 30 19:57:14.991091 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:57:14.991070    4892 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/33098b51-3c25-4761-a108-844516b8d2fa-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/33098b51-3c25-4761-a108-844516b8d2fa-istiod-ca-cert\") pod \"33098b51-3c25-4761-a108-844516b8d2fa\" (UID: \"33098b51-3c25-4761-a108-844516b8d2fa\") "
Oct 30 19:57:14.991214 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:57:14.991148    4892 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/33098b51-3c25-4761-a108-844516b8d2fa-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/33098b51-3c25-4761-a108-844516b8d2fa-istio-data\") pod \"33098b51-3c25-4761-a108-844516b8d2fa\" (UID: \"33098b51-3c25-4761-a108-844516b8d2fa\") "
Oct 30 19:57:14.991282 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:57:14.991259    4892 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/33098b51-3c25-4761-a108-844516b8d2fa-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/33098b51-3c25-4761-a108-844516b8d2fa-istio-envoy\") pod \"33098b51-3c25-4761-a108-844516b8d2fa\" (UID: \"33098b51-3c25-4761-a108-844516b8d2fa\") "
Oct 30 19:57:14.991369 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:57:14.991354    4892 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/33098b51-3c25-4761-a108-844516b8d2fa-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/33098b51-3c25-4761-a108-844516b8d2fa-workload-certs\") pod \"33098b51-3c25-4761-a108-844516b8d2fa\" (UID: \"33098b51-3c25-4761-a108-844516b8d2fa\") "
Oct 30 19:57:14.991441 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:57:14.991393    4892 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/33098b51-3c25-4761-a108-844516b8d2fa-istio-token\" (UniqueName: \"kubernetes.io/projected/33098b51-3c25-4761-a108-844516b8d2fa-istio-token\") pod \"33098b51-3c25-4761-a108-844516b8d2fa\" (UID: \"33098b51-3c25-4761-a108-844516b8d2fa\") "
Oct 30 19:57:14.991506 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:57:14.991447    4892 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/33098b51-3c25-4761-a108-844516b8d2fa-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/33098b51-3c25-4761-a108-844516b8d2fa-istio-podinfo\") pod \"33098b51-3c25-4761-a108-844516b8d2fa\" (UID: \"33098b51-3c25-4761-a108-844516b8d2fa\") "
Oct 30 19:57:14.991506 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:57:14.991446    4892 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/33098b51-3c25-4761-a108-844516b8d2fa-istio-data" pod "33098b51-3c25-4761-a108-844516b8d2fa" (UID: "33098b51-3c25-4761-a108-844516b8d2fa"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:57:14.991506 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:57:14.991469    4892 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/33098b51-3c25-4761-a108-844516b8d2fa-istiod-ca-cert" pod "33098b51-3c25-4761-a108-844516b8d2fa" (UID: "33098b51-3c25-4761-a108-844516b8d2fa"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 19:57:14.991506 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:57:14.991484    4892 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/33098b51-3c25-4761-a108-844516b8d2fa-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/33098b51-3c25-4761-a108-844516b8d2fa-credential-socket\") pod \"33098b51-3c25-4761-a108-844516b8d2fa\" (UID: \"33098b51-3c25-4761-a108-844516b8d2fa\") "
Oct 30 19:57:14.991708 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:57:14.991555    4892 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/33098b51-3c25-4761-a108-844516b8d2fa-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/33098b51-3c25-4761-a108-844516b8d2fa-workload-socket\") pod \"33098b51-3c25-4761-a108-844516b8d2fa\" (UID: \"33098b51-3c25-4761-a108-844516b8d2fa\") "
Oct 30 19:57:14.991708 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:57:14.991584    4892 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/33098b51-3c25-4761-a108-844516b8d2fa-kube-api-access-vlrmm\" (UniqueName: \"kubernetes.io/projected/33098b51-3c25-4761-a108-844516b8d2fa-kube-api-access-vlrmm\") pod \"33098b51-3c25-4761-a108-844516b8d2fa\" (UID: \"33098b51-3c25-4761-a108-844516b8d2fa\") "
Oct 30 19:57:14.991708 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:57:14.991627    4892 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/33098b51-3c25-4761-a108-844516b8d2fa-workload-certs" pod "33098b51-3c25-4761-a108-844516b8d2fa" (UID: "33098b51-3c25-4761-a108-844516b8d2fa"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:57:14.991863 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:57:14.991835    4892 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/33098b51-3c25-4761-a108-844516b8d2fa-credential-socket" pod "33098b51-3c25-4761-a108-844516b8d2fa" (UID: "33098b51-3c25-4761-a108-844516b8d2fa"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:57:14.992063 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:57:14.992041    4892 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/33098b51-3c25-4761-a108-844516b8d2fa-workload-socket" pod "33098b51-3c25-4761-a108-844516b8d2fa" (UID: "33098b51-3c25-4761-a108-844516b8d2fa"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:57:14.992384 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:57:14.992360    4892 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/33098b51-3c25-4761-a108-844516b8d2fa-istiod-ca-cert\") on node \"ip-10-0-132-82.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 19:57:14.992497 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:57:14.992388    4892 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/33098b51-3c25-4761-a108-844516b8d2fa-istio-data\") on node \"ip-10-0-132-82.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 19:57:14.992497 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:57:14.992413    4892 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/33098b51-3c25-4761-a108-844516b8d2fa-workload-certs\") on node \"ip-10-0-132-82.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 19:57:14.992497 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:57:14.992427    4892 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/33098b51-3c25-4761-a108-844516b8d2fa-credential-socket\") on node \"ip-10-0-132-82.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 19:57:14.992497 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:57:14.992440    4892 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/33098b51-3c25-4761-a108-844516b8d2fa-workload-socket\") on node \"ip-10-0-132-82.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 19:57:14.993892 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:57:14.993867    4892 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/33098b51-3c25-4761-a108-844516b8d2fa-istio-podinfo" pod "33098b51-3c25-4761-a108-844516b8d2fa" (UID: "33098b51-3c25-4761-a108-844516b8d2fa"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 19:57:14.994131 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:57:14.994109    4892 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/33098b51-3c25-4761-a108-844516b8d2fa-istio-envoy" pod "33098b51-3c25-4761-a108-844516b8d2fa" (UID: "33098b51-3c25-4761-a108-844516b8d2fa"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:57:14.994196 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:57:14.994136    4892 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/33098b51-3c25-4761-a108-844516b8d2fa-istio-token" pod "33098b51-3c25-4761-a108-844516b8d2fa" (UID: "33098b51-3c25-4761-a108-844516b8d2fa"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:57:14.994368 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:57:14.994348    4892 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/33098b51-3c25-4761-a108-844516b8d2fa-kube-api-access-vlrmm" pod "33098b51-3c25-4761-a108-844516b8d2fa" (UID: "33098b51-3c25-4761-a108-844516b8d2fa"). InnerVolumeSpecName "kube-api-access-vlrmm". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:57:15.093986 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:57:15.093933    4892 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/33098b51-3c25-4761-a108-844516b8d2fa-istio-envoy\") on node \"ip-10-0-132-82.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 19:57:15.093986 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:57:15.093975    4892 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/33098b51-3c25-4761-a108-844516b8d2fa-istio-token\") on node \"ip-10-0-132-82.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 19:57:15.093986 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:57:15.093990    4892 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/33098b51-3c25-4761-a108-844516b8d2fa-istio-podinfo\") on node \"ip-10-0-132-82.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 19:57:15.094287 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:57:15.094001    4892 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vlrmm\" (UniqueName: \"kubernetes.io/projected/33098b51-3c25-4761-a108-844516b8d2fa-kube-api-access-vlrmm\") on node \"ip-10-0-132-82.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 19:57:15.950450 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:57:15.950420    4892 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-6caff5-openshift-default-665946589f-hn4qk"
Oct 30 19:57:16.114606 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:57:16.114581    4892 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-6873e0-6f6mc"
Oct 30 19:57:16.165508 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:57:16.165473    4892 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-a-6caff5-openshift-default-665946589f-hn4qk"]
Oct 30 19:57:16.207918 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:57:16.207854    4892 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/561a463b-f457-4bc5-bcdc-aa4b520df908-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/561a463b-f457-4bc5-bcdc-aa4b520df908-maas-api-openshift-ca\") pod \"561a463b-f457-4bc5-bcdc-aa4b520df908\" (UID: \"561a463b-f457-4bc5-bcdc-aa4b520df908\") "
Oct 30 19:57:16.207918 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:57:16.207896    4892 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/561a463b-f457-4bc5-bcdc-aa4b520df908-kube-api-access-4skbz\" (UniqueName: \"kubernetes.io/projected/561a463b-f457-4bc5-bcdc-aa4b520df908-kube-api-access-4skbz\") pod \"561a463b-f457-4bc5-bcdc-aa4b520df908\" (UID: \"561a463b-f457-4bc5-bcdc-aa4b520df908\") "
Oct 30 19:57:16.208118 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:57:16.208050    4892 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/561a463b-f457-4bc5-bcdc-aa4b520df908-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/561a463b-f457-4bc5-bcdc-aa4b520df908-maas-api-xks-ca\") pod \"561a463b-f457-4bc5-bcdc-aa4b520df908\" (UID: \"561a463b-f457-4bc5-bcdc-aa4b520df908\") "
Oct 30 19:57:16.208228 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:57:16.208206    4892 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/561a463b-f457-4bc5-bcdc-aa4b520df908-maas-api-openshift-ca" pod "561a463b-f457-4bc5-bcdc-aa4b520df908" (UID: "561a463b-f457-4bc5-bcdc-aa4b520df908"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 19:57:16.208586 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:57:16.208540    4892 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/561a463b-f457-4bc5-bcdc-aa4b520df908-maas-api-openshift-ca\") on node \"ip-10-0-132-82.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 19:57:16.210312 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:57:16.210277    4892 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/561a463b-f457-4bc5-bcdc-aa4b520df908-kube-api-access-4skbz" pod "561a463b-f457-4bc5-bcdc-aa4b520df908" (UID: "561a463b-f457-4bc5-bcdc-aa4b520df908"). InnerVolumeSpecName "kube-api-access-4skbz". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:57:16.210431 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:57:16.210410    4892 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/561a463b-f457-4bc5-bcdc-aa4b520df908-maas-api-xks-ca" pod "561a463b-f457-4bc5-bcdc-aa4b520df908" (UID: "561a463b-f457-4bc5-bcdc-aa4b520df908"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 19:57:16.214346 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:57:16.214322    4892 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-api-a-6caff5-openshift-default-665946589f-hn4qk"]
Oct 30 19:57:16.310164 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:57:16.310126    4892 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4skbz\" (UniqueName: \"kubernetes.io/projected/561a463b-f457-4bc5-bcdc-aa4b520df908-kube-api-access-4skbz\") on node \"ip-10-0-132-82.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 19:57:16.310164 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:57:16.310159    4892 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/561a463b-f457-4bc5-bcdc-aa4b520df908-maas-api-xks-ca\") on node \"ip-10-0-132-82.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 19:57:16.575251 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:57:16.575154    4892 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="33098b51-3c25-4761-a108-844516b8d2fa" path="/var/lib/kubelet/pods/33098b51-3c25-4761-a108-844516b8d2fa/volumes"
Oct 30 19:57:16.961366 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:57:16.961324    4892 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-6873e0-6f6mc"
Oct 30 19:57:16.961366 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:57:16.961348    4892 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-6873e0-6f6mc" event={"ID":"561a463b-f457-4bc5-bcdc-aa4b520df908","Type":"ContainerDied","Data":"c3ddad455d996c7c181dc93598c7bba4db3498ce5f2d550f4393eae8c1df757d"}
Oct 30 19:57:16.961805 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:57:16.961401    4892 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="c3ddad455d996c7c181dc93598c7bba4db3498ce5f2d550f4393eae8c1df757d"
Oct 30 19:57:22.388717 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:57:22.388687    4892 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-inf-a-132119-openshift-default-849b46588f-29f2x"]
Oct 30 19:57:22.389339 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:57:22.389325    4892 cpu_manager.go:414] "containerMap: removing container" podUID="561a463b-f457-4bc5-bcdc-aa4b520df908" containerName="revoke-keys"
Oct 30 19:57:22.389387 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:57:22.389340    4892 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="561a463b-f457-4bc5-bcdc-aa4b520df908" containerName="revoke-keys"
Oct 30 19:57:22.389387 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:57:22.389356    4892 cpu_manager.go:414] "containerMap: removing container" podUID="33098b51-3c25-4761-a108-844516b8d2fa" containerName="istio-proxy"
Oct 30 19:57:22.389387 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:57:22.389361    4892 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="33098b51-3c25-4761-a108-844516b8d2fa" containerName="istio-proxy"
Oct 30 19:57:22.389485 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:57:22.389392    4892 cpu_manager.go:414] "containerMap: removing container" podUID="33098b51-3c25-4761-a108-844516b8d2fa" containerName="istio-proxy"
Oct 30 19:57:22.389485 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:57:22.389397    4892 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="33098b51-3c25-4761-a108-844516b8d2fa" containerName="istio-proxy"
Oct 30 19:57:22.389548 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:57:22.389500    4892 memory_manager.go:365] "RemoveStaleState removing state" podUID="33098b51-3c25-4761-a108-844516b8d2fa" containerName="istio-proxy"
Oct 30 19:57:22.389548 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:57:22.389512    4892 memory_manager.go:365] "RemoveStaleState removing state" podUID="33098b51-3c25-4761-a108-844516b8d2fa" containerName="istio-proxy"
Oct 30 19:57:22.389548 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:57:22.389523    4892 memory_manager.go:365] "RemoveStaleState removing state" podUID="561a463b-f457-4bc5-bcdc-aa4b520df908" containerName="revoke-keys"
Oct 30 19:57:22.396251 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:57:22.396226    4892 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-132119-openshift-default-849b46588f-29f2x"
Oct 30 19:57:22.459346 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:57:22.457113    4892 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-inf-a-132119-openshift-default-dockercfg-76xvd\""
Oct 30 19:57:22.493818 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:57:22.493777    4892 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c76fc24b-0fe2-490e-8967-9a5eb4e9ce12-istio-envoy\") pod \"e2e-inf-a-132119-openshift-default-849b46588f-29f2x\" (UID: \"c76fc24b-0fe2-490e-8967-9a5eb4e9ce12\") " pod="openshift-ingress/e2e-inf-a-132119-openshift-default-849b46588f-29f2x"
Oct 30 19:57:22.494513 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:57:22.494485    4892 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c76fc24b-0fe2-490e-8967-9a5eb4e9ce12-istiod-ca-cert\") pod \"e2e-inf-a-132119-openshift-default-849b46588f-29f2x\" (UID: \"c76fc24b-0fe2-490e-8967-9a5eb4e9ce12\") " pod="openshift-ingress/e2e-inf-a-132119-openshift-default-849b46588f-29f2x"
Oct 30 19:57:22.494712 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:57:22.494694    4892 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c76fc24b-0fe2-490e-8967-9a5eb4e9ce12-workload-socket\") pod \"e2e-inf-a-132119-openshift-default-849b46588f-29f2x\" (UID: \"c76fc24b-0fe2-490e-8967-9a5eb4e9ce12\") " pod="openshift-ingress/e2e-inf-a-132119-openshift-default-849b46588f-29f2x"
Oct 30 19:57:22.494858 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:57:22.494842    4892 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c76fc24b-0fe2-490e-8967-9a5eb4e9ce12-workload-certs\") pod \"e2e-inf-a-132119-openshift-default-849b46588f-29f2x\" (UID: \"c76fc24b-0fe2-490e-8967-9a5eb4e9ce12\") " pod="openshift-ingress/e2e-inf-a-132119-openshift-default-849b46588f-29f2x"
Oct 30 19:57:22.495013 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:57:22.494996    4892 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c76fc24b-0fe2-490e-8967-9a5eb4e9ce12-istio-token\") pod \"e2e-inf-a-132119-openshift-default-849b46588f-29f2x\" (UID: \"c76fc24b-0fe2-490e-8967-9a5eb4e9ce12\") " pod="openshift-ingress/e2e-inf-a-132119-openshift-default-849b46588f-29f2x"
Oct 30 19:57:22.495205 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:57:22.495187    4892 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c76fc24b-0fe2-490e-8967-9a5eb4e9ce12-istio-data\") pod \"e2e-inf-a-132119-openshift-default-849b46588f-29f2x\" (UID: \"c76fc24b-0fe2-490e-8967-9a5eb4e9ce12\") " pod="openshift-ingress/e2e-inf-a-132119-openshift-default-849b46588f-29f2x"
Oct 30 19:57:22.495339 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:57:22.495322    4892 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c76fc24b-0fe2-490e-8967-9a5eb4e9ce12-credential-socket\") pod \"e2e-inf-a-132119-openshift-default-849b46588f-29f2x\" (UID: \"c76fc24b-0fe2-490e-8967-9a5eb4e9ce12\") " pod="openshift-ingress/e2e-inf-a-132119-openshift-default-849b46588f-29f2x"
Oct 30 19:57:22.495487 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:57:22.495473    4892 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8snqf\" (UniqueName: \"kubernetes.io/projected/c76fc24b-0fe2-490e-8967-9a5eb4e9ce12-kube-api-access-8snqf\") pod \"e2e-inf-a-132119-openshift-default-849b46588f-29f2x\" (UID: \"c76fc24b-0fe2-490e-8967-9a5eb4e9ce12\") " pod="openshift-ingress/e2e-inf-a-132119-openshift-default-849b46588f-29f2x"
Oct 30 19:57:22.495588 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:57:22.495574    4892 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c76fc24b-0fe2-490e-8967-9a5eb4e9ce12-istio-podinfo\") pod \"e2e-inf-a-132119-openshift-default-849b46588f-29f2x\" (UID: \"c76fc24b-0fe2-490e-8967-9a5eb4e9ce12\") " pod="openshift-ingress/e2e-inf-a-132119-openshift-default-849b46588f-29f2x"
Oct 30 19:57:22.555998 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:57:22.551848    4892 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-a-132119-openshift-default-849b46588f-29f2x"]
Oct 30 19:57:22.621865 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:57:22.620345    4892 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c76fc24b-0fe2-490e-8967-9a5eb4e9ce12-istiod-ca-cert\") pod \"e2e-inf-a-132119-openshift-default-849b46588f-29f2x\" (UID: \"c76fc24b-0fe2-490e-8967-9a5eb4e9ce12\") " pod="openshift-ingress/e2e-inf-a-132119-openshift-default-849b46588f-29f2x"
Oct 30 19:57:22.622164 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:57:22.622139    4892 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c76fc24b-0fe2-490e-8967-9a5eb4e9ce12-workload-socket\") pod \"e2e-inf-a-132119-openshift-default-849b46588f-29f2x\" (UID: \"c76fc24b-0fe2-490e-8967-9a5eb4e9ce12\") " pod="openshift-ingress/e2e-inf-a-132119-openshift-default-849b46588f-29f2x"
Oct 30 19:57:22.622335 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:57:22.622314    4892 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c76fc24b-0fe2-490e-8967-9a5eb4e9ce12-workload-certs\") pod \"e2e-inf-a-132119-openshift-default-849b46588f-29f2x\" (UID: \"c76fc24b-0fe2-490e-8967-9a5eb4e9ce12\") " pod="openshift-ingress/e2e-inf-a-132119-openshift-default-849b46588f-29f2x"
Oct 30 19:57:22.622473 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:57:22.622456    4892 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c76fc24b-0fe2-490e-8967-9a5eb4e9ce12-istio-token\") pod \"e2e-inf-a-132119-openshift-default-849b46588f-29f2x\" (UID: \"c76fc24b-0fe2-490e-8967-9a5eb4e9ce12\") " pod="openshift-ingress/e2e-inf-a-132119-openshift-default-849b46588f-29f2x"
Oct 30 19:57:22.622646 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:57:22.622628    4892 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c76fc24b-0fe2-490e-8967-9a5eb4e9ce12-istio-data\") pod \"e2e-inf-a-132119-openshift-default-849b46588f-29f2x\" (UID: \"c76fc24b-0fe2-490e-8967-9a5eb4e9ce12\") " pod="openshift-ingress/e2e-inf-a-132119-openshift-default-849b46588f-29f2x"
Oct 30 19:57:22.622778 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:57:22.622762    4892 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c76fc24b-0fe2-490e-8967-9a5eb4e9ce12-credential-socket\") pod \"e2e-inf-a-132119-openshift-default-849b46588f-29f2x\" (UID: \"c76fc24b-0fe2-490e-8967-9a5eb4e9ce12\") " pod="openshift-ingress/e2e-inf-a-132119-openshift-default-849b46588f-29f2x"
Oct 30 19:57:22.622997 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:57:22.622942    4892 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8snqf\" (UniqueName: \"kubernetes.io/projected/c76fc24b-0fe2-490e-8967-9a5eb4e9ce12-kube-api-access-8snqf\") pod \"e2e-inf-a-132119-openshift-default-849b46588f-29f2x\" (UID: \"c76fc24b-0fe2-490e-8967-9a5eb4e9ce12\") " pod="openshift-ingress/e2e-inf-a-132119-openshift-default-849b46588f-29f2x"
Oct 30 19:57:22.624194 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:57:22.623126    4892 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c76fc24b-0fe2-490e-8967-9a5eb4e9ce12-istio-podinfo\") pod \"e2e-inf-a-132119-openshift-default-849b46588f-29f2x\" (UID: \"c76fc24b-0fe2-490e-8967-9a5eb4e9ce12\") " pod="openshift-ingress/e2e-inf-a-132119-openshift-default-849b46588f-29f2x"
Oct 30 19:57:22.624417 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:57:22.624395    4892 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c76fc24b-0fe2-490e-8967-9a5eb4e9ce12-istio-envoy\") pod \"e2e-inf-a-132119-openshift-default-849b46588f-29f2x\" (UID: \"c76fc24b-0fe2-490e-8967-9a5eb4e9ce12\") " pod="openshift-ingress/e2e-inf-a-132119-openshift-default-849b46588f-29f2x"
Oct 30 19:57:22.634064 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:57:22.634035    4892 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c76fc24b-0fe2-490e-8967-9a5eb4e9ce12-istiod-ca-cert\") pod \"e2e-inf-a-132119-openshift-default-849b46588f-29f2x\" (UID: \"c76fc24b-0fe2-490e-8967-9a5eb4e9ce12\") " pod="openshift-ingress/e2e-inf-a-132119-openshift-default-849b46588f-29f2x"
Oct 30 19:57:22.635359 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:57:22.634359    4892 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c76fc24b-0fe2-490e-8967-9a5eb4e9ce12-workload-socket\") pod \"e2e-inf-a-132119-openshift-default-849b46588f-29f2x\" (UID: \"c76fc24b-0fe2-490e-8967-9a5eb4e9ce12\") " pod="openshift-ingress/e2e-inf-a-132119-openshift-default-849b46588f-29f2x"
Oct 30 19:57:22.635508 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:57:22.634594    4892 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c76fc24b-0fe2-490e-8967-9a5eb4e9ce12-credential-socket\") pod \"e2e-inf-a-132119-openshift-default-849b46588f-29f2x\" (UID: \"c76fc24b-0fe2-490e-8967-9a5eb4e9ce12\") " pod="openshift-ingress/e2e-inf-a-132119-openshift-default-849b46588f-29f2x"
Oct 30 19:57:22.635614 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:57:22.634630    4892 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c76fc24b-0fe2-490e-8967-9a5eb4e9ce12-istio-data\") pod \"e2e-inf-a-132119-openshift-default-849b46588f-29f2x\" (UID: \"c76fc24b-0fe2-490e-8967-9a5eb4e9ce12\") " pod="openshift-ingress/e2e-inf-a-132119-openshift-default-849b46588f-29f2x"
Oct 30 19:57:22.635711 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:57:22.634868    4892 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c76fc24b-0fe2-490e-8967-9a5eb4e9ce12-workload-certs\") pod \"e2e-inf-a-132119-openshift-default-849b46588f-29f2x\" (UID: \"c76fc24b-0fe2-490e-8967-9a5eb4e9ce12\") " pod="openshift-ingress/e2e-inf-a-132119-openshift-default-849b46588f-29f2x"
Oct 30 19:57:22.665920 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:57:22.661469    4892 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c76fc24b-0fe2-490e-8967-9a5eb4e9ce12-istio-envoy\") pod \"e2e-inf-a-132119-openshift-default-849b46588f-29f2x\" (UID: \"c76fc24b-0fe2-490e-8967-9a5eb4e9ce12\") " pod="openshift-ingress/e2e-inf-a-132119-openshift-default-849b46588f-29f2x"
Oct 30 19:57:22.697932 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:57:22.693652    4892 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c76fc24b-0fe2-490e-8967-9a5eb4e9ce12-istio-podinfo\") pod \"e2e-inf-a-132119-openshift-default-849b46588f-29f2x\" (UID: \"c76fc24b-0fe2-490e-8967-9a5eb4e9ce12\") " pod="openshift-ingress/e2e-inf-a-132119-openshift-default-849b46588f-29f2x"
Oct 30 19:57:22.708113 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:57:22.708090    4892 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c76fc24b-0fe2-490e-8967-9a5eb4e9ce12-istio-token\") pod \"e2e-inf-a-132119-openshift-default-849b46588f-29f2x\" (UID: \"c76fc24b-0fe2-490e-8967-9a5eb4e9ce12\") " pod="openshift-ingress/e2e-inf-a-132119-openshift-default-849b46588f-29f2x"
Oct 30 19:57:22.711616 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:57:22.711128    4892 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8snqf\" (UniqueName: \"kubernetes.io/projected/c76fc24b-0fe2-490e-8967-9a5eb4e9ce12-kube-api-access-8snqf\") pod \"e2e-inf-a-132119-openshift-default-849b46588f-29f2x\" (UID: \"c76fc24b-0fe2-490e-8967-9a5eb4e9ce12\") " pod="openshift-ingress/e2e-inf-a-132119-openshift-default-849b46588f-29f2x"
Oct 30 19:57:22.770208 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:57:22.770179    4892 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-132119-openshift-default-849b46588f-29f2x"
Oct 30 19:57:23.205894 ip-10-0-132-82 kubenswrapper[4892]: W1030 19:57:23.205864    4892 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc76fc24b_0fe2_490e_8967_9a5eb4e9ce12.slice/crio-218e97154c0674a68bc13085ac7061bbe47bb5693484891d67a20ed4762b356e WatchSource:0}: Error finding container 218e97154c0674a68bc13085ac7061bbe47bb5693484891d67a20ed4762b356e: Status 404 returned error can't find the container with id 218e97154c0674a68bc13085ac7061bbe47bb5693484891d67a20ed4762b356e
Oct 30 19:57:23.206498 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:57:23.206475    4892 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-a-132119-openshift-default-849b46588f-29f2x"]
Oct 30 19:57:23.208591 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:57:23.208558    4892 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892000Ki","pods":"250"}
Oct 30 19:57:23.208670 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:57:23.208623    4892 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892000Ki","pods":"250"}
Oct 30 19:57:23.208670 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:57:23.208652    4892 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892000Ki","pods":"250"}
Oct 30 19:57:23.284025 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:57:23.283997    4892 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-6873e0-5ddd7bfd58-cbrnz"]
Oct 30 19:57:23.364886 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:57:23.364848    4892 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-6873e0-78bf88f8b7-hhml5"]
Oct 30 19:57:24.020894 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:57:24.020855    4892 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-132119-openshift-default-849b46588f-29f2x" event={"ID":"c76fc24b-0fe2-490e-8967-9a5eb4e9ce12","Type":"ContainerStarted","Data":"8104d1feef8b7a9b4f6b6cf7646ea8bfc8897bc7178a480aa99cb6ce22992041"}
Oct 30 19:57:24.021342 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:57:24.020922    4892 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-132119-openshift-default-849b46588f-29f2x" event={"ID":"c76fc24b-0fe2-490e-8967-9a5eb4e9ce12","Type":"ContainerStarted","Data":"218e97154c0674a68bc13085ac7061bbe47bb5693484891d67a20ed4762b356e"}
Oct 30 19:57:24.140135 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:57:24.140067    4892 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-inf-a-132119-openshift-default-849b46588f-29f2x" podStartSLOduration=2.140046525 podStartE2EDuration="2.140046525s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:57:22 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:57:24.138346839 +0000 UTC m=+2491.708812438" watchObservedRunningTime="2026-10-30 19:57:24.140046525 +0000 UTC m=+2491.710512125"
Oct 30 19:57:24.771379 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:57:24.771344    4892 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-a-132119-openshift-default-849b46588f-29f2x"
Oct 30 19:57:24.865362 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:57:24.865333    4892 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-a-132119-openshift-default-849b46588f-29f2x"
Oct 30 19:57:24.874446 ip-10-0-132-82 kubenswrapper[4892]: E1030 19:57:24.874409    4892 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-jrm87" containerName="extractor"
Oct 30 19:57:25.036632 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:57:25.036567    4892 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-a-132119-openshift-default-849b46588f-29f2x"
Oct 30 19:57:25.047998 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:57:25.038894    4892 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-a-132119-openshift-default-849b46588f-29f2x"
Oct 30 19:57:25.235379 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:57:25.235350    4892 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-fsfvk"]
Oct 30 19:57:28.284598 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:57:28.284555    4892 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-inf-a-6873e0-5ddd7bfd58-cbrnz" podUID="cc52cc8e-3920-45b4-9807-b78a778537a0" containerName="payload-processing" containerID="cri-o://c1c3c5c30831784587be0fb455ed4839366c861e63ee2ae57e5c70f301313221" gracePeriod=25
Oct 30 19:57:28.365589 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:57:28.365557    4892 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-6873e0-78bf88f8b7-hhml5" podUID="9c8d8a28-2b91-4b8f-a582-f8059580f6e0" containerName="payload-pre-processing" containerID="cri-o://f1507e6e223fe2e8daf7e8ca514ca76fcee879dae9d0d3cc2226945c440304b8" gracePeriod=25
Oct 30 19:57:28.665105 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:57:28.665083    4892 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-a-6873e0-5ddd7bfd58-cbrnz"
Oct 30 19:57:28.685022 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:57:28.684999    4892 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-6873e0-78bf88f8b7-hhml5"
Oct 30 19:57:28.733669 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:57:28.733640    4892 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/cc52cc8e-3920-45b4-9807-b78a778537a0-kube-api-access-phh7k\" (UniqueName: \"kubernetes.io/projected/cc52cc8e-3920-45b4-9807-b78a778537a0-kube-api-access-phh7k\") pod \"cc52cc8e-3920-45b4-9807-b78a778537a0\" (UID: \"cc52cc8e-3920-45b4-9807-b78a778537a0\") "
Oct 30 19:57:28.733840 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:57:28.733727    4892 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/cc52cc8e-3920-45b4-9807-b78a778537a0-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/cc52cc8e-3920-45b4-9807-b78a778537a0-plugins-config-volume\") pod \"cc52cc8e-3920-45b4-9807-b78a778537a0\" (UID: \"cc52cc8e-3920-45b4-9807-b78a778537a0\") "
Oct 30 19:57:28.734122 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:57:28.734102    4892 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/cc52cc8e-3920-45b4-9807-b78a778537a0-plugins-config-volume" pod "cc52cc8e-3920-45b4-9807-b78a778537a0" (UID: "cc52cc8e-3920-45b4-9807-b78a778537a0"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 19:57:28.735823 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:57:28.735805    4892 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/cc52cc8e-3920-45b4-9807-b78a778537a0-kube-api-access-phh7k" pod "cc52cc8e-3920-45b4-9807-b78a778537a0" (UID: "cc52cc8e-3920-45b4-9807-b78a778537a0"). InnerVolumeSpecName "kube-api-access-phh7k". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:57:28.834796 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:57:28.834706    4892 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9c8d8a28-2b91-4b8f-a582-f8059580f6e0-kube-api-access-9vfbh\" (UniqueName: \"kubernetes.io/projected/9c8d8a28-2b91-4b8f-a582-f8059580f6e0-kube-api-access-9vfbh\") pod \"9c8d8a28-2b91-4b8f-a582-f8059580f6e0\" (UID: \"9c8d8a28-2b91-4b8f-a582-f8059580f6e0\") "
Oct 30 19:57:28.834796 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:57:28.834790    4892 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/9c8d8a28-2b91-4b8f-a582-f8059580f6e0-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9c8d8a28-2b91-4b8f-a582-f8059580f6e0-plugins-config-volume\") pod \"9c8d8a28-2b91-4b8f-a582-f8059580f6e0\" (UID: \"9c8d8a28-2b91-4b8f-a582-f8059580f6e0\") "
Oct 30 19:57:28.835248 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:57:28.835221    4892 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/cc52cc8e-3920-45b4-9807-b78a778537a0-plugins-config-volume\") on node \"ip-10-0-132-82.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 19:57:28.835336 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:57:28.835254    4892 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-phh7k\" (UniqueName: \"kubernetes.io/projected/cc52cc8e-3920-45b4-9807-b78a778537a0-kube-api-access-phh7k\") on node \"ip-10-0-132-82.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 19:57:28.835336 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:57:28.835286    4892 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9c8d8a28-2b91-4b8f-a582-f8059580f6e0-plugins-config-volume" pod "9c8d8a28-2b91-4b8f-a582-f8059580f6e0" (UID: "9c8d8a28-2b91-4b8f-a582-f8059580f6e0"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 19:57:28.837119 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:57:28.837087    4892 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9c8d8a28-2b91-4b8f-a582-f8059580f6e0-kube-api-access-9vfbh" pod "9c8d8a28-2b91-4b8f-a582-f8059580f6e0" (UID: "9c8d8a28-2b91-4b8f-a582-f8059580f6e0"). InnerVolumeSpecName "kube-api-access-9vfbh". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:57:28.937135 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:57:28.937104    4892 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9vfbh\" (UniqueName: \"kubernetes.io/projected/9c8d8a28-2b91-4b8f-a582-f8059580f6e0-kube-api-access-9vfbh\") on node \"ip-10-0-132-82.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 19:57:28.937135 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:57:28.937135    4892 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9c8d8a28-2b91-4b8f-a582-f8059580f6e0-plugins-config-volume\") on node \"ip-10-0-132-82.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 19:57:29.065184 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:57:29.065154    4892 generic.go:356] "Generic (PLEG): container finished" podID="9c8d8a28-2b91-4b8f-a582-f8059580f6e0" containerID="f1507e6e223fe2e8daf7e8ca514ca76fcee879dae9d0d3cc2226945c440304b8" exitCode=0
Oct 30 19:57:29.065346 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:57:29.065228    4892 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-6873e0-78bf88f8b7-hhml5"
Oct 30 19:57:29.065346 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:57:29.065250    4892 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-6873e0-78bf88f8b7-hhml5" event={"ID":"9c8d8a28-2b91-4b8f-a582-f8059580f6e0","Type":"ContainerDied","Data":"f1507e6e223fe2e8daf7e8ca514ca76fcee879dae9d0d3cc2226945c440304b8"}
Oct 30 19:57:29.065346 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:57:29.065328    4892 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-6873e0-78bf88f8b7-hhml5" event={"ID":"9c8d8a28-2b91-4b8f-a582-f8059580f6e0","Type":"ContainerDied","Data":"1534beaecb6928e04c6696f925f6a169f2ef9d61644b623312879897d0821271"}
Oct 30 19:57:29.065483 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:57:29.065354    4892 scope.go:122] "RemoveContainer" containerID="f1507e6e223fe2e8daf7e8ca514ca76fcee879dae9d0d3cc2226945c440304b8"
Oct 30 19:57:29.067140 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:57:29.067097    4892 generic.go:356] "Generic (PLEG): container finished" podID="cc52cc8e-3920-45b4-9807-b78a778537a0" containerID="c1c3c5c30831784587be0fb455ed4839366c861e63ee2ae57e5c70f301313221" exitCode=0
Oct 30 19:57:29.067277 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:57:29.067144    4892 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-a-6873e0-5ddd7bfd58-cbrnz" event={"ID":"cc52cc8e-3920-45b4-9807-b78a778537a0","Type":"ContainerDied","Data":"c1c3c5c30831784587be0fb455ed4839366c861e63ee2ae57e5c70f301313221"}
Oct 30 19:57:29.067277 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:57:29.067157    4892 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-a-6873e0-5ddd7bfd58-cbrnz"
Oct 30 19:57:29.067277 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:57:29.067183    4892 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-a-6873e0-5ddd7bfd58-cbrnz" event={"ID":"cc52cc8e-3920-45b4-9807-b78a778537a0","Type":"ContainerDied","Data":"aed3dd0f99b28c0ccec37b5df3fd213322f75009e0caf0edbeb2963deaac7479"}
Oct 30 19:57:29.078214 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:57:29.078194    4892 scope.go:122] "RemoveContainer" containerID="f1507e6e223fe2e8daf7e8ca514ca76fcee879dae9d0d3cc2226945c440304b8"
Oct 30 19:57:29.078470 ip-10-0-132-82 kubenswrapper[4892]: E1030 19:57:29.078454    4892 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f1507e6e223fe2e8daf7e8ca514ca76fcee879dae9d0d3cc2226945c440304b8\": container with ID starting with f1507e6e223fe2e8daf7e8ca514ca76fcee879dae9d0d3cc2226945c440304b8 not found: ID does not exist" containerID="f1507e6e223fe2e8daf7e8ca514ca76fcee879dae9d0d3cc2226945c440304b8"
Oct 30 19:57:29.078526 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:57:29.078477    4892 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f1507e6e223fe2e8daf7e8ca514ca76fcee879dae9d0d3cc2226945c440304b8"} err="failed to get container status \"f1507e6e223fe2e8daf7e8ca514ca76fcee879dae9d0d3cc2226945c440304b8\": rpc error: code = NotFound desc = could not find container \"f1507e6e223fe2e8daf7e8ca514ca76fcee879dae9d0d3cc2226945c440304b8\": container with ID starting with f1507e6e223fe2e8daf7e8ca514ca76fcee879dae9d0d3cc2226945c440304b8 not found: ID does not exist"
Oct 30 19:57:29.078526 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:57:29.078495    4892 scope.go:122] "RemoveContainer" containerID="c1c3c5c30831784587be0fb455ed4839366c861e63ee2ae57e5c70f301313221"
Oct 30 19:57:29.088639 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:57:29.088623    4892 scope.go:122] "RemoveContainer" containerID="c1c3c5c30831784587be0fb455ed4839366c861e63ee2ae57e5c70f301313221"
Oct 30 19:57:29.088887 ip-10-0-132-82 kubenswrapper[4892]: E1030 19:57:29.088870    4892 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c1c3c5c30831784587be0fb455ed4839366c861e63ee2ae57e5c70f301313221\": container with ID starting with c1c3c5c30831784587be0fb455ed4839366c861e63ee2ae57e5c70f301313221 not found: ID does not exist" containerID="c1c3c5c30831784587be0fb455ed4839366c861e63ee2ae57e5c70f301313221"
Oct 30 19:57:29.088937 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:57:29.088892    4892 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c1c3c5c30831784587be0fb455ed4839366c861e63ee2ae57e5c70f301313221"} err="failed to get container status \"c1c3c5c30831784587be0fb455ed4839366c861e63ee2ae57e5c70f301313221\": rpc error: code = NotFound desc = could not find container \"c1c3c5c30831784587be0fb455ed4839366c861e63ee2ae57e5c70f301313221\": container with ID starting with c1c3c5c30831784587be0fb455ed4839366c861e63ee2ae57e5c70f301313221 not found: ID does not exist"
Oct 30 19:57:29.229663 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:57:29.229635    4892 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-6873e0-5ddd7bfd58-cbrnz"]
Oct 30 19:57:29.233231 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:57:29.233207    4892 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-6873e0-5ddd7bfd58-cbrnz"]
Oct 30 19:57:29.341209 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:57:29.341154    4892 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-6873e0-78bf88f8b7-hhml5"]
Oct 30 19:57:29.344589 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:57:29.344569    4892 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-6873e0-78bf88f8b7-hhml5"]
Oct 30 19:57:30.569848 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:57:30.569816    4892 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="9c8d8a28-2b91-4b8f-a582-f8059580f6e0" path="/var/lib/kubelet/pods/9c8d8a28-2b91-4b8f-a582-f8059580f6e0/volumes"
Oct 30 19:57:30.570255 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:57:30.570242    4892 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="cc52cc8e-3920-45b4-9807-b78a778537a0" path="/var/lib/kubelet/pods/cc52cc8e-3920-45b4-9807-b78a778537a0/volumes"
Oct 30 19:57:31.100344 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:57:31.100310    4892 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-132119-6b76cf689f-tk59n"]
Oct 30 19:57:31.101312 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:57:31.101284    4892 cpu_manager.go:414] "containerMap: removing container" podUID="cc52cc8e-3920-45b4-9807-b78a778537a0" containerName="payload-processing"
Oct 30 19:57:31.101312 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:57:31.101311    4892 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="cc52cc8e-3920-45b4-9807-b78a778537a0" containerName="payload-processing"
Oct 30 19:57:31.101509 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:57:31.101382    4892 cpu_manager.go:414] "containerMap: removing container" podUID="9c8d8a28-2b91-4b8f-a582-f8059580f6e0" containerName="payload-pre-processing"
Oct 30 19:57:31.101509 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:57:31.101390    4892 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9c8d8a28-2b91-4b8f-a582-f8059580f6e0" containerName="payload-pre-processing"
Oct 30 19:57:31.101578 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:57:31.101538    4892 memory_manager.go:365] "RemoveStaleState removing state" podUID="9c8d8a28-2b91-4b8f-a582-f8059580f6e0" containerName="payload-pre-processing"
Oct 30 19:57:31.101578 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:57:31.101555    4892 memory_manager.go:365] "RemoveStaleState removing state" podUID="cc52cc8e-3920-45b4-9807-b78a778537a0" containerName="payload-processing"
Oct 30 19:57:31.107339 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:57:31.107318    4892 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-132119-6b76cf689f-tk59n"
Oct 30 19:57:31.214990 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:57:31.204243    4892 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-inf-a-132119\""
Oct 30 19:57:31.214990 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:57:31.204892    4892 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-inf-a-132119-dockercfg-qrmlc\""
Oct 30 19:57:31.214990 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:57:31.206174    4892 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-132119-6b76cf689f-tk59n"]
Oct 30 19:57:31.296992 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:57:31.293012    4892 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-s6kk6\" (UniqueName: \"kubernetes.io/projected/60856d7a-fd76-42e9-ac9f-c28861d49701-kube-api-access-s6kk6\") pod \"payload-pre-processing-e2e-inf-a-132119-6b76cf689f-tk59n\" (UID: \"60856d7a-fd76-42e9-ac9f-c28861d49701\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-132119-6b76cf689f-tk59n"
Oct 30 19:57:31.296992 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:57:31.293153    4892 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/60856d7a-fd76-42e9-ac9f-c28861d49701-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-a-132119-6b76cf689f-tk59n\" (UID: \"60856d7a-fd76-42e9-ac9f-c28861d49701\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-132119-6b76cf689f-tk59n"
Oct 30 19:57:31.296992 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:57:31.296722    4892 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-132119-79464cf78b-w2p5b"]
Oct 30 19:57:31.304856 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:57:31.304825    4892 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-a-132119-79464cf78b-w2p5b"
Oct 30 19:57:31.332311 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:57:31.331982    4892 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-132119-79464cf78b-w2p5b"]
Oct 30 19:57:31.394095 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:57:31.394060    4892 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-s6kk6\" (UniqueName: \"kubernetes.io/projected/60856d7a-fd76-42e9-ac9f-c28861d49701-kube-api-access-s6kk6\") pod \"payload-pre-processing-e2e-inf-a-132119-6b76cf689f-tk59n\" (UID: \"60856d7a-fd76-42e9-ac9f-c28861d49701\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-132119-6b76cf689f-tk59n"
Oct 30 19:57:31.394278 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:57:31.394183    4892 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/60856d7a-fd76-42e9-ac9f-c28861d49701-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-a-132119-6b76cf689f-tk59n\" (UID: \"60856d7a-fd76-42e9-ac9f-c28861d49701\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-132119-6b76cf689f-tk59n"
Oct 30 19:57:31.394857 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:57:31.394836    4892 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/60856d7a-fd76-42e9-ac9f-c28861d49701-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-a-132119-6b76cf689f-tk59n\" (UID: \"60856d7a-fd76-42e9-ac9f-c28861d49701\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-132119-6b76cf689f-tk59n"
Oct 30 19:57:31.451207 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:57:31.451175    4892 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-s6kk6\" (UniqueName: \"kubernetes.io/projected/60856d7a-fd76-42e9-ac9f-c28861d49701-kube-api-access-s6kk6\") pod \"payload-pre-processing-e2e-inf-a-132119-6b76cf689f-tk59n\" (UID: \"60856d7a-fd76-42e9-ac9f-c28861d49701\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-132119-6b76cf689f-tk59n"
Oct 30 19:57:31.495588 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:57:31.495556    4892 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-v9ssq\" (UniqueName: \"kubernetes.io/projected/c63fdd71-1e7d-4c3d-8946-a20291b4f6b8-kube-api-access-v9ssq\") pod \"payload-processing-e2e-inf-a-132119-79464cf78b-w2p5b\" (UID: \"c63fdd71-1e7d-4c3d-8946-a20291b4f6b8\") " pod="openshift-ingress/payload-processing-e2e-inf-a-132119-79464cf78b-w2p5b"
Oct 30 19:57:31.495727 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:57:31.495684    4892 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c63fdd71-1e7d-4c3d-8946-a20291b4f6b8-plugins-config-volume\") pod \"payload-processing-e2e-inf-a-132119-79464cf78b-w2p5b\" (UID: \"c63fdd71-1e7d-4c3d-8946-a20291b4f6b8\") " pod="openshift-ingress/payload-processing-e2e-inf-a-132119-79464cf78b-w2p5b"
Oct 30 19:57:31.596564 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:57:31.596534    4892 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-v9ssq\" (UniqueName: \"kubernetes.io/projected/c63fdd71-1e7d-4c3d-8946-a20291b4f6b8-kube-api-access-v9ssq\") pod \"payload-processing-e2e-inf-a-132119-79464cf78b-w2p5b\" (UID: \"c63fdd71-1e7d-4c3d-8946-a20291b4f6b8\") " pod="openshift-ingress/payload-processing-e2e-inf-a-132119-79464cf78b-w2p5b"
Oct 30 19:57:31.597045 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:57:31.596617    4892 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c63fdd71-1e7d-4c3d-8946-a20291b4f6b8-plugins-config-volume\") pod \"payload-processing-e2e-inf-a-132119-79464cf78b-w2p5b\" (UID: \"c63fdd71-1e7d-4c3d-8946-a20291b4f6b8\") " pod="openshift-ingress/payload-processing-e2e-inf-a-132119-79464cf78b-w2p5b"
Oct 30 19:57:31.597361 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:57:31.597340    4892 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c63fdd71-1e7d-4c3d-8946-a20291b4f6b8-plugins-config-volume\") pod \"payload-processing-e2e-inf-a-132119-79464cf78b-w2p5b\" (UID: \"c63fdd71-1e7d-4c3d-8946-a20291b4f6b8\") " pod="openshift-ingress/payload-processing-e2e-inf-a-132119-79464cf78b-w2p5b"
Oct 30 19:57:31.652989 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:57:31.652924    4892 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-v9ssq\" (UniqueName: \"kubernetes.io/projected/c63fdd71-1e7d-4c3d-8946-a20291b4f6b8-kube-api-access-v9ssq\") pod \"payload-processing-e2e-inf-a-132119-79464cf78b-w2p5b\" (UID: \"c63fdd71-1e7d-4c3d-8946-a20291b4f6b8\") " pod="openshift-ingress/payload-processing-e2e-inf-a-132119-79464cf78b-w2p5b"
Oct 30 19:57:31.725423 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:57:31.725389    4892 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-132119-6b76cf689f-tk59n"
Oct 30 19:57:31.935340 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:57:31.935254    4892 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-a-132119-79464cf78b-w2p5b"
Oct 30 19:57:32.107363 ip-10-0-132-82 kubenswrapper[4892]: W1030 19:57:32.107327    4892 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod60856d7a_fd76_42e9_ac9f_c28861d49701.slice/crio-1975dc48a88d0e50c89aab6af7c856ef96b1c0b055beb84348d5ddcdd1f77146 WatchSource:0}: Error finding container 1975dc48a88d0e50c89aab6af7c856ef96b1c0b055beb84348d5ddcdd1f77146: Status 404 returned error can't find the container with id 1975dc48a88d0e50c89aab6af7c856ef96b1c0b055beb84348d5ddcdd1f77146
Oct 30 19:57:32.107595 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:57:32.107571    4892 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-132119-6b76cf689f-tk59n"]
Oct 30 19:57:32.320009 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:57:32.319984    4892 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-132119-79464cf78b-w2p5b"]
Oct 30 19:57:32.320462 ip-10-0-132-82 kubenswrapper[4892]: W1030 19:57:32.320434    4892 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc63fdd71_1e7d_4c3d_8946_a20291b4f6b8.slice/crio-392c8e9ebba7fae354ffcd589b4a87ddc9c3b79f4b9ea946a2ded0402c8df98a WatchSource:0}: Error finding container 392c8e9ebba7fae354ffcd589b4a87ddc9c3b79f4b9ea946a2ded0402c8df98a: Status 404 returned error can't find the container with id 392c8e9ebba7fae354ffcd589b4a87ddc9c3b79f4b9ea946a2ded0402c8df98a
Oct 30 19:57:33.107178 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:57:33.107136    4892 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-132119-6b76cf689f-tk59n" event={"ID":"60856d7a-fd76-42e9-ac9f-c28861d49701","Type":"ContainerStarted","Data":"5814939db6bc91961288ac2cef5bf2d36b4045fd5b06af2fccc45d3bb6032ac6"}
Oct 30 19:57:33.107635 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:57:33.107205    4892 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-132119-6b76cf689f-tk59n" event={"ID":"60856d7a-fd76-42e9-ac9f-c28861d49701","Type":"ContainerStarted","Data":"1975dc48a88d0e50c89aab6af7c856ef96b1c0b055beb84348d5ddcdd1f77146"}
Oct 30 19:57:33.107635 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:57:33.107237    4892 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-132119-6b76cf689f-tk59n"
Oct 30 19:57:33.109055 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:57:33.109031    4892 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-a-132119-79464cf78b-w2p5b" event={"ID":"c63fdd71-1e7d-4c3d-8946-a20291b4f6b8","Type":"ContainerStarted","Data":"ddc641c3b05923c27aa337967673403bad9627b19af544efb0d32025429bbb67"}
Oct 30 19:57:33.109170 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:57:33.109071    4892 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-a-132119-79464cf78b-w2p5b" event={"ID":"c63fdd71-1e7d-4c3d-8946-a20291b4f6b8","Type":"ContainerStarted","Data":"392c8e9ebba7fae354ffcd589b4a87ddc9c3b79f4b9ea946a2ded0402c8df98a"}
Oct 30 19:57:33.109170 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:57:33.109106    4892 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-inf-a-132119-79464cf78b-w2p5b"
Oct 30 19:57:33.222313 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:57:33.222266    4892 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-132119-6b76cf689f-tk59n" podStartSLOduration=2.222253165 podStartE2EDuration="2.222253165s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:57:31 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:57:33.220399568 +0000 UTC m=+2500.790865181" watchObservedRunningTime="2026-10-30 19:57:33.222253165 +0000 UTC m=+2500.792718763"
Oct 30 19:57:33.359058 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:57:33.358904    4892 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-inf-a-132119-79464cf78b-w2p5b" podStartSLOduration=2.358883089 podStartE2EDuration="2.358883089s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:57:31 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:57:33.356654038 +0000 UTC m=+2500.927119662" watchObservedRunningTime="2026-10-30 19:57:33.358883089 +0000 UTC m=+2500.929348689"
Oct 30 19:57:44.119086 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:57:44.119058    4892 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-inf-a-132119-79464cf78b-w2p5b"
Oct 30 19:57:44.119603 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:57:44.119125    4892 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-132119-6b76cf689f-tk59n"
Oct 30 19:57:54.846667 ip-10-0-132-82 kubenswrapper[4892]: E1030 19:57:54.846625    4892 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-jrm87" containerName="extractor"
Oct 30 19:57:58.757190 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:57:58.757150    4892 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-898968-kbwn2"]
Oct 30 19:57:58.803829 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:57:58.803798    4892 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-898968-kbwn2"]
Oct 30 19:58:00.571623 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:58:00.571587    4892 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="adb0c593-1248-4d0e-bae9-2f6ac76233a9" path="/var/lib/kubelet/pods/adb0c593-1248-4d0e-bae9-2f6ac76233a9/volumes"
Oct 30 19:58:05.087021 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:58:05.086986    4892 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-daa1ac-swtts"]
Oct 30 19:58:05.136075 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:58:05.136048    4892 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-daa1ac-swtts"]
Oct 30 19:58:05.752118 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:58:05.752079    4892 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-a-898968-openshift-default-b55dfc9c9-gmck5"]
Oct 30 19:58:05.752363 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:58:05.752337    4892 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-inf-a-898968-openshift-default-b55dfc9c9-gmck5" podUID="95cba503-314a-4b92-a90b-c23b551ea329" containerName="istio-proxy" containerID="cri-o://c406d824e6f76efb5e69812a9ad3ee202c4adb8ee77210c44ab23930d4f19069" gracePeriod=30
Oct 30 19:58:06.572523 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:58:06.572486    4892 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="39e00956-193c-4dc5-9ccc-4ba8ba94584c" path="/var/lib/kubelet/pods/39e00956-193c-4dc5-9ccc-4ba8ba94584c/volumes"
Oct 30 19:58:07.982017 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:58:07.981949    4892 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w1-d52df0-9q2ht"]
Oct 30 19:58:07.987819 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:58:07.987797    4892 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w1-d52df0-9q2ht"
Oct 30 19:58:08.045025 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:58:08.044991    4892 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w1-d52df0-9q2ht"]
Oct 30 19:58:08.088660 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:58:08.088624    4892 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 19:58:08.088660 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:58:08.088652    4892 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 30 19:58:08.088926 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:58:08.088674    4892 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 30 19:58:08.089408 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:58:08.089387    4892 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-nlxp4\""
Oct 30 19:58:08.096035 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:58:08.096006    4892 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8b7c4bca-c523-4094-a2d7-3bfee37be2e3-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-w1-d52df0-9q2ht\" (UID: \"8b7c4bca-c523-4094-a2d7-3bfee37be2e3\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w1-d52df0-9q2ht"
Oct 30 19:58:08.096163 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:58:08.096079    4892 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mk2gc\" (UniqueName: \"kubernetes.io/projected/8b7c4bca-c523-4094-a2d7-3bfee37be2e3-kube-api-access-mk2gc\") pod \"maas-api-revoke-keys-e2e-worker-w1-d52df0-9q2ht\" (UID: \"8b7c4bca-c523-4094-a2d7-3bfee37be2e3\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w1-d52df0-9q2ht"
Oct 30 19:58:08.096288 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:58:08.096218    4892 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8b7c4bca-c523-4094-a2d7-3bfee37be2e3-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-w1-d52df0-9q2ht\" (UID: \"8b7c4bca-c523-4094-a2d7-3bfee37be2e3\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w1-d52df0-9q2ht"
Oct 30 19:58:08.197041 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:58:08.196997    4892 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8b7c4bca-c523-4094-a2d7-3bfee37be2e3-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-w1-d52df0-9q2ht\" (UID: \"8b7c4bca-c523-4094-a2d7-3bfee37be2e3\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w1-d52df0-9q2ht"
Oct 30 19:58:08.197233 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:58:08.197084    4892 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mk2gc\" (UniqueName: \"kubernetes.io/projected/8b7c4bca-c523-4094-a2d7-3bfee37be2e3-kube-api-access-mk2gc\") pod \"maas-api-revoke-keys-e2e-worker-w1-d52df0-9q2ht\" (UID: \"8b7c4bca-c523-4094-a2d7-3bfee37be2e3\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w1-d52df0-9q2ht"
Oct 30 19:58:08.197233 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:58:08.197144    4892 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8b7c4bca-c523-4094-a2d7-3bfee37be2e3-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-w1-d52df0-9q2ht\" (UID: \"8b7c4bca-c523-4094-a2d7-3bfee37be2e3\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w1-d52df0-9q2ht"
Oct 30 19:58:08.197780 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:58:08.197760    4892 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8b7c4bca-c523-4094-a2d7-3bfee37be2e3-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-w1-d52df0-9q2ht\" (UID: \"8b7c4bca-c523-4094-a2d7-3bfee37be2e3\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w1-d52df0-9q2ht"
Oct 30 19:58:08.199485 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:58:08.199459    4892 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8b7c4bca-c523-4094-a2d7-3bfee37be2e3-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-w1-d52df0-9q2ht\" (UID: \"8b7c4bca-c523-4094-a2d7-3bfee37be2e3\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w1-d52df0-9q2ht"
Oct 30 19:58:08.253624 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:58:08.253562    4892 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mk2gc\" (UniqueName: \"kubernetes.io/projected/8b7c4bca-c523-4094-a2d7-3bfee37be2e3-kube-api-access-mk2gc\") pod \"maas-api-revoke-keys-e2e-worker-w1-d52df0-9q2ht\" (UID: \"8b7c4bca-c523-4094-a2d7-3bfee37be2e3\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w1-d52df0-9q2ht"
Oct 30 19:58:08.301903 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:58:08.301877    4892 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w1-d52df0-9q2ht"
Oct 30 19:58:08.686051 ip-10-0-132-82 kubenswrapper[4892]: W1030 19:58:08.686018    4892 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8b7c4bca_c523_4094_a2d7_3bfee37be2e3.slice/crio-7d20b8a0f89588744ee8acb488a1120d862524baa956008405dceb7b33ff2ff3 WatchSource:0}: Error finding container 7d20b8a0f89588744ee8acb488a1120d862524baa956008405dceb7b33ff2ff3: Status 404 returned error can't find the container with id 7d20b8a0f89588744ee8acb488a1120d862524baa956008405dceb7b33ff2ff3
Oct 30 19:58:08.686190 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:58:08.686173    4892 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w1-d52df0-9q2ht"]
Oct 30 19:58:09.451607 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:58:09.451572    4892 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w1-d52df0-9q2ht" event={"ID":"8b7c4bca-c523-4094-a2d7-3bfee37be2e3","Type":"ContainerStarted","Data":"55a5d749961896f71efa97ff668654dd9fc0cbd86d1d05b392cdeaa11b803014"}
Oct 30 19:58:09.452057 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:58:09.451621    4892 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w1-d52df0-9q2ht" event={"ID":"8b7c4bca-c523-4094-a2d7-3bfee37be2e3","Type":"ContainerStarted","Data":"7d20b8a0f89588744ee8acb488a1120d862524baa956008405dceb7b33ff2ff3"}
Oct 30 19:58:09.575626 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:58:09.575578    4892 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w1-d52df0-9q2ht" podStartSLOduration=2.575562582 podStartE2EDuration="2.575562582s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:58:07 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:58:09.575536998 +0000 UTC m=+2537.146002598" watchObservedRunningTime="2026-10-30 19:58:09.575562582 +0000 UTC m=+2537.146028183"
Oct 30 19:58:10.461185 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:58:10.461145    4892 generic.go:356] "Generic (PLEG): container finished" podID="8b7c4bca-c523-4094-a2d7-3bfee37be2e3" containerID="55a5d749961896f71efa97ff668654dd9fc0cbd86d1d05b392cdeaa11b803014" exitCode=0
Oct 30 19:58:10.461579 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:58:10.461224    4892 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w1-d52df0-9q2ht" event={"ID":"8b7c4bca-c523-4094-a2d7-3bfee37be2e3","Type":"ContainerDied","Data":"55a5d749961896f71efa97ff668654dd9fc0cbd86d1d05b392cdeaa11b803014"}
Oct 30 19:58:11.054277 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:58:11.054254    4892 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-898968-openshift-default-b55dfc9c9-gmck5"
Oct 30 19:58:11.135372 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:58:11.135347    4892 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/95cba503-314a-4b92-a90b-c23b551ea329-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/95cba503-314a-4b92-a90b-c23b551ea329-workload-socket\") pod \"95cba503-314a-4b92-a90b-c23b551ea329\" (UID: \"95cba503-314a-4b92-a90b-c23b551ea329\") "
Oct 30 19:58:11.135508 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:58:11.135420    4892 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/95cba503-314a-4b92-a90b-c23b551ea329-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/95cba503-314a-4b92-a90b-c23b551ea329-credential-socket\") pod \"95cba503-314a-4b92-a90b-c23b551ea329\" (UID: \"95cba503-314a-4b92-a90b-c23b551ea329\") "
Oct 30 19:58:11.135553 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:58:11.135515    4892 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/95cba503-314a-4b92-a90b-c23b551ea329-kube-api-access-dcqqd\" (UniqueName: \"kubernetes.io/projected/95cba503-314a-4b92-a90b-c23b551ea329-kube-api-access-dcqqd\") pod \"95cba503-314a-4b92-a90b-c23b551ea329\" (UID: \"95cba503-314a-4b92-a90b-c23b551ea329\") "
Oct 30 19:58:11.135631 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:58:11.135615    4892 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/95cba503-314a-4b92-a90b-c23b551ea329-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/95cba503-314a-4b92-a90b-c23b551ea329-istio-envoy\") pod \"95cba503-314a-4b92-a90b-c23b551ea329\" (UID: \"95cba503-314a-4b92-a90b-c23b551ea329\") "
Oct 30 19:58:11.135669 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:58:11.135631    4892 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/95cba503-314a-4b92-a90b-c23b551ea329-workload-socket" pod "95cba503-314a-4b92-a90b-c23b551ea329" (UID: "95cba503-314a-4b92-a90b-c23b551ea329"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:58:11.135708 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:58:11.135664    4892 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/95cba503-314a-4b92-a90b-c23b551ea329-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/95cba503-314a-4b92-a90b-c23b551ea329-istio-podinfo\") pod \"95cba503-314a-4b92-a90b-c23b551ea329\" (UID: \"95cba503-314a-4b92-a90b-c23b551ea329\") "
Oct 30 19:58:11.135708 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:58:11.135700    4892 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/95cba503-314a-4b92-a90b-c23b551ea329-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/95cba503-314a-4b92-a90b-c23b551ea329-istio-data\") pod \"95cba503-314a-4b92-a90b-c23b551ea329\" (UID: \"95cba503-314a-4b92-a90b-c23b551ea329\") "
Oct 30 19:58:11.135789 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:58:11.135710    4892 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/95cba503-314a-4b92-a90b-c23b551ea329-credential-socket" pod "95cba503-314a-4b92-a90b-c23b551ea329" (UID: "95cba503-314a-4b92-a90b-c23b551ea329"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:58:11.135789 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:58:11.135736    4892 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/95cba503-314a-4b92-a90b-c23b551ea329-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/95cba503-314a-4b92-a90b-c23b551ea329-istiod-ca-cert\") pod \"95cba503-314a-4b92-a90b-c23b551ea329\" (UID: \"95cba503-314a-4b92-a90b-c23b551ea329\") "
Oct 30 19:58:11.135789 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:58:11.135779    4892 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/95cba503-314a-4b92-a90b-c23b551ea329-istio-token\" (UniqueName: \"kubernetes.io/projected/95cba503-314a-4b92-a90b-c23b551ea329-istio-token\") pod \"95cba503-314a-4b92-a90b-c23b551ea329\" (UID: \"95cba503-314a-4b92-a90b-c23b551ea329\") "
Oct 30 19:58:11.135934 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:58:11.135840    4892 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/95cba503-314a-4b92-a90b-c23b551ea329-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/95cba503-314a-4b92-a90b-c23b551ea329-workload-certs\") pod \"95cba503-314a-4b92-a90b-c23b551ea329\" (UID: \"95cba503-314a-4b92-a90b-c23b551ea329\") "
Oct 30 19:58:11.136283 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:58:11.136012    4892 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/95cba503-314a-4b92-a90b-c23b551ea329-istio-data" pod "95cba503-314a-4b92-a90b-c23b551ea329" (UID: "95cba503-314a-4b92-a90b-c23b551ea329"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:58:11.136283 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:58:11.136252    4892 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/95cba503-314a-4b92-a90b-c23b551ea329-workload-certs" pod "95cba503-314a-4b92-a90b-c23b551ea329" (UID: "95cba503-314a-4b92-a90b-c23b551ea329"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:58:11.136283 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:58:11.136272    4892 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/95cba503-314a-4b92-a90b-c23b551ea329-istiod-ca-cert" pod "95cba503-314a-4b92-a90b-c23b551ea329" (UID: "95cba503-314a-4b92-a90b-c23b551ea329"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 19:58:11.136643 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:58:11.136616    4892 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/95cba503-314a-4b92-a90b-c23b551ea329-istio-data\") on node \"ip-10-0-132-82.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 19:58:11.136831 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:58:11.136647    4892 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/95cba503-314a-4b92-a90b-c23b551ea329-istiod-ca-cert\") on node \"ip-10-0-132-82.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 19:58:11.136831 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:58:11.136664    4892 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/95cba503-314a-4b92-a90b-c23b551ea329-workload-certs\") on node \"ip-10-0-132-82.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 19:58:11.136831 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:58:11.136678    4892 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/95cba503-314a-4b92-a90b-c23b551ea329-workload-socket\") on node \"ip-10-0-132-82.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 19:58:11.136831 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:58:11.136692    4892 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/95cba503-314a-4b92-a90b-c23b551ea329-credential-socket\") on node \"ip-10-0-132-82.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 19:58:11.137977 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:58:11.137935    4892 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/95cba503-314a-4b92-a90b-c23b551ea329-istio-envoy" pod "95cba503-314a-4b92-a90b-c23b551ea329" (UID: "95cba503-314a-4b92-a90b-c23b551ea329"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:58:11.138077 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:58:11.138023    4892 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/95cba503-314a-4b92-a90b-c23b551ea329-kube-api-access-dcqqd" pod "95cba503-314a-4b92-a90b-c23b551ea329" (UID: "95cba503-314a-4b92-a90b-c23b551ea329"). InnerVolumeSpecName "kube-api-access-dcqqd". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:58:11.138133 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:58:11.138075    4892 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/95cba503-314a-4b92-a90b-c23b551ea329-istio-podinfo" pod "95cba503-314a-4b92-a90b-c23b551ea329" (UID: "95cba503-314a-4b92-a90b-c23b551ea329"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 19:58:11.138319 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:58:11.138284    4892 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/95cba503-314a-4b92-a90b-c23b551ea329-istio-token" pod "95cba503-314a-4b92-a90b-c23b551ea329" (UID: "95cba503-314a-4b92-a90b-c23b551ea329"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:58:11.237872 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:58:11.237843    4892 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dcqqd\" (UniqueName: \"kubernetes.io/projected/95cba503-314a-4b92-a90b-c23b551ea329-kube-api-access-dcqqd\") on node \"ip-10-0-132-82.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 19:58:11.237872 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:58:11.237871    4892 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/95cba503-314a-4b92-a90b-c23b551ea329-istio-envoy\") on node \"ip-10-0-132-82.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 19:58:11.238077 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:58:11.237881    4892 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/95cba503-314a-4b92-a90b-c23b551ea329-istio-podinfo\") on node \"ip-10-0-132-82.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 19:58:11.238077 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:58:11.237891    4892 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/95cba503-314a-4b92-a90b-c23b551ea329-istio-token\") on node \"ip-10-0-132-82.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 19:58:11.470804 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:58:11.470767    4892 generic.go:356] "Generic (PLEG): container finished" podID="95cba503-314a-4b92-a90b-c23b551ea329" containerID="c406d824e6f76efb5e69812a9ad3ee202c4adb8ee77210c44ab23930d4f19069" exitCode=0
Oct 30 19:58:11.471264 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:58:11.470814    4892 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-898968-openshift-default-b55dfc9c9-gmck5"
Oct 30 19:58:11.471264 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:58:11.470842    4892 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-898968-openshift-default-b55dfc9c9-gmck5" event={"ID":"95cba503-314a-4b92-a90b-c23b551ea329","Type":"ContainerDied","Data":"c406d824e6f76efb5e69812a9ad3ee202c4adb8ee77210c44ab23930d4f19069"}
Oct 30 19:58:11.471264 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:58:11.470891    4892 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-898968-openshift-default-b55dfc9c9-gmck5" event={"ID":"95cba503-314a-4b92-a90b-c23b551ea329","Type":"ContainerDied","Data":"e45e1352a94cc464f084d485f363341efd16ccd6e76f9d80586719b3a0960805"}
Oct 30 19:58:11.471264 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:58:11.470912    4892 scope.go:122] "RemoveContainer" containerID="c406d824e6f76efb5e69812a9ad3ee202c4adb8ee77210c44ab23930d4f19069"
Oct 30 19:58:11.483421 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:58:11.483399    4892 scope.go:122] "RemoveContainer" containerID="30d80d93579e16c3d80d44735723524b0cb4e15e7b0540f54d72969c46e9ec1e"
Oct 30 19:58:11.494854 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:58:11.494833    4892 scope.go:122] "RemoveContainer" containerID="c406d824e6f76efb5e69812a9ad3ee202c4adb8ee77210c44ab23930d4f19069"
Oct 30 19:58:11.495160 ip-10-0-132-82 kubenswrapper[4892]: E1030 19:58:11.495141    4892 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c406d824e6f76efb5e69812a9ad3ee202c4adb8ee77210c44ab23930d4f19069\": container with ID starting with c406d824e6f76efb5e69812a9ad3ee202c4adb8ee77210c44ab23930d4f19069 not found: ID does not exist" containerID="c406d824e6f76efb5e69812a9ad3ee202c4adb8ee77210c44ab23930d4f19069"
Oct 30 19:58:11.495226 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:58:11.495170    4892 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c406d824e6f76efb5e69812a9ad3ee202c4adb8ee77210c44ab23930d4f19069"} err="failed to get container status \"c406d824e6f76efb5e69812a9ad3ee202c4adb8ee77210c44ab23930d4f19069\": rpc error: code = NotFound desc = could not find container \"c406d824e6f76efb5e69812a9ad3ee202c4adb8ee77210c44ab23930d4f19069\": container with ID starting with c406d824e6f76efb5e69812a9ad3ee202c4adb8ee77210c44ab23930d4f19069 not found: ID does not exist"
Oct 30 19:58:11.495226 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:58:11.495188    4892 scope.go:122] "RemoveContainer" containerID="30d80d93579e16c3d80d44735723524b0cb4e15e7b0540f54d72969c46e9ec1e"
Oct 30 19:58:11.495461 ip-10-0-132-82 kubenswrapper[4892]: E1030 19:58:11.495442    4892 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"30d80d93579e16c3d80d44735723524b0cb4e15e7b0540f54d72969c46e9ec1e\": container with ID starting with 30d80d93579e16c3d80d44735723524b0cb4e15e7b0540f54d72969c46e9ec1e not found: ID does not exist" containerID="30d80d93579e16c3d80d44735723524b0cb4e15e7b0540f54d72969c46e9ec1e"
Oct 30 19:58:11.495520 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:58:11.495466    4892 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"30d80d93579e16c3d80d44735723524b0cb4e15e7b0540f54d72969c46e9ec1e"} err="failed to get container status \"30d80d93579e16c3d80d44735723524b0cb4e15e7b0540f54d72969c46e9ec1e\": rpc error: code = NotFound desc = could not find container \"30d80d93579e16c3d80d44735723524b0cb4e15e7b0540f54d72969c46e9ec1e\": container with ID starting with 30d80d93579e16c3d80d44735723524b0cb4e15e7b0540f54d72969c46e9ec1e not found: ID does not exist"
Oct 30 19:58:11.622511 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:58:11.622482    4892 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w1-d52df0-9q2ht"
Oct 30 19:58:11.636721 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:58:11.636693    4892 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-a-898968-openshift-default-b55dfc9c9-gmck5"]
Oct 30 19:58:11.655614 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:58:11.655589    4892 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/8b7c4bca-c523-4094-a2d7-3bfee37be2e3-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8b7c4bca-c523-4094-a2d7-3bfee37be2e3-maas-api-openshift-ca\") pod \"8b7c4bca-c523-4094-a2d7-3bfee37be2e3\" (UID: \"8b7c4bca-c523-4094-a2d7-3bfee37be2e3\") "
Oct 30 19:58:11.655749 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:58:11.655728    4892 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8b7c4bca-c523-4094-a2d7-3bfee37be2e3-kube-api-access-mk2gc\" (UniqueName: \"kubernetes.io/projected/8b7c4bca-c523-4094-a2d7-3bfee37be2e3-kube-api-access-mk2gc\") pod \"8b7c4bca-c523-4094-a2d7-3bfee37be2e3\" (UID: \"8b7c4bca-c523-4094-a2d7-3bfee37be2e3\") "
Oct 30 19:58:11.655791 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:58:11.655776    4892 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/8b7c4bca-c523-4094-a2d7-3bfee37be2e3-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8b7c4bca-c523-4094-a2d7-3bfee37be2e3-maas-api-xks-ca\") pod \"8b7c4bca-c523-4094-a2d7-3bfee37be2e3\" (UID: \"8b7c4bca-c523-4094-a2d7-3bfee37be2e3\") "
Oct 30 19:58:11.655914 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:58:11.655894    4892 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8b7c4bca-c523-4094-a2d7-3bfee37be2e3-maas-api-openshift-ca" pod "8b7c4bca-c523-4094-a2d7-3bfee37be2e3" (UID: "8b7c4bca-c523-4094-a2d7-3bfee37be2e3"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 19:58:11.656750 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:58:11.656726    4892 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8b7c4bca-c523-4094-a2d7-3bfee37be2e3-maas-api-openshift-ca\") on node \"ip-10-0-132-82.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 19:58:11.657957 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:58:11.657932    4892 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8b7c4bca-c523-4094-a2d7-3bfee37be2e3-maas-api-xks-ca" pod "8b7c4bca-c523-4094-a2d7-3bfee37be2e3" (UID: "8b7c4bca-c523-4094-a2d7-3bfee37be2e3"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 19:58:11.657957 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:58:11.657988    4892 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8b7c4bca-c523-4094-a2d7-3bfee37be2e3-kube-api-access-mk2gc" pod "8b7c4bca-c523-4094-a2d7-3bfee37be2e3" (UID: "8b7c4bca-c523-4094-a2d7-3bfee37be2e3"). InnerVolumeSpecName "kube-api-access-mk2gc". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:58:11.685824 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:58:11.685800    4892 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-inf-a-898968-openshift-default-b55dfc9c9-gmck5"]
Oct 30 19:58:11.757629 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:58:11.757567    4892 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mk2gc\" (UniqueName: \"kubernetes.io/projected/8b7c4bca-c523-4094-a2d7-3bfee37be2e3-kube-api-access-mk2gc\") on node \"ip-10-0-132-82.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 19:58:11.757629 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:58:11.757593    4892 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8b7c4bca-c523-4094-a2d7-3bfee37be2e3-maas-api-xks-ca\") on node \"ip-10-0-132-82.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 19:58:12.487986 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:58:12.487877    4892 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w1-d52df0-9q2ht" event={"ID":"8b7c4bca-c523-4094-a2d7-3bfee37be2e3","Type":"ContainerDied","Data":"7d20b8a0f89588744ee8acb488a1120d862524baa956008405dceb7b33ff2ff3"}
Oct 30 19:58:12.487986 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:58:12.487959    4892 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="7d20b8a0f89588744ee8acb488a1120d862524baa956008405dceb7b33ff2ff3"
Oct 30 19:58:12.488524 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:58:12.488082    4892 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w1-d52df0-9q2ht"
Oct 30 19:58:12.576069 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:58:12.576034    4892 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="95cba503-314a-4b92-a90b-c23b551ea329" path="/var/lib/kubelet/pods/95cba503-314a-4b92-a90b-c23b551ea329/volumes"
Oct 30 19:58:12.708579 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:58:12.708481    4892 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w1-d52df0/distinct-2-w1-d52df0-kserve-7846558464-xrkdc"]
Oct 30 19:58:12.708869 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:58:12.708843    4892 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w1-d52df0/simulator-w1-d52df0-kserve-67f74bc577-q5bjz"]
Oct 30 19:58:23.165440 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:58:23.165400    4892 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-b0c827-npkmz"]
Oct 30 19:58:23.166352 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:58:23.166330    4892 cpu_manager.go:414] "containerMap: removing container" podUID="95cba503-314a-4b92-a90b-c23b551ea329" containerName="istio-proxy"
Oct 30 19:58:23.166469 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:58:23.166353    4892 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="95cba503-314a-4b92-a90b-c23b551ea329" containerName="istio-proxy"
Oct 30 19:58:23.166469 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:58:23.166376    4892 cpu_manager.go:414] "containerMap: removing container" podUID="95cba503-314a-4b92-a90b-c23b551ea329" containerName="istio-proxy"
Oct 30 19:58:23.166469 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:58:23.166382    4892 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="95cba503-314a-4b92-a90b-c23b551ea329" containerName="istio-proxy"
Oct 30 19:58:23.166469 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:58:23.166399    4892 cpu_manager.go:414] "containerMap: removing container" podUID="8b7c4bca-c523-4094-a2d7-3bfee37be2e3" containerName="revoke-keys"
Oct 30 19:58:23.166469 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:58:23.166406    4892 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8b7c4bca-c523-4094-a2d7-3bfee37be2e3" containerName="revoke-keys"
Oct 30 19:58:23.166737 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:58:23.166582    4892 memory_manager.go:365] "RemoveStaleState removing state" podUID="95cba503-314a-4b92-a90b-c23b551ea329" containerName="istio-proxy"
Oct 30 19:58:23.166737 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:58:23.166600    4892 memory_manager.go:365] "RemoveStaleState removing state" podUID="95cba503-314a-4b92-a90b-c23b551ea329" containerName="istio-proxy"
Oct 30 19:58:23.166737 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:58:23.166613    4892 memory_manager.go:365] "RemoveStaleState removing state" podUID="8b7c4bca-c523-4094-a2d7-3bfee37be2e3" containerName="revoke-keys"
Oct 30 19:58:23.170276 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:58:23.170254    4892 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-b0c827-npkmz"
Oct 30 19:58:23.227693 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:58:23.227669    4892 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-b0c827-npkmz"]
Oct 30 19:58:23.233130 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:58:23.233104    4892 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6f18d694-a7ca-449e-8ba5-0b07d06cea98-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-b0c827-npkmz\" (UID: \"6f18d694-a7ca-449e-8ba5-0b07d06cea98\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-b0c827-npkmz"
Oct 30 19:58:23.233327 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:58:23.233307    4892 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7b6b4\" (UniqueName: \"kubernetes.io/projected/6f18d694-a7ca-449e-8ba5-0b07d06cea98-kube-api-access-7b6b4\") pod \"maas-api-revoke-keys-e2e-inf-b-b0c827-npkmz\" (UID: \"6f18d694-a7ca-449e-8ba5-0b07d06cea98\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-b0c827-npkmz"
Oct 30 19:58:23.233503 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:58:23.233487    4892 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6f18d694-a7ca-449e-8ba5-0b07d06cea98-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-b0c827-npkmz\" (UID: \"6f18d694-a7ca-449e-8ba5-0b07d06cea98\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-b0c827-npkmz"
Oct 30 19:58:23.271547 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:58:23.271514    4892 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 30 19:58:23.271547 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:58:23.271518    4892 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 19:58:23.271732 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:58:23.271589    4892 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-nlxp4\""
Oct 30 19:58:23.271732 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:58:23.271619    4892 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 30 19:58:23.335084 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:58:23.335055    4892 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6f18d694-a7ca-449e-8ba5-0b07d06cea98-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-b0c827-npkmz\" (UID: \"6f18d694-a7ca-449e-8ba5-0b07d06cea98\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-b0c827-npkmz"
Oct 30 19:58:23.335288 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:58:23.335269    4892 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6f18d694-a7ca-449e-8ba5-0b07d06cea98-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-b0c827-npkmz\" (UID: \"6f18d694-a7ca-449e-8ba5-0b07d06cea98\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-b0c827-npkmz"
Oct 30 19:58:23.335332 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:58:23.335317    4892 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7b6b4\" (UniqueName: \"kubernetes.io/projected/6f18d694-a7ca-449e-8ba5-0b07d06cea98-kube-api-access-7b6b4\") pod \"maas-api-revoke-keys-e2e-inf-b-b0c827-npkmz\" (UID: \"6f18d694-a7ca-449e-8ba5-0b07d06cea98\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-b0c827-npkmz"
Oct 30 19:58:23.335830 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:58:23.335801    4892 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6f18d694-a7ca-449e-8ba5-0b07d06cea98-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-b0c827-npkmz\" (UID: \"6f18d694-a7ca-449e-8ba5-0b07d06cea98\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-b0c827-npkmz"
Oct 30 19:58:23.337757 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:58:23.337736    4892 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6f18d694-a7ca-449e-8ba5-0b07d06cea98-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-b0c827-npkmz\" (UID: \"6f18d694-a7ca-449e-8ba5-0b07d06cea98\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-b0c827-npkmz"
Oct 30 19:58:23.392340 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:58:23.392313    4892 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7b6b4\" (UniqueName: \"kubernetes.io/projected/6f18d694-a7ca-449e-8ba5-0b07d06cea98-kube-api-access-7b6b4\") pod \"maas-api-revoke-keys-e2e-inf-b-b0c827-npkmz\" (UID: \"6f18d694-a7ca-449e-8ba5-0b07d06cea98\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-b0c827-npkmz"
Oct 30 19:58:23.485859 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:58:23.485795    4892 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-b0c827-npkmz"
Oct 30 19:58:23.879027 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:58:23.869601    4892 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-b0c827-npkmz"]
Oct 30 19:58:23.880790 ip-10-0-132-82 kubenswrapper[4892]: W1030 19:58:23.880762    4892 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6f18d694_a7ca_449e_8ba5_0b07d06cea98.slice/crio-89af4debfcaab67c79e1f7749123c040e5cae7499105ddd86ee57ac33ca155e2 WatchSource:0}: Error finding container 89af4debfcaab67c79e1f7749123c040e5cae7499105ddd86ee57ac33ca155e2: Status 404 returned error can't find the container with id 89af4debfcaab67c79e1f7749123c040e5cae7499105ddd86ee57ac33ca155e2
Oct 30 19:58:24.011439 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:58:24.011408    4892 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-mt-dd9b8d79-openshift-default-857b46944f-4zd2h"]
Oct 30 19:58:24.016982 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:58:24.016949    4892 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-dd9b8d79-openshift-default-857b46944f-4zd2h"
Oct 30 19:58:24.029719 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:58:24.029688    4892 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-mt-dd9b8d79-openshift-default-857b46944f-5wjxt"]
Oct 30 19:58:24.033676 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:58:24.033656    4892 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-dd9b8d79-openshift-default-857b46944f-5wjxt"
Oct 30 19:58:24.043351 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:58:24.043329    4892 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/332cc10b-8e86-49c3-8381-d48fa8bd4782-credential-socket\") pod \"e2e-mt-dd9b8d79-openshift-default-857b46944f-4zd2h\" (UID: \"332cc10b-8e86-49c3-8381-d48fa8bd4782\") " pod="openshift-ingress/e2e-mt-dd9b8d79-openshift-default-857b46944f-4zd2h"
Oct 30 19:58:24.043463 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:58:24.043368    4892 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0d3f37ec-9b98-4a8c-a38c-d02ffd5417b1-istiod-ca-cert\") pod \"e2e-mt-dd9b8d79-openshift-default-857b46944f-5wjxt\" (UID: \"0d3f37ec-9b98-4a8c-a38c-d02ffd5417b1\") " pod="openshift-ingress/e2e-mt-dd9b8d79-openshift-default-857b46944f-5wjxt"
Oct 30 19:58:24.043463 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:58:24.043454    4892 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pzvgb\" (UniqueName: \"kubernetes.io/projected/0d3f37ec-9b98-4a8c-a38c-d02ffd5417b1-kube-api-access-pzvgb\") pod \"e2e-mt-dd9b8d79-openshift-default-857b46944f-5wjxt\" (UID: \"0d3f37ec-9b98-4a8c-a38c-d02ffd5417b1\") " pod="openshift-ingress/e2e-mt-dd9b8d79-openshift-default-857b46944f-5wjxt"
Oct 30 19:58:24.043589 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:58:24.043485    4892 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0d3f37ec-9b98-4a8c-a38c-d02ffd5417b1-istio-data\") pod \"e2e-mt-dd9b8d79-openshift-default-857b46944f-5wjxt\" (UID: \"0d3f37ec-9b98-4a8c-a38c-d02ffd5417b1\") " pod="openshift-ingress/e2e-mt-dd9b8d79-openshift-default-857b46944f-5wjxt"
Oct 30 19:58:24.043589 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:58:24.043508    4892 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0d3f37ec-9b98-4a8c-a38c-d02ffd5417b1-istio-token\") pod \"e2e-mt-dd9b8d79-openshift-default-857b46944f-5wjxt\" (UID: \"0d3f37ec-9b98-4a8c-a38c-d02ffd5417b1\") " pod="openshift-ingress/e2e-mt-dd9b8d79-openshift-default-857b46944f-5wjxt"
Oct 30 19:58:24.043677 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:58:24.043603    4892 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0d3f37ec-9b98-4a8c-a38c-d02ffd5417b1-istio-podinfo\") pod \"e2e-mt-dd9b8d79-openshift-default-857b46944f-5wjxt\" (UID: \"0d3f37ec-9b98-4a8c-a38c-d02ffd5417b1\") " pod="openshift-ingress/e2e-mt-dd9b8d79-openshift-default-857b46944f-5wjxt"
Oct 30 19:58:24.043715 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:58:24.043680    4892 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/332cc10b-8e86-49c3-8381-d48fa8bd4782-workload-certs\") pod \"e2e-mt-dd9b8d79-openshift-default-857b46944f-4zd2h\" (UID: \"332cc10b-8e86-49c3-8381-d48fa8bd4782\") " pod="openshift-ingress/e2e-mt-dd9b8d79-openshift-default-857b46944f-4zd2h"
Oct 30 19:58:24.043715 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:58:24.043709    4892 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0d3f37ec-9b98-4a8c-a38c-d02ffd5417b1-credential-socket\") pod \"e2e-mt-dd9b8d79-openshift-default-857b46944f-5wjxt\" (UID: \"0d3f37ec-9b98-4a8c-a38c-d02ffd5417b1\") " pod="openshift-ingress/e2e-mt-dd9b8d79-openshift-default-857b46944f-5wjxt"
Oct 30 19:58:24.043845 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:58:24.043824    4892 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/332cc10b-8e86-49c3-8381-d48fa8bd4782-istio-data\") pod \"e2e-mt-dd9b8d79-openshift-default-857b46944f-4zd2h\" (UID: \"332cc10b-8e86-49c3-8381-d48fa8bd4782\") " pod="openshift-ingress/e2e-mt-dd9b8d79-openshift-default-857b46944f-4zd2h"
Oct 30 19:58:24.043928 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:58:24.043895    4892 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/332cc10b-8e86-49c3-8381-d48fa8bd4782-workload-socket\") pod \"e2e-mt-dd9b8d79-openshift-default-857b46944f-4zd2h\" (UID: \"332cc10b-8e86-49c3-8381-d48fa8bd4782\") " pod="openshift-ingress/e2e-mt-dd9b8d79-openshift-default-857b46944f-4zd2h"
Oct 30 19:58:24.044068 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:58:24.043943    4892 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0d3f37ec-9b98-4a8c-a38c-d02ffd5417b1-istio-envoy\") pod \"e2e-mt-dd9b8d79-openshift-default-857b46944f-5wjxt\" (UID: \"0d3f37ec-9b98-4a8c-a38c-d02ffd5417b1\") " pod="openshift-ingress/e2e-mt-dd9b8d79-openshift-default-857b46944f-5wjxt"
Oct 30 19:58:24.044068 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:58:24.044035    4892 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8hxk9\" (UniqueName: \"kubernetes.io/projected/332cc10b-8e86-49c3-8381-d48fa8bd4782-kube-api-access-8hxk9\") pod \"e2e-mt-dd9b8d79-openshift-default-857b46944f-4zd2h\" (UID: \"332cc10b-8e86-49c3-8381-d48fa8bd4782\") " pod="openshift-ingress/e2e-mt-dd9b8d79-openshift-default-857b46944f-4zd2h"
Oct 30 19:58:24.044068 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:58:24.044060    4892 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0d3f37ec-9b98-4a8c-a38c-d02ffd5417b1-workload-socket\") pod \"e2e-mt-dd9b8d79-openshift-default-857b46944f-5wjxt\" (UID: \"0d3f37ec-9b98-4a8c-a38c-d02ffd5417b1\") " pod="openshift-ingress/e2e-mt-dd9b8d79-openshift-default-857b46944f-5wjxt"
Oct 30 19:58:24.044244 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:58:24.044139    4892 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/332cc10b-8e86-49c3-8381-d48fa8bd4782-istio-podinfo\") pod \"e2e-mt-dd9b8d79-openshift-default-857b46944f-4zd2h\" (UID: \"332cc10b-8e86-49c3-8381-d48fa8bd4782\") " pod="openshift-ingress/e2e-mt-dd9b8d79-openshift-default-857b46944f-4zd2h"
Oct 30 19:58:24.044244 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:58:24.044217    4892 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0d3f37ec-9b98-4a8c-a38c-d02ffd5417b1-workload-certs\") pod \"e2e-mt-dd9b8d79-openshift-default-857b46944f-5wjxt\" (UID: \"0d3f37ec-9b98-4a8c-a38c-d02ffd5417b1\") " pod="openshift-ingress/e2e-mt-dd9b8d79-openshift-default-857b46944f-5wjxt"
Oct 30 19:58:24.044360 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:58:24.044247    4892 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/332cc10b-8e86-49c3-8381-d48fa8bd4782-istio-token\") pod \"e2e-mt-dd9b8d79-openshift-default-857b46944f-4zd2h\" (UID: \"332cc10b-8e86-49c3-8381-d48fa8bd4782\") " pod="openshift-ingress/e2e-mt-dd9b8d79-openshift-default-857b46944f-4zd2h"
Oct 30 19:58:24.044360 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:58:24.044305    4892 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/332cc10b-8e86-49c3-8381-d48fa8bd4782-istio-envoy\") pod \"e2e-mt-dd9b8d79-openshift-default-857b46944f-4zd2h\" (UID: \"332cc10b-8e86-49c3-8381-d48fa8bd4782\") " pod="openshift-ingress/e2e-mt-dd9b8d79-openshift-default-857b46944f-4zd2h"
Oct 30 19:58:24.044360 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:58:24.044344    4892 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/332cc10b-8e86-49c3-8381-d48fa8bd4782-istiod-ca-cert\") pod \"e2e-mt-dd9b8d79-openshift-default-857b46944f-4zd2h\" (UID: \"332cc10b-8e86-49c3-8381-d48fa8bd4782\") " pod="openshift-ingress/e2e-mt-dd9b8d79-openshift-default-857b46944f-4zd2h"
Oct 30 19:58:24.075263 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:58:24.075237    4892 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-mt-dd9b8d79-openshift-default-dockercfg-nk77p\""
Oct 30 19:58:24.079500 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:58:24.079481    4892 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-mt-dd9b8d79-openshift-default-857b46944f-4zd2h"]
Oct 30 19:58:24.128468 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:58:24.128383    4892 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-mt-dd9b8d79-openshift-default-857b46944f-5wjxt"]
Oct 30 19:58:24.145806 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:58:24.145771    4892 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0d3f37ec-9b98-4a8c-a38c-d02ffd5417b1-istio-data\") pod \"e2e-mt-dd9b8d79-openshift-default-857b46944f-5wjxt\" (UID: \"0d3f37ec-9b98-4a8c-a38c-d02ffd5417b1\") " pod="openshift-ingress/e2e-mt-dd9b8d79-openshift-default-857b46944f-5wjxt"
Oct 30 19:58:24.145944 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:58:24.145814    4892 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0d3f37ec-9b98-4a8c-a38c-d02ffd5417b1-istio-token\") pod \"e2e-mt-dd9b8d79-openshift-default-857b46944f-5wjxt\" (UID: \"0d3f37ec-9b98-4a8c-a38c-d02ffd5417b1\") " pod="openshift-ingress/e2e-mt-dd9b8d79-openshift-default-857b46944f-5wjxt"
Oct 30 19:58:24.145944 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:58:24.145846    4892 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0d3f37ec-9b98-4a8c-a38c-d02ffd5417b1-istio-podinfo\") pod \"e2e-mt-dd9b8d79-openshift-default-857b46944f-5wjxt\" (UID: \"0d3f37ec-9b98-4a8c-a38c-d02ffd5417b1\") " pod="openshift-ingress/e2e-mt-dd9b8d79-openshift-default-857b46944f-5wjxt"
Oct 30 19:58:24.145944 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:58:24.145886    4892 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/332cc10b-8e86-49c3-8381-d48fa8bd4782-workload-certs\") pod \"e2e-mt-dd9b8d79-openshift-default-857b46944f-4zd2h\" (UID: \"332cc10b-8e86-49c3-8381-d48fa8bd4782\") " pod="openshift-ingress/e2e-mt-dd9b8d79-openshift-default-857b46944f-4zd2h"
Oct 30 19:58:24.145944 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:58:24.145910    4892 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0d3f37ec-9b98-4a8c-a38c-d02ffd5417b1-credential-socket\") pod \"e2e-mt-dd9b8d79-openshift-default-857b46944f-5wjxt\" (UID: \"0d3f37ec-9b98-4a8c-a38c-d02ffd5417b1\") " pod="openshift-ingress/e2e-mt-dd9b8d79-openshift-default-857b46944f-5wjxt"
Oct 30 19:58:24.146229 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:58:24.145947    4892 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/332cc10b-8e86-49c3-8381-d48fa8bd4782-istio-data\") pod \"e2e-mt-dd9b8d79-openshift-default-857b46944f-4zd2h\" (UID: \"332cc10b-8e86-49c3-8381-d48fa8bd4782\") " pod="openshift-ingress/e2e-mt-dd9b8d79-openshift-default-857b46944f-4zd2h"
Oct 30 19:58:24.146229 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:58:24.145992    4892 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/332cc10b-8e86-49c3-8381-d48fa8bd4782-workload-socket\") pod \"e2e-mt-dd9b8d79-openshift-default-857b46944f-4zd2h\" (UID: \"332cc10b-8e86-49c3-8381-d48fa8bd4782\") " pod="openshift-ingress/e2e-mt-dd9b8d79-openshift-default-857b46944f-4zd2h"
Oct 30 19:58:24.146229 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:58:24.146011    4892 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0d3f37ec-9b98-4a8c-a38c-d02ffd5417b1-istio-envoy\") pod \"e2e-mt-dd9b8d79-openshift-default-857b46944f-5wjxt\" (UID: \"0d3f37ec-9b98-4a8c-a38c-d02ffd5417b1\") " pod="openshift-ingress/e2e-mt-dd9b8d79-openshift-default-857b46944f-5wjxt"
Oct 30 19:58:24.146229 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:58:24.146037    4892 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8hxk9\" (UniqueName: \"kubernetes.io/projected/332cc10b-8e86-49c3-8381-d48fa8bd4782-kube-api-access-8hxk9\") pod \"e2e-mt-dd9b8d79-openshift-default-857b46944f-4zd2h\" (UID: \"332cc10b-8e86-49c3-8381-d48fa8bd4782\") " pod="openshift-ingress/e2e-mt-dd9b8d79-openshift-default-857b46944f-4zd2h"
Oct 30 19:58:24.146229 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:58:24.146058    4892 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0d3f37ec-9b98-4a8c-a38c-d02ffd5417b1-workload-socket\") pod \"e2e-mt-dd9b8d79-openshift-default-857b46944f-5wjxt\" (UID: \"0d3f37ec-9b98-4a8c-a38c-d02ffd5417b1\") " pod="openshift-ingress/e2e-mt-dd9b8d79-openshift-default-857b46944f-5wjxt"
Oct 30 19:58:24.146229 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:58:24.146093    4892 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/332cc10b-8e86-49c3-8381-d48fa8bd4782-istio-podinfo\") pod \"e2e-mt-dd9b8d79-openshift-default-857b46944f-4zd2h\" (UID: \"332cc10b-8e86-49c3-8381-d48fa8bd4782\") " pod="openshift-ingress/e2e-mt-dd9b8d79-openshift-default-857b46944f-4zd2h"
Oct 30 19:58:24.146229 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:58:24.146129    4892 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0d3f37ec-9b98-4a8c-a38c-d02ffd5417b1-workload-certs\") pod \"e2e-mt-dd9b8d79-openshift-default-857b46944f-5wjxt\" (UID: \"0d3f37ec-9b98-4a8c-a38c-d02ffd5417b1\") " pod="openshift-ingress/e2e-mt-dd9b8d79-openshift-default-857b46944f-5wjxt"
Oct 30 19:58:24.146229 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:58:24.146148    4892 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/332cc10b-8e86-49c3-8381-d48fa8bd4782-istio-token\") pod \"e2e-mt-dd9b8d79-openshift-default-857b46944f-4zd2h\" (UID: \"332cc10b-8e86-49c3-8381-d48fa8bd4782\") " pod="openshift-ingress/e2e-mt-dd9b8d79-openshift-default-857b46944f-4zd2h"
Oct 30 19:58:24.146229 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:58:24.146177    4892 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/332cc10b-8e86-49c3-8381-d48fa8bd4782-istio-envoy\") pod \"e2e-mt-dd9b8d79-openshift-default-857b46944f-4zd2h\" (UID: \"332cc10b-8e86-49c3-8381-d48fa8bd4782\") " pod="openshift-ingress/e2e-mt-dd9b8d79-openshift-default-857b46944f-4zd2h"
Oct 30 19:58:24.146229 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:58:24.146200    4892 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/332cc10b-8e86-49c3-8381-d48fa8bd4782-istiod-ca-cert\") pod \"e2e-mt-dd9b8d79-openshift-default-857b46944f-4zd2h\" (UID: \"332cc10b-8e86-49c3-8381-d48fa8bd4782\") " pod="openshift-ingress/e2e-mt-dd9b8d79-openshift-default-857b46944f-4zd2h"
Oct 30 19:58:24.146229 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:58:24.146226    4892 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/332cc10b-8e86-49c3-8381-d48fa8bd4782-credential-socket\") pod \"e2e-mt-dd9b8d79-openshift-default-857b46944f-4zd2h\" (UID: \"332cc10b-8e86-49c3-8381-d48fa8bd4782\") " pod="openshift-ingress/e2e-mt-dd9b8d79-openshift-default-857b46944f-4zd2h"
Oct 30 19:58:24.146849 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:58:24.146244    4892 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0d3f37ec-9b98-4a8c-a38c-d02ffd5417b1-istiod-ca-cert\") pod \"e2e-mt-dd9b8d79-openshift-default-857b46944f-5wjxt\" (UID: \"0d3f37ec-9b98-4a8c-a38c-d02ffd5417b1\") " pod="openshift-ingress/e2e-mt-dd9b8d79-openshift-default-857b46944f-5wjxt"
Oct 30 19:58:24.146849 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:58:24.146274    4892 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pzvgb\" (UniqueName: \"kubernetes.io/projected/0d3f37ec-9b98-4a8c-a38c-d02ffd5417b1-kube-api-access-pzvgb\") pod \"e2e-mt-dd9b8d79-openshift-default-857b46944f-5wjxt\" (UID: \"0d3f37ec-9b98-4a8c-a38c-d02ffd5417b1\") " pod="openshift-ingress/e2e-mt-dd9b8d79-openshift-default-857b46944f-5wjxt"
Oct 30 19:58:24.146849 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:58:24.146765    4892 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0d3f37ec-9b98-4a8c-a38c-d02ffd5417b1-istio-data\") pod \"e2e-mt-dd9b8d79-openshift-default-857b46944f-5wjxt\" (UID: \"0d3f37ec-9b98-4a8c-a38c-d02ffd5417b1\") " pod="openshift-ingress/e2e-mt-dd9b8d79-openshift-default-857b46944f-5wjxt"
Oct 30 19:58:24.148239 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:58:24.148217    4892 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/332cc10b-8e86-49c3-8381-d48fa8bd4782-workload-certs\") pod \"e2e-mt-dd9b8d79-openshift-default-857b46944f-4zd2h\" (UID: \"332cc10b-8e86-49c3-8381-d48fa8bd4782\") " pod="openshift-ingress/e2e-mt-dd9b8d79-openshift-default-857b46944f-4zd2h"
Oct 30 19:58:24.148587 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:58:24.148551    4892 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0d3f37ec-9b98-4a8c-a38c-d02ffd5417b1-credential-socket\") pod \"e2e-mt-dd9b8d79-openshift-default-857b46944f-5wjxt\" (UID: \"0d3f37ec-9b98-4a8c-a38c-d02ffd5417b1\") " pod="openshift-ingress/e2e-mt-dd9b8d79-openshift-default-857b46944f-5wjxt"
Oct 30 19:58:24.148861 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:58:24.148827    4892 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/332cc10b-8e86-49c3-8381-d48fa8bd4782-istio-data\") pod \"e2e-mt-dd9b8d79-openshift-default-857b46944f-4zd2h\" (UID: \"332cc10b-8e86-49c3-8381-d48fa8bd4782\") " pod="openshift-ingress/e2e-mt-dd9b8d79-openshift-default-857b46944f-4zd2h"
Oct 30 19:58:24.149114 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:58:24.149097    4892 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/332cc10b-8e86-49c3-8381-d48fa8bd4782-workload-socket\") pod \"e2e-mt-dd9b8d79-openshift-default-857b46944f-4zd2h\" (UID: \"332cc10b-8e86-49c3-8381-d48fa8bd4782\") " pod="openshift-ingress/e2e-mt-dd9b8d79-openshift-default-857b46944f-4zd2h"
Oct 30 19:58:24.150178 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:58:24.150156    4892 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0d3f37ec-9b98-4a8c-a38c-d02ffd5417b1-workload-socket\") pod \"e2e-mt-dd9b8d79-openshift-default-857b46944f-5wjxt\" (UID: \"0d3f37ec-9b98-4a8c-a38c-d02ffd5417b1\") " pod="openshift-ingress/e2e-mt-dd9b8d79-openshift-default-857b46944f-5wjxt"
Oct 30 19:58:24.150493 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:58:24.150466    4892 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/332cc10b-8e86-49c3-8381-d48fa8bd4782-istiod-ca-cert\") pod \"e2e-mt-dd9b8d79-openshift-default-857b46944f-4zd2h\" (UID: \"332cc10b-8e86-49c3-8381-d48fa8bd4782\") " pod="openshift-ingress/e2e-mt-dd9b8d79-openshift-default-857b46944f-4zd2h"
Oct 30 19:58:24.150808 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:58:24.150783    4892 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0d3f37ec-9b98-4a8c-a38c-d02ffd5417b1-istio-podinfo\") pod \"e2e-mt-dd9b8d79-openshift-default-857b46944f-5wjxt\" (UID: \"0d3f37ec-9b98-4a8c-a38c-d02ffd5417b1\") " pod="openshift-ingress/e2e-mt-dd9b8d79-openshift-default-857b46944f-5wjxt"
Oct 30 19:58:24.150914 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:58:24.150866    4892 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0d3f37ec-9b98-4a8c-a38c-d02ffd5417b1-workload-certs\") pod \"e2e-mt-dd9b8d79-openshift-default-857b46944f-5wjxt\" (UID: \"0d3f37ec-9b98-4a8c-a38c-d02ffd5417b1\") " pod="openshift-ingress/e2e-mt-dd9b8d79-openshift-default-857b46944f-5wjxt"
Oct 30 19:58:24.151123 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:58:24.151100    4892 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/332cc10b-8e86-49c3-8381-d48fa8bd4782-credential-socket\") pod \"e2e-mt-dd9b8d79-openshift-default-857b46944f-4zd2h\" (UID: \"332cc10b-8e86-49c3-8381-d48fa8bd4782\") " pod="openshift-ingress/e2e-mt-dd9b8d79-openshift-default-857b46944f-4zd2h"
Oct 30 19:58:24.151541 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:58:24.151520    4892 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0d3f37ec-9b98-4a8c-a38c-d02ffd5417b1-istio-envoy\") pod \"e2e-mt-dd9b8d79-openshift-default-857b46944f-5wjxt\" (UID: \"0d3f37ec-9b98-4a8c-a38c-d02ffd5417b1\") " pod="openshift-ingress/e2e-mt-dd9b8d79-openshift-default-857b46944f-5wjxt"
Oct 30 19:58:24.151629 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:58:24.151599    4892 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0d3f37ec-9b98-4a8c-a38c-d02ffd5417b1-istiod-ca-cert\") pod \"e2e-mt-dd9b8d79-openshift-default-857b46944f-5wjxt\" (UID: \"0d3f37ec-9b98-4a8c-a38c-d02ffd5417b1\") " pod="openshift-ingress/e2e-mt-dd9b8d79-openshift-default-857b46944f-5wjxt"
Oct 30 19:58:24.152142 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:58:24.152122    4892 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/332cc10b-8e86-49c3-8381-d48fa8bd4782-istio-envoy\") pod \"e2e-mt-dd9b8d79-openshift-default-857b46944f-4zd2h\" (UID: \"332cc10b-8e86-49c3-8381-d48fa8bd4782\") " pod="openshift-ingress/e2e-mt-dd9b8d79-openshift-default-857b46944f-4zd2h"
Oct 30 19:58:24.153723 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:58:24.153701    4892 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/332cc10b-8e86-49c3-8381-d48fa8bd4782-istio-podinfo\") pod \"e2e-mt-dd9b8d79-openshift-default-857b46944f-4zd2h\" (UID: \"332cc10b-8e86-49c3-8381-d48fa8bd4782\") " pod="openshift-ingress/e2e-mt-dd9b8d79-openshift-default-857b46944f-4zd2h"
Oct 30 19:58:24.202805 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:58:24.202782    4892 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0d3f37ec-9b98-4a8c-a38c-d02ffd5417b1-istio-token\") pod \"e2e-mt-dd9b8d79-openshift-default-857b46944f-5wjxt\" (UID: \"0d3f37ec-9b98-4a8c-a38c-d02ffd5417b1\") " pod="openshift-ingress/e2e-mt-dd9b8d79-openshift-default-857b46944f-5wjxt"
Oct 30 19:58:24.203215 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:58:24.203108    4892 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-pzvgb\" (UniqueName: \"kubernetes.io/projected/0d3f37ec-9b98-4a8c-a38c-d02ffd5417b1-kube-api-access-pzvgb\") pod \"e2e-mt-dd9b8d79-openshift-default-857b46944f-5wjxt\" (UID: \"0d3f37ec-9b98-4a8c-a38c-d02ffd5417b1\") " pod="openshift-ingress/e2e-mt-dd9b8d79-openshift-default-857b46944f-5wjxt"
Oct 30 19:58:24.204983 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:58:24.204948    4892 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/332cc10b-8e86-49c3-8381-d48fa8bd4782-istio-token\") pod \"e2e-mt-dd9b8d79-openshift-default-857b46944f-4zd2h\" (UID: \"332cc10b-8e86-49c3-8381-d48fa8bd4782\") " pod="openshift-ingress/e2e-mt-dd9b8d79-openshift-default-857b46944f-4zd2h"
Oct 30 19:58:24.205378 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:58:24.205360    4892 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8hxk9\" (UniqueName: \"kubernetes.io/projected/332cc10b-8e86-49c3-8381-d48fa8bd4782-kube-api-access-8hxk9\") pod \"e2e-mt-dd9b8d79-openshift-default-857b46944f-4zd2h\" (UID: \"332cc10b-8e86-49c3-8381-d48fa8bd4782\") " pod="openshift-ingress/e2e-mt-dd9b8d79-openshift-default-857b46944f-4zd2h"
Oct 30 19:58:24.331723 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:58:24.331693    4892 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-dd9b8d79-openshift-default-857b46944f-4zd2h"
Oct 30 19:58:24.350401 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:58:24.350369    4892 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-dd9b8d79-openshift-default-857b46944f-5wjxt"
Oct 30 19:58:24.592652 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:58:24.592618    4892 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-b0c827-npkmz" event={"ID":"6f18d694-a7ca-449e-8ba5-0b07d06cea98","Type":"ContainerStarted","Data":"4c714a6cddb76596b4c14a02ef169db62458392f8d937ca7941d018953404f73"}
Oct 30 19:58:24.592827 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:58:24.592674    4892 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-b0c827-npkmz" event={"ID":"6f18d694-a7ca-449e-8ba5-0b07d06cea98","Type":"ContainerStarted","Data":"89af4debfcaab67c79e1f7749123c040e5cae7499105ddd86ee57ac33ca155e2"}
Oct 30 19:58:24.718062 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:58:24.717943    4892 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-b0c827-npkmz" podStartSLOduration=1.717929246 podStartE2EDuration="1.717929246s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:58:23 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:58:24.716097793 +0000 UTC m=+2552.286563392" watchObservedRunningTime="2026-10-30 19:58:24.717929246 +0000 UTC m=+2552.288394844"
Oct 30 19:58:24.787281 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:58:24.787253    4892 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-mt-dd9b8d79-openshift-default-857b46944f-4zd2h"]
Oct 30 19:58:24.789729 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:58:24.789695    4892 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892000Ki","pods":"250"}
Oct 30 19:58:24.789865 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:58:24.789770    4892 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892000Ki","pods":"250"}
Oct 30 19:58:24.789865 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:58:24.789695    4892 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892000Ki","pods":"250"}
Oct 30 19:58:24.789865 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:58:24.789807    4892 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892000Ki","pods":"250"}
Oct 30 19:58:24.790094 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:58:24.789860    4892 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892000Ki","pods":"250"}
Oct 30 19:58:24.790094 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:58:24.789907    4892 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892000Ki","pods":"250"}
Oct 30 19:58:24.793470 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:58:24.793445    4892 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-mt-dd9b8d79-openshift-default-857b46944f-5wjxt"]
Oct 30 19:58:24.857072 ip-10-0-132-82 kubenswrapper[4892]: E1030 19:58:24.857037    4892 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-jrm87" containerName="extractor"
Oct 30 19:58:25.608780 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:58:25.603769    4892 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-dd9b8d79-openshift-default-857b46944f-5wjxt" event={"ID":"0d3f37ec-9b98-4a8c-a38c-d02ffd5417b1","Type":"ContainerStarted","Data":"8362b168aab1751b927549e61df4fbf77947cd9d1642f02367196ef1356ac97b"}
Oct 30 19:58:25.608780 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:58:25.603839    4892 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-dd9b8d79-openshift-default-857b46944f-5wjxt" event={"ID":"0d3f37ec-9b98-4a8c-a38c-d02ffd5417b1","Type":"ContainerStarted","Data":"2959d833a96babb3ad079c1ec4102ffeaaad3744c4e0cb4254b1e5157b215f02"}
Oct 30 19:58:25.608780 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:58:25.607840    4892 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-dd9b8d79-openshift-default-857b46944f-4zd2h" event={"ID":"332cc10b-8e86-49c3-8381-d48fa8bd4782","Type":"ContainerStarted","Data":"4696f8f805d751f4d0130504b60b6a5b82a6f0d4157e02818fdaf7fdb041d9e6"}
Oct 30 19:58:25.608780 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:58:25.607912    4892 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-dd9b8d79-openshift-default-857b46944f-4zd2h" event={"ID":"332cc10b-8e86-49c3-8381-d48fa8bd4782","Type":"ContainerStarted","Data":"df828233221d53ec7aedb240efaa7c9d2085b6789d78a26e771f531dfb8d943d"}
Oct 30 19:58:25.613220 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:58:25.611162    4892 generic.go:356] "Generic (PLEG): container finished" podID="6f18d694-a7ca-449e-8ba5-0b07d06cea98" containerID="4c714a6cddb76596b4c14a02ef169db62458392f8d937ca7941d018953404f73" exitCode=0
Oct 30 19:58:25.613220 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:58:25.611349    4892 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-b0c827-npkmz" event={"ID":"6f18d694-a7ca-449e-8ba5-0b07d06cea98","Type":"ContainerDied","Data":"4c714a6cddb76596b4c14a02ef169db62458392f8d937ca7941d018953404f73"}
Oct 30 19:58:25.756988 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:58:25.747473    4892 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-mt-dd9b8d79-openshift-default-857b46944f-5wjxt" podStartSLOduration=2.747451226 podStartE2EDuration="2.747451226s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:58:23 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:58:25.734468147 +0000 UTC m=+2553.304933747" watchObservedRunningTime="2026-10-30 19:58:25.747451226 +0000 UTC m=+2553.317916826"
Oct 30 19:58:26.003767 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:58:26.003700    4892 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-mt-dd9b8d79-openshift-default-857b46944f-4zd2h" podStartSLOduration=3.003663087 podStartE2EDuration="3.003663087s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:58:23 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:58:26.002547481 +0000 UTC m=+2553.573013082" watchObservedRunningTime="2026-10-30 19:58:26.003663087 +0000 UTC m=+2553.574128692"
Oct 30 19:58:26.336135 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:58:26.336035    4892 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-mt-dd9b8d79-openshift-default-857b46944f-4zd2h"
Oct 30 19:58:26.343988 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:58:26.340067    4892 patch_prober.go:29] interesting pod/e2e-mt-dd9b8d79-openshift-default-857b46944f-4zd2h container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.219:15021/healthz/ready\": dial tcp 10.132.0.219:15021: connect: connection refused" start-of-body=
Oct 30 19:58:26.343988 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:58:26.340129    4892 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-dd9b8d79-openshift-default-857b46944f-4zd2h" podUID="332cc10b-8e86-49c3-8381-d48fa8bd4782" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.219:15021/healthz/ready\": dial tcp 10.132.0.219:15021: connect: connection refused"
Oct 30 19:58:26.354987 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:58:26.351339    4892 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-mt-dd9b8d79-openshift-default-857b46944f-5wjxt"
Oct 30 19:58:26.359990 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:58:26.359078    4892 patch_prober.go:29] interesting pod/e2e-mt-dd9b8d79-openshift-default-857b46944f-5wjxt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.220:15021/healthz/ready\": dial tcp 10.132.0.220:15021: connect: connection refused" start-of-body=
Oct 30 19:58:26.359990 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:58:26.359138    4892 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-dd9b8d79-openshift-default-857b46944f-5wjxt" podUID="0d3f37ec-9b98-4a8c-a38c-d02ffd5417b1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.220:15021/healthz/ready\": dial tcp 10.132.0.220:15021: connect: connection refused"
Oct 30 19:58:26.958001 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:58:26.955286    4892 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-b0c827-npkmz"
Oct 30 19:58:27.019051 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:58:27.017786    4892 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/6f18d694-a7ca-449e-8ba5-0b07d06cea98-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6f18d694-a7ca-449e-8ba5-0b07d06cea98-maas-api-openshift-ca\") pod \"6f18d694-a7ca-449e-8ba5-0b07d06cea98\" (UID: \"6f18d694-a7ca-449e-8ba5-0b07d06cea98\") "
Oct 30 19:58:27.019051 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:58:27.017887    4892 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6f18d694-a7ca-449e-8ba5-0b07d06cea98-kube-api-access-7b6b4\" (UniqueName: \"kubernetes.io/projected/6f18d694-a7ca-449e-8ba5-0b07d06cea98-kube-api-access-7b6b4\") pod \"6f18d694-a7ca-449e-8ba5-0b07d06cea98\" (UID: \"6f18d694-a7ca-449e-8ba5-0b07d06cea98\") "
Oct 30 19:58:27.019051 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:58:27.017955    4892 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/6f18d694-a7ca-449e-8ba5-0b07d06cea98-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6f18d694-a7ca-449e-8ba5-0b07d06cea98-maas-api-xks-ca\") pod \"6f18d694-a7ca-449e-8ba5-0b07d06cea98\" (UID: \"6f18d694-a7ca-449e-8ba5-0b07d06cea98\") "
Oct 30 19:58:27.019051 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:58:27.018680    4892 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6f18d694-a7ca-449e-8ba5-0b07d06cea98-maas-api-openshift-ca" pod "6f18d694-a7ca-449e-8ba5-0b07d06cea98" (UID: "6f18d694-a7ca-449e-8ba5-0b07d06cea98"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 19:58:27.020837 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:58:27.020804    4892 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6f18d694-a7ca-449e-8ba5-0b07d06cea98-kube-api-access-7b6b4" pod "6f18d694-a7ca-449e-8ba5-0b07d06cea98" (UID: "6f18d694-a7ca-449e-8ba5-0b07d06cea98"). InnerVolumeSpecName "kube-api-access-7b6b4". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:58:27.021010 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:58:27.020998    4892 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6f18d694-a7ca-449e-8ba5-0b07d06cea98-maas-api-xks-ca" pod "6f18d694-a7ca-449e-8ba5-0b07d06cea98" (UID: "6f18d694-a7ca-449e-8ba5-0b07d06cea98"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 19:58:27.119006 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:58:27.118951    4892 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6f18d694-a7ca-449e-8ba5-0b07d06cea98-maas-api-openshift-ca\") on node \"ip-10-0-132-82.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 19:58:27.119206 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:58:27.119194    4892 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7b6b4\" (UniqueName: \"kubernetes.io/projected/6f18d694-a7ca-449e-8ba5-0b07d06cea98-kube-api-access-7b6b4\") on node \"ip-10-0-132-82.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 19:58:27.119274 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:58:27.119266    4892 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6f18d694-a7ca-449e-8ba5-0b07d06cea98-maas-api-xks-ca\") on node \"ip-10-0-132-82.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 19:58:27.335050 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:58:27.333409    4892 patch_prober.go:29] interesting pod/e2e-mt-dd9b8d79-openshift-default-857b46944f-4zd2h container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.219:15021/healthz/ready\": dial tcp 10.132.0.219:15021: connect: connection refused" start-of-body=
Oct 30 19:58:27.335050 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:58:27.333486    4892 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-dd9b8d79-openshift-default-857b46944f-4zd2h" podUID="332cc10b-8e86-49c3-8381-d48fa8bd4782" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.219:15021/healthz/ready\": dial tcp 10.132.0.219:15021: connect: connection refused"
Oct 30 19:58:27.354995 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:58:27.354396    4892 patch_prober.go:29] interesting pod/e2e-mt-dd9b8d79-openshift-default-857b46944f-5wjxt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.220:15021/healthz/ready\": dial tcp 10.132.0.220:15021: connect: connection refused" start-of-body=
Oct 30 19:58:27.354995 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:58:27.354466    4892 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-dd9b8d79-openshift-default-857b46944f-5wjxt" podUID="0d3f37ec-9b98-4a8c-a38c-d02ffd5417b1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.220:15021/healthz/ready\": dial tcp 10.132.0.220:15021: connect: connection refused"
Oct 30 19:58:27.626788 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:58:27.626714    4892 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-b0c827-npkmz"
Oct 30 19:58:27.626788 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:58:27.626747    4892 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-b0c827-npkmz" event={"ID":"6f18d694-a7ca-449e-8ba5-0b07d06cea98","Type":"ContainerDied","Data":"89af4debfcaab67c79e1f7749123c040e5cae7499105ddd86ee57ac33ca155e2"}
Oct 30 19:58:27.627055 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:58:27.626796    4892 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="89af4debfcaab67c79e1f7749123c040e5cae7499105ddd86ee57ac33ca155e2"
Oct 30 19:58:27.778992 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:58:27.775579    4892 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w1-d52df0/distinct-2-w1-d52df0-kserve-7846558464-xrkdc" podUID="9c614ab6-30e2-4fe1-8609-d5e2dadc198a" containerName="main" containerID="cri-o://b228e7b15dedc2b43fcd3cfd5c1fadc04c9de4564d3fe81f8d34d8831242a1a9" gracePeriod=45
Oct 30 19:58:27.793941 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:58:27.793488    4892 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w1-d52df0/simulator-w1-d52df0-kserve-67f74bc577-q5bjz" podUID="81b7bb84-1003-4452-b7e0-b4aaba6fefa4" containerName="main" containerID="cri-o://0b4b42a7b4b61a403cc70c8100f05eff14970c55a7a43951f2c8d8ecda92219c" gracePeriod=45
Oct 30 19:58:28.396008 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:58:28.342112    4892 patch_prober.go:29] interesting pod/e2e-mt-dd9b8d79-openshift-default-857b46944f-4zd2h container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.219:15021/healthz/ready\": dial tcp 10.132.0.219:15021: connect: connection refused" start-of-body=
Oct 30 19:58:28.412992 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:58:28.398339    4892 patch_prober.go:29] interesting pod/e2e-mt-dd9b8d79-openshift-default-857b46944f-5wjxt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.220:15021/healthz/ready\": dial tcp 10.132.0.220:15021: connect: connection refused" start-of-body=
Oct 30 19:58:28.412992 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:58:28.398399    4892 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-dd9b8d79-openshift-default-857b46944f-5wjxt" podUID="0d3f37ec-9b98-4a8c-a38c-d02ffd5417b1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.220:15021/healthz/ready\": dial tcp 10.132.0.220:15021: connect: connection refused"
Oct 30 19:58:28.412992 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:58:28.350030    4892 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-dd9b8d79-openshift-default-857b46944f-4zd2h" podUID="332cc10b-8e86-49c3-8381-d48fa8bd4782" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.219:15021/healthz/ready\": dial tcp 10.132.0.219:15021: connect: connection refused"
Oct 30 19:58:28.523482 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:58:28.523451    4892 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w1-d52df0/simulator-w1-d52df0-kserve-67f74bc577-q5bjz"
Oct 30 19:58:28.526198 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:58:28.525601    4892 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w1-d52df0/distinct-2-w1-d52df0-kserve-7846558464-xrkdc"
Oct 30 19:58:28.620033 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:58:28.619949    4892 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/81b7bb84-1003-4452-b7e0-b4aaba6fefa4-dshm\" (UniqueName: \"kubernetes.io/empty-dir/81b7bb84-1003-4452-b7e0-b4aaba6fefa4-dshm\") pod \"81b7bb84-1003-4452-b7e0-b4aaba6fefa4\" (UID: \"81b7bb84-1003-4452-b7e0-b4aaba6fefa4\") "
Oct 30 19:58:28.620195 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:58:28.620088    4892 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/81b7bb84-1003-4452-b7e0-b4aaba6fefa4-kube-api-access-l2tmb\" (UniqueName: \"kubernetes.io/projected/81b7bb84-1003-4452-b7e0-b4aaba6fefa4-kube-api-access-l2tmb\") pod \"81b7bb84-1003-4452-b7e0-b4aaba6fefa4\" (UID: \"81b7bb84-1003-4452-b7e0-b4aaba6fefa4\") "
Oct 30 19:58:28.620195 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:58:28.620162    4892 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9c614ab6-30e2-4fe1-8609-d5e2dadc198a-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9c614ab6-30e2-4fe1-8609-d5e2dadc198a-tmp-dir\") pod \"9c614ab6-30e2-4fe1-8609-d5e2dadc198a\" (UID: \"9c614ab6-30e2-4fe1-8609-d5e2dadc198a\") "
Oct 30 19:58:28.620195 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:58:28.620188    4892 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/81b7bb84-1003-4452-b7e0-b4aaba6fefa4-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/81b7bb84-1003-4452-b7e0-b4aaba6fefa4-tmp-dir\") pod \"81b7bb84-1003-4452-b7e0-b4aaba6fefa4\" (UID: \"81b7bb84-1003-4452-b7e0-b4aaba6fefa4\") "
Oct 30 19:58:28.620301 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:58:28.620250    4892 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9c614ab6-30e2-4fe1-8609-d5e2dadc198a-home\" (UniqueName: \"kubernetes.io/empty-dir/9c614ab6-30e2-4fe1-8609-d5e2dadc198a-home\") pod \"9c614ab6-30e2-4fe1-8609-d5e2dadc198a\" (UID: \"9c614ab6-30e2-4fe1-8609-d5e2dadc198a\") "
Oct 30 19:58:28.620336 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:58:28.620305    4892 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/81b7bb84-1003-4452-b7e0-b4aaba6fefa4-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/81b7bb84-1003-4452-b7e0-b4aaba6fefa4-model-cache\") pod \"81b7bb84-1003-4452-b7e0-b4aaba6fefa4\" (UID: \"81b7bb84-1003-4452-b7e0-b4aaba6fefa4\") "
Oct 30 19:58:28.620336 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:58:28.620324    4892 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/9c614ab6-30e2-4fe1-8609-d5e2dadc198a-tls-certs\" (UniqueName: \"kubernetes.io/secret/9c614ab6-30e2-4fe1-8609-d5e2dadc198a-tls-certs\") pod \"9c614ab6-30e2-4fe1-8609-d5e2dadc198a\" (UID: \"9c614ab6-30e2-4fe1-8609-d5e2dadc198a\") "
Oct 30 19:58:28.620430 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:58:28.620417    4892 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/81b7bb84-1003-4452-b7e0-b4aaba6fefa4-home\" (UniqueName: \"kubernetes.io/empty-dir/81b7bb84-1003-4452-b7e0-b4aaba6fefa4-home\") pod \"81b7bb84-1003-4452-b7e0-b4aaba6fefa4\" (UID: \"81b7bb84-1003-4452-b7e0-b4aaba6fefa4\") "
Oct 30 19:58:28.620467 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:58:28.620444    4892 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9c614ab6-30e2-4fe1-8609-d5e2dadc198a-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/9c614ab6-30e2-4fe1-8609-d5e2dadc198a-model-cache\") pod \"9c614ab6-30e2-4fe1-8609-d5e2dadc198a\" (UID: \"9c614ab6-30e2-4fe1-8609-d5e2dadc198a\") "
Oct 30 19:58:28.620505 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:58:28.620492    4892 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9c614ab6-30e2-4fe1-8609-d5e2dadc198a-kube-api-access-hw5fw\" (UniqueName: \"kubernetes.io/projected/9c614ab6-30e2-4fe1-8609-d5e2dadc198a-kube-api-access-hw5fw\") pod \"9c614ab6-30e2-4fe1-8609-d5e2dadc198a\" (UID: \"9c614ab6-30e2-4fe1-8609-d5e2dadc198a\") "
Oct 30 19:58:28.620540 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:58:28.620513    4892 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/81b7bb84-1003-4452-b7e0-b4aaba6fefa4-tls-certs\" (UniqueName: \"kubernetes.io/secret/81b7bb84-1003-4452-b7e0-b4aaba6fefa4-tls-certs\") pod \"81b7bb84-1003-4452-b7e0-b4aaba6fefa4\" (UID: \"81b7bb84-1003-4452-b7e0-b4aaba6fefa4\") "
Oct 30 19:58:28.620540 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:58:28.620529    4892 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9c614ab6-30e2-4fe1-8609-d5e2dadc198a-dshm\" (UniqueName: \"kubernetes.io/empty-dir/9c614ab6-30e2-4fe1-8609-d5e2dadc198a-dshm\") pod \"9c614ab6-30e2-4fe1-8609-d5e2dadc198a\" (UID: \"9c614ab6-30e2-4fe1-8609-d5e2dadc198a\") "
Oct 30 19:58:28.621230 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:58:28.621207    4892 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/81b7bb84-1003-4452-b7e0-b4aaba6fefa4-model-cache" pod "81b7bb84-1003-4452-b7e0-b4aaba6fefa4" (UID: "81b7bb84-1003-4452-b7e0-b4aaba6fefa4"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:58:28.621382 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:58:28.621354    4892 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9c614ab6-30e2-4fe1-8609-d5e2dadc198a-tmp-dir" pod "9c614ab6-30e2-4fe1-8609-d5e2dadc198a" (UID: "9c614ab6-30e2-4fe1-8609-d5e2dadc198a"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:58:28.621494 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:58:28.621470    4892 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9c614ab6-30e2-4fe1-8609-d5e2dadc198a-home" pod "9c614ab6-30e2-4fe1-8609-d5e2dadc198a" (UID: "9c614ab6-30e2-4fe1-8609-d5e2dadc198a"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:58:28.621909 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:58:28.621680    4892 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9c614ab6-30e2-4fe1-8609-d5e2dadc198a-model-cache" pod "9c614ab6-30e2-4fe1-8609-d5e2dadc198a" (UID: "9c614ab6-30e2-4fe1-8609-d5e2dadc198a"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:58:28.621909 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:58:28.621689    4892 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/81b7bb84-1003-4452-b7e0-b4aaba6fefa4-home" pod "81b7bb84-1003-4452-b7e0-b4aaba6fefa4" (UID: "81b7bb84-1003-4452-b7e0-b4aaba6fefa4"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:58:28.621909 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:58:28.621811    4892 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/81b7bb84-1003-4452-b7e0-b4aaba6fefa4-tmp-dir" pod "81b7bb84-1003-4452-b7e0-b4aaba6fefa4" (UID: "81b7bb84-1003-4452-b7e0-b4aaba6fefa4"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:58:28.623847 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:58:28.623826    4892 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/81b7bb84-1003-4452-b7e0-b4aaba6fefa4-dshm" pod "81b7bb84-1003-4452-b7e0-b4aaba6fefa4" (UID: "81b7bb84-1003-4452-b7e0-b4aaba6fefa4"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:58:28.624256 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:58:28.624208    4892 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/81b7bb84-1003-4452-b7e0-b4aaba6fefa4-kube-api-access-l2tmb" pod "81b7bb84-1003-4452-b7e0-b4aaba6fefa4" (UID: "81b7bb84-1003-4452-b7e0-b4aaba6fefa4"). InnerVolumeSpecName "kube-api-access-l2tmb". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:58:28.624335 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:58:28.624308    4892 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9c614ab6-30e2-4fe1-8609-d5e2dadc198a-tls-certs" pod "9c614ab6-30e2-4fe1-8609-d5e2dadc198a" (UID: "9c614ab6-30e2-4fe1-8609-d5e2dadc198a"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 19:58:28.624552 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:58:28.624537    4892 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9c614ab6-30e2-4fe1-8609-d5e2dadc198a-dshm" pod "9c614ab6-30e2-4fe1-8609-d5e2dadc198a" (UID: "9c614ab6-30e2-4fe1-8609-d5e2dadc198a"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:58:28.624627 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:58:28.624582    4892 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9c614ab6-30e2-4fe1-8609-d5e2dadc198a-kube-api-access-hw5fw" pod "9c614ab6-30e2-4fe1-8609-d5e2dadc198a" (UID: "9c614ab6-30e2-4fe1-8609-d5e2dadc198a"). InnerVolumeSpecName "kube-api-access-hw5fw". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:58:28.625873 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:58:28.625855    4892 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/81b7bb84-1003-4452-b7e0-b4aaba6fefa4-tls-certs" pod "81b7bb84-1003-4452-b7e0-b4aaba6fefa4" (UID: "81b7bb84-1003-4452-b7e0-b4aaba6fefa4"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 19:58:28.713128 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:58:28.711057    4892 generic.go:356] "Generic (PLEG): container finished" podID="81b7bb84-1003-4452-b7e0-b4aaba6fefa4" containerID="0b4b42a7b4b61a403cc70c8100f05eff14970c55a7a43951f2c8d8ecda92219c" exitCode=0
Oct 30 19:58:28.713128 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:58:28.711196    4892 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w1-d52df0/simulator-w1-d52df0-kserve-67f74bc577-q5bjz" event={"ID":"81b7bb84-1003-4452-b7e0-b4aaba6fefa4","Type":"ContainerDied","Data":"0b4b42a7b4b61a403cc70c8100f05eff14970c55a7a43951f2c8d8ecda92219c"}
Oct 30 19:58:28.713128 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:58:28.711245    4892 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w1-d52df0/simulator-w1-d52df0-kserve-67f74bc577-q5bjz" event={"ID":"81b7bb84-1003-4452-b7e0-b4aaba6fefa4","Type":"ContainerDied","Data":"def8789d082e6d6ef7854065da675231870b8fb10903ad1f890cebcec9aead50"}
Oct 30 19:58:28.713128 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:58:28.711268    4892 scope.go:122] "RemoveContainer" containerID="0b4b42a7b4b61a403cc70c8100f05eff14970c55a7a43951f2c8d8ecda92219c"
Oct 30 19:58:28.713128 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:58:28.711424    4892 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w1-d52df0/simulator-w1-d52df0-kserve-67f74bc577-q5bjz"
Oct 30 19:58:28.716457 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:58:28.716416    4892 generic.go:356] "Generic (PLEG): container finished" podID="9c614ab6-30e2-4fe1-8609-d5e2dadc198a" containerID="b228e7b15dedc2b43fcd3cfd5c1fadc04c9de4564d3fe81f8d34d8831242a1a9" exitCode=0
Oct 30 19:58:28.716589 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:58:28.716503    4892 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w1-d52df0/distinct-2-w1-d52df0-kserve-7846558464-xrkdc" event={"ID":"9c614ab6-30e2-4fe1-8609-d5e2dadc198a","Type":"ContainerDied","Data":"b228e7b15dedc2b43fcd3cfd5c1fadc04c9de4564d3fe81f8d34d8831242a1a9"}
Oct 30 19:58:28.716589 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:58:28.716526    4892 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w1-d52df0/distinct-2-w1-d52df0-kserve-7846558464-xrkdc" event={"ID":"9c614ab6-30e2-4fe1-8609-d5e2dadc198a","Type":"ContainerDied","Data":"1bab29e42ab285badb43d510b55ed3f68d2ae2b109925a479c8d5ad9c3828d6c"}
Oct 30 19:58:28.716589 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:58:28.716572    4892 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w1-d52df0/distinct-2-w1-d52df0-kserve-7846558464-xrkdc"
Oct 30 19:58:28.725594 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:58:28.724156    4892 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/81b7bb84-1003-4452-b7e0-b4aaba6fefa4-tmp-dir\") on node \"ip-10-0-132-82.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 19:58:28.725594 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:58:28.724179    4892 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/9c614ab6-30e2-4fe1-8609-d5e2dadc198a-home\") on node \"ip-10-0-132-82.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 19:58:28.725594 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:58:28.724192    4892 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/81b7bb84-1003-4452-b7e0-b4aaba6fefa4-model-cache\") on node \"ip-10-0-132-82.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 19:58:28.725594 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:58:28.724206    4892 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/9c614ab6-30e2-4fe1-8609-d5e2dadc198a-tls-certs\") on node \"ip-10-0-132-82.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 19:58:28.725594 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:58:28.724219    4892 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/81b7bb84-1003-4452-b7e0-b4aaba6fefa4-home\") on node \"ip-10-0-132-82.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 19:58:28.725594 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:58:28.724232    4892 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/9c614ab6-30e2-4fe1-8609-d5e2dadc198a-model-cache\") on node \"ip-10-0-132-82.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 19:58:28.725594 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:58:28.724248    4892 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hw5fw\" (UniqueName: \"kubernetes.io/projected/9c614ab6-30e2-4fe1-8609-d5e2dadc198a-kube-api-access-hw5fw\") on node \"ip-10-0-132-82.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 19:58:28.725594 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:58:28.724264    4892 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/81b7bb84-1003-4452-b7e0-b4aaba6fefa4-tls-certs\") on node \"ip-10-0-132-82.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 19:58:28.725594 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:58:28.724279    4892 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/9c614ab6-30e2-4fe1-8609-d5e2dadc198a-dshm\") on node \"ip-10-0-132-82.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 19:58:28.725594 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:58:28.724291    4892 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/81b7bb84-1003-4452-b7e0-b4aaba6fefa4-dshm\") on node \"ip-10-0-132-82.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 19:58:28.725594 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:58:28.724304    4892 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-l2tmb\" (UniqueName: \"kubernetes.io/projected/81b7bb84-1003-4452-b7e0-b4aaba6fefa4-kube-api-access-l2tmb\") on node \"ip-10-0-132-82.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 19:58:28.725594 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:58:28.724318    4892 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9c614ab6-30e2-4fe1-8609-d5e2dadc198a-tmp-dir\") on node \"ip-10-0-132-82.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 19:58:28.815485 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:58:28.815365    4892 scope.go:122] "RemoveContainer" containerID="0b4b42a7b4b61a403cc70c8100f05eff14970c55a7a43951f2c8d8ecda92219c"
Oct 30 19:58:28.816455 ip-10-0-132-82 kubenswrapper[4892]: E1030 19:58:28.816423    4892 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0b4b42a7b4b61a403cc70c8100f05eff14970c55a7a43951f2c8d8ecda92219c\": container with ID starting with 0b4b42a7b4b61a403cc70c8100f05eff14970c55a7a43951f2c8d8ecda92219c not found: ID does not exist" containerID="0b4b42a7b4b61a403cc70c8100f05eff14970c55a7a43951f2c8d8ecda92219c"
Oct 30 19:58:28.816604 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:58:28.816464    4892 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0b4b42a7b4b61a403cc70c8100f05eff14970c55a7a43951f2c8d8ecda92219c"} err="failed to get container status \"0b4b42a7b4b61a403cc70c8100f05eff14970c55a7a43951f2c8d8ecda92219c\": rpc error: code = NotFound desc = could not find container \"0b4b42a7b4b61a403cc70c8100f05eff14970c55a7a43951f2c8d8ecda92219c\": container with ID starting with 0b4b42a7b4b61a403cc70c8100f05eff14970c55a7a43951f2c8d8ecda92219c not found: ID does not exist"
Oct 30 19:58:28.816604 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:58:28.816494    4892 scope.go:122] "RemoveContainer" containerID="b228e7b15dedc2b43fcd3cfd5c1fadc04c9de4564d3fe81f8d34d8831242a1a9"
Oct 30 19:58:28.909384 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:58:28.909360    4892 scope.go:122] "RemoveContainer" containerID="b228e7b15dedc2b43fcd3cfd5c1fadc04c9de4564d3fe81f8d34d8831242a1a9"
Oct 30 19:58:28.909663 ip-10-0-132-82 kubenswrapper[4892]: E1030 19:58:28.909643    4892 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b228e7b15dedc2b43fcd3cfd5c1fadc04c9de4564d3fe81f8d34d8831242a1a9\": container with ID starting with b228e7b15dedc2b43fcd3cfd5c1fadc04c9de4564d3fe81f8d34d8831242a1a9 not found: ID does not exist" containerID="b228e7b15dedc2b43fcd3cfd5c1fadc04c9de4564d3fe81f8d34d8831242a1a9"
Oct 30 19:58:28.909761 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:58:28.909670    4892 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b228e7b15dedc2b43fcd3cfd5c1fadc04c9de4564d3fe81f8d34d8831242a1a9"} err="failed to get container status \"b228e7b15dedc2b43fcd3cfd5c1fadc04c9de4564d3fe81f8d34d8831242a1a9\": rpc error: code = NotFound desc = could not find container \"b228e7b15dedc2b43fcd3cfd5c1fadc04c9de4564d3fe81f8d34d8831242a1a9\": container with ID starting with b228e7b15dedc2b43fcd3cfd5c1fadc04c9de4564d3fe81f8d34d8831242a1a9 not found: ID does not exist"
Oct 30 19:58:29.007925 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:58:29.007837    4892 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w1-d52df0/simulator-w1-d52df0-kserve-67f74bc577-q5bjz"]
Oct 30 19:58:29.011906 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:58:29.011881    4892 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w1-d52df0/simulator-w1-d52df0-kserve-67f74bc577-q5bjz"]
Oct 30 19:58:29.105156 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:58:29.105107    4892 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-d52df0/simulator-w1-d52df0-kserve-67f74bc577-q5bjz" podUID="81b7bb84-1003-4452-b7e0-b4aaba6fefa4" containerName="main" probeResult="failure" output="Get \"https://10.132.0.57:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:58:29.112997 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:58:29.112950    4892 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w1-d52df0/distinct-2-w1-d52df0-kserve-7846558464-xrkdc"]
Oct 30 19:58:29.117004 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:58:29.116683    4892 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w1-d52df0/distinct-2-w1-d52df0-kserve-7846558464-xrkdc"]
Oct 30 19:58:29.335044 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:58:29.333279    4892 patch_prober.go:29] interesting pod/e2e-mt-dd9b8d79-openshift-default-857b46944f-4zd2h container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.219:15021/healthz/ready\": dial tcp 10.132.0.219:15021: connect: connection refused" start-of-body=
Oct 30 19:58:29.335044 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:58:29.333334    4892 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-dd9b8d79-openshift-default-857b46944f-4zd2h" podUID="332cc10b-8e86-49c3-8381-d48fa8bd4782" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.219:15021/healthz/ready\": dial tcp 10.132.0.219:15021: connect: connection refused"
Oct 30 19:58:29.352308 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:58:29.352274    4892 patch_prober.go:29] interesting pod/e2e-mt-dd9b8d79-openshift-default-857b46944f-5wjxt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.220:15021/healthz/ready\": dial tcp 10.132.0.220:15021: connect: connection refused" start-of-body=
Oct 30 19:58:29.352648 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:58:29.352581    4892 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-dd9b8d79-openshift-default-857b46944f-5wjxt" podUID="0d3f37ec-9b98-4a8c-a38c-d02ffd5417b1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.220:15021/healthz/ready\": dial tcp 10.132.0.220:15021: connect: connection refused"
Oct 30 19:58:30.350989 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:58:30.350513    4892 patch_prober.go:29] interesting pod/e2e-mt-dd9b8d79-openshift-default-857b46944f-4zd2h container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.219:15021/healthz/ready\": dial tcp 10.132.0.219:15021: connect: connection refused" start-of-body=
Oct 30 19:58:30.350989 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:58:30.350593    4892 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-dd9b8d79-openshift-default-857b46944f-4zd2h" podUID="332cc10b-8e86-49c3-8381-d48fa8bd4782" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.219:15021/healthz/ready\": dial tcp 10.132.0.219:15021: connect: connection refused"
Oct 30 19:58:30.375212 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:58:30.352338    4892 patch_prober.go:29] interesting pod/e2e-mt-dd9b8d79-openshift-default-857b46944f-5wjxt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.220:15021/healthz/ready\": dial tcp 10.132.0.220:15021: connect: connection refused" start-of-body=
Oct 30 19:58:30.375212 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:58:30.352400    4892 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-dd9b8d79-openshift-default-857b46944f-5wjxt" podUID="0d3f37ec-9b98-4a8c-a38c-d02ffd5417b1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.220:15021/healthz/ready\": dial tcp 10.132.0.220:15021: connect: connection refused"
Oct 30 19:58:30.605563 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:58:30.605472    4892 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="81b7bb84-1003-4452-b7e0-b4aaba6fefa4" path="/var/lib/kubelet/pods/81b7bb84-1003-4452-b7e0-b4aaba6fefa4/volumes"
Oct 30 19:58:30.606319 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:58:30.606293    4892 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="9c614ab6-30e2-4fe1-8609-d5e2dadc198a" path="/var/lib/kubelet/pods/9c614ab6-30e2-4fe1-8609-d5e2dadc198a/volumes"
Oct 30 19:58:31.332312 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:58:31.332220    4892 patch_prober.go:29] interesting pod/e2e-mt-dd9b8d79-openshift-default-857b46944f-4zd2h container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.219:15021/healthz/ready\": dial tcp 10.132.0.219:15021: connect: connection refused" start-of-body=
Oct 30 19:58:31.332312 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:58:31.332289    4892 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-dd9b8d79-openshift-default-857b46944f-4zd2h" podUID="332cc10b-8e86-49c3-8381-d48fa8bd4782" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.219:15021/healthz/ready\": dial tcp 10.132.0.219:15021: connect: connection refused"
Oct 30 19:58:31.350719 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:58:31.350685    4892 patch_prober.go:29] interesting pod/e2e-mt-dd9b8d79-openshift-default-857b46944f-5wjxt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.220:15021/healthz/ready\": dial tcp 10.132.0.220:15021: connect: connection refused" start-of-body=
Oct 30 19:58:31.350875 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:58:31.350729    4892 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-dd9b8d79-openshift-default-857b46944f-5wjxt" podUID="0d3f37ec-9b98-4a8c-a38c-d02ffd5417b1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.220:15021/healthz/ready\": dial tcp 10.132.0.220:15021: connect: connection refused"
Oct 30 19:58:32.337618 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:58:32.337582    4892 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-mt-dd9b8d79-openshift-default-857b46944f-4zd2h"
Oct 30 19:58:32.338127 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:58:32.338077    4892 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-mt-dd9b8d79-openshift-default-857b46944f-4zd2h"
Oct 30 19:58:32.339003 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:58:32.338978    4892 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-mt-dd9b8d79-openshift-default-857b46944f-4zd2h"
Oct 30 19:58:32.356133 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:58:32.356108    4892 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-mt-dd9b8d79-openshift-default-857b46944f-5wjxt"
Oct 30 19:58:32.356427 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:58:32.356409    4892 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-mt-dd9b8d79-openshift-default-857b46944f-5wjxt"
Oct 30 19:58:32.357623 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:58:32.357593    4892 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-mt-dd9b8d79-openshift-default-857b46944f-5wjxt"
Oct 30 19:58:34.861985 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:58:34.861817    4892 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-b0c827-fc5bcbb66-q72h8"]
Oct 30 19:58:34.983761 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:58:34.983718    4892 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-b0c827-7b674bdfbc-vrpw8"]
Oct 30 19:58:36.983040 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:58:36.983002    4892 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-6873e0-6f6mc"]
Oct 30 19:58:37.031763 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:58:37.031725    4892 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-6873e0-6f6mc"]
Oct 30 19:58:37.311907 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:58:37.311825    4892 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-fsfvk"]
Oct 30 19:58:38.571773 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:58:38.571730    4892 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="561a463b-f457-4bc5-bcdc-aa4b520df908" path="/var/lib/kubelet/pods/561a463b-f457-4bc5-bcdc-aa4b520df908/volumes"
Oct 30 19:58:39.863356 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:58:39.863315    4892 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-inf-b-b0c827-fc5bcbb66-q72h8" podUID="e1a27576-9ec0-4d96-9d2f-3e8df58f9852" containerName="payload-processing" containerID="cri-o://c49b7da76d0d4843579b5ea620190e267168a4a845eedfd6472eb277027bdddb" gracePeriod=25
Oct 30 19:58:39.984265 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:58:39.984200    4892 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-b0c827-7b674bdfbc-vrpw8" podUID="d31c06b2-df7e-4d8a-a386-7afc365e6f8a" containerName="payload-pre-processing" containerID="cri-o://03dd2e98a14f38631f9f32f5814418a232586ecd97afd10aa43cd9a56927511a" gracePeriod=25
Oct 30 19:58:40.149158 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:58:40.149126    4892 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-b-b0c827-fc5bcbb66-q72h8"
Oct 30 19:58:40.196124 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:58:40.196091    4892 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e1a27576-9ec0-4d96-9d2f-3e8df58f9852-kube-api-access-mq8bv\" (UniqueName: \"kubernetes.io/projected/e1a27576-9ec0-4d96-9d2f-3e8df58f9852-kube-api-access-mq8bv\") pod \"e1a27576-9ec0-4d96-9d2f-3e8df58f9852\" (UID: \"e1a27576-9ec0-4d96-9d2f-3e8df58f9852\") "
Oct 30 19:58:40.196295 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:58:40.196134    4892 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e1a27576-9ec0-4d96-9d2f-3e8df58f9852-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e1a27576-9ec0-4d96-9d2f-3e8df58f9852-plugins-config-volume\") pod \"e1a27576-9ec0-4d96-9d2f-3e8df58f9852\" (UID: \"e1a27576-9ec0-4d96-9d2f-3e8df58f9852\") "
Oct 30 19:58:40.196549 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:58:40.196519    4892 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e1a27576-9ec0-4d96-9d2f-3e8df58f9852-plugins-config-volume" pod "e1a27576-9ec0-4d96-9d2f-3e8df58f9852" (UID: "e1a27576-9ec0-4d96-9d2f-3e8df58f9852"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 19:58:40.196741 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:58:40.196726    4892 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e1a27576-9ec0-4d96-9d2f-3e8df58f9852-plugins-config-volume\") on node \"ip-10-0-132-82.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 19:58:40.199244 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:58:40.198896    4892 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e1a27576-9ec0-4d96-9d2f-3e8df58f9852-kube-api-access-mq8bv" pod "e1a27576-9ec0-4d96-9d2f-3e8df58f9852" (UID: "e1a27576-9ec0-4d96-9d2f-3e8df58f9852"). InnerVolumeSpecName "kube-api-access-mq8bv". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:58:40.253434 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:58:40.253405    4892 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-b0c827-7b674bdfbc-vrpw8"
Oct 30 19:58:40.297824 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:58:40.297795    4892 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d31c06b2-df7e-4d8a-a386-7afc365e6f8a-kube-api-access-dhqrt\" (UniqueName: \"kubernetes.io/projected/d31c06b2-df7e-4d8a-a386-7afc365e6f8a-kube-api-access-dhqrt\") pod \"d31c06b2-df7e-4d8a-a386-7afc365e6f8a\" (UID: \"d31c06b2-df7e-4d8a-a386-7afc365e6f8a\") "
Oct 30 19:58:40.298047 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:58:40.297944    4892 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d31c06b2-df7e-4d8a-a386-7afc365e6f8a-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d31c06b2-df7e-4d8a-a386-7afc365e6f8a-plugins-config-volume\") pod \"d31c06b2-df7e-4d8a-a386-7afc365e6f8a\" (UID: \"d31c06b2-df7e-4d8a-a386-7afc365e6f8a\") "
Oct 30 19:58:40.298419 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:58:40.298391    4892 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d31c06b2-df7e-4d8a-a386-7afc365e6f8a-plugins-config-volume" pod "d31c06b2-df7e-4d8a-a386-7afc365e6f8a" (UID: "d31c06b2-df7e-4d8a-a386-7afc365e6f8a"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 19:58:40.298508 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:58:40.298458    4892 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mq8bv\" (UniqueName: \"kubernetes.io/projected/e1a27576-9ec0-4d96-9d2f-3e8df58f9852-kube-api-access-mq8bv\") on node \"ip-10-0-132-82.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 19:58:40.300262 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:58:40.300236    4892 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d31c06b2-df7e-4d8a-a386-7afc365e6f8a-kube-api-access-dhqrt" pod "d31c06b2-df7e-4d8a-a386-7afc365e6f8a" (UID: "d31c06b2-df7e-4d8a-a386-7afc365e6f8a"). InnerVolumeSpecName "kube-api-access-dhqrt". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:58:40.399525 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:58:40.399498    4892 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dhqrt\" (UniqueName: \"kubernetes.io/projected/d31c06b2-df7e-4d8a-a386-7afc365e6f8a-kube-api-access-dhqrt\") on node \"ip-10-0-132-82.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 19:58:40.399525 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:58:40.399525    4892 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d31c06b2-df7e-4d8a-a386-7afc365e6f8a-plugins-config-volume\") on node \"ip-10-0-132-82.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 19:58:40.886799 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:58:40.886767    4892 generic.go:356] "Generic (PLEG): container finished" podID="e1a27576-9ec0-4d96-9d2f-3e8df58f9852" containerID="c49b7da76d0d4843579b5ea620190e267168a4a845eedfd6472eb277027bdddb" exitCode=0
Oct 30 19:58:40.887178 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:58:40.886818    4892 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-b-b0c827-fc5bcbb66-q72h8"
Oct 30 19:58:40.887178 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:58:40.886855    4892 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-b-b0c827-fc5bcbb66-q72h8" event={"ID":"e1a27576-9ec0-4d96-9d2f-3e8df58f9852","Type":"ContainerDied","Data":"c49b7da76d0d4843579b5ea620190e267168a4a845eedfd6472eb277027bdddb"}
Oct 30 19:58:40.887178 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:58:40.886916    4892 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-b-b0c827-fc5bcbb66-q72h8" event={"ID":"e1a27576-9ec0-4d96-9d2f-3e8df58f9852","Type":"ContainerDied","Data":"b5b2e1a3a2693af72b5deb767e024d7b6c607efab8730df4dbb3195972cad287"}
Oct 30 19:58:40.887178 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:58:40.886939    4892 scope.go:122] "RemoveContainer" containerID="c49b7da76d0d4843579b5ea620190e267168a4a845eedfd6472eb277027bdddb"
Oct 30 19:58:40.888759 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:58:40.888735    4892 generic.go:356] "Generic (PLEG): container finished" podID="d31c06b2-df7e-4d8a-a386-7afc365e6f8a" containerID="03dd2e98a14f38631f9f32f5814418a232586ecd97afd10aa43cd9a56927511a" exitCode=0
Oct 30 19:58:40.888866 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:58:40.888769    4892 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-b0c827-7b674bdfbc-vrpw8"
Oct 30 19:58:40.888866 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:58:40.888815    4892 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-b0c827-7b674bdfbc-vrpw8" event={"ID":"d31c06b2-df7e-4d8a-a386-7afc365e6f8a","Type":"ContainerDied","Data":"03dd2e98a14f38631f9f32f5814418a232586ecd97afd10aa43cd9a56927511a"}
Oct 30 19:58:40.888866 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:58:40.888858    4892 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-b0c827-7b674bdfbc-vrpw8" event={"ID":"d31c06b2-df7e-4d8a-a386-7afc365e6f8a","Type":"ContainerDied","Data":"415f78bee68aae355bf1e87b04aa60818dc758d8762205de7c5487b487635b84"}
Oct 30 19:58:40.898289 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:58:40.898271    4892 scope.go:122] "RemoveContainer" containerID="c49b7da76d0d4843579b5ea620190e267168a4a845eedfd6472eb277027bdddb"
Oct 30 19:58:40.898549 ip-10-0-132-82 kubenswrapper[4892]: E1030 19:58:40.898530    4892 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c49b7da76d0d4843579b5ea620190e267168a4a845eedfd6472eb277027bdddb\": container with ID starting with c49b7da76d0d4843579b5ea620190e267168a4a845eedfd6472eb277027bdddb not found: ID does not exist" containerID="c49b7da76d0d4843579b5ea620190e267168a4a845eedfd6472eb277027bdddb"
Oct 30 19:58:40.898598 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:58:40.898556    4892 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c49b7da76d0d4843579b5ea620190e267168a4a845eedfd6472eb277027bdddb"} err="failed to get container status \"c49b7da76d0d4843579b5ea620190e267168a4a845eedfd6472eb277027bdddb\": rpc error: code = NotFound desc = could not find container \"c49b7da76d0d4843579b5ea620190e267168a4a845eedfd6472eb277027bdddb\": container with ID starting with c49b7da76d0d4843579b5ea620190e267168a4a845eedfd6472eb277027bdddb not found: ID does not exist"
Oct 30 19:58:40.898598 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:58:40.898573    4892 scope.go:122] "RemoveContainer" containerID="03dd2e98a14f38631f9f32f5814418a232586ecd97afd10aa43cd9a56927511a"
Oct 30 19:58:40.908812 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:58:40.908795    4892 scope.go:122] "RemoveContainer" containerID="03dd2e98a14f38631f9f32f5814418a232586ecd97afd10aa43cd9a56927511a"
Oct 30 19:58:40.909114 ip-10-0-132-82 kubenswrapper[4892]: E1030 19:58:40.909092    4892 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"03dd2e98a14f38631f9f32f5814418a232586ecd97afd10aa43cd9a56927511a\": container with ID starting with 03dd2e98a14f38631f9f32f5814418a232586ecd97afd10aa43cd9a56927511a not found: ID does not exist" containerID="03dd2e98a14f38631f9f32f5814418a232586ecd97afd10aa43cd9a56927511a"
Oct 30 19:58:40.909166 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:58:40.909123    4892 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"03dd2e98a14f38631f9f32f5814418a232586ecd97afd10aa43cd9a56927511a"} err="failed to get container status \"03dd2e98a14f38631f9f32f5814418a232586ecd97afd10aa43cd9a56927511a\": rpc error: code = NotFound desc = could not find container \"03dd2e98a14f38631f9f32f5814418a232586ecd97afd10aa43cd9a56927511a\": container with ID starting with 03dd2e98a14f38631f9f32f5814418a232586ecd97afd10aa43cd9a56927511a not found: ID does not exist"
Oct 30 19:58:41.061418 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:58:41.061386    4892 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-b0c827-7b674bdfbc-vrpw8"]
Oct 30 19:58:41.110568 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:58:41.110539    4892 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-b0c827-7b674bdfbc-vrpw8"]
Oct 30 19:58:41.154358 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:58:41.149556    4892 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-aa5947e6-mzxvf"]
Oct 30 19:58:41.154358 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:58:41.151202    4892 cpu_manager.go:414] "containerMap: removing container" podUID="6f18d694-a7ca-449e-8ba5-0b07d06cea98" containerName="revoke-keys"
Oct 30 19:58:41.154358 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:58:41.151223    4892 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6f18d694-a7ca-449e-8ba5-0b07d06cea98" containerName="revoke-keys"
Oct 30 19:58:41.154358 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:58:41.151247    4892 cpu_manager.go:414] "containerMap: removing container" podUID="e1a27576-9ec0-4d96-9d2f-3e8df58f9852" containerName="payload-processing"
Oct 30 19:58:41.154358 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:58:41.151255    4892 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e1a27576-9ec0-4d96-9d2f-3e8df58f9852" containerName="payload-processing"
Oct 30 19:58:41.154358 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:58:41.151269    4892 cpu_manager.go:414] "containerMap: removing container" podUID="9c614ab6-30e2-4fe1-8609-d5e2dadc198a" containerName="main"
Oct 30 19:58:41.154358 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:58:41.151276    4892 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9c614ab6-30e2-4fe1-8609-d5e2dadc198a" containerName="main"
Oct 30 19:58:41.154358 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:58:41.151312    4892 cpu_manager.go:414] "containerMap: removing container" podUID="d31c06b2-df7e-4d8a-a386-7afc365e6f8a" containerName="payload-pre-processing"
Oct 30 19:58:41.154358 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:58:41.151320    4892 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d31c06b2-df7e-4d8a-a386-7afc365e6f8a" containerName="payload-pre-processing"
Oct 30 19:58:41.154358 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:58:41.151364    4892 cpu_manager.go:414] "containerMap: removing container" podUID="81b7bb84-1003-4452-b7e0-b4aaba6fefa4" containerName="main"
Oct 30 19:58:41.154358 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:58:41.151371    4892 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="81b7bb84-1003-4452-b7e0-b4aaba6fefa4" containerName="main"
Oct 30 19:58:41.154358 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:58:41.151656    4892 memory_manager.go:365] "RemoveStaleState removing state" podUID="6f18d694-a7ca-449e-8ba5-0b07d06cea98" containerName="revoke-keys"
Oct 30 19:58:41.154358 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:58:41.151676    4892 memory_manager.go:365] "RemoveStaleState removing state" podUID="e1a27576-9ec0-4d96-9d2f-3e8df58f9852" containerName="payload-processing"
Oct 30 19:58:41.154358 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:58:41.151695    4892 memory_manager.go:365] "RemoveStaleState removing state" podUID="9c614ab6-30e2-4fe1-8609-d5e2dadc198a" containerName="main"
Oct 30 19:58:41.154358 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:58:41.151706    4892 memory_manager.go:365] "RemoveStaleState removing state" podUID="d31c06b2-df7e-4d8a-a386-7afc365e6f8a" containerName="payload-pre-processing"
Oct 30 19:58:41.154358 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:58:41.151725    4892 memory_manager.go:365] "RemoveStaleState removing state" podUID="81b7bb84-1003-4452-b7e0-b4aaba6fefa4" containerName="main"
Oct 30 19:58:41.158434 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:58:41.158411    4892 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-aa5947e6-mzxvf"
Oct 30 19:58:41.200306 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:58:41.200280    4892 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-4ad5eb9b-czdvj"]
Oct 30 19:58:41.204378 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:58:41.204358    4892 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4ad5eb9b-czdvj"
Oct 30 19:58:41.208819 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:58:41.208791    4892 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a6645cd6-a8c2-42ea-9284-aa7bc3dcac48-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-aa5947e6-mzxvf\" (UID: \"a6645cd6-a8c2-42ea-9284-aa7bc3dcac48\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-aa5947e6-mzxvf"
Oct 30 19:58:41.208953 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:58:41.208931    4892 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a6645cd6-a8c2-42ea-9284-aa7bc3dcac48-maas-api-xks-ca\") pod \"maas-api-revoke-sub-aa5947e6-mzxvf\" (UID: \"a6645cd6-a8c2-42ea-9284-aa7bc3dcac48\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-aa5947e6-mzxvf"
Oct 30 19:58:41.209072 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:58:41.209018    4892 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5jv9s\" (UniqueName: \"kubernetes.io/projected/a6645cd6-a8c2-42ea-9284-aa7bc3dcac48-kube-api-access-5jv9s\") pod \"maas-api-revoke-sub-aa5947e6-mzxvf\" (UID: \"a6645cd6-a8c2-42ea-9284-aa7bc3dcac48\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-aa5947e6-mzxvf"
Oct 30 19:58:41.209592 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:58:41.209574    4892 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-b0c827-fc5bcbb66-q72h8"]
Oct 30 19:58:41.217998 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:58:41.216006    4892 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 19:58:41.217998 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:58:41.216147    4892 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-nlxp4\""
Oct 30 19:58:41.218345 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:58:41.218327    4892 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 30 19:58:41.221982 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:58:41.221077    4892 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-aa5947e6-mzxvf"]
Oct 30 19:58:41.250071 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:58:41.250045    4892 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-b0c827-fc5bcbb66-q72h8"]
Oct 30 19:58:41.264748 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:58:41.264727    4892 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-4ad5eb9b-czdvj"]
Oct 30 19:58:41.310287 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:58:41.310264    4892 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-42qzs\" (UniqueName: \"kubernetes.io/projected/9cea7b8f-5b17-4130-b392-61640b58fa52-kube-api-access-42qzs\") pod \"maas-api-revoke-sub-4ad5eb9b-czdvj\" (UID: \"9cea7b8f-5b17-4130-b392-61640b58fa52\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4ad5eb9b-czdvj"
Oct 30 19:58:41.310417 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:58:41.310376    4892 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9cea7b8f-5b17-4130-b392-61640b58fa52-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-4ad5eb9b-czdvj\" (UID: \"9cea7b8f-5b17-4130-b392-61640b58fa52\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4ad5eb9b-czdvj"
Oct 30 19:58:41.310417 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:58:41.310409    4892 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a6645cd6-a8c2-42ea-9284-aa7bc3dcac48-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-aa5947e6-mzxvf\" (UID: \"a6645cd6-a8c2-42ea-9284-aa7bc3dcac48\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-aa5947e6-mzxvf"
Oct 30 19:58:41.310494 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:58:41.310468    4892 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9cea7b8f-5b17-4130-b392-61640b58fa52-maas-api-xks-ca\") pod \"maas-api-revoke-sub-4ad5eb9b-czdvj\" (UID: \"9cea7b8f-5b17-4130-b392-61640b58fa52\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4ad5eb9b-czdvj"
Oct 30 19:58:41.310528 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:58:41.310510    4892 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a6645cd6-a8c2-42ea-9284-aa7bc3dcac48-maas-api-xks-ca\") pod \"maas-api-revoke-sub-aa5947e6-mzxvf\" (UID: \"a6645cd6-a8c2-42ea-9284-aa7bc3dcac48\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-aa5947e6-mzxvf"
Oct 30 19:58:41.310590 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:58:41.310572    4892 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5jv9s\" (UniqueName: \"kubernetes.io/projected/a6645cd6-a8c2-42ea-9284-aa7bc3dcac48-kube-api-access-5jv9s\") pod \"maas-api-revoke-sub-aa5947e6-mzxvf\" (UID: \"a6645cd6-a8c2-42ea-9284-aa7bc3dcac48\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-aa5947e6-mzxvf"
Oct 30 19:58:41.310985 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:58:41.310945    4892 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a6645cd6-a8c2-42ea-9284-aa7bc3dcac48-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-aa5947e6-mzxvf\" (UID: \"a6645cd6-a8c2-42ea-9284-aa7bc3dcac48\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-aa5947e6-mzxvf"
Oct 30 19:58:41.313040 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:58:41.313021    4892 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a6645cd6-a8c2-42ea-9284-aa7bc3dcac48-maas-api-xks-ca\") pod \"maas-api-revoke-sub-aa5947e6-mzxvf\" (UID: \"a6645cd6-a8c2-42ea-9284-aa7bc3dcac48\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-aa5947e6-mzxvf"
Oct 30 19:58:41.318820 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:58:41.318801    4892 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 30 19:58:41.369497 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:58:41.369474    4892 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5jv9s\" (UniqueName: \"kubernetes.io/projected/a6645cd6-a8c2-42ea-9284-aa7bc3dcac48-kube-api-access-5jv9s\") pod \"maas-api-revoke-sub-aa5947e6-mzxvf\" (UID: \"a6645cd6-a8c2-42ea-9284-aa7bc3dcac48\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-aa5947e6-mzxvf"
Oct 30 19:58:41.412103 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:58:41.412038    4892 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9cea7b8f-5b17-4130-b392-61640b58fa52-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-4ad5eb9b-czdvj\" (UID: \"9cea7b8f-5b17-4130-b392-61640b58fa52\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4ad5eb9b-czdvj"
Oct 30 19:58:41.412103 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:58:41.412081    4892 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9cea7b8f-5b17-4130-b392-61640b58fa52-maas-api-xks-ca\") pod \"maas-api-revoke-sub-4ad5eb9b-czdvj\" (UID: \"9cea7b8f-5b17-4130-b392-61640b58fa52\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4ad5eb9b-czdvj"
Oct 30 19:58:41.412249 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:58:41.412168    4892 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-42qzs\" (UniqueName: \"kubernetes.io/projected/9cea7b8f-5b17-4130-b392-61640b58fa52-kube-api-access-42qzs\") pod \"maas-api-revoke-sub-4ad5eb9b-czdvj\" (UID: \"9cea7b8f-5b17-4130-b392-61640b58fa52\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4ad5eb9b-czdvj"
Oct 30 19:58:41.412738 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:58:41.412719    4892 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9cea7b8f-5b17-4130-b392-61640b58fa52-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-4ad5eb9b-czdvj\" (UID: \"9cea7b8f-5b17-4130-b392-61640b58fa52\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4ad5eb9b-czdvj"
Oct 30 19:58:41.414473 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:58:41.414453    4892 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9cea7b8f-5b17-4130-b392-61640b58fa52-maas-api-xks-ca\") pod \"maas-api-revoke-sub-4ad5eb9b-czdvj\" (UID: \"9cea7b8f-5b17-4130-b392-61640b58fa52\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4ad5eb9b-czdvj"
Oct 30 19:58:41.473092 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:58:41.473069    4892 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-aa5947e6-mzxvf"
Oct 30 19:58:41.474481 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:58:41.474459    4892 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-42qzs\" (UniqueName: \"kubernetes.io/projected/9cea7b8f-5b17-4130-b392-61640b58fa52-kube-api-access-42qzs\") pod \"maas-api-revoke-sub-4ad5eb9b-czdvj\" (UID: \"9cea7b8f-5b17-4130-b392-61640b58fa52\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4ad5eb9b-czdvj"
Oct 30 19:58:41.518685 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:58:41.518428    4892 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4ad5eb9b-czdvj"
Oct 30 19:58:41.868058 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:58:41.868030    4892 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-aa5947e6-mzxvf"]
Oct 30 19:58:41.869811 ip-10-0-132-82 kubenswrapper[4892]: W1030 19:58:41.869779    4892 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda6645cd6_a8c2_42ea_9284_aa7bc3dcac48.slice/crio-a10cd6018f42cc45329a991400e2653fd66b2119dd1d95d3975b0d3c15a6673f WatchSource:0}: Error finding container a10cd6018f42cc45329a991400e2653fd66b2119dd1d95d3975b0d3c15a6673f: Status 404 returned error can't find the container with id a10cd6018f42cc45329a991400e2653fd66b2119dd1d95d3975b0d3c15a6673f
Oct 30 19:58:41.899081 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:58:41.899051    4892 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-aa5947e6-mzxvf" event={"ID":"a6645cd6-a8c2-42ea-9284-aa7bc3dcac48","Type":"ContainerStarted","Data":"a10cd6018f42cc45329a991400e2653fd66b2119dd1d95d3975b0d3c15a6673f"}
Oct 30 19:58:41.939291 ip-10-0-132-82 kubenswrapper[4892]: W1030 19:58:41.939266    4892 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9cea7b8f_5b17_4130_b392_61640b58fa52.slice/crio-8bd0d290c8520cbb340141c93278fb87356b2ac2c462aaa6851444f7f897d2ec WatchSource:0}: Error finding container 8bd0d290c8520cbb340141c93278fb87356b2ac2c462aaa6851444f7f897d2ec: Status 404 returned error can't find the container with id 8bd0d290c8520cbb340141c93278fb87356b2ac2c462aaa6851444f7f897d2ec
Oct 30 19:58:41.940525 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:58:41.940490    4892 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-4ad5eb9b-czdvj"]
Oct 30 19:58:42.570814 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:58:42.570742    4892 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d31c06b2-df7e-4d8a-a386-7afc365e6f8a" path="/var/lib/kubelet/pods/d31c06b2-df7e-4d8a-a386-7afc365e6f8a/volumes"
Oct 30 19:58:42.571173 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:58:42.571156    4892 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e1a27576-9ec0-4d96-9d2f-3e8df58f9852" path="/var/lib/kubelet/pods/e1a27576-9ec0-4d96-9d2f-3e8df58f9852/volumes"
Oct 30 19:58:42.910421 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:58:42.910389    4892 generic.go:356] "Generic (PLEG): container finished" podID="a6645cd6-a8c2-42ea-9284-aa7bc3dcac48" containerID="5ceb7015507274cf3f097cdae638b98af2af71ed4bf059c264bdffa1e54c706a" exitCode=0
Oct 30 19:58:42.910857 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:58:42.910437    4892 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-aa5947e6-mzxvf" event={"ID":"a6645cd6-a8c2-42ea-9284-aa7bc3dcac48","Type":"ContainerDied","Data":"5ceb7015507274cf3f097cdae638b98af2af71ed4bf059c264bdffa1e54c706a"}
Oct 30 19:58:42.912229 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:58:42.912208    4892 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4ad5eb9b-czdvj" event={"ID":"9cea7b8f-5b17-4130-b392-61640b58fa52","Type":"ContainerStarted","Data":"24f73394c85811dcf684ed6d069342beafb02e29b236513cd228bac26990d25c"}
Oct 30 19:58:42.912319 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:58:42.912245    4892 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4ad5eb9b-czdvj" event={"ID":"9cea7b8f-5b17-4130-b392-61640b58fa52","Type":"ContainerStarted","Data":"8bd0d290c8520cbb340141c93278fb87356b2ac2c462aaa6851444f7f897d2ec"}
Oct 30 19:58:43.177799 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:58:43.177736    4892 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4ad5eb9b-czdvj" podStartSLOduration=2.177718077 podStartE2EDuration="2.177718077s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:58:41 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:58:43.175804267 +0000 UTC m=+2570.746269873" watchObservedRunningTime="2026-10-30 19:58:43.177718077 +0000 UTC m=+2570.748183677"
Oct 30 19:58:43.753268 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:58:43.753233    4892 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-dd9b8d79-dlbxt"]
Oct 30 19:58:43.756942 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:58:43.756923    4892 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-dd9b8d79-dlbxt"
Oct 30 19:58:43.819056 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:58:43.819032    4892 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-dd9b8d79-dlbxt"]
Oct 30 19:58:43.845698 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:58:43.845664    4892 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d126159b-6468-4c98-808f-a4243e83f11b-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-mt-dd9b8d79-dlbxt\" (UID: \"d126159b-6468-4c98-808f-a4243e83f11b\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-dd9b8d79-dlbxt"
Oct 30 19:58:43.845912 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:58:43.845885    4892 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d126159b-6468-4c98-808f-a4243e83f11b-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-mt-dd9b8d79-dlbxt\" (UID: \"d126159b-6468-4c98-808f-a4243e83f11b\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-dd9b8d79-dlbxt"
Oct 30 19:58:43.846070 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:58:43.846050    4892 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mw5qr\" (UniqueName: \"kubernetes.io/projected/d126159b-6468-4c98-808f-a4243e83f11b-kube-api-access-mw5qr\") pod \"maas-api-revoke-keys-e2e-mt-dd9b8d79-dlbxt\" (UID: \"d126159b-6468-4c98-808f-a4243e83f11b\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-dd9b8d79-dlbxt"
Oct 30 19:58:43.921948 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:58:43.921913    4892 generic.go:356] "Generic (PLEG): container finished" podID="9cea7b8f-5b17-4130-b392-61640b58fa52" containerID="24f73394c85811dcf684ed6d069342beafb02e29b236513cd228bac26990d25c" exitCode=0
Oct 30 19:58:43.922381 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:58:43.921957    4892 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4ad5eb9b-czdvj" event={"ID":"9cea7b8f-5b17-4130-b392-61640b58fa52","Type":"ContainerDied","Data":"24f73394c85811dcf684ed6d069342beafb02e29b236513cd228bac26990d25c"}
Oct 30 19:58:43.947317 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:58:43.947284    4892 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d126159b-6468-4c98-808f-a4243e83f11b-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-mt-dd9b8d79-dlbxt\" (UID: \"d126159b-6468-4c98-808f-a4243e83f11b\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-dd9b8d79-dlbxt"
Oct 30 19:58:43.947499 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:58:43.947362    4892 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mw5qr\" (UniqueName: \"kubernetes.io/projected/d126159b-6468-4c98-808f-a4243e83f11b-kube-api-access-mw5qr\") pod \"maas-api-revoke-keys-e2e-mt-dd9b8d79-dlbxt\" (UID: \"d126159b-6468-4c98-808f-a4243e83f11b\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-dd9b8d79-dlbxt"
Oct 30 19:58:43.947499 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:58:43.947475    4892 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d126159b-6468-4c98-808f-a4243e83f11b-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-mt-dd9b8d79-dlbxt\" (UID: \"d126159b-6468-4c98-808f-a4243e83f11b\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-dd9b8d79-dlbxt"
Oct 30 19:58:43.948118 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:58:43.948091    4892 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d126159b-6468-4c98-808f-a4243e83f11b-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-mt-dd9b8d79-dlbxt\" (UID: \"d126159b-6468-4c98-808f-a4243e83f11b\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-dd9b8d79-dlbxt"
Oct 30 19:58:43.949782 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:58:43.949761    4892 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d126159b-6468-4c98-808f-a4243e83f11b-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-mt-dd9b8d79-dlbxt\" (UID: \"d126159b-6468-4c98-808f-a4243e83f11b\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-dd9b8d79-dlbxt"
Oct 30 19:58:44.005623 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:58:44.005456    4892 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mw5qr\" (UniqueName: \"kubernetes.io/projected/d126159b-6468-4c98-808f-a4243e83f11b-kube-api-access-mw5qr\") pod \"maas-api-revoke-keys-e2e-mt-dd9b8d79-dlbxt\" (UID: \"d126159b-6468-4c98-808f-a4243e83f11b\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-dd9b8d79-dlbxt"
Oct 30 19:58:44.072122 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:58:44.072099    4892 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-dd9b8d79-dlbxt"
Oct 30 19:58:44.104250 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:58:44.104224    4892 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-aa5947e6-mzxvf"
Oct 30 19:58:44.149490 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:58:44.149462    4892 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a6645cd6-a8c2-42ea-9284-aa7bc3dcac48-kube-api-access-5jv9s\" (UniqueName: \"kubernetes.io/projected/a6645cd6-a8c2-42ea-9284-aa7bc3dcac48-kube-api-access-5jv9s\") pod \"a6645cd6-a8c2-42ea-9284-aa7bc3dcac48\" (UID: \"a6645cd6-a8c2-42ea-9284-aa7bc3dcac48\") "
Oct 30 19:58:44.149626 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:58:44.149514    4892 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/a6645cd6-a8c2-42ea-9284-aa7bc3dcac48-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a6645cd6-a8c2-42ea-9284-aa7bc3dcac48-maas-api-xks-ca\") pod \"a6645cd6-a8c2-42ea-9284-aa7bc3dcac48\" (UID: \"a6645cd6-a8c2-42ea-9284-aa7bc3dcac48\") "
Oct 30 19:58:44.149818 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:58:44.149792    4892 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a6645cd6-a8c2-42ea-9284-aa7bc3dcac48-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a6645cd6-a8c2-42ea-9284-aa7bc3dcac48-maas-api-openshift-ca\") pod \"a6645cd6-a8c2-42ea-9284-aa7bc3dcac48\" (UID: \"a6645cd6-a8c2-42ea-9284-aa7bc3dcac48\") "
Oct 30 19:58:44.150438 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:58:44.150385    4892 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a6645cd6-a8c2-42ea-9284-aa7bc3dcac48-maas-api-openshift-ca" pod "a6645cd6-a8c2-42ea-9284-aa7bc3dcac48" (UID: "a6645cd6-a8c2-42ea-9284-aa7bc3dcac48"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 19:58:44.162015 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:58:44.153363    4892 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a6645cd6-a8c2-42ea-9284-aa7bc3dcac48-kube-api-access-5jv9s" pod "a6645cd6-a8c2-42ea-9284-aa7bc3dcac48" (UID: "a6645cd6-a8c2-42ea-9284-aa7bc3dcac48"). InnerVolumeSpecName "kube-api-access-5jv9s". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:58:44.162015 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:58:44.155175    4892 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a6645cd6-a8c2-42ea-9284-aa7bc3dcac48-maas-api-xks-ca" pod "a6645cd6-a8c2-42ea-9284-aa7bc3dcac48" (UID: "a6645cd6-a8c2-42ea-9284-aa7bc3dcac48"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 19:58:44.252036 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:58:44.252004    4892 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a6645cd6-a8c2-42ea-9284-aa7bc3dcac48-maas-api-openshift-ca\") on node \"ip-10-0-132-82.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 19:58:44.252036 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:58:44.252030    4892 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5jv9s\" (UniqueName: \"kubernetes.io/projected/a6645cd6-a8c2-42ea-9284-aa7bc3dcac48-kube-api-access-5jv9s\") on node \"ip-10-0-132-82.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 19:58:44.252036 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:58:44.252043    4892 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a6645cd6-a8c2-42ea-9284-aa7bc3dcac48-maas-api-xks-ca\") on node \"ip-10-0-132-82.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 19:58:44.458955 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:58:44.458926    4892 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-dd9b8d79-dlbxt"]
Oct 30 19:58:44.460268 ip-10-0-132-82 kubenswrapper[4892]: W1030 19:58:44.460247    4892 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd126159b_6468_4c98_808f_a4243e83f11b.slice/crio-bc5d26f01de46c3a1b9eb84ef18432aeb4ebb6652a82ed2c915d5eaf5228d724 WatchSource:0}: Error finding container bc5d26f01de46c3a1b9eb84ef18432aeb4ebb6652a82ed2c915d5eaf5228d724: Status 404 returned error can't find the container with id bc5d26f01de46c3a1b9eb84ef18432aeb4ebb6652a82ed2c915d5eaf5228d724
Oct 30 19:58:44.931406 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:58:44.931379    4892 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-aa5947e6-mzxvf"
Oct 30 19:58:44.931406 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:58:44.931388    4892 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-aa5947e6-mzxvf" event={"ID":"a6645cd6-a8c2-42ea-9284-aa7bc3dcac48","Type":"ContainerDied","Data":"a10cd6018f42cc45329a991400e2653fd66b2119dd1d95d3975b0d3c15a6673f"}
Oct 30 19:58:44.931926 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:58:44.931431    4892 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="a10cd6018f42cc45329a991400e2653fd66b2119dd1d95d3975b0d3c15a6673f"
Oct 30 19:58:44.933494 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:58:44.933413    4892 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-dd9b8d79-dlbxt" event={"ID":"d126159b-6468-4c98-808f-a4243e83f11b","Type":"ContainerStarted","Data":"671d0bb085a20c8e3af6c8ccaedfe450644082e5b869d96f66122f9a135e06ff"}
Oct 30 19:58:44.933494 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:58:44.933478    4892 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-dd9b8d79-dlbxt" event={"ID":"d126159b-6468-4c98-808f-a4243e83f11b","Type":"ContainerStarted","Data":"bc5d26f01de46c3a1b9eb84ef18432aeb4ebb6652a82ed2c915d5eaf5228d724"}
Oct 30 19:58:45.052279 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:58:45.052229    4892 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-dd9b8d79-dlbxt" podStartSLOduration=2.052213167 podStartE2EDuration="2.052213167s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:58:43 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:58:45.04966233 +0000 UTC m=+2572.620127951" watchObservedRunningTime="2026-10-30 19:58:45.052213167 +0000 UTC m=+2572.622678767"
Oct 30 19:58:45.085774 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:58:45.085750    4892 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4ad5eb9b-czdvj"
Oct 30 19:58:45.162808 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:58:45.162778    4892 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/9cea7b8f-5b17-4130-b392-61640b58fa52-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9cea7b8f-5b17-4130-b392-61640b58fa52-maas-api-openshift-ca\") pod \"9cea7b8f-5b17-4130-b392-61640b58fa52\" (UID: \"9cea7b8f-5b17-4130-b392-61640b58fa52\") "
Oct 30 19:58:45.162993 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:58:45.162863    4892 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/9cea7b8f-5b17-4130-b392-61640b58fa52-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9cea7b8f-5b17-4130-b392-61640b58fa52-maas-api-xks-ca\") pod \"9cea7b8f-5b17-4130-b392-61640b58fa52\" (UID: \"9cea7b8f-5b17-4130-b392-61640b58fa52\") "
Oct 30 19:58:45.163094 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:58:45.163077    4892 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9cea7b8f-5b17-4130-b392-61640b58fa52-kube-api-access-42qzs\" (UniqueName: \"kubernetes.io/projected/9cea7b8f-5b17-4130-b392-61640b58fa52-kube-api-access-42qzs\") pod \"9cea7b8f-5b17-4130-b392-61640b58fa52\" (UID: \"9cea7b8f-5b17-4130-b392-61640b58fa52\") "
Oct 30 19:58:45.163161 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:58:45.163140    4892 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9cea7b8f-5b17-4130-b392-61640b58fa52-maas-api-openshift-ca" pod "9cea7b8f-5b17-4130-b392-61640b58fa52" (UID: "9cea7b8f-5b17-4130-b392-61640b58fa52"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 19:58:45.163441 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:58:45.163422    4892 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9cea7b8f-5b17-4130-b392-61640b58fa52-maas-api-openshift-ca\") on node \"ip-10-0-132-82.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 19:58:45.165277 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:58:45.165256    4892 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9cea7b8f-5b17-4130-b392-61640b58fa52-kube-api-access-42qzs" pod "9cea7b8f-5b17-4130-b392-61640b58fa52" (UID: "9cea7b8f-5b17-4130-b392-61640b58fa52"). InnerVolumeSpecName "kube-api-access-42qzs". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:58:45.165360 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:58:45.165283    4892 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9cea7b8f-5b17-4130-b392-61640b58fa52-maas-api-xks-ca" pod "9cea7b8f-5b17-4130-b392-61640b58fa52" (UID: "9cea7b8f-5b17-4130-b392-61640b58fa52"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 19:58:45.265140 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:58:45.265110    4892 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9cea7b8f-5b17-4130-b392-61640b58fa52-maas-api-xks-ca\") on node \"ip-10-0-132-82.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 19:58:45.265140 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:58:45.265137    4892 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-42qzs\" (UniqueName: \"kubernetes.io/projected/9cea7b8f-5b17-4130-b392-61640b58fa52-kube-api-access-42qzs\") on node \"ip-10-0-132-82.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 19:58:45.943941 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:58:45.943903    4892 generic.go:356] "Generic (PLEG): container finished" podID="d126159b-6468-4c98-808f-a4243e83f11b" containerID="671d0bb085a20c8e3af6c8ccaedfe450644082e5b869d96f66122f9a135e06ff" exitCode=0
Oct 30 19:58:45.944415 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:58:45.944003    4892 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-dd9b8d79-dlbxt" event={"ID":"d126159b-6468-4c98-808f-a4243e83f11b","Type":"ContainerDied","Data":"671d0bb085a20c8e3af6c8ccaedfe450644082e5b869d96f66122f9a135e06ff"}
Oct 30 19:58:45.945807 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:58:45.945788    4892 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4ad5eb9b-czdvj"
Oct 30 19:58:45.945923 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:58:45.945832    4892 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4ad5eb9b-czdvj" event={"ID":"9cea7b8f-5b17-4130-b392-61640b58fa52","Type":"ContainerDied","Data":"8bd0d290c8520cbb340141c93278fb87356b2ac2c462aaa6851444f7f897d2ec"}
Oct 30 19:58:45.945923 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:58:45.945871    4892 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="8bd0d290c8520cbb340141c93278fb87356b2ac2c462aaa6851444f7f897d2ec"
Oct 30 19:58:47.128985 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:58:47.128922    4892 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-75b38d-rhqx2"]
Oct 30 19:58:47.130051 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:58:47.130022    4892 cpu_manager.go:414] "containerMap: removing container" podUID="a6645cd6-a8c2-42ea-9284-aa7bc3dcac48" containerName="revoke-keys"
Oct 30 19:58:47.130051 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:58:47.130049    4892 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a6645cd6-a8c2-42ea-9284-aa7bc3dcac48" containerName="revoke-keys"
Oct 30 19:58:47.130224 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:58:47.130086    4892 cpu_manager.go:414] "containerMap: removing container" podUID="9cea7b8f-5b17-4130-b392-61640b58fa52" containerName="revoke-keys"
Oct 30 19:58:47.130224 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:58:47.130095    4892 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9cea7b8f-5b17-4130-b392-61640b58fa52" containerName="revoke-keys"
Oct 30 19:58:47.130352 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:58:47.130301    4892 memory_manager.go:365] "RemoveStaleState removing state" podUID="a6645cd6-a8c2-42ea-9284-aa7bc3dcac48" containerName="revoke-keys"
Oct 30 19:58:47.130352 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:58:47.130318    4892 memory_manager.go:365] "RemoveStaleState removing state" podUID="9cea7b8f-5b17-4130-b392-61640b58fa52" containerName="revoke-keys"
Oct 30 19:58:47.134606 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:58:47.134584    4892 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-75b38d-rhqx2"
Oct 30 19:58:47.192897 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:58:47.192828    4892 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-db6qf\" (UniqueName: \"kubernetes.io/projected/51ed721a-e066-4f02-bea5-27d6598591bc-kube-api-access-db6qf\") pod \"maas-api-revoke-keys-e2e-inf-b-75b38d-rhqx2\" (UID: \"51ed721a-e066-4f02-bea5-27d6598591bc\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-75b38d-rhqx2"
Oct 30 19:58:47.193380 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:58:47.193299    4892 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/51ed721a-e066-4f02-bea5-27d6598591bc-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-75b38d-rhqx2\" (UID: \"51ed721a-e066-4f02-bea5-27d6598591bc\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-75b38d-rhqx2"
Oct 30 19:58:47.193710 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:58:47.193688    4892 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/51ed721a-e066-4f02-bea5-27d6598591bc-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-75b38d-rhqx2\" (UID: \"51ed721a-e066-4f02-bea5-27d6598591bc\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-75b38d-rhqx2"
Oct 30 19:58:47.207280 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:58:47.207252    4892 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-75b38d-rhqx2"]
Oct 30 19:58:47.282805 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:58:47.282772    4892 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-dd9b8d79-dlbxt"
Oct 30 19:58:47.295369 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:58:47.295337    4892 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/51ed721a-e066-4f02-bea5-27d6598591bc-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-75b38d-rhqx2\" (UID: \"51ed721a-e066-4f02-bea5-27d6598591bc\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-75b38d-rhqx2"
Oct 30 19:58:47.295522 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:58:47.295464    4892 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-db6qf\" (UniqueName: \"kubernetes.io/projected/51ed721a-e066-4f02-bea5-27d6598591bc-kube-api-access-db6qf\") pod \"maas-api-revoke-keys-e2e-inf-b-75b38d-rhqx2\" (UID: \"51ed721a-e066-4f02-bea5-27d6598591bc\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-75b38d-rhqx2"
Oct 30 19:58:47.295629 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:58:47.295600    4892 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/51ed721a-e066-4f02-bea5-27d6598591bc-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-75b38d-rhqx2\" (UID: \"51ed721a-e066-4f02-bea5-27d6598591bc\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-75b38d-rhqx2"
Oct 30 19:58:47.296125 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:58:47.296092    4892 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/51ed721a-e066-4f02-bea5-27d6598591bc-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-75b38d-rhqx2\" (UID: \"51ed721a-e066-4f02-bea5-27d6598591bc\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-75b38d-rhqx2"
Oct 30 19:58:47.298752 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:58:47.298695    4892 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/51ed721a-e066-4f02-bea5-27d6598591bc-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-75b38d-rhqx2\" (UID: \"51ed721a-e066-4f02-bea5-27d6598591bc\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-75b38d-rhqx2"
Oct 30 19:58:47.355855 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:58:47.355690    4892 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-db6qf\" (UniqueName: \"kubernetes.io/projected/51ed721a-e066-4f02-bea5-27d6598591bc-kube-api-access-db6qf\") pod \"maas-api-revoke-keys-e2e-inf-b-75b38d-rhqx2\" (UID: \"51ed721a-e066-4f02-bea5-27d6598591bc\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-75b38d-rhqx2"
Oct 30 19:58:47.397187 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:58:47.397150    4892 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/d126159b-6468-4c98-808f-a4243e83f11b-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d126159b-6468-4c98-808f-a4243e83f11b-maas-api-xks-ca\") pod \"d126159b-6468-4c98-808f-a4243e83f11b\" (UID: \"d126159b-6468-4c98-808f-a4243e83f11b\") "
Oct 30 19:58:47.397357 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:58:47.397285    4892 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d126159b-6468-4c98-808f-a4243e83f11b-kube-api-access-mw5qr\" (UniqueName: \"kubernetes.io/projected/d126159b-6468-4c98-808f-a4243e83f11b-kube-api-access-mw5qr\") pod \"d126159b-6468-4c98-808f-a4243e83f11b\" (UID: \"d126159b-6468-4c98-808f-a4243e83f11b\") "
Oct 30 19:58:47.397427 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:58:47.397360    4892 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d126159b-6468-4c98-808f-a4243e83f11b-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d126159b-6468-4c98-808f-a4243e83f11b-maas-api-openshift-ca\") pod \"d126159b-6468-4c98-808f-a4243e83f11b\" (UID: \"d126159b-6468-4c98-808f-a4243e83f11b\") "
Oct 30 19:58:47.397673 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:58:47.397648    4892 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d126159b-6468-4c98-808f-a4243e83f11b-maas-api-openshift-ca" pod "d126159b-6468-4c98-808f-a4243e83f11b" (UID: "d126159b-6468-4c98-808f-a4243e83f11b"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 19:58:47.397837 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:58:47.397821    4892 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d126159b-6468-4c98-808f-a4243e83f11b-maas-api-openshift-ca\") on node \"ip-10-0-132-82.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 19:58:47.400060 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:58:47.400024    4892 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d126159b-6468-4c98-808f-a4243e83f11b-kube-api-access-mw5qr" pod "d126159b-6468-4c98-808f-a4243e83f11b" (UID: "d126159b-6468-4c98-808f-a4243e83f11b"). InnerVolumeSpecName "kube-api-access-mw5qr". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:58:47.400474 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:58:47.400450    4892 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d126159b-6468-4c98-808f-a4243e83f11b-maas-api-xks-ca" pod "d126159b-6468-4c98-808f-a4243e83f11b" (UID: "d126159b-6468-4c98-808f-a4243e83f11b"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 19:58:47.463030 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:58:47.455173    4892 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-75b38d-rhqx2"
Oct 30 19:58:47.499416 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:58:47.499325    4892 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mw5qr\" (UniqueName: \"kubernetes.io/projected/d126159b-6468-4c98-808f-a4243e83f11b-kube-api-access-mw5qr\") on node \"ip-10-0-132-82.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 19:58:47.499416 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:58:47.499375    4892 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d126159b-6468-4c98-808f-a4243e83f11b-maas-api-xks-ca\") on node \"ip-10-0-132-82.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 19:58:47.911516 ip-10-0-132-82 kubenswrapper[4892]: W1030 19:58:47.911483    4892 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod51ed721a_e066_4f02_bea5_27d6598591bc.slice/crio-a52cafa96bec6f39a908ce0397097cdb81e9ced72ef67ec0c3bbd70acac9981c WatchSource:0}: Error finding container a52cafa96bec6f39a908ce0397097cdb81e9ced72ef67ec0c3bbd70acac9981c: Status 404 returned error can't find the container with id a52cafa96bec6f39a908ce0397097cdb81e9ced72ef67ec0c3bbd70acac9981c
Oct 30 19:58:47.913855 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:58:47.913831    4892 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-75b38d-rhqx2"]
Oct 30 19:58:48.065634 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:58:48.065595    4892 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-75b38d-rhqx2" event={"ID":"51ed721a-e066-4f02-bea5-27d6598591bc","Type":"ContainerStarted","Data":"a52cafa96bec6f39a908ce0397097cdb81e9ced72ef67ec0c3bbd70acac9981c"}
Oct 30 19:58:48.067206 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:58:48.067183    4892 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-dd9b8d79-dlbxt"
Oct 30 19:58:48.067343 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:58:48.067216    4892 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-dd9b8d79-dlbxt" event={"ID":"d126159b-6468-4c98-808f-a4243e83f11b","Type":"ContainerDied","Data":"bc5d26f01de46c3a1b9eb84ef18432aeb4ebb6652a82ed2c915d5eaf5228d724"}
Oct 30 19:58:48.067343 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:58:48.067264    4892 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="bc5d26f01de46c3a1b9eb84ef18432aeb4ebb6652a82ed2c915d5eaf5228d724"
Oct 30 19:58:49.077161 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:58:49.077124    4892 generic.go:356] "Generic (PLEG): container finished" podID="51ed721a-e066-4f02-bea5-27d6598591bc" containerID="463a0af2ea6209b33957bc2bd5d1479e0d6e5e2bb760c91a99e705037c7e615e" exitCode=0
Oct 30 19:58:49.077538 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:58:49.077217    4892 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-75b38d-rhqx2" event={"ID":"51ed721a-e066-4f02-bea5-27d6598591bc","Type":"ContainerDied","Data":"463a0af2ea6209b33957bc2bd5d1479e0d6e5e2bb760c91a99e705037c7e615e"}
Oct 30 19:58:50.257822 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:58:50.257796    4892 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-75b38d-rhqx2"
Oct 30 19:58:50.342935 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:58:50.342841    4892 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/51ed721a-e066-4f02-bea5-27d6598591bc-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/51ed721a-e066-4f02-bea5-27d6598591bc-maas-api-openshift-ca\") pod \"51ed721a-e066-4f02-bea5-27d6598591bc\" (UID: \"51ed721a-e066-4f02-bea5-27d6598591bc\") "
Oct 30 19:58:50.342935 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:58:50.342894    4892 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/51ed721a-e066-4f02-bea5-27d6598591bc-kube-api-access-db6qf\" (UniqueName: \"kubernetes.io/projected/51ed721a-e066-4f02-bea5-27d6598591bc-kube-api-access-db6qf\") pod \"51ed721a-e066-4f02-bea5-27d6598591bc\" (UID: \"51ed721a-e066-4f02-bea5-27d6598591bc\") "
Oct 30 19:58:50.343206 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:58:50.343035    4892 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/51ed721a-e066-4f02-bea5-27d6598591bc-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/51ed721a-e066-4f02-bea5-27d6598591bc-maas-api-xks-ca\") pod \"51ed721a-e066-4f02-bea5-27d6598591bc\" (UID: \"51ed721a-e066-4f02-bea5-27d6598591bc\") "
Oct 30 19:58:50.343308 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:58:50.343282    4892 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/51ed721a-e066-4f02-bea5-27d6598591bc-maas-api-openshift-ca" pod "51ed721a-e066-4f02-bea5-27d6598591bc" (UID: "51ed721a-e066-4f02-bea5-27d6598591bc"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 19:58:50.343858 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:58:50.343833    4892 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/51ed721a-e066-4f02-bea5-27d6598591bc-maas-api-openshift-ca\") on node \"ip-10-0-132-82.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 19:58:50.345374 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:58:50.345343    4892 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/51ed721a-e066-4f02-bea5-27d6598591bc-maas-api-xks-ca" pod "51ed721a-e066-4f02-bea5-27d6598591bc" (UID: "51ed721a-e066-4f02-bea5-27d6598591bc"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 19:58:50.345786 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:58:50.345389    4892 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/51ed721a-e066-4f02-bea5-27d6598591bc-kube-api-access-db6qf" pod "51ed721a-e066-4f02-bea5-27d6598591bc" (UID: "51ed721a-e066-4f02-bea5-27d6598591bc"). InnerVolumeSpecName "kube-api-access-db6qf". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:58:50.444572 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:58:50.444536    4892 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-db6qf\" (UniqueName: \"kubernetes.io/projected/51ed721a-e066-4f02-bea5-27d6598591bc-kube-api-access-db6qf\") on node \"ip-10-0-132-82.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 19:58:50.444572 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:58:50.444568    4892 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/51ed721a-e066-4f02-bea5-27d6598591bc-maas-api-xks-ca\") on node \"ip-10-0-132-82.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 19:58:51.095535 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:58:51.095503    4892 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-75b38d-rhqx2"
Oct 30 19:58:51.095695 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:58:51.095540    4892 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-75b38d-rhqx2" event={"ID":"51ed721a-e066-4f02-bea5-27d6598591bc","Type":"ContainerDied","Data":"a52cafa96bec6f39a908ce0397097cdb81e9ced72ef67ec0c3bbd70acac9981c"}
Oct 30 19:58:51.095695 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:58:51.095609    4892 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="a52cafa96bec6f39a908ce0397097cdb81e9ced72ef67ec0c3bbd70acac9981c"
Oct 30 19:58:53.055878 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:58:53.055842    4892 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-80518402-mssm4"]
Oct 30 19:58:53.105128 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:58:53.105098    4892 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-80518402-mssm4"]
Oct 30 19:58:54.371698 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:58:54.371668    4892 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-a3f9b7-openshift-default-7d6d8b8fb5-5vgch"]
Oct 30 19:58:54.372236 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:58:54.371880    4892 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w2-a3f9b7-openshift-default-7d6d8b8fb5-5vgch" podUID="78ea3297-af25-446c-b7d0-3da5ae88dd15" containerName="istio-proxy" containerID="cri-o://31f71be1a3f974094d49502d6d3ef62771bc9911e1fee1d0ae54798ae5f87cde" gracePeriod=30
Oct 30 19:58:54.573243 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:58:54.573204    4892 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="af9f361e-cebe-4309-93cb-c609ef835889" path="/var/lib/kubelet/pods/af9f361e-cebe-4309-93cb-c609ef835889/volumes"
Oct 30 19:58:54.573609 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:58:54.573590    4892 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-2d9861-openshift-default-58ddf574b9-8pwn9"]
Oct 30 19:58:54.573812 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:58:54.573790    4892 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w0-2d9861-openshift-default-58ddf574b9-8pwn9" podUID="abea5659-2a38-406f-8221-66aefbc769b7" containerName="istio-proxy" containerID="cri-o://f8042e8de7eb3bfd7ddbca7161920b7d44b7965b6e8358ded7f787f0f714009b" gracePeriod=30
Oct 30 19:58:54.844150 ip-10-0-132-82 kubenswrapper[4892]: E1030 19:58:54.844113    4892 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-jrm87" containerName="extractor"
Oct 30 19:58:56.670664 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:58:56.670614    4892 patch_prober.go:29] interesting pod/e2e-worker-w2-a3f9b7-openshift-default-7d6d8b8fb5-5vgch container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Oct 30 19:58:56.671231 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:58:56.670712    4892 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w2-a3f9b7-openshift-default-7d6d8b8fb5-5vgch" podUID="78ea3297-af25-446c-b7d0-3da5ae88dd15" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Oct 30 19:58:57.681885 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:58:57.681856    4892 scope.go:122] "RemoveContainer" containerID="168cedc919b24607d50684cb50ed7c4b8fc1fd9016cff3ef8af293b6ccd1df5c"
Oct 30 19:58:57.692723 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:58:57.692698    4892 scope.go:122] "RemoveContainer" containerID="46a4cf6ce878ac3694540a5cb6b2de4b9842f55ae0be01a49cb7669fbccb6364"
Oct 30 19:58:57.821220 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:58:57.821193    4892 patch_prober.go:29] interesting pod/e2e-worker-w0-2d9861-openshift-default-58ddf574b9-8pwn9 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Oct 30 19:58:57.821360 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:58:57.821251    4892 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w0-2d9861-openshift-default-58ddf574b9-8pwn9" podUID="abea5659-2a38-406f-8221-66aefbc769b7" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Oct 30 19:58:58.780674 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:58:58.780639    4892 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-75b38d-9b874d7b7-z56js"]
Oct 30 19:58:59.708645 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:58:59.708612    4892 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-a3f9b7-openshift-default-7d6d8b8fb5-5vgch"
Oct 30 19:58:59.786991 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:58:59.786397    4892 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/78ea3297-af25-446c-b7d0-3da5ae88dd15-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/78ea3297-af25-446c-b7d0-3da5ae88dd15-workload-certs\") pod \"78ea3297-af25-446c-b7d0-3da5ae88dd15\" (UID: \"78ea3297-af25-446c-b7d0-3da5ae88dd15\") "
Oct 30 19:58:59.786991 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:58:59.786459    4892 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/78ea3297-af25-446c-b7d0-3da5ae88dd15-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/78ea3297-af25-446c-b7d0-3da5ae88dd15-istio-podinfo\") pod \"78ea3297-af25-446c-b7d0-3da5ae88dd15\" (UID: \"78ea3297-af25-446c-b7d0-3da5ae88dd15\") "
Oct 30 19:58:59.786991 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:58:59.786500    4892 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/78ea3297-af25-446c-b7d0-3da5ae88dd15-kube-api-access-r7qbx\" (UniqueName: \"kubernetes.io/projected/78ea3297-af25-446c-b7d0-3da5ae88dd15-kube-api-access-r7qbx\") pod \"78ea3297-af25-446c-b7d0-3da5ae88dd15\" (UID: \"78ea3297-af25-446c-b7d0-3da5ae88dd15\") "
Oct 30 19:58:59.786991 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:58:59.786529    4892 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/78ea3297-af25-446c-b7d0-3da5ae88dd15-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/78ea3297-af25-446c-b7d0-3da5ae88dd15-workload-socket\") pod \"78ea3297-af25-446c-b7d0-3da5ae88dd15\" (UID: \"78ea3297-af25-446c-b7d0-3da5ae88dd15\") "
Oct 30 19:58:59.786991 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:58:59.786588    4892 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/78ea3297-af25-446c-b7d0-3da5ae88dd15-istio-token\" (UniqueName: \"kubernetes.io/projected/78ea3297-af25-446c-b7d0-3da5ae88dd15-istio-token\") pod \"78ea3297-af25-446c-b7d0-3da5ae88dd15\" (UID: \"78ea3297-af25-446c-b7d0-3da5ae88dd15\") "
Oct 30 19:58:59.786991 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:58:59.786650    4892 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/78ea3297-af25-446c-b7d0-3da5ae88dd15-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/78ea3297-af25-446c-b7d0-3da5ae88dd15-istio-data\") pod \"78ea3297-af25-446c-b7d0-3da5ae88dd15\" (UID: \"78ea3297-af25-446c-b7d0-3da5ae88dd15\") "
Oct 30 19:58:59.786991 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:58:59.786793    4892 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/78ea3297-af25-446c-b7d0-3da5ae88dd15-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/78ea3297-af25-446c-b7d0-3da5ae88dd15-istiod-ca-cert\") pod \"78ea3297-af25-446c-b7d0-3da5ae88dd15\" (UID: \"78ea3297-af25-446c-b7d0-3da5ae88dd15\") "
Oct 30 19:58:59.786991 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:58:59.786853    4892 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/78ea3297-af25-446c-b7d0-3da5ae88dd15-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/78ea3297-af25-446c-b7d0-3da5ae88dd15-istio-envoy\") pod \"78ea3297-af25-446c-b7d0-3da5ae88dd15\" (UID: \"78ea3297-af25-446c-b7d0-3da5ae88dd15\") "
Oct 30 19:58:59.786991 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:58:59.786897    4892 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/78ea3297-af25-446c-b7d0-3da5ae88dd15-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/78ea3297-af25-446c-b7d0-3da5ae88dd15-credential-socket\") pod \"78ea3297-af25-446c-b7d0-3da5ae88dd15\" (UID: \"78ea3297-af25-446c-b7d0-3da5ae88dd15\") "
Oct 30 19:58:59.787914 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:58:59.787286    4892 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/78ea3297-af25-446c-b7d0-3da5ae88dd15-workload-socket" pod "78ea3297-af25-446c-b7d0-3da5ae88dd15" (UID: "78ea3297-af25-446c-b7d0-3da5ae88dd15"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:58:59.787914 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:58:59.787487    4892 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/78ea3297-af25-446c-b7d0-3da5ae88dd15-workload-socket\") on node \"ip-10-0-132-82.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 19:58:59.787914 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:58:59.787490    4892 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/78ea3297-af25-446c-b7d0-3da5ae88dd15-workload-certs" pod "78ea3297-af25-446c-b7d0-3da5ae88dd15" (UID: "78ea3297-af25-446c-b7d0-3da5ae88dd15"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:58:59.789749 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:58:59.788169    4892 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/78ea3297-af25-446c-b7d0-3da5ae88dd15-istio-data" pod "78ea3297-af25-446c-b7d0-3da5ae88dd15" (UID: "78ea3297-af25-446c-b7d0-3da5ae88dd15"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:58:59.789749 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:58:59.788346    4892 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/78ea3297-af25-446c-b7d0-3da5ae88dd15-credential-socket" pod "78ea3297-af25-446c-b7d0-3da5ae88dd15" (UID: "78ea3297-af25-446c-b7d0-3da5ae88dd15"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:58:59.791470 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:58:59.791408    4892 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/78ea3297-af25-446c-b7d0-3da5ae88dd15-istio-envoy" pod "78ea3297-af25-446c-b7d0-3da5ae88dd15" (UID: "78ea3297-af25-446c-b7d0-3da5ae88dd15"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:58:59.793349 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:58:59.793324    4892 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/78ea3297-af25-446c-b7d0-3da5ae88dd15-istio-podinfo" pod "78ea3297-af25-446c-b7d0-3da5ae88dd15" (UID: "78ea3297-af25-446c-b7d0-3da5ae88dd15"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 19:58:59.795152 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:58:59.795127    4892 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/78ea3297-af25-446c-b7d0-3da5ae88dd15-istio-token" pod "78ea3297-af25-446c-b7d0-3da5ae88dd15" (UID: "78ea3297-af25-446c-b7d0-3da5ae88dd15"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:58:59.796484 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:58:59.796463    4892 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/78ea3297-af25-446c-b7d0-3da5ae88dd15-kube-api-access-r7qbx" pod "78ea3297-af25-446c-b7d0-3da5ae88dd15" (UID: "78ea3297-af25-446c-b7d0-3da5ae88dd15"). InnerVolumeSpecName "kube-api-access-r7qbx". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:58:59.799178 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:58:59.799154    4892 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/78ea3297-af25-446c-b7d0-3da5ae88dd15-istiod-ca-cert" pod "78ea3297-af25-446c-b7d0-3da5ae88dd15" (UID: "78ea3297-af25-446c-b7d0-3da5ae88dd15"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 19:58:59.825947 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:58:59.825916    4892 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-86e4a7-686g8"]
Oct 30 19:58:59.832255 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:58:59.832235    4892 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-86e4a7-686g8"]
Oct 30 19:58:59.845223 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:58:59.843843    4892 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-2d9861-openshift-default-58ddf574b9-8pwn9"
Oct 30 19:58:59.898005 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:58:59.890378    4892 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/abea5659-2a38-406f-8221-66aefbc769b7-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/abea5659-2a38-406f-8221-66aefbc769b7-istiod-ca-cert\") pod \"abea5659-2a38-406f-8221-66aefbc769b7\" (UID: \"abea5659-2a38-406f-8221-66aefbc769b7\") "
Oct 30 19:58:59.898005 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:58:59.890455    4892 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/abea5659-2a38-406f-8221-66aefbc769b7-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/abea5659-2a38-406f-8221-66aefbc769b7-istio-data\") pod \"abea5659-2a38-406f-8221-66aefbc769b7\" (UID: \"abea5659-2a38-406f-8221-66aefbc769b7\") "
Oct 30 19:58:59.898005 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:58:59.890497    4892 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/abea5659-2a38-406f-8221-66aefbc769b7-istio-token\" (UniqueName: \"kubernetes.io/projected/abea5659-2a38-406f-8221-66aefbc769b7-istio-token\") pod \"abea5659-2a38-406f-8221-66aefbc769b7\" (UID: \"abea5659-2a38-406f-8221-66aefbc769b7\") "
Oct 30 19:58:59.898005 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:58:59.890595    4892 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/abea5659-2a38-406f-8221-66aefbc769b7-kube-api-access-fv5d6\" (UniqueName: \"kubernetes.io/projected/abea5659-2a38-406f-8221-66aefbc769b7-kube-api-access-fv5d6\") pod \"abea5659-2a38-406f-8221-66aefbc769b7\" (UID: \"abea5659-2a38-406f-8221-66aefbc769b7\") "
Oct 30 19:58:59.898005 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:58:59.890660    4892 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/abea5659-2a38-406f-8221-66aefbc769b7-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/abea5659-2a38-406f-8221-66aefbc769b7-istio-podinfo\") pod \"abea5659-2a38-406f-8221-66aefbc769b7\" (UID: \"abea5659-2a38-406f-8221-66aefbc769b7\") "
Oct 30 19:58:59.898005 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:58:59.890702    4892 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/abea5659-2a38-406f-8221-66aefbc769b7-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/abea5659-2a38-406f-8221-66aefbc769b7-istio-envoy\") pod \"abea5659-2a38-406f-8221-66aefbc769b7\" (UID: \"abea5659-2a38-406f-8221-66aefbc769b7\") "
Oct 30 19:58:59.898005 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:58:59.890797    4892 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/abea5659-2a38-406f-8221-66aefbc769b7-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/abea5659-2a38-406f-8221-66aefbc769b7-credential-socket\") pod \"abea5659-2a38-406f-8221-66aefbc769b7\" (UID: \"abea5659-2a38-406f-8221-66aefbc769b7\") "
Oct 30 19:58:59.898005 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:58:59.890852    4892 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/abea5659-2a38-406f-8221-66aefbc769b7-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/abea5659-2a38-406f-8221-66aefbc769b7-workload-certs\") pod \"abea5659-2a38-406f-8221-66aefbc769b7\" (UID: \"abea5659-2a38-406f-8221-66aefbc769b7\") "
Oct 30 19:58:59.898005 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:58:59.890922    4892 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/abea5659-2a38-406f-8221-66aefbc769b7-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/abea5659-2a38-406f-8221-66aefbc769b7-workload-socket\") pod \"abea5659-2a38-406f-8221-66aefbc769b7\" (UID: \"abea5659-2a38-406f-8221-66aefbc769b7\") "
Oct 30 19:58:59.898005 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:58:59.891479    4892 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/78ea3297-af25-446c-b7d0-3da5ae88dd15-workload-certs\") on node \"ip-10-0-132-82.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 19:58:59.898005 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:58:59.891501    4892 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/78ea3297-af25-446c-b7d0-3da5ae88dd15-istio-podinfo\") on node \"ip-10-0-132-82.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 19:58:59.898005 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:58:59.891518    4892 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-r7qbx\" (UniqueName: \"kubernetes.io/projected/78ea3297-af25-446c-b7d0-3da5ae88dd15-kube-api-access-r7qbx\") on node \"ip-10-0-132-82.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 19:58:59.898005 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:58:59.891533    4892 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/78ea3297-af25-446c-b7d0-3da5ae88dd15-istio-token\") on node \"ip-10-0-132-82.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 19:58:59.898005 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:58:59.891546    4892 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/78ea3297-af25-446c-b7d0-3da5ae88dd15-istio-data\") on node \"ip-10-0-132-82.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 19:58:59.898005 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:58:59.891560    4892 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/78ea3297-af25-446c-b7d0-3da5ae88dd15-istiod-ca-cert\") on node \"ip-10-0-132-82.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 19:58:59.898005 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:58:59.891574    4892 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/78ea3297-af25-446c-b7d0-3da5ae88dd15-istio-envoy\") on node \"ip-10-0-132-82.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 19:58:59.898005 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:58:59.891588    4892 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/78ea3297-af25-446c-b7d0-3da5ae88dd15-credential-socket\") on node \"ip-10-0-132-82.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 19:58:59.898005 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:58:59.892188    4892 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/abea5659-2a38-406f-8221-66aefbc769b7-workload-socket" pod "abea5659-2a38-406f-8221-66aefbc769b7" (UID: "abea5659-2a38-406f-8221-66aefbc769b7"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:58:59.903108 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:58:59.903031    4892 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/abea5659-2a38-406f-8221-66aefbc769b7-kube-api-access-fv5d6" pod "abea5659-2a38-406f-8221-66aefbc769b7" (UID: "abea5659-2a38-406f-8221-66aefbc769b7"). InnerVolumeSpecName "kube-api-access-fv5d6". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:58:59.904139 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:58:59.904115    4892 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/abea5659-2a38-406f-8221-66aefbc769b7-istio-podinfo" pod "abea5659-2a38-406f-8221-66aefbc769b7" (UID: "abea5659-2a38-406f-8221-66aefbc769b7"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 19:58:59.905875 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:58:59.905841    4892 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/abea5659-2a38-406f-8221-66aefbc769b7-credential-socket" pod "abea5659-2a38-406f-8221-66aefbc769b7" (UID: "abea5659-2a38-406f-8221-66aefbc769b7"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:58:59.906226 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:58:59.906189    4892 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/abea5659-2a38-406f-8221-66aefbc769b7-istio-data" pod "abea5659-2a38-406f-8221-66aefbc769b7" (UID: "abea5659-2a38-406f-8221-66aefbc769b7"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:58:59.906433 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:58:59.906411    4892 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/abea5659-2a38-406f-8221-66aefbc769b7-workload-certs" pod "abea5659-2a38-406f-8221-66aefbc769b7" (UID: "abea5659-2a38-406f-8221-66aefbc769b7"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:58:59.909048 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:58:59.908071    4892 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/abea5659-2a38-406f-8221-66aefbc769b7-istio-envoy" pod "abea5659-2a38-406f-8221-66aefbc769b7" (UID: "abea5659-2a38-406f-8221-66aefbc769b7"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:58:59.909319 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:58:59.909229    4892 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/abea5659-2a38-406f-8221-66aefbc769b7-istiod-ca-cert" pod "abea5659-2a38-406f-8221-66aefbc769b7" (UID: "abea5659-2a38-406f-8221-66aefbc769b7"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 19:58:59.912216 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:58:59.912189    4892 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/abea5659-2a38-406f-8221-66aefbc769b7-istio-token" pod "abea5659-2a38-406f-8221-66aefbc769b7" (UID: "abea5659-2a38-406f-8221-66aefbc769b7"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:58:59.992620 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:58:59.992586    4892 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/abea5659-2a38-406f-8221-66aefbc769b7-workload-certs\") on node \"ip-10-0-132-82.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 19:58:59.992620 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:58:59.992616    4892 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/abea5659-2a38-406f-8221-66aefbc769b7-workload-socket\") on node \"ip-10-0-132-82.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 19:58:59.992620 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:58:59.992626    4892 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/abea5659-2a38-406f-8221-66aefbc769b7-istiod-ca-cert\") on node \"ip-10-0-132-82.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 19:58:59.992853 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:58:59.992634    4892 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/abea5659-2a38-406f-8221-66aefbc769b7-istio-data\") on node \"ip-10-0-132-82.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 19:58:59.992853 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:58:59.992644    4892 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/abea5659-2a38-406f-8221-66aefbc769b7-istio-token\") on node \"ip-10-0-132-82.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 19:58:59.992853 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:58:59.992653    4892 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fv5d6\" (UniqueName: \"kubernetes.io/projected/abea5659-2a38-406f-8221-66aefbc769b7-kube-api-access-fv5d6\") on node \"ip-10-0-132-82.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 19:58:59.992853 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:58:59.992663    4892 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/abea5659-2a38-406f-8221-66aefbc769b7-istio-podinfo\") on node \"ip-10-0-132-82.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 19:58:59.992853 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:58:59.992671    4892 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/abea5659-2a38-406f-8221-66aefbc769b7-istio-envoy\") on node \"ip-10-0-132-82.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 19:58:59.992853 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:58:59.992679    4892 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/abea5659-2a38-406f-8221-66aefbc769b7-credential-socket\") on node \"ip-10-0-132-82.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 19:59:00.225049 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:59:00.220181    4892 generic.go:356] "Generic (PLEG): container finished" podID="abea5659-2a38-406f-8221-66aefbc769b7" containerID="f8042e8de7eb3bfd7ddbca7161920b7d44b7965b6e8358ded7f787f0f714009b" exitCode=0
Oct 30 19:59:00.225049 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:59:00.220466    4892 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-2d9861-openshift-default-58ddf574b9-8pwn9" event={"ID":"abea5659-2a38-406f-8221-66aefbc769b7","Type":"ContainerDied","Data":"f8042e8de7eb3bfd7ddbca7161920b7d44b7965b6e8358ded7f787f0f714009b"}
Oct 30 19:59:00.225049 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:59:00.220526    4892 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-2d9861-openshift-default-58ddf574b9-8pwn9" event={"ID":"abea5659-2a38-406f-8221-66aefbc769b7","Type":"ContainerDied","Data":"44556d16a4998e8e9546aa97be4a412f85102c05ced870571e9c67dddcdf8592"}
Oct 30 19:59:00.225049 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:59:00.220556    4892 scope.go:122] "RemoveContainer" containerID="f8042e8de7eb3bfd7ddbca7161920b7d44b7965b6e8358ded7f787f0f714009b"
Oct 30 19:59:00.225049 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:59:00.220749    4892 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-2d9861-openshift-default-58ddf574b9-8pwn9"
Oct 30 19:59:00.236806 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:59:00.233592    4892 generic.go:356] "Generic (PLEG): container finished" podID="78ea3297-af25-446c-b7d0-3da5ae88dd15" containerID="31f71be1a3f974094d49502d6d3ef62771bc9911e1fee1d0ae54798ae5f87cde" exitCode=0
Oct 30 19:59:00.236806 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:59:00.233680    4892 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-a3f9b7-openshift-default-7d6d8b8fb5-5vgch" event={"ID":"78ea3297-af25-446c-b7d0-3da5ae88dd15","Type":"ContainerDied","Data":"31f71be1a3f974094d49502d6d3ef62771bc9911e1fee1d0ae54798ae5f87cde"}
Oct 30 19:59:00.236806 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:59:00.233726    4892 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-a3f9b7-openshift-default-7d6d8b8fb5-5vgch" event={"ID":"78ea3297-af25-446c-b7d0-3da5ae88dd15","Type":"ContainerDied","Data":"e33f3e6d2da0ee55a7d1eef0cff4fb158fdf52c67915d5bbb592670f152f4c5d"}
Oct 30 19:59:00.236806 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:59:00.233801    4892 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-a3f9b7-openshift-default-7d6d8b8fb5-5vgch"
Oct 30 19:59:00.249326 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:59:00.248864    4892 scope.go:122] "RemoveContainer" containerID="f8042e8de7eb3bfd7ddbca7161920b7d44b7965b6e8358ded7f787f0f714009b"
Oct 30 19:59:00.250799 ip-10-0-132-82 kubenswrapper[4892]: E1030 19:59:00.250773    4892 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f8042e8de7eb3bfd7ddbca7161920b7d44b7965b6e8358ded7f787f0f714009b\": container with ID starting with f8042e8de7eb3bfd7ddbca7161920b7d44b7965b6e8358ded7f787f0f714009b not found: ID does not exist" containerID="f8042e8de7eb3bfd7ddbca7161920b7d44b7965b6e8358ded7f787f0f714009b"
Oct 30 19:59:00.250928 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:59:00.250812    4892 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f8042e8de7eb3bfd7ddbca7161920b7d44b7965b6e8358ded7f787f0f714009b"} err="failed to get container status \"f8042e8de7eb3bfd7ddbca7161920b7d44b7965b6e8358ded7f787f0f714009b\": rpc error: code = NotFound desc = could not find container \"f8042e8de7eb3bfd7ddbca7161920b7d44b7965b6e8358ded7f787f0f714009b\": container with ID starting with f8042e8de7eb3bfd7ddbca7161920b7d44b7965b6e8358ded7f787f0f714009b not found: ID does not exist"
Oct 30 19:59:00.250928 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:59:00.250831    4892 scope.go:122] "RemoveContainer" containerID="31f71be1a3f974094d49502d6d3ef62771bc9911e1fee1d0ae54798ae5f87cde"
Oct 30 19:59:00.265160 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:59:00.265135    4892 scope.go:122] "RemoveContainer" containerID="31f71be1a3f974094d49502d6d3ef62771bc9911e1fee1d0ae54798ae5f87cde"
Oct 30 19:59:00.265452 ip-10-0-132-82 kubenswrapper[4892]: E1030 19:59:00.265429    4892 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"31f71be1a3f974094d49502d6d3ef62771bc9911e1fee1d0ae54798ae5f87cde\": container with ID starting with 31f71be1a3f974094d49502d6d3ef62771bc9911e1fee1d0ae54798ae5f87cde not found: ID does not exist" containerID="31f71be1a3f974094d49502d6d3ef62771bc9911e1fee1d0ae54798ae5f87cde"
Oct 30 19:59:00.265538 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:59:00.265461    4892 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"31f71be1a3f974094d49502d6d3ef62771bc9911e1fee1d0ae54798ae5f87cde"} err="failed to get container status \"31f71be1a3f974094d49502d6d3ef62771bc9911e1fee1d0ae54798ae5f87cde\": rpc error: code = NotFound desc = could not find container \"31f71be1a3f974094d49502d6d3ef62771bc9911e1fee1d0ae54798ae5f87cde\": container with ID starting with 31f71be1a3f974094d49502d6d3ef62771bc9911e1fee1d0ae54798ae5f87cde not found: ID does not exist"
Oct 30 19:59:00.413673 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:59:00.413588    4892 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-2d9861-openshift-default-58ddf574b9-8pwn9"]
Oct 30 19:59:00.427549 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:59:00.427520    4892 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w0-2d9861-openshift-default-58ddf574b9-8pwn9"]
Oct 30 19:59:00.559189 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:59:00.559101    4892 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-a3f9b7-openshift-default-7d6d8b8fb5-5vgch"]
Oct 30 19:59:00.572151 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:59:00.572117    4892 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="96460de9-c817-4b3a-b08a-6adb3d836530" path="/var/lib/kubelet/pods/96460de9-c817-4b3a-b08a-6adb3d836530/volumes"
Oct 30 19:59:00.572744 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:59:00.572723    4892 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="abea5659-2a38-406f-8221-66aefbc769b7" path="/var/lib/kubelet/pods/abea5659-2a38-406f-8221-66aefbc769b7/volumes"
Oct 30 19:59:00.573142 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:59:00.573124    4892 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w2-a3f9b7-openshift-default-7d6d8b8fb5-5vgch"]
Oct 30 19:59:01.841910 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:59:01.841863    4892 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-b5178f-openshift-default-c8fbdff8c-bjzz7"]
Oct 30 19:59:01.842525 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:59:01.842133    4892 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w4-b5178f-openshift-default-c8fbdff8c-bjzz7" podUID="98aae325-07a2-49fc-a4ba-3e6709dc7c31" containerName="istio-proxy" containerID="cri-o://7ef76347a62902de8860eacaff47801435076f847731251161c12a26d4f90b60" gracePeriod=30
Oct 30 19:59:02.570581 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:59:02.570537    4892 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="78ea3297-af25-446c-b7d0-3da5ae88dd15" path="/var/lib/kubelet/pods/78ea3297-af25-446c-b7d0-3da5ae88dd15/volumes"
Oct 30 19:59:02.632764 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:59:02.632729    4892 patch_prober.go:29] interesting pod/e2e-worker-w4-b5178f-openshift-default-c8fbdff8c-bjzz7 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Oct 30 19:59:02.632952 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:59:02.632788    4892 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w4-b5178f-openshift-default-c8fbdff8c-bjzz7" podUID="98aae325-07a2-49fc-a4ba-3e6709dc7c31" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Oct 30 19:59:03.781347 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:59:03.781307    4892 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-75b38d-9b874d7b7-z56js" podUID="558323d9-9532-440c-a95b-f367d8d0df78" containerName="payload-pre-processing" containerID="cri-o://ba6aa44eddee4783a33fa700389847bca36090d8a680c12dc0d7f1f80e06c634" gracePeriod=25
Oct 30 19:59:03.847039 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:59:03.847005    4892 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-dd9b8d79-67c8d6d46-mvgdr"]
Oct 30 19:59:03.953370 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:59:03.953334    4892 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-dd9b8d79-867946f667-2p8kf"]
Oct 30 19:59:04.090827 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:59:04.090801    4892 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-75b38d-9b874d7b7-z56js"
Oct 30 19:59:04.170190 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:59:04.170011    4892 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/558323d9-9532-440c-a95b-f367d8d0df78-kube-api-access-sgrvk\" (UniqueName: \"kubernetes.io/projected/558323d9-9532-440c-a95b-f367d8d0df78-kube-api-access-sgrvk\") pod \"558323d9-9532-440c-a95b-f367d8d0df78\" (UID: \"558323d9-9532-440c-a95b-f367d8d0df78\") "
Oct 30 19:59:04.170779 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:59:04.170667    4892 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/558323d9-9532-440c-a95b-f367d8d0df78-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/558323d9-9532-440c-a95b-f367d8d0df78-plugins-config-volume\") pod \"558323d9-9532-440c-a95b-f367d8d0df78\" (UID: \"558323d9-9532-440c-a95b-f367d8d0df78\") "
Oct 30 19:59:04.170779 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:59:04.170738    4892 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/558323d9-9532-440c-a95b-f367d8d0df78-plugins-config-volume" pod "558323d9-9532-440c-a95b-f367d8d0df78" (UID: "558323d9-9532-440c-a95b-f367d8d0df78"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 19:59:04.171246 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:59:04.171224    4892 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/558323d9-9532-440c-a95b-f367d8d0df78-plugins-config-volume\") on node \"ip-10-0-132-82.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 19:59:04.172922 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:59:04.172885    4892 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/558323d9-9532-440c-a95b-f367d8d0df78-kube-api-access-sgrvk" pod "558323d9-9532-440c-a95b-f367d8d0df78" (UID: "558323d9-9532-440c-a95b-f367d8d0df78"). InnerVolumeSpecName "kube-api-access-sgrvk". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:59:04.296955 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:59:04.273953    4892 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-sgrvk\" (UniqueName: \"kubernetes.io/projected/558323d9-9532-440c-a95b-f367d8d0df78-kube-api-access-sgrvk\") on node \"ip-10-0-132-82.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 19:59:04.314459 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:59:04.314417    4892 generic.go:356] "Generic (PLEG): container finished" podID="558323d9-9532-440c-a95b-f367d8d0df78" containerID="ba6aa44eddee4783a33fa700389847bca36090d8a680c12dc0d7f1f80e06c634" exitCode=0
Oct 30 19:59:04.314665 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:59:04.314645    4892 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-75b38d-9b874d7b7-z56js" event={"ID":"558323d9-9532-440c-a95b-f367d8d0df78","Type":"ContainerDied","Data":"ba6aa44eddee4783a33fa700389847bca36090d8a680c12dc0d7f1f80e06c634"}
Oct 30 19:59:04.314731 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:59:04.314706    4892 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-75b38d-9b874d7b7-z56js" event={"ID":"558323d9-9532-440c-a95b-f367d8d0df78","Type":"ContainerDied","Data":"e0cd12b1ae632dc97f71f3f2ff0b534dae0cb778511c219adc0fdc69f4677c8d"}
Oct 30 19:59:04.314810 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:59:04.314740    4892 scope.go:122] "RemoveContainer" containerID="ba6aa44eddee4783a33fa700389847bca36090d8a680c12dc0d7f1f80e06c634"
Oct 30 19:59:04.314904 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:59:04.314887    4892 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-75b38d-9b874d7b7-z56js"
Oct 30 19:59:04.334845 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:59:04.334823    4892 scope.go:122] "RemoveContainer" containerID="ba6aa44eddee4783a33fa700389847bca36090d8a680c12dc0d7f1f80e06c634"
Oct 30 19:59:04.335240 ip-10-0-132-82 kubenswrapper[4892]: E1030 19:59:04.335212    4892 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ba6aa44eddee4783a33fa700389847bca36090d8a680c12dc0d7f1f80e06c634\": container with ID starting with ba6aa44eddee4783a33fa700389847bca36090d8a680c12dc0d7f1f80e06c634 not found: ID does not exist" containerID="ba6aa44eddee4783a33fa700389847bca36090d8a680c12dc0d7f1f80e06c634"
Oct 30 19:59:04.335413 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:59:04.335248    4892 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ba6aa44eddee4783a33fa700389847bca36090d8a680c12dc0d7f1f80e06c634"} err="failed to get container status \"ba6aa44eddee4783a33fa700389847bca36090d8a680c12dc0d7f1f80e06c634\": rpc error: code = NotFound desc = could not find container \"ba6aa44eddee4783a33fa700389847bca36090d8a680c12dc0d7f1f80e06c634\": container with ID starting with ba6aa44eddee4783a33fa700389847bca36090d8a680c12dc0d7f1f80e06c634 not found: ID does not exist"
Oct 30 19:59:04.485207 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:59:04.485175    4892 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-75b38d-9b874d7b7-z56js"]
Oct 30 19:59:04.495320 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:59:04.495287    4892 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-75b38d-9b874d7b7-z56js"]
Oct 30 19:59:04.571891 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:59:04.571811    4892 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="558323d9-9532-440c-a95b-f367d8d0df78" path="/var/lib/kubelet/pods/558323d9-9532-440c-a95b-f367d8d0df78/volumes"
Oct 30 19:59:07.133058 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:59:07.132387    4892 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-b5178f-openshift-default-c8fbdff8c-bjzz7"
Oct 30 19:59:07.209530 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:59:07.209496    4892 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/98aae325-07a2-49fc-a4ba-3e6709dc7c31-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/98aae325-07a2-49fc-a4ba-3e6709dc7c31-istio-podinfo\") pod \"98aae325-07a2-49fc-a4ba-3e6709dc7c31\" (UID: \"98aae325-07a2-49fc-a4ba-3e6709dc7c31\") "
Oct 30 19:59:07.209731 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:59:07.209551    4892 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/98aae325-07a2-49fc-a4ba-3e6709dc7c31-istio-token\" (UniqueName: \"kubernetes.io/projected/98aae325-07a2-49fc-a4ba-3e6709dc7c31-istio-token\") pod \"98aae325-07a2-49fc-a4ba-3e6709dc7c31\" (UID: \"98aae325-07a2-49fc-a4ba-3e6709dc7c31\") "
Oct 30 19:59:07.209731 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:59:07.209579    4892 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/98aae325-07a2-49fc-a4ba-3e6709dc7c31-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/98aae325-07a2-49fc-a4ba-3e6709dc7c31-istiod-ca-cert\") pod \"98aae325-07a2-49fc-a4ba-3e6709dc7c31\" (UID: \"98aae325-07a2-49fc-a4ba-3e6709dc7c31\") "
Oct 30 19:59:07.209731 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:59:07.209603    4892 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/98aae325-07a2-49fc-a4ba-3e6709dc7c31-kube-api-access-8l2c6\" (UniqueName: \"kubernetes.io/projected/98aae325-07a2-49fc-a4ba-3e6709dc7c31-kube-api-access-8l2c6\") pod \"98aae325-07a2-49fc-a4ba-3e6709dc7c31\" (UID: \"98aae325-07a2-49fc-a4ba-3e6709dc7c31\") "
Oct 30 19:59:07.209731 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:59:07.209642    4892 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/98aae325-07a2-49fc-a4ba-3e6709dc7c31-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/98aae325-07a2-49fc-a4ba-3e6709dc7c31-credential-socket\") pod \"98aae325-07a2-49fc-a4ba-3e6709dc7c31\" (UID: \"98aae325-07a2-49fc-a4ba-3e6709dc7c31\") "
Oct 30 19:59:07.209731 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:59:07.209676    4892 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/98aae325-07a2-49fc-a4ba-3e6709dc7c31-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/98aae325-07a2-49fc-a4ba-3e6709dc7c31-istio-envoy\") pod \"98aae325-07a2-49fc-a4ba-3e6709dc7c31\" (UID: \"98aae325-07a2-49fc-a4ba-3e6709dc7c31\") "
Oct 30 19:59:07.210047 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:59:07.209742    4892 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/98aae325-07a2-49fc-a4ba-3e6709dc7c31-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/98aae325-07a2-49fc-a4ba-3e6709dc7c31-workload-socket\") pod \"98aae325-07a2-49fc-a4ba-3e6709dc7c31\" (UID: \"98aae325-07a2-49fc-a4ba-3e6709dc7c31\") "
Oct 30 19:59:07.210047 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:59:07.209784    4892 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/98aae325-07a2-49fc-a4ba-3e6709dc7c31-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/98aae325-07a2-49fc-a4ba-3e6709dc7c31-istio-data\") pod \"98aae325-07a2-49fc-a4ba-3e6709dc7c31\" (UID: \"98aae325-07a2-49fc-a4ba-3e6709dc7c31\") "
Oct 30 19:59:07.210047 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:59:07.209904    4892 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/98aae325-07a2-49fc-a4ba-3e6709dc7c31-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/98aae325-07a2-49fc-a4ba-3e6709dc7c31-workload-certs\") pod \"98aae325-07a2-49fc-a4ba-3e6709dc7c31\" (UID: \"98aae325-07a2-49fc-a4ba-3e6709dc7c31\") "
Oct 30 19:59:07.210217 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:59:07.210115    4892 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/98aae325-07a2-49fc-a4ba-3e6709dc7c31-credential-socket" pod "98aae325-07a2-49fc-a4ba-3e6709dc7c31" (UID: "98aae325-07a2-49fc-a4ba-3e6709dc7c31"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:59:07.210333 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:59:07.210298    4892 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/98aae325-07a2-49fc-a4ba-3e6709dc7c31-istiod-ca-cert" pod "98aae325-07a2-49fc-a4ba-3e6709dc7c31" (UID: "98aae325-07a2-49fc-a4ba-3e6709dc7c31"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 19:59:07.210624 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:59:07.210547    4892 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/98aae325-07a2-49fc-a4ba-3e6709dc7c31-workload-socket" pod "98aae325-07a2-49fc-a4ba-3e6709dc7c31" (UID: "98aae325-07a2-49fc-a4ba-3e6709dc7c31"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:59:07.210744 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:59:07.210700    4892 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/98aae325-07a2-49fc-a4ba-3e6709dc7c31-istiod-ca-cert\") on node \"ip-10-0-132-82.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 19:59:07.210744 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:59:07.210723    4892 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/98aae325-07a2-49fc-a4ba-3e6709dc7c31-credential-socket\") on node \"ip-10-0-132-82.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 19:59:07.211187 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:59:07.211144    4892 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/98aae325-07a2-49fc-a4ba-3e6709dc7c31-workload-certs" pod "98aae325-07a2-49fc-a4ba-3e6709dc7c31" (UID: "98aae325-07a2-49fc-a4ba-3e6709dc7c31"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:59:07.212210 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:59:07.212183    4892 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/98aae325-07a2-49fc-a4ba-3e6709dc7c31-istio-data" pod "98aae325-07a2-49fc-a4ba-3e6709dc7c31" (UID: "98aae325-07a2-49fc-a4ba-3e6709dc7c31"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:59:07.213507 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:59:07.213483    4892 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/98aae325-07a2-49fc-a4ba-3e6709dc7c31-kube-api-access-8l2c6" pod "98aae325-07a2-49fc-a4ba-3e6709dc7c31" (UID: "98aae325-07a2-49fc-a4ba-3e6709dc7c31"). InnerVolumeSpecName "kube-api-access-8l2c6". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:59:07.213630 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:59:07.213521    4892 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/98aae325-07a2-49fc-a4ba-3e6709dc7c31-istio-token" pod "98aae325-07a2-49fc-a4ba-3e6709dc7c31" (UID: "98aae325-07a2-49fc-a4ba-3e6709dc7c31"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:59:07.215119 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:59:07.215064    4892 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/98aae325-07a2-49fc-a4ba-3e6709dc7c31-istio-envoy" pod "98aae325-07a2-49fc-a4ba-3e6709dc7c31" (UID: "98aae325-07a2-49fc-a4ba-3e6709dc7c31"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:59:07.215677 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:59:07.215641    4892 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/98aae325-07a2-49fc-a4ba-3e6709dc7c31-istio-podinfo" pod "98aae325-07a2-49fc-a4ba-3e6709dc7c31" (UID: "98aae325-07a2-49fc-a4ba-3e6709dc7c31"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 19:59:07.315916 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:59:07.311793    4892 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/98aae325-07a2-49fc-a4ba-3e6709dc7c31-workload-certs\") on node \"ip-10-0-132-82.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 19:59:07.315916 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:59:07.311831    4892 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/98aae325-07a2-49fc-a4ba-3e6709dc7c31-istio-podinfo\") on node \"ip-10-0-132-82.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 19:59:07.315916 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:59:07.311846    4892 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/98aae325-07a2-49fc-a4ba-3e6709dc7c31-istio-token\") on node \"ip-10-0-132-82.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 19:59:07.315916 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:59:07.311860    4892 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8l2c6\" (UniqueName: \"kubernetes.io/projected/98aae325-07a2-49fc-a4ba-3e6709dc7c31-kube-api-access-8l2c6\") on node \"ip-10-0-132-82.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 19:59:07.315916 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:59:07.311874    4892 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/98aae325-07a2-49fc-a4ba-3e6709dc7c31-istio-envoy\") on node \"ip-10-0-132-82.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 19:59:07.315916 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:59:07.311886    4892 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/98aae325-07a2-49fc-a4ba-3e6709dc7c31-workload-socket\") on node \"ip-10-0-132-82.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 19:59:07.315916 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:59:07.311900    4892 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/98aae325-07a2-49fc-a4ba-3e6709dc7c31-istio-data\") on node \"ip-10-0-132-82.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 19:59:07.359622 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:59:07.359586    4892 generic.go:356] "Generic (PLEG): container finished" podID="98aae325-07a2-49fc-a4ba-3e6709dc7c31" containerID="7ef76347a62902de8860eacaff47801435076f847731251161c12a26d4f90b60" exitCode=0
Oct 30 19:59:07.359843 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:59:07.359708    4892 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-b5178f-openshift-default-c8fbdff8c-bjzz7" event={"ID":"98aae325-07a2-49fc-a4ba-3e6709dc7c31","Type":"ContainerDied","Data":"7ef76347a62902de8860eacaff47801435076f847731251161c12a26d4f90b60"}
Oct 30 19:59:07.359843 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:59:07.359793    4892 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-b5178f-openshift-default-c8fbdff8c-bjzz7" event={"ID":"98aae325-07a2-49fc-a4ba-3e6709dc7c31","Type":"ContainerDied","Data":"970b23dd993461e6cf6b45bf7d9f34c1e22de65696203557aa76aaf20474b6f6"}
Oct 30 19:59:07.359843 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:59:07.359819    4892 scope.go:122] "RemoveContainer" containerID="7ef76347a62902de8860eacaff47801435076f847731251161c12a26d4f90b60"
Oct 30 19:59:07.360072 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:59:07.359941    4892 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-b5178f-openshift-default-c8fbdff8c-bjzz7"
Oct 30 19:59:07.374273 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:59:07.374252    4892 scope.go:122] "RemoveContainer" containerID="7ef76347a62902de8860eacaff47801435076f847731251161c12a26d4f90b60"
Oct 30 19:59:07.374543 ip-10-0-132-82 kubenswrapper[4892]: E1030 19:59:07.374519    4892 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7ef76347a62902de8860eacaff47801435076f847731251161c12a26d4f90b60\": container with ID starting with 7ef76347a62902de8860eacaff47801435076f847731251161c12a26d4f90b60 not found: ID does not exist" containerID="7ef76347a62902de8860eacaff47801435076f847731251161c12a26d4f90b60"
Oct 30 19:59:07.374614 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:59:07.374556    4892 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7ef76347a62902de8860eacaff47801435076f847731251161c12a26d4f90b60"} err="failed to get container status \"7ef76347a62902de8860eacaff47801435076f847731251161c12a26d4f90b60\": rpc error: code = NotFound desc = could not find container \"7ef76347a62902de8860eacaff47801435076f847731251161c12a26d4f90b60\": container with ID starting with 7ef76347a62902de8860eacaff47801435076f847731251161c12a26d4f90b60 not found: ID does not exist"
Oct 30 19:59:07.527132 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:59:07.527047    4892 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-b5178f-openshift-default-c8fbdff8c-bjzz7"]
Oct 30 19:59:07.537871 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:59:07.537840    4892 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w4-b5178f-openshift-default-c8fbdff8c-bjzz7"]
Oct 30 19:59:07.714660 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:59:07.714623    4892 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-1c5537-8xk7g"]
Oct 30 19:59:07.715459 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:59:07.715434    4892 cpu_manager.go:414] "containerMap: removing container" podUID="abea5659-2a38-406f-8221-66aefbc769b7" containerName="istio-proxy"
Oct 30 19:59:07.715459 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:59:07.715457    4892 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="abea5659-2a38-406f-8221-66aefbc769b7" containerName="istio-proxy"
Oct 30 19:59:07.715647 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:59:07.715484    4892 cpu_manager.go:414] "containerMap: removing container" podUID="558323d9-9532-440c-a95b-f367d8d0df78" containerName="payload-pre-processing"
Oct 30 19:59:07.715647 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:59:07.715490    4892 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="558323d9-9532-440c-a95b-f367d8d0df78" containerName="payload-pre-processing"
Oct 30 19:59:07.715647 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:59:07.715511    4892 cpu_manager.go:414] "containerMap: removing container" podUID="98aae325-07a2-49fc-a4ba-3e6709dc7c31" containerName="istio-proxy"
Oct 30 19:59:07.715647 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:59:07.715517    4892 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="98aae325-07a2-49fc-a4ba-3e6709dc7c31" containerName="istio-proxy"
Oct 30 19:59:07.715647 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:59:07.715539    4892 cpu_manager.go:414] "containerMap: removing container" podUID="51ed721a-e066-4f02-bea5-27d6598591bc" containerName="revoke-keys"
Oct 30 19:59:07.715647 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:59:07.715546    4892 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="51ed721a-e066-4f02-bea5-27d6598591bc" containerName="revoke-keys"
Oct 30 19:59:07.715647 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:59:07.715557    4892 cpu_manager.go:414] "containerMap: removing container" podUID="d126159b-6468-4c98-808f-a4243e83f11b" containerName="revoke-keys"
Oct 30 19:59:07.715647 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:59:07.715561    4892 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d126159b-6468-4c98-808f-a4243e83f11b" containerName="revoke-keys"
Oct 30 19:59:07.715647 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:59:07.715572    4892 cpu_manager.go:414] "containerMap: removing container" podUID="78ea3297-af25-446c-b7d0-3da5ae88dd15" containerName="istio-proxy"
Oct 30 19:59:07.715647 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:59:07.715576    4892 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="78ea3297-af25-446c-b7d0-3da5ae88dd15" containerName="istio-proxy"
Oct 30 19:59:07.716088 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:59:07.715674    4892 memory_manager.go:365] "RemoveStaleState removing state" podUID="558323d9-9532-440c-a95b-f367d8d0df78" containerName="payload-pre-processing"
Oct 30 19:59:07.716088 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:59:07.715690    4892 memory_manager.go:365] "RemoveStaleState removing state" podUID="98aae325-07a2-49fc-a4ba-3e6709dc7c31" containerName="istio-proxy"
Oct 30 19:59:07.716088 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:59:07.715702    4892 memory_manager.go:365] "RemoveStaleState removing state" podUID="51ed721a-e066-4f02-bea5-27d6598591bc" containerName="revoke-keys"
Oct 30 19:59:07.716088 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:59:07.715710    4892 memory_manager.go:365] "RemoveStaleState removing state" podUID="d126159b-6468-4c98-808f-a4243e83f11b" containerName="revoke-keys"
Oct 30 19:59:07.716088 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:59:07.715717    4892 memory_manager.go:365] "RemoveStaleState removing state" podUID="78ea3297-af25-446c-b7d0-3da5ae88dd15" containerName="istio-proxy"
Oct 30 19:59:07.716088 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:59:07.715723    4892 memory_manager.go:365] "RemoveStaleState removing state" podUID="abea5659-2a38-406f-8221-66aefbc769b7" containerName="istio-proxy"
Oct 30 19:59:07.721352 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:59:07.721313    4892 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-1c5537-8xk7g"
Oct 30 19:59:07.778995 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:59:07.774185    4892 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 30 19:59:07.778995 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:59:07.774819    4892 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 30 19:59:07.778995 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:59:07.778845    4892 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-1c5537-8xk7g"]
Oct 30 19:59:07.819122 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:59:07.819080    4892 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-b6jmm\" (UniqueName: \"kubernetes.io/projected/6cd3a5c6-0c31-49f6-ba5b-e1e02d756f1a-kube-api-access-b6jmm\") pod \"maas-api-revoke-keys-e2e-api-b-1c5537-8xk7g\" (UID: \"6cd3a5c6-0c31-49f6-ba5b-e1e02d756f1a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-1c5537-8xk7g"
Oct 30 19:59:07.819300 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:59:07.819147    4892 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6cd3a5c6-0c31-49f6-ba5b-e1e02d756f1a-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-api-b-1c5537-8xk7g\" (UID: \"6cd3a5c6-0c31-49f6-ba5b-e1e02d756f1a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-1c5537-8xk7g"
Oct 30 19:59:07.819300 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:59:07.819235    4892 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6cd3a5c6-0c31-49f6-ba5b-e1e02d756f1a-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-api-b-1c5537-8xk7g\" (UID: \"6cd3a5c6-0c31-49f6-ba5b-e1e02d756f1a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-1c5537-8xk7g"
Oct 30 19:59:07.859684 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:59:07.859656    4892 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 19:59:07.859868 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:59:07.859850    4892 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-nlxp4\""
Oct 30 19:59:07.920516 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:59:07.920484    4892 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6cd3a5c6-0c31-49f6-ba5b-e1e02d756f1a-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-api-b-1c5537-8xk7g\" (UID: \"6cd3a5c6-0c31-49f6-ba5b-e1e02d756f1a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-1c5537-8xk7g"
Oct 30 19:59:07.920694 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:59:07.920581    4892 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6cd3a5c6-0c31-49f6-ba5b-e1e02d756f1a-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-api-b-1c5537-8xk7g\" (UID: \"6cd3a5c6-0c31-49f6-ba5b-e1e02d756f1a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-1c5537-8xk7g"
Oct 30 19:59:07.920790 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:59:07.920774    4892 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-b6jmm\" (UniqueName: \"kubernetes.io/projected/6cd3a5c6-0c31-49f6-ba5b-e1e02d756f1a-kube-api-access-b6jmm\") pod \"maas-api-revoke-keys-e2e-api-b-1c5537-8xk7g\" (UID: \"6cd3a5c6-0c31-49f6-ba5b-e1e02d756f1a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-1c5537-8xk7g"
Oct 30 19:59:07.921181 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:59:07.921159    4892 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6cd3a5c6-0c31-49f6-ba5b-e1e02d756f1a-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-api-b-1c5537-8xk7g\" (UID: \"6cd3a5c6-0c31-49f6-ba5b-e1e02d756f1a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-1c5537-8xk7g"
Oct 30 19:59:07.923101 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:59:07.923077    4892 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6cd3a5c6-0c31-49f6-ba5b-e1e02d756f1a-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-api-b-1c5537-8xk7g\" (UID: \"6cd3a5c6-0c31-49f6-ba5b-e1e02d756f1a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-1c5537-8xk7g"
Oct 30 19:59:07.981157 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:59:07.981130    4892 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-b6jmm\" (UniqueName: \"kubernetes.io/projected/6cd3a5c6-0c31-49f6-ba5b-e1e02d756f1a-kube-api-access-b6jmm\") pod \"maas-api-revoke-keys-e2e-api-b-1c5537-8xk7g\" (UID: \"6cd3a5c6-0c31-49f6-ba5b-e1e02d756f1a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-1c5537-8xk7g"
Oct 30 19:59:08.036803 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:59:08.036735    4892 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-1c5537-8xk7g"
Oct 30 19:59:08.417797 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:59:08.417771    4892 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-1c5537-8xk7g"]
Oct 30 19:59:08.418815 ip-10-0-132-82 kubenswrapper[4892]: W1030 19:59:08.418780    4892 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6cd3a5c6_0c31_49f6_ba5b_e1e02d756f1a.slice/crio-954282c332a5a4207f802e790b2598c1c028404c61a2cd618fd1dff9d8e8f39c WatchSource:0}: Error finding container 954282c332a5a4207f802e790b2598c1c028404c61a2cd618fd1dff9d8e8f39c: Status 404 returned error can't find the container with id 954282c332a5a4207f802e790b2598c1c028404c61a2cd618fd1dff9d8e8f39c
Oct 30 19:59:08.570185 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:59:08.570154    4892 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="98aae325-07a2-49fc-a4ba-3e6709dc7c31" path="/var/lib/kubelet/pods/98aae325-07a2-49fc-a4ba-3e6709dc7c31/volumes"
Oct 30 19:59:08.847748 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:59:08.847691    4892 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-mt-dd9b8d79-67c8d6d46-mvgdr" podUID="4a87b754-ce9f-4ff5-855e-c27f1db74eb9" containerName="payload-processing" containerID="cri-o://9b9c0d8c510aec854417af981420eb4c6a3b43e9552f8349707e75d867bb8528" gracePeriod=25
Oct 30 19:59:08.954044 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:59:08.954005    4892 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-mt-dd9b8d79-867946f667-2p8kf" podUID="27ecc98d-6201-4d1c-b9b2-7df1fabbd48c" containerName="payload-pre-processing" containerID="cri-o://0ae383fca745efdaee5e56af30e55a8eed90e9b01a3afbbc6af0de259ca9dedd" gracePeriod=25
Oct 30 19:59:09.122519 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:59:09.122493    4892 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-dd9b8d79-67c8d6d46-mvgdr"
Oct 30 19:59:09.236403 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:59:09.236377    4892 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4a87b754-ce9f-4ff5-855e-c27f1db74eb9-kube-api-access-h86bt\" (UniqueName: \"kubernetes.io/projected/4a87b754-ce9f-4ff5-855e-c27f1db74eb9-kube-api-access-h86bt\") pod \"4a87b754-ce9f-4ff5-855e-c27f1db74eb9\" (UID: \"4a87b754-ce9f-4ff5-855e-c27f1db74eb9\") "
Oct 30 19:59:09.236567 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:59:09.236551    4892 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/4a87b754-ce9f-4ff5-855e-c27f1db74eb9-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4a87b754-ce9f-4ff5-855e-c27f1db74eb9-plugins-config-volume\") pod \"4a87b754-ce9f-4ff5-855e-c27f1db74eb9\" (UID: \"4a87b754-ce9f-4ff5-855e-c27f1db74eb9\") "
Oct 30 19:59:09.236979 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:59:09.236934    4892 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4a87b754-ce9f-4ff5-855e-c27f1db74eb9-plugins-config-volume" pod "4a87b754-ce9f-4ff5-855e-c27f1db74eb9" (UID: "4a87b754-ce9f-4ff5-855e-c27f1db74eb9"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 19:59:09.237281 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:59:09.237049    4892 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4a87b754-ce9f-4ff5-855e-c27f1db74eb9-plugins-config-volume\") on node \"ip-10-0-132-82.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 19:59:09.238675 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:59:09.238640    4892 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4a87b754-ce9f-4ff5-855e-c27f1db74eb9-kube-api-access-h86bt" pod "4a87b754-ce9f-4ff5-855e-c27f1db74eb9" (UID: "4a87b754-ce9f-4ff5-855e-c27f1db74eb9"). InnerVolumeSpecName "kube-api-access-h86bt". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:59:09.254420 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:59:09.254391    4892 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-dd9b8d79-867946f667-2p8kf"
Oct 30 19:59:09.351163 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:59:09.344867    4892 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/27ecc98d-6201-4d1c-b9b2-7df1fabbd48c-kube-api-access-xpnvp\" (UniqueName: \"kubernetes.io/projected/27ecc98d-6201-4d1c-b9b2-7df1fabbd48c-kube-api-access-xpnvp\") pod \"27ecc98d-6201-4d1c-b9b2-7df1fabbd48c\" (UID: \"27ecc98d-6201-4d1c-b9b2-7df1fabbd48c\") "
Oct 30 19:59:09.351163 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:59:09.345203    4892 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/27ecc98d-6201-4d1c-b9b2-7df1fabbd48c-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/27ecc98d-6201-4d1c-b9b2-7df1fabbd48c-plugins-config-volume\") pod \"27ecc98d-6201-4d1c-b9b2-7df1fabbd48c\" (UID: \"27ecc98d-6201-4d1c-b9b2-7df1fabbd48c\") "
Oct 30 19:59:09.351163 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:59:09.345761    4892 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-h86bt\" (UniqueName: \"kubernetes.io/projected/4a87b754-ce9f-4ff5-855e-c27f1db74eb9-kube-api-access-h86bt\") on node \"ip-10-0-132-82.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 19:59:09.351163 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:59:09.347552    4892 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/27ecc98d-6201-4d1c-b9b2-7df1fabbd48c-plugins-config-volume" pod "27ecc98d-6201-4d1c-b9b2-7df1fabbd48c" (UID: "27ecc98d-6201-4d1c-b9b2-7df1fabbd48c"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 19:59:09.354600 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:59:09.354567    4892 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/27ecc98d-6201-4d1c-b9b2-7df1fabbd48c-kube-api-access-xpnvp" pod "27ecc98d-6201-4d1c-b9b2-7df1fabbd48c" (UID: "27ecc98d-6201-4d1c-b9b2-7df1fabbd48c"). InnerVolumeSpecName "kube-api-access-xpnvp". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:59:09.388060 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:59:09.388021    4892 generic.go:356] "Generic (PLEG): container finished" podID="4a87b754-ce9f-4ff5-855e-c27f1db74eb9" containerID="9b9c0d8c510aec854417af981420eb4c6a3b43e9552f8349707e75d867bb8528" exitCode=0
Oct 30 19:59:09.388902 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:59:09.388117    4892 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-dd9b8d79-67c8d6d46-mvgdr"
Oct 30 19:59:09.390377 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:59:09.388135    4892 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-dd9b8d79-67c8d6d46-mvgdr" event={"ID":"4a87b754-ce9f-4ff5-855e-c27f1db74eb9","Type":"ContainerDied","Data":"9b9c0d8c510aec854417af981420eb4c6a3b43e9552f8349707e75d867bb8528"}
Oct 30 19:59:09.391162 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:59:09.391133    4892 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-dd9b8d79-67c8d6d46-mvgdr" event={"ID":"4a87b754-ce9f-4ff5-855e-c27f1db74eb9","Type":"ContainerDied","Data":"84df25dfea9df9107ff4ac1eae2c6b9a27c7139f5c016482c28758e4016f0fff"}
Oct 30 19:59:09.391772 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:59:09.391312    4892 scope.go:122] "RemoveContainer" containerID="9b9c0d8c510aec854417af981420eb4c6a3b43e9552f8349707e75d867bb8528"
Oct 30 19:59:09.393399 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:59:09.393143    4892 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-1c5537-8xk7g" event={"ID":"6cd3a5c6-0c31-49f6-ba5b-e1e02d756f1a","Type":"ContainerStarted","Data":"db349bb08440b61c12da3989ce700c20f39a84d1a9ad1874f6abd0144a6f7827"}
Oct 30 19:59:09.393399 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:59:09.393178    4892 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-1c5537-8xk7g" event={"ID":"6cd3a5c6-0c31-49f6-ba5b-e1e02d756f1a","Type":"ContainerStarted","Data":"954282c332a5a4207f802e790b2598c1c028404c61a2cd618fd1dff9d8e8f39c"}
Oct 30 19:59:09.395342 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:59:09.395238    4892 generic.go:356] "Generic (PLEG): container finished" podID="27ecc98d-6201-4d1c-b9b2-7df1fabbd48c" containerID="0ae383fca745efdaee5e56af30e55a8eed90e9b01a3afbbc6af0de259ca9dedd" exitCode=0
Oct 30 19:59:09.395611 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:59:09.395553    4892 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-dd9b8d79-867946f667-2p8kf" event={"ID":"27ecc98d-6201-4d1c-b9b2-7df1fabbd48c","Type":"ContainerDied","Data":"0ae383fca745efdaee5e56af30e55a8eed90e9b01a3afbbc6af0de259ca9dedd"}
Oct 30 19:59:09.395807 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:59:09.395733    4892 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-dd9b8d79-867946f667-2p8kf" event={"ID":"27ecc98d-6201-4d1c-b9b2-7df1fabbd48c","Type":"ContainerDied","Data":"fea1d9ae0fbbc227a577e5a2e46fa9f23a5eab009aa820f0a5171b8cac33608d"}
Oct 30 19:59:09.395984 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:59:09.395940    4892 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-dd9b8d79-867946f667-2p8kf"
Oct 30 19:59:09.407306 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:59:09.407085    4892 scope.go:122] "RemoveContainer" containerID="9b9c0d8c510aec854417af981420eb4c6a3b43e9552f8349707e75d867bb8528"
Oct 30 19:59:09.407845 ip-10-0-132-82 kubenswrapper[4892]: E1030 19:59:09.407536    4892 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9b9c0d8c510aec854417af981420eb4c6a3b43e9552f8349707e75d867bb8528\": container with ID starting with 9b9c0d8c510aec854417af981420eb4c6a3b43e9552f8349707e75d867bb8528 not found: ID does not exist" containerID="9b9c0d8c510aec854417af981420eb4c6a3b43e9552f8349707e75d867bb8528"
Oct 30 19:59:09.407845 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:59:09.407574    4892 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9b9c0d8c510aec854417af981420eb4c6a3b43e9552f8349707e75d867bb8528"} err="failed to get container status \"9b9c0d8c510aec854417af981420eb4c6a3b43e9552f8349707e75d867bb8528\": rpc error: code = NotFound desc = could not find container \"9b9c0d8c510aec854417af981420eb4c6a3b43e9552f8349707e75d867bb8528\": container with ID starting with 9b9c0d8c510aec854417af981420eb4c6a3b43e9552f8349707e75d867bb8528 not found: ID does not exist"
Oct 30 19:59:09.407845 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:59:09.407601    4892 scope.go:122] "RemoveContainer" containerID="0ae383fca745efdaee5e56af30e55a8eed90e9b01a3afbbc6af0de259ca9dedd"
Oct 30 19:59:09.422154 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:59:09.422130    4892 scope.go:122] "RemoveContainer" containerID="0ae383fca745efdaee5e56af30e55a8eed90e9b01a3afbbc6af0de259ca9dedd"
Oct 30 19:59:09.422812 ip-10-0-132-82 kubenswrapper[4892]: E1030 19:59:09.422790    4892 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0ae383fca745efdaee5e56af30e55a8eed90e9b01a3afbbc6af0de259ca9dedd\": container with ID starting with 0ae383fca745efdaee5e56af30e55a8eed90e9b01a3afbbc6af0de259ca9dedd not found: ID does not exist" containerID="0ae383fca745efdaee5e56af30e55a8eed90e9b01a3afbbc6af0de259ca9dedd"
Oct 30 19:59:09.423332 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:59:09.423303    4892 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0ae383fca745efdaee5e56af30e55a8eed90e9b01a3afbbc6af0de259ca9dedd"} err="failed to get container status \"0ae383fca745efdaee5e56af30e55a8eed90e9b01a3afbbc6af0de259ca9dedd\": rpc error: code = NotFound desc = could not find container \"0ae383fca745efdaee5e56af30e55a8eed90e9b01a3afbbc6af0de259ca9dedd\": container with ID starting with 0ae383fca745efdaee5e56af30e55a8eed90e9b01a3afbbc6af0de259ca9dedd not found: ID does not exist"
Oct 30 19:59:09.446498 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:59:09.446478    4892 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xpnvp\" (UniqueName: \"kubernetes.io/projected/27ecc98d-6201-4d1c-b9b2-7df1fabbd48c-kube-api-access-xpnvp\") on node \"ip-10-0-132-82.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 19:59:09.446498 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:59:09.446501    4892 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/27ecc98d-6201-4d1c-b9b2-7df1fabbd48c-plugins-config-volume\") on node \"ip-10-0-132-82.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 19:59:09.591054 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:59:09.581796    4892 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-1c5537-8xk7g" podStartSLOduration=2.5817770490000003 podStartE2EDuration="2.581777049s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:59:07 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:59:09.563403715 +0000 UTC m=+2597.133869327" watchObservedRunningTime="2026-10-30 19:59:09.581777049 +0000 UTC m=+2597.152242649"
Oct 30 19:59:09.698532 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:59:09.697061    4892 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-dd9b8d79-867946f667-2p8kf"]
Oct 30 19:59:09.707395 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:59:09.707360    4892 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-dd9b8d79-867946f667-2p8kf"]
Oct 30 19:59:09.802365 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:59:09.802334    4892 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-dd9b8d79-67c8d6d46-mvgdr"]
Oct 30 19:59:09.805798 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:59:09.805777    4892 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-dd9b8d79-67c8d6d46-mvgdr"]
Oct 30 19:59:10.405158 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:59:10.405123    4892 generic.go:356] "Generic (PLEG): container finished" podID="6cd3a5c6-0c31-49f6-ba5b-e1e02d756f1a" containerID="db349bb08440b61c12da3989ce700c20f39a84d1a9ad1874f6abd0144a6f7827" exitCode=0
Oct 30 19:59:10.405359 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:59:10.405212    4892 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-1c5537-8xk7g" event={"ID":"6cd3a5c6-0c31-49f6-ba5b-e1e02d756f1a","Type":"ContainerDied","Data":"db349bb08440b61c12da3989ce700c20f39a84d1a9ad1874f6abd0144a6f7827"}
Oct 30 19:59:10.570898 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:59:10.570854    4892 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="27ecc98d-6201-4d1c-b9b2-7df1fabbd48c" path="/var/lib/kubelet/pods/27ecc98d-6201-4d1c-b9b2-7df1fabbd48c/volumes"
Oct 30 19:59:10.571426 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:59:10.571397    4892 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="4a87b754-ce9f-4ff5-855e-c27f1db74eb9" path="/var/lib/kubelet/pods/4a87b754-ce9f-4ff5-855e-c27f1db74eb9/volumes"
Oct 30 19:59:11.450986 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:59:11.445491    4892 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-75b38d-rhqx2"]
Oct 30 19:59:11.460041 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:59:11.460007    4892 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-75b38d-rhqx2"]
Oct 30 19:59:11.639945 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:59:11.639920    4892 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-1c5537-8xk7g"
Oct 30 19:59:11.782262 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:59:11.782174    4892 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/6cd3a5c6-0c31-49f6-ba5b-e1e02d756f1a-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6cd3a5c6-0c31-49f6-ba5b-e1e02d756f1a-maas-api-xks-ca\") pod \"6cd3a5c6-0c31-49f6-ba5b-e1e02d756f1a\" (UID: \"6cd3a5c6-0c31-49f6-ba5b-e1e02d756f1a\") "
Oct 30 19:59:11.782427 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:59:11.782350    4892 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/6cd3a5c6-0c31-49f6-ba5b-e1e02d756f1a-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6cd3a5c6-0c31-49f6-ba5b-e1e02d756f1a-maas-api-openshift-ca\") pod \"6cd3a5c6-0c31-49f6-ba5b-e1e02d756f1a\" (UID: \"6cd3a5c6-0c31-49f6-ba5b-e1e02d756f1a\") "
Oct 30 19:59:11.782427 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:59:11.782391    4892 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6cd3a5c6-0c31-49f6-ba5b-e1e02d756f1a-kube-api-access-b6jmm\" (UniqueName: \"kubernetes.io/projected/6cd3a5c6-0c31-49f6-ba5b-e1e02d756f1a-kube-api-access-b6jmm\") pod \"6cd3a5c6-0c31-49f6-ba5b-e1e02d756f1a\" (UID: \"6cd3a5c6-0c31-49f6-ba5b-e1e02d756f1a\") "
Oct 30 19:59:11.782717 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:59:11.782694    4892 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6cd3a5c6-0c31-49f6-ba5b-e1e02d756f1a-maas-api-openshift-ca" pod "6cd3a5c6-0c31-49f6-ba5b-e1e02d756f1a" (UID: "6cd3a5c6-0c31-49f6-ba5b-e1e02d756f1a"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 19:59:11.782921 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:59:11.782904    4892 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6cd3a5c6-0c31-49f6-ba5b-e1e02d756f1a-maas-api-openshift-ca\") on node \"ip-10-0-132-82.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 19:59:11.784724 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:59:11.784701    4892 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6cd3a5c6-0c31-49f6-ba5b-e1e02d756f1a-maas-api-xks-ca" pod "6cd3a5c6-0c31-49f6-ba5b-e1e02d756f1a" (UID: "6cd3a5c6-0c31-49f6-ba5b-e1e02d756f1a"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 19:59:11.784852 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:59:11.784837    4892 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6cd3a5c6-0c31-49f6-ba5b-e1e02d756f1a-kube-api-access-b6jmm" pod "6cd3a5c6-0c31-49f6-ba5b-e1e02d756f1a" (UID: "6cd3a5c6-0c31-49f6-ba5b-e1e02d756f1a"). InnerVolumeSpecName "kube-api-access-b6jmm". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:59:11.884547 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:59:11.884518    4892 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6cd3a5c6-0c31-49f6-ba5b-e1e02d756f1a-maas-api-xks-ca\") on node \"ip-10-0-132-82.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 19:59:11.884547 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:59:11.884545    4892 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-b6jmm\" (UniqueName: \"kubernetes.io/projected/6cd3a5c6-0c31-49f6-ba5b-e1e02d756f1a-kube-api-access-b6jmm\") on node \"ip-10-0-132-82.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 19:59:12.471562 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:59:12.471519    4892 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-1c5537-8xk7g" event={"ID":"6cd3a5c6-0c31-49f6-ba5b-e1e02d756f1a","Type":"ContainerDied","Data":"954282c332a5a4207f802e790b2598c1c028404c61a2cd618fd1dff9d8e8f39c"}
Oct 30 19:59:12.471755 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:59:12.471597    4892 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="954282c332a5a4207f802e790b2598c1c028404c61a2cd618fd1dff9d8e8f39c"
Oct 30 19:59:12.471755 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:59:12.471536    4892 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-1c5537-8xk7g"
Oct 30 19:59:12.571386 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:59:12.571345    4892 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="51ed721a-e066-4f02-bea5-27d6598591bc" path="/var/lib/kubelet/pods/51ed721a-e066-4f02-bea5-27d6598591bc/volumes"
Oct 30 19:59:12.739514 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:59:12.739437    4892 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-cb14ed-2sl4x"]
Oct 30 19:59:12.744249 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:59:12.744226    4892 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-cb14ed-2sl4x"]
Oct 30 19:59:14.584014 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:59:14.583977    4892 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="10fe3864-9417-4f6d-84ad-019943adf796" path="/var/lib/kubelet/pods/10fe3864-9417-4f6d-84ad-019943adf796/volumes"
Oct 30 19:59:19.575284 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:59:19.575244    4892 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w0-2e5977-67qhn"]
Oct 30 19:59:19.576611 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:59:19.576575    4892 cpu_manager.go:414] "containerMap: removing container" podUID="27ecc98d-6201-4d1c-b9b2-7df1fabbd48c" containerName="payload-pre-processing"
Oct 30 19:59:19.576611 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:59:19.576612    4892 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="27ecc98d-6201-4d1c-b9b2-7df1fabbd48c" containerName="payload-pre-processing"
Oct 30 19:59:19.576807 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:59:19.576675    4892 cpu_manager.go:414] "containerMap: removing container" podUID="6cd3a5c6-0c31-49f6-ba5b-e1e02d756f1a" containerName="revoke-keys"
Oct 30 19:59:19.576807 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:59:19.576684    4892 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6cd3a5c6-0c31-49f6-ba5b-e1e02d756f1a" containerName="revoke-keys"
Oct 30 19:59:19.576807 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:59:19.576710    4892 cpu_manager.go:414] "containerMap: removing container" podUID="4a87b754-ce9f-4ff5-855e-c27f1db74eb9" containerName="payload-processing"
Oct 30 19:59:19.576807 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:59:19.576718    4892 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4a87b754-ce9f-4ff5-855e-c27f1db74eb9" containerName="payload-processing"
Oct 30 19:59:19.577466 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:59:19.577441    4892 memory_manager.go:365] "RemoveStaleState removing state" podUID="27ecc98d-6201-4d1c-b9b2-7df1fabbd48c" containerName="payload-pre-processing"
Oct 30 19:59:19.577570 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:59:19.577487    4892 memory_manager.go:365] "RemoveStaleState removing state" podUID="6cd3a5c6-0c31-49f6-ba5b-e1e02d756f1a" containerName="revoke-keys"
Oct 30 19:59:19.577570 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:59:19.577507    4892 memory_manager.go:365] "RemoveStaleState removing state" podUID="4a87b754-ce9f-4ff5-855e-c27f1db74eb9" containerName="payload-processing"
Oct 30 19:59:19.588257 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:59:19.587734    4892 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w0-2e5977-67qhn"
Oct 30 19:59:19.638003 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:59:19.637955    4892 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w0-2e5977-67qhn"]
Oct 30 19:59:19.682343 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:59:19.682315    4892 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 19:59:19.682343 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:59:19.682335    4892 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 30 19:59:19.682508 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:59:19.682371    4892 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 30 19:59:19.687216 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:59:19.687191    4892 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/135c03e8-7715-418e-96b8-6257bc7023d0-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w0-2e5977-67qhn\" (UID: \"135c03e8-7715-418e-96b8-6257bc7023d0\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w0-2e5977-67qhn"
Oct 30 19:59:19.687342 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:59:19.687298    4892 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2n5l2\" (UniqueName: \"kubernetes.io/projected/135c03e8-7715-418e-96b8-6257bc7023d0-kube-api-access-2n5l2\") pod \"maas-api-revoke-keys-e2e-shared-a-w0-2e5977-67qhn\" (UID: \"135c03e8-7715-418e-96b8-6257bc7023d0\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w0-2e5977-67qhn"
Oct 30 19:59:19.687398 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:59:19.687338    4892 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/135c03e8-7715-418e-96b8-6257bc7023d0-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w0-2e5977-67qhn\" (UID: \"135c03e8-7715-418e-96b8-6257bc7023d0\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w0-2e5977-67qhn"
Oct 30 19:59:19.689627 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:59:19.689607    4892 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-nlxp4\""
Oct 30 19:59:19.788499 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:59:19.788473    4892 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2n5l2\" (UniqueName: \"kubernetes.io/projected/135c03e8-7715-418e-96b8-6257bc7023d0-kube-api-access-2n5l2\") pod \"maas-api-revoke-keys-e2e-shared-a-w0-2e5977-67qhn\" (UID: \"135c03e8-7715-418e-96b8-6257bc7023d0\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w0-2e5977-67qhn"
Oct 30 19:59:19.788654 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:59:19.788512    4892 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/135c03e8-7715-418e-96b8-6257bc7023d0-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w0-2e5977-67qhn\" (UID: \"135c03e8-7715-418e-96b8-6257bc7023d0\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w0-2e5977-67qhn"
Oct 30 19:59:19.788654 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:59:19.788635    4892 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/135c03e8-7715-418e-96b8-6257bc7023d0-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w0-2e5977-67qhn\" (UID: \"135c03e8-7715-418e-96b8-6257bc7023d0\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w0-2e5977-67qhn"
Oct 30 19:59:19.789270 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:59:19.789238    4892 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/135c03e8-7715-418e-96b8-6257bc7023d0-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w0-2e5977-67qhn\" (UID: \"135c03e8-7715-418e-96b8-6257bc7023d0\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w0-2e5977-67qhn"
Oct 30 19:59:19.791207 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:59:19.791184    4892 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/135c03e8-7715-418e-96b8-6257bc7023d0-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w0-2e5977-67qhn\" (UID: \"135c03e8-7715-418e-96b8-6257bc7023d0\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w0-2e5977-67qhn"
Oct 30 19:59:19.845868 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:59:19.845801    4892 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2n5l2\" (UniqueName: \"kubernetes.io/projected/135c03e8-7715-418e-96b8-6257bc7023d0-kube-api-access-2n5l2\") pod \"maas-api-revoke-keys-e2e-shared-a-w0-2e5977-67qhn\" (UID: \"135c03e8-7715-418e-96b8-6257bc7023d0\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w0-2e5977-67qhn"
Oct 30 19:59:19.902325 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:59:19.902301    4892 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w0-2e5977-67qhn"
Oct 30 19:59:20.283423 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:59:20.283396    4892 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w0-2e5977-67qhn"]
Oct 30 19:59:20.284810 ip-10-0-132-82 kubenswrapper[4892]: W1030 19:59:20.284778    4892 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod135c03e8_7715_418e_96b8_6257bc7023d0.slice/crio-cf3e9118ab139dd62b68894a1b307390382f0dfedff400cdd6f4b7131c6a3637 WatchSource:0}: Error finding container cf3e9118ab139dd62b68894a1b307390382f0dfedff400cdd6f4b7131c6a3637: Status 404 returned error can't find the container with id cf3e9118ab139dd62b68894a1b307390382f0dfedff400cdd6f4b7131c6a3637
Oct 30 19:59:20.533495 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:59:20.533425    4892 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w0-2e5977-67qhn" event={"ID":"135c03e8-7715-418e-96b8-6257bc7023d0","Type":"ContainerStarted","Data":"cf3e9118ab139dd62b68894a1b307390382f0dfedff400cdd6f4b7131c6a3637"}
Oct 30 19:59:21.170315 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:59:21.170283    4892 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-1c5537-7f86d78844-ljthj"]
Oct 30 19:59:21.275559 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:59:21.275528    4892 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-1c5537-6d5ffd475f-8jtnh"]
Oct 30 19:59:21.547660 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:59:21.547570    4892 generic.go:356] "Generic (PLEG): container finished" podID="135c03e8-7715-418e-96b8-6257bc7023d0" containerID="ab6ce328922b158ffae34102b4c2e083ef1f419a8e5fbe377997ffd3c6e9cb2a" exitCode=0
Oct 30 19:59:21.547849 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:59:21.547656    4892 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w0-2e5977-67qhn" event={"ID":"135c03e8-7715-418e-96b8-6257bc7023d0","Type":"ContainerDied","Data":"ab6ce328922b158ffae34102b4c2e083ef1f419a8e5fbe377997ffd3c6e9cb2a"}
Oct 30 19:59:22.342490 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:59:22.342443    4892 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-da4c62-mbvvh"]
Oct 30 19:59:22.347803 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:59:22.347776    4892 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-da4c62-mbvvh"
Oct 30 19:59:22.409652 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:59:22.409620    4892 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-da4c62-mbvvh"]
Oct 30 19:59:22.453417 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:59:22.453382    4892 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/833f7eac-9bb0-4ba9-9192-158d25933978-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-api-b-da4c62-mbvvh\" (UID: \"833f7eac-9bb0-4ba9-9192-158d25933978\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-da4c62-mbvvh"
Oct 30 19:59:22.453599 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:59:22.453435    4892 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/833f7eac-9bb0-4ba9-9192-158d25933978-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-api-b-da4c62-mbvvh\" (UID: \"833f7eac-9bb0-4ba9-9192-158d25933978\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-da4c62-mbvvh"
Oct 30 19:59:22.453743 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:59:22.453716    4892 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7ndkl\" (UniqueName: \"kubernetes.io/projected/833f7eac-9bb0-4ba9-9192-158d25933978-kube-api-access-7ndkl\") pod \"maas-api-revoke-keys-e2e-api-b-da4c62-mbvvh\" (UID: \"833f7eac-9bb0-4ba9-9192-158d25933978\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-da4c62-mbvvh"
Oct 30 19:59:22.555255 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:59:22.555222    4892 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7ndkl\" (UniqueName: \"kubernetes.io/projected/833f7eac-9bb0-4ba9-9192-158d25933978-kube-api-access-7ndkl\") pod \"maas-api-revoke-keys-e2e-api-b-da4c62-mbvvh\" (UID: \"833f7eac-9bb0-4ba9-9192-158d25933978\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-da4c62-mbvvh"
Oct 30 19:59:22.555461 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:59:22.555291    4892 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/833f7eac-9bb0-4ba9-9192-158d25933978-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-api-b-da4c62-mbvvh\" (UID: \"833f7eac-9bb0-4ba9-9192-158d25933978\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-da4c62-mbvvh"
Oct 30 19:59:22.555461 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:59:22.555355    4892 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/833f7eac-9bb0-4ba9-9192-158d25933978-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-api-b-da4c62-mbvvh\" (UID: \"833f7eac-9bb0-4ba9-9192-158d25933978\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-da4c62-mbvvh"
Oct 30 19:59:22.556200 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:59:22.556107    4892 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/833f7eac-9bb0-4ba9-9192-158d25933978-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-api-b-da4c62-mbvvh\" (UID: \"833f7eac-9bb0-4ba9-9192-158d25933978\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-da4c62-mbvvh"
Oct 30 19:59:22.557978 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:59:22.557941    4892 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/833f7eac-9bb0-4ba9-9192-158d25933978-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-api-b-da4c62-mbvvh\" (UID: \"833f7eac-9bb0-4ba9-9192-158d25933978\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-da4c62-mbvvh"
Oct 30 19:59:22.612514 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:59:22.612448    4892 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7ndkl\" (UniqueName: \"kubernetes.io/projected/833f7eac-9bb0-4ba9-9192-158d25933978-kube-api-access-7ndkl\") pod \"maas-api-revoke-keys-e2e-api-b-da4c62-mbvvh\" (UID: \"833f7eac-9bb0-4ba9-9192-158d25933978\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-da4c62-mbvvh"
Oct 30 19:59:22.665377 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:59:22.665339    4892 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-da4c62-mbvvh"
Oct 30 19:59:22.766982 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:59:22.766927    4892 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w0-2e5977-67qhn"
Oct 30 19:59:22.859490 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:59:22.859377    4892 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/135c03e8-7715-418e-96b8-6257bc7023d0-kube-api-access-2n5l2\" (UniqueName: \"kubernetes.io/projected/135c03e8-7715-418e-96b8-6257bc7023d0-kube-api-access-2n5l2\") pod \"135c03e8-7715-418e-96b8-6257bc7023d0\" (UID: \"135c03e8-7715-418e-96b8-6257bc7023d0\") "
Oct 30 19:59:22.859646 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:59:22.859518    4892 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/135c03e8-7715-418e-96b8-6257bc7023d0-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/135c03e8-7715-418e-96b8-6257bc7023d0-maas-api-xks-ca\") pod \"135c03e8-7715-418e-96b8-6257bc7023d0\" (UID: \"135c03e8-7715-418e-96b8-6257bc7023d0\") "
Oct 30 19:59:22.859646 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:59:22.859589    4892 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/135c03e8-7715-418e-96b8-6257bc7023d0-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/135c03e8-7715-418e-96b8-6257bc7023d0-maas-api-openshift-ca\") pod \"135c03e8-7715-418e-96b8-6257bc7023d0\" (UID: \"135c03e8-7715-418e-96b8-6257bc7023d0\") "
Oct 30 19:59:22.860439 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:59:22.860411    4892 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/135c03e8-7715-418e-96b8-6257bc7023d0-maas-api-openshift-ca" pod "135c03e8-7715-418e-96b8-6257bc7023d0" (UID: "135c03e8-7715-418e-96b8-6257bc7023d0"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 19:59:22.862556 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:59:22.862484    4892 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/135c03e8-7715-418e-96b8-6257bc7023d0-maas-api-xks-ca" pod "135c03e8-7715-418e-96b8-6257bc7023d0" (UID: "135c03e8-7715-418e-96b8-6257bc7023d0"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 19:59:22.862820 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:59:22.862790    4892 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/135c03e8-7715-418e-96b8-6257bc7023d0-kube-api-access-2n5l2" pod "135c03e8-7715-418e-96b8-6257bc7023d0" (UID: "135c03e8-7715-418e-96b8-6257bc7023d0"). InnerVolumeSpecName "kube-api-access-2n5l2". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:59:22.961000 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:59:22.960927    4892 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2n5l2\" (UniqueName: \"kubernetes.io/projected/135c03e8-7715-418e-96b8-6257bc7023d0-kube-api-access-2n5l2\") on node \"ip-10-0-132-82.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 19:59:22.961000 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:59:22.960996    4892 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/135c03e8-7715-418e-96b8-6257bc7023d0-maas-api-xks-ca\") on node \"ip-10-0-132-82.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 19:59:22.961000 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:59:22.961011    4892 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/135c03e8-7715-418e-96b8-6257bc7023d0-maas-api-openshift-ca\") on node \"ip-10-0-132-82.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 19:59:23.169397 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:59:23.169368    4892 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-da4c62-mbvvh"]
Oct 30 19:59:23.170157 ip-10-0-132-82 kubenswrapper[4892]: W1030 19:59:23.170127    4892 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod833f7eac_9bb0_4ba9_9192_158d25933978.slice/crio-6f091ffa9a5d6409c77862f1a87fde4bb4bbdd70736c6bbc52f6a4e39e92aec2 WatchSource:0}: Error finding container 6f091ffa9a5d6409c77862f1a87fde4bb4bbdd70736c6bbc52f6a4e39e92aec2: Status 404 returned error can't find the container with id 6f091ffa9a5d6409c77862f1a87fde4bb4bbdd70736c6bbc52f6a4e39e92aec2
Oct 30 19:59:23.566311 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:59:23.566277    4892 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w0-2e5977-67qhn"
Oct 30 19:59:23.566311 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:59:23.566302    4892 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w0-2e5977-67qhn" event={"ID":"135c03e8-7715-418e-96b8-6257bc7023d0","Type":"ContainerDied","Data":"cf3e9118ab139dd62b68894a1b307390382f0dfedff400cdd6f4b7131c6a3637"}
Oct 30 19:59:23.566819 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:59:23.566350    4892 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="cf3e9118ab139dd62b68894a1b307390382f0dfedff400cdd6f4b7131c6a3637"
Oct 30 19:59:23.568398 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:59:23.568377    4892 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-da4c62-mbvvh" event={"ID":"833f7eac-9bb0-4ba9-9192-158d25933978","Type":"ContainerStarted","Data":"1e8d5e7453ec2ed5c8cb8c32fc0dd1685d911fa58453abb0baa70ee7fb0dd2db"}
Oct 30 19:59:23.568530 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:59:23.568417    4892 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-da4c62-mbvvh" event={"ID":"833f7eac-9bb0-4ba9-9192-158d25933978","Type":"ContainerStarted","Data":"6f091ffa9a5d6409c77862f1a87fde4bb4bbdd70736c6bbc52f6a4e39e92aec2"}
Oct 30 19:59:23.685847 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:59:23.685789    4892 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-da4c62-mbvvh" podStartSLOduration=1.685775022 podStartE2EDuration="1.685775022s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:59:22 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:59:23.683127172 +0000 UTC m=+2611.253592750" watchObservedRunningTime="2026-10-30 19:59:23.685775022 +0000 UTC m=+2611.256240620"
Oct 30 19:59:24.577604 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:59:24.577525    4892 generic.go:356] "Generic (PLEG): container finished" podID="833f7eac-9bb0-4ba9-9192-158d25933978" containerID="1e8d5e7453ec2ed5c8cb8c32fc0dd1685d911fa58453abb0baa70ee7fb0dd2db" exitCode=0
Oct 30 19:59:24.577604 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:59:24.577590    4892 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-da4c62-mbvvh" event={"ID":"833f7eac-9bb0-4ba9-9192-158d25933978","Type":"ContainerDied","Data":"1e8d5e7453ec2ed5c8cb8c32fc0dd1685d911fa58453abb0baa70ee7fb0dd2db"}
Oct 30 19:59:24.844540 ip-10-0-132-82 kubenswrapper[4892]: E1030 19:59:24.844452    4892 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-jrm87" containerName="extractor"
Oct 30 19:59:25.736325 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:59:25.736300    4892 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-da4c62-mbvvh"
Oct 30 19:59:25.793350 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:59:25.793319    4892 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/833f7eac-9bb0-4ba9-9192-158d25933978-kube-api-access-7ndkl\" (UniqueName: \"kubernetes.io/projected/833f7eac-9bb0-4ba9-9192-158d25933978-kube-api-access-7ndkl\") pod \"833f7eac-9bb0-4ba9-9192-158d25933978\" (UID: \"833f7eac-9bb0-4ba9-9192-158d25933978\") "
Oct 30 19:59:25.793493 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:59:25.793426    4892 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/833f7eac-9bb0-4ba9-9192-158d25933978-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/833f7eac-9bb0-4ba9-9192-158d25933978-maas-api-openshift-ca\") pod \"833f7eac-9bb0-4ba9-9192-158d25933978\" (UID: \"833f7eac-9bb0-4ba9-9192-158d25933978\") "
Oct 30 19:59:25.793493 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:59:25.793457    4892 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/833f7eac-9bb0-4ba9-9192-158d25933978-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/833f7eac-9bb0-4ba9-9192-158d25933978-maas-api-xks-ca\") pod \"833f7eac-9bb0-4ba9-9192-158d25933978\" (UID: \"833f7eac-9bb0-4ba9-9192-158d25933978\") "
Oct 30 19:59:25.793788 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:59:25.793766    4892 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/833f7eac-9bb0-4ba9-9192-158d25933978-maas-api-openshift-ca" pod "833f7eac-9bb0-4ba9-9192-158d25933978" (UID: "833f7eac-9bb0-4ba9-9192-158d25933978"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 19:59:25.793959 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:59:25.793939    4892 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/833f7eac-9bb0-4ba9-9192-158d25933978-maas-api-openshift-ca\") on node \"ip-10-0-132-82.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 19:59:25.795653 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:59:25.795625    4892 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/833f7eac-9bb0-4ba9-9192-158d25933978-kube-api-access-7ndkl" pod "833f7eac-9bb0-4ba9-9192-158d25933978" (UID: "833f7eac-9bb0-4ba9-9192-158d25933978"). InnerVolumeSpecName "kube-api-access-7ndkl". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:59:25.795653 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:59:25.795635    4892 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/833f7eac-9bb0-4ba9-9192-158d25933978-maas-api-xks-ca" pod "833f7eac-9bb0-4ba9-9192-158d25933978" (UID: "833f7eac-9bb0-4ba9-9192-158d25933978"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 19:59:25.895230 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:59:25.895198    4892 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7ndkl\" (UniqueName: \"kubernetes.io/projected/833f7eac-9bb0-4ba9-9192-158d25933978-kube-api-access-7ndkl\") on node \"ip-10-0-132-82.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 19:59:25.895230 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:59:25.895226    4892 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/833f7eac-9bb0-4ba9-9192-158d25933978-maas-api-xks-ca\") on node \"ip-10-0-132-82.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 19:59:26.171538 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:59:26.171453    4892 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-api-b-1c5537-7f86d78844-ljthj" podUID="1ca3885d-5fb3-4c84-b1fc-2e9138885566" containerName="payload-processing" containerID="cri-o://1646839ada220254ef1045f78b020e60b97cf9fdea4608a14a738e1817b08f8c" gracePeriod=25
Oct 30 19:59:26.276882 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:59:26.276816    4892 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-api-b-1c5537-6d5ffd475f-8jtnh" podUID="81a5c6ce-725d-42b3-8bba-1f80fb5edfad" containerName="payload-pre-processing" containerID="cri-o://693139186d056a664db68dfaa475f1d09fbb7de7e57bc6ff817b6109ec03f6d4" gracePeriod=25
Oct 30 19:59:26.421851 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:59:26.421787    4892 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-b-1c5537-7f86d78844-ljthj"
Oct 30 19:59:26.502800 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:59:26.501838    4892 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/1ca3885d-5fb3-4c84-b1fc-2e9138885566-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1ca3885d-5fb3-4c84-b1fc-2e9138885566-plugins-config-volume\") pod \"1ca3885d-5fb3-4c84-b1fc-2e9138885566\" (UID: \"1ca3885d-5fb3-4c84-b1fc-2e9138885566\") "
Oct 30 19:59:26.502800 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:59:26.502103    4892 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1ca3885d-5fb3-4c84-b1fc-2e9138885566-kube-api-access-brwx6\" (UniqueName: \"kubernetes.io/projected/1ca3885d-5fb3-4c84-b1fc-2e9138885566-kube-api-access-brwx6\") pod \"1ca3885d-5fb3-4c84-b1fc-2e9138885566\" (UID: \"1ca3885d-5fb3-4c84-b1fc-2e9138885566\") "
Oct 30 19:59:26.503066 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:59:26.502941    4892 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1ca3885d-5fb3-4c84-b1fc-2e9138885566-plugins-config-volume" pod "1ca3885d-5fb3-4c84-b1fc-2e9138885566" (UID: "1ca3885d-5fb3-4c84-b1fc-2e9138885566"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 19:59:26.505395 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:59:26.505365    4892 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1ca3885d-5fb3-4c84-b1fc-2e9138885566-kube-api-access-brwx6" pod "1ca3885d-5fb3-4c84-b1fc-2e9138885566" (UID: "1ca3885d-5fb3-4c84-b1fc-2e9138885566"). InnerVolumeSpecName "kube-api-access-brwx6". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:59:26.552189 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:59:26.552162    4892 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-b-1c5537-6d5ffd475f-8jtnh"
Oct 30 19:59:26.602182 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:59:26.602152    4892 generic.go:356] "Generic (PLEG): container finished" podID="81a5c6ce-725d-42b3-8bba-1f80fb5edfad" containerID="693139186d056a664db68dfaa475f1d09fbb7de7e57bc6ff817b6109ec03f6d4" exitCode=0
Oct 30 19:59:26.602319 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:59:26.602206    4892 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-b-1c5537-6d5ffd475f-8jtnh"
Oct 30 19:59:26.602319 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:59:26.602233    4892 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-b-1c5537-6d5ffd475f-8jtnh" event={"ID":"81a5c6ce-725d-42b3-8bba-1f80fb5edfad","Type":"ContainerDied","Data":"693139186d056a664db68dfaa475f1d09fbb7de7e57bc6ff817b6109ec03f6d4"}
Oct 30 19:59:26.602319 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:59:26.602315    4892 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-b-1c5537-6d5ffd475f-8jtnh" event={"ID":"81a5c6ce-725d-42b3-8bba-1f80fb5edfad","Type":"ContainerDied","Data":"768b4e2d6742c5af526e3fdb2b1941959b9b704942403541f5417dfab6f94690"}
Oct 30 19:59:26.602453 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:59:26.602339    4892 scope.go:122] "RemoveContainer" containerID="693139186d056a664db68dfaa475f1d09fbb7de7e57bc6ff817b6109ec03f6d4"
Oct 30 19:59:26.603022 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:59:26.603003    4892 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/81a5c6ce-725d-42b3-8bba-1f80fb5edfad-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/81a5c6ce-725d-42b3-8bba-1f80fb5edfad-plugins-config-volume\") pod \"81a5c6ce-725d-42b3-8bba-1f80fb5edfad\" (UID: \"81a5c6ce-725d-42b3-8bba-1f80fb5edfad\") "
Oct 30 19:59:26.603164 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:59:26.603079    4892 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/81a5c6ce-725d-42b3-8bba-1f80fb5edfad-kube-api-access-2cb52\" (UniqueName: \"kubernetes.io/projected/81a5c6ce-725d-42b3-8bba-1f80fb5edfad-kube-api-access-2cb52\") pod \"81a5c6ce-725d-42b3-8bba-1f80fb5edfad\" (UID: \"81a5c6ce-725d-42b3-8bba-1f80fb5edfad\") "
Oct 30 19:59:26.603339 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:59:26.603318    4892 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/81a5c6ce-725d-42b3-8bba-1f80fb5edfad-plugins-config-volume" pod "81a5c6ce-725d-42b3-8bba-1f80fb5edfad" (UID: "81a5c6ce-725d-42b3-8bba-1f80fb5edfad"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 19:59:26.603621 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:59:26.603600    4892 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1ca3885d-5fb3-4c84-b1fc-2e9138885566-plugins-config-volume\") on node \"ip-10-0-132-82.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 19:59:26.603725 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:59:26.603626    4892 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/81a5c6ce-725d-42b3-8bba-1f80fb5edfad-plugins-config-volume\") on node \"ip-10-0-132-82.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 19:59:26.603725 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:59:26.603640    4892 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-brwx6\" (UniqueName: \"kubernetes.io/projected/1ca3885d-5fb3-4c84-b1fc-2e9138885566-kube-api-access-brwx6\") on node \"ip-10-0-132-82.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 19:59:26.604253 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:59:26.604230    4892 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-da4c62-mbvvh"
Oct 30 19:59:26.604520 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:59:26.604495    4892 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-da4c62-mbvvh" event={"ID":"833f7eac-9bb0-4ba9-9192-158d25933978","Type":"ContainerDied","Data":"6f091ffa9a5d6409c77862f1a87fde4bb4bbdd70736c6bbc52f6a4e39e92aec2"}
Oct 30 19:59:26.604611 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:59:26.604542    4892 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="6f091ffa9a5d6409c77862f1a87fde4bb4bbdd70736c6bbc52f6a4e39e92aec2"
Oct 30 19:59:26.606474 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:59:26.606446    4892 generic.go:356] "Generic (PLEG): container finished" podID="1ca3885d-5fb3-4c84-b1fc-2e9138885566" containerID="1646839ada220254ef1045f78b020e60b97cf9fdea4608a14a738e1817b08f8c" exitCode=0
Oct 30 19:59:26.606592 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:59:26.606531    4892 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-b-1c5537-7f86d78844-ljthj" event={"ID":"1ca3885d-5fb3-4c84-b1fc-2e9138885566","Type":"ContainerDied","Data":"1646839ada220254ef1045f78b020e60b97cf9fdea4608a14a738e1817b08f8c"}
Oct 30 19:59:26.606592 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:59:26.606545    4892 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/81a5c6ce-725d-42b3-8bba-1f80fb5edfad-kube-api-access-2cb52" pod "81a5c6ce-725d-42b3-8bba-1f80fb5edfad" (UID: "81a5c6ce-725d-42b3-8bba-1f80fb5edfad"). InnerVolumeSpecName "kube-api-access-2cb52". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:59:26.606592 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:59:26.606573    4892 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-b-1c5537-7f86d78844-ljthj" event={"ID":"1ca3885d-5fb3-4c84-b1fc-2e9138885566","Type":"ContainerDied","Data":"ae9366dd0d9e939c5a6b0ca53fd958bbac0b83a35ca20dd846f757f75a59c631"}
Oct 30 19:59:26.606779 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:59:26.606653    4892 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-b-1c5537-7f86d78844-ljthj"
Oct 30 19:59:26.614652 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:59:26.614633    4892 scope.go:122] "RemoveContainer" containerID="693139186d056a664db68dfaa475f1d09fbb7de7e57bc6ff817b6109ec03f6d4"
Oct 30 19:59:26.614921 ip-10-0-132-82 kubenswrapper[4892]: E1030 19:59:26.614901    4892 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"693139186d056a664db68dfaa475f1d09fbb7de7e57bc6ff817b6109ec03f6d4\": container with ID starting with 693139186d056a664db68dfaa475f1d09fbb7de7e57bc6ff817b6109ec03f6d4 not found: ID does not exist" containerID="693139186d056a664db68dfaa475f1d09fbb7de7e57bc6ff817b6109ec03f6d4"
Oct 30 19:59:26.615036 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:59:26.614932    4892 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"693139186d056a664db68dfaa475f1d09fbb7de7e57bc6ff817b6109ec03f6d4"} err="failed to get container status \"693139186d056a664db68dfaa475f1d09fbb7de7e57bc6ff817b6109ec03f6d4\": rpc error: code = NotFound desc = could not find container \"693139186d056a664db68dfaa475f1d09fbb7de7e57bc6ff817b6109ec03f6d4\": container with ID starting with 693139186d056a664db68dfaa475f1d09fbb7de7e57bc6ff817b6109ec03f6d4 not found: ID does not exist"
Oct 30 19:59:26.615036 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:59:26.614958    4892 scope.go:122] "RemoveContainer" containerID="1646839ada220254ef1045f78b020e60b97cf9fdea4608a14a738e1817b08f8c"
Oct 30 19:59:26.624560 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:59:26.624544    4892 scope.go:122] "RemoveContainer" containerID="1646839ada220254ef1045f78b020e60b97cf9fdea4608a14a738e1817b08f8c"
Oct 30 19:59:26.624826 ip-10-0-132-82 kubenswrapper[4892]: E1030 19:59:26.624807    4892 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1646839ada220254ef1045f78b020e60b97cf9fdea4608a14a738e1817b08f8c\": container with ID starting with 1646839ada220254ef1045f78b020e60b97cf9fdea4608a14a738e1817b08f8c not found: ID does not exist" containerID="1646839ada220254ef1045f78b020e60b97cf9fdea4608a14a738e1817b08f8c"
Oct 30 19:59:26.624885 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:59:26.624835    4892 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1646839ada220254ef1045f78b020e60b97cf9fdea4608a14a738e1817b08f8c"} err="failed to get container status \"1646839ada220254ef1045f78b020e60b97cf9fdea4608a14a738e1817b08f8c\": rpc error: code = NotFound desc = could not find container \"1646839ada220254ef1045f78b020e60b97cf9fdea4608a14a738e1817b08f8c\": container with ID starting with 1646839ada220254ef1045f78b020e60b97cf9fdea4608a14a738e1817b08f8c not found: ID does not exist"
Oct 30 19:59:26.705280 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:59:26.705218    4892 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2cb52\" (UniqueName: \"kubernetes.io/projected/81a5c6ce-725d-42b3-8bba-1f80fb5edfad-kube-api-access-2cb52\") on node \"ip-10-0-132-82.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 19:59:26.888766 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:59:26.888727    4892 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-1c5537-7f86d78844-ljthj"]
Oct 30 19:59:26.892331 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:59:26.892305    4892 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-1c5537-7f86d78844-ljthj"]
Oct 30 19:59:27.028272 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:59:27.028194    4892 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-1c5537-6d5ffd475f-8jtnh"]
Oct 30 19:59:27.031694 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:59:27.031671    4892 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-1c5537-6d5ffd475f-8jtnh"]
Oct 30 19:59:28.570564 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:59:28.570529    4892 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="1ca3885d-5fb3-4c84-b1fc-2e9138885566" path="/var/lib/kubelet/pods/1ca3885d-5fb3-4c84-b1fc-2e9138885566/volumes"
Oct 30 19:59:28.571109 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:59:28.571092    4892 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="81a5c6ce-725d-42b3-8bba-1f80fb5edfad" path="/var/lib/kubelet/pods/81a5c6ce-725d-42b3-8bba-1f80fb5edfad/volumes"
Oct 30 19:59:31.299422 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:59:31.299390    4892 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w0-2e5977-55d59bd866-nxr6c"]
Oct 30 19:59:31.405178 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:59:31.405147    4892 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w0-2e5977-5b974cf99f-mrcrv"]
Oct 30 19:59:33.936613 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:59:33.936582    4892 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-da4c62-5899dfc94d-2s5tq"]
Oct 30 19:59:34.068123 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:59:34.068087    4892 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-da4c62-79568d7967-nmj4w"]
Oct 30 19:59:36.300347 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:59:36.300292    4892 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-shared-a-w0-2e5977-55d59bd866-nxr6c" podUID="e0fa9dd2-6f90-485c-aaff-e343ccbe428a" containerName="payload-processing" containerID="cri-o://aa006a086c6d984fb4d0ec4d0b735358620913b45c55459cc930d37463706fb9" gracePeriod=25
Oct 30 19:59:36.407529 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:59:36.407476    4892 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w0-2e5977-5b974cf99f-mrcrv" podUID="f07835ed-2bb8-4dcc-8d53-9da5b4e2a14f" containerName="payload-pre-processing" containerID="cri-o://6bb17959ba06b84ecca4792abc36bbafdce0cc7c41775b877efabc7de0f40420" gracePeriod=25
Oct 30 19:59:36.456406 ip-10-0-132-82 kubenswrapper[4892]: W1030 19:59:36.456379    4892 logging.go:55] [core] [Channel #6240 SubChannel #6242]grpc: addrConn.createTransport failed to connect to {Addr: "10.132.0.155:50052", ServerName: "10.132.0.155:50052", BalancerAttributes: {"<%!p(pickfirstleaf.managedByPickfirstKeyType={})>": "<%!p(bool=true)>" }}. Err: connection error: desc = "transport: Error while dialing: dial tcp 10.132.0.155:50052: connect: connection refused"
Oct 30 19:59:36.456406 ip-10-0-132-82 kubenswrapper[4892]: W1030 19:59:36.456402    4892 logging.go:55] [core] [Channel #6241 SubChannel #6243]grpc: addrConn.createTransport failed to connect to {Addr: "10.132.0.156:50052", ServerName: "10.132.0.156:50052", BalancerAttributes: {"<%!p(pickfirstleaf.managedByPickfirstKeyType={})>": "<%!p(bool=true)>" }}. Err: connection error: desc = "transport: Error while dialing: dial tcp 10.132.0.156:50052: connect: connection refused"
Oct 30 19:59:36.587977 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:59:36.587935    4892 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w0-2e5977-55d59bd866-nxr6c"
Oct 30 19:59:36.677185 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:59:36.677154    4892 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w0-2e5977-5b974cf99f-mrcrv"
Oct 30 19:59:36.689009 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:59:36.688955    4892 generic.go:356] "Generic (PLEG): container finished" podID="f07835ed-2bb8-4dcc-8d53-9da5b4e2a14f" containerID="6bb17959ba06b84ecca4792abc36bbafdce0cc7c41775b877efabc7de0f40420" exitCode=0
Oct 30 19:59:36.689009 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:59:36.688989    4892 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w0-2e5977-5b974cf99f-mrcrv" event={"ID":"f07835ed-2bb8-4dcc-8d53-9da5b4e2a14f","Type":"ContainerDied","Data":"6bb17959ba06b84ecca4792abc36bbafdce0cc7c41775b877efabc7de0f40420"}
Oct 30 19:59:36.689262 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:59:36.689013    4892 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w0-2e5977-5b974cf99f-mrcrv"
Oct 30 19:59:36.689262 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:59:36.689041    4892 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w0-2e5977-5b974cf99f-mrcrv" event={"ID":"f07835ed-2bb8-4dcc-8d53-9da5b4e2a14f","Type":"ContainerDied","Data":"d2283134c0675507621f9a8df66997e141902b9b4a72c19ff72c7c07e8dba0cc"}
Oct 30 19:59:36.689262 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:59:36.689063    4892 scope.go:122] "RemoveContainer" containerID="6bb17959ba06b84ecca4792abc36bbafdce0cc7c41775b877efabc7de0f40420"
Oct 30 19:59:36.690508 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:59:36.690488    4892 generic.go:356] "Generic (PLEG): container finished" podID="e0fa9dd2-6f90-485c-aaff-e343ccbe428a" containerID="aa006a086c6d984fb4d0ec4d0b735358620913b45c55459cc930d37463706fb9" exitCode=0
Oct 30 19:59:36.690797 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:59:36.690527    4892 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w0-2e5977-55d59bd866-nxr6c"
Oct 30 19:59:36.690797 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:59:36.690549    4892 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w0-2e5977-55d59bd866-nxr6c" event={"ID":"e0fa9dd2-6f90-485c-aaff-e343ccbe428a","Type":"ContainerDied","Data":"aa006a086c6d984fb4d0ec4d0b735358620913b45c55459cc930d37463706fb9"}
Oct 30 19:59:36.690797 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:59:36.690578    4892 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w0-2e5977-55d59bd866-nxr6c" event={"ID":"e0fa9dd2-6f90-485c-aaff-e343ccbe428a","Type":"ContainerDied","Data":"e60f5dbe938a9236e81d950e9cd06640f499ae6a9e201d0cd6d2e39445dbc9fe"}
Oct 30 19:59:36.699079 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:59:36.699063    4892 scope.go:122] "RemoveContainer" containerID="6bb17959ba06b84ecca4792abc36bbafdce0cc7c41775b877efabc7de0f40420"
Oct 30 19:59:36.699324 ip-10-0-132-82 kubenswrapper[4892]: E1030 19:59:36.699305    4892 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6bb17959ba06b84ecca4792abc36bbafdce0cc7c41775b877efabc7de0f40420\": container with ID starting with 6bb17959ba06b84ecca4792abc36bbafdce0cc7c41775b877efabc7de0f40420 not found: ID does not exist" containerID="6bb17959ba06b84ecca4792abc36bbafdce0cc7c41775b877efabc7de0f40420"
Oct 30 19:59:36.699376 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:59:36.699332    4892 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6bb17959ba06b84ecca4792abc36bbafdce0cc7c41775b877efabc7de0f40420"} err="failed to get container status \"6bb17959ba06b84ecca4792abc36bbafdce0cc7c41775b877efabc7de0f40420\": rpc error: code = NotFound desc = could not find container \"6bb17959ba06b84ecca4792abc36bbafdce0cc7c41775b877efabc7de0f40420\": container with ID starting with 6bb17959ba06b84ecca4792abc36bbafdce0cc7c41775b877efabc7de0f40420 not found: ID does not exist"
Oct 30 19:59:36.699376 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:59:36.699349    4892 scope.go:122] "RemoveContainer" containerID="aa006a086c6d984fb4d0ec4d0b735358620913b45c55459cc930d37463706fb9"
Oct 30 19:59:36.710138 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:59:36.710120    4892 scope.go:122] "RemoveContainer" containerID="aa006a086c6d984fb4d0ec4d0b735358620913b45c55459cc930d37463706fb9"
Oct 30 19:59:36.710413 ip-10-0-132-82 kubenswrapper[4892]: E1030 19:59:36.710394    4892 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"aa006a086c6d984fb4d0ec4d0b735358620913b45c55459cc930d37463706fb9\": container with ID starting with aa006a086c6d984fb4d0ec4d0b735358620913b45c55459cc930d37463706fb9 not found: ID does not exist" containerID="aa006a086c6d984fb4d0ec4d0b735358620913b45c55459cc930d37463706fb9"
Oct 30 19:59:36.710493 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:59:36.710425    4892 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"aa006a086c6d984fb4d0ec4d0b735358620913b45c55459cc930d37463706fb9"} err="failed to get container status \"aa006a086c6d984fb4d0ec4d0b735358620913b45c55459cc930d37463706fb9\": rpc error: code = NotFound desc = could not find container \"aa006a086c6d984fb4d0ec4d0b735358620913b45c55459cc930d37463706fb9\": container with ID starting with aa006a086c6d984fb4d0ec4d0b735358620913b45c55459cc930d37463706fb9 not found: ID does not exist"
Oct 30 19:59:36.767594 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:59:36.767565    4892 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e0fa9dd2-6f90-485c-aaff-e343ccbe428a-kube-api-access-d5vh5\" (UniqueName: \"kubernetes.io/projected/e0fa9dd2-6f90-485c-aaff-e343ccbe428a-kube-api-access-d5vh5\") pod \"e0fa9dd2-6f90-485c-aaff-e343ccbe428a\" (UID: \"e0fa9dd2-6f90-485c-aaff-e343ccbe428a\") "
Oct 30 19:59:36.767780 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:59:36.767648    4892 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f07835ed-2bb8-4dcc-8d53-9da5b4e2a14f-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f07835ed-2bb8-4dcc-8d53-9da5b4e2a14f-plugins-config-volume\") pod \"f07835ed-2bb8-4dcc-8d53-9da5b4e2a14f\" (UID: \"f07835ed-2bb8-4dcc-8d53-9da5b4e2a14f\") "
Oct 30 19:59:36.767780 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:59:36.767706    4892 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f07835ed-2bb8-4dcc-8d53-9da5b4e2a14f-kube-api-access-qnp77\" (UniqueName: \"kubernetes.io/projected/f07835ed-2bb8-4dcc-8d53-9da5b4e2a14f-kube-api-access-qnp77\") pod \"f07835ed-2bb8-4dcc-8d53-9da5b4e2a14f\" (UID: \"f07835ed-2bb8-4dcc-8d53-9da5b4e2a14f\") "
Oct 30 19:59:36.767780 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:59:36.767755    4892 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e0fa9dd2-6f90-485c-aaff-e343ccbe428a-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e0fa9dd2-6f90-485c-aaff-e343ccbe428a-plugins-config-volume\") pod \"e0fa9dd2-6f90-485c-aaff-e343ccbe428a\" (UID: \"e0fa9dd2-6f90-485c-aaff-e343ccbe428a\") "
Oct 30 19:59:36.768118 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:59:36.768089    4892 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f07835ed-2bb8-4dcc-8d53-9da5b4e2a14f-plugins-config-volume" pod "f07835ed-2bb8-4dcc-8d53-9da5b4e2a14f" (UID: "f07835ed-2bb8-4dcc-8d53-9da5b4e2a14f"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 19:59:36.768190 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:59:36.768174    4892 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e0fa9dd2-6f90-485c-aaff-e343ccbe428a-plugins-config-volume" pod "e0fa9dd2-6f90-485c-aaff-e343ccbe428a" (UID: "e0fa9dd2-6f90-485c-aaff-e343ccbe428a"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 19:59:36.768286 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:59:36.768265    4892 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f07835ed-2bb8-4dcc-8d53-9da5b4e2a14f-plugins-config-volume\") on node \"ip-10-0-132-82.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 19:59:36.768286 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:59:36.768288    4892 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e0fa9dd2-6f90-485c-aaff-e343ccbe428a-plugins-config-volume\") on node \"ip-10-0-132-82.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 19:59:36.769880 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:59:36.769859    4892 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e0fa9dd2-6f90-485c-aaff-e343ccbe428a-kube-api-access-d5vh5" pod "e0fa9dd2-6f90-485c-aaff-e343ccbe428a" (UID: "e0fa9dd2-6f90-485c-aaff-e343ccbe428a"). InnerVolumeSpecName "kube-api-access-d5vh5". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:59:36.770513 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:59:36.770498    4892 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f07835ed-2bb8-4dcc-8d53-9da5b4e2a14f-kube-api-access-qnp77" pod "f07835ed-2bb8-4dcc-8d53-9da5b4e2a14f" (UID: "f07835ed-2bb8-4dcc-8d53-9da5b4e2a14f"). InnerVolumeSpecName "kube-api-access-qnp77". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:59:36.869083 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:59:36.869006    4892 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qnp77\" (UniqueName: \"kubernetes.io/projected/f07835ed-2bb8-4dcc-8d53-9da5b4e2a14f-kube-api-access-qnp77\") on node \"ip-10-0-132-82.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 19:59:36.869083 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:59:36.869038    4892 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-d5vh5\" (UniqueName: \"kubernetes.io/projected/e0fa9dd2-6f90-485c-aaff-e343ccbe428a-kube-api-access-d5vh5\") on node \"ip-10-0-132-82.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 19:59:37.113618 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:59:37.113581    4892 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w0-2e5977-55d59bd866-nxr6c"]
Oct 30 19:59:37.117232 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:59:37.117205    4892 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w0-2e5977-55d59bd866-nxr6c"]
Oct 30 19:59:37.226506 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:59:37.226475    4892 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w0-2e5977-5b974cf99f-mrcrv"]
Oct 30 19:59:37.230857 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:59:37.230831    4892 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w0-2e5977-5b974cf99f-mrcrv"]
Oct 30 19:59:37.456341 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:59:37.456295    4892 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w0-2e5977-5b974cf99f-mrcrv" podUID="f07835ed-2bb8-4dcc-8d53-9da5b4e2a14f" containerName="payload-pre-processing" probeResult="failure" output="timeout: failed to connect service \"10.132.0.155:50052\" within 1s: context deadline exceeded"
Oct 30 19:59:37.456730 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:59:37.456295    4892 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-shared-a-w0-2e5977-55d59bd866-nxr6c" podUID="e0fa9dd2-6f90-485c-aaff-e343ccbe428a" containerName="payload-processing" probeResult="failure" output="timeout: failed to connect service \"10.132.0.156:50052\" within 1s: context deadline exceeded"
Oct 30 19:59:38.572171 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:59:38.572130    4892 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e0fa9dd2-6f90-485c-aaff-e343ccbe428a" path="/var/lib/kubelet/pods/e0fa9dd2-6f90-485c-aaff-e343ccbe428a/volumes"
Oct 30 19:59:38.572743 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:59:38.572723    4892 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f07835ed-2bb8-4dcc-8d53-9da5b4e2a14f" path="/var/lib/kubelet/pods/f07835ed-2bb8-4dcc-8d53-9da5b4e2a14f/volumes"
Oct 30 19:59:38.937464 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:59:38.937420    4892 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-api-b-da4c62-5899dfc94d-2s5tq" podUID="c2d43160-e748-4f65-81e1-0e9895b11593" containerName="payload-processing" containerID="cri-o://13ce3ff9e65fb78c33156c11db5f3bd4ae6096c17b535a5e9b6e25c43cc850f9" gracePeriod=25
Oct 30 19:59:39.069040 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:59:39.068995    4892 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-api-b-da4c62-79568d7967-nmj4w" podUID="abff79c3-d35b-48bf-86ca-6329ee026b07" containerName="payload-pre-processing" containerID="cri-o://fc25cf446ce8f4bc7c6a1906171dee830df43635139fb6594740cc9c46cd30e4" gracePeriod=25
Oct 30 19:59:39.215130 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:59:39.215104    4892 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-b-da4c62-5899dfc94d-2s5tq"
Oct 30 19:59:39.299427 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:59:39.299397    4892 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c2d43160-e748-4f65-81e1-0e9895b11593-kube-api-access-qz6p5\" (UniqueName: \"kubernetes.io/projected/c2d43160-e748-4f65-81e1-0e9895b11593-kube-api-access-qz6p5\") pod \"c2d43160-e748-4f65-81e1-0e9895b11593\" (UID: \"c2d43160-e748-4f65-81e1-0e9895b11593\") "
Oct 30 19:59:39.299556 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:59:39.299461    4892 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c2d43160-e748-4f65-81e1-0e9895b11593-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c2d43160-e748-4f65-81e1-0e9895b11593-plugins-config-volume\") pod \"c2d43160-e748-4f65-81e1-0e9895b11593\" (UID: \"c2d43160-e748-4f65-81e1-0e9895b11593\") "
Oct 30 19:59:39.300109 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:59:39.300081    4892 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c2d43160-e748-4f65-81e1-0e9895b11593-plugins-config-volume" pod "c2d43160-e748-4f65-81e1-0e9895b11593" (UID: "c2d43160-e748-4f65-81e1-0e9895b11593"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 19:59:39.302084 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:59:39.301935    4892 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c2d43160-e748-4f65-81e1-0e9895b11593-kube-api-access-qz6p5" pod "c2d43160-e748-4f65-81e1-0e9895b11593" (UID: "c2d43160-e748-4f65-81e1-0e9895b11593"). InnerVolumeSpecName "kube-api-access-qz6p5". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:59:39.321405 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:59:39.321385    4892 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-b-da4c62-79568d7967-nmj4w"
Oct 30 19:59:39.400498 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:59:39.400469    4892 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/abff79c3-d35b-48bf-86ca-6329ee026b07-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/abff79c3-d35b-48bf-86ca-6329ee026b07-plugins-config-volume\") pod \"abff79c3-d35b-48bf-86ca-6329ee026b07\" (UID: \"abff79c3-d35b-48bf-86ca-6329ee026b07\") "
Oct 30 19:59:39.400636 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:59:39.400557    4892 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/abff79c3-d35b-48bf-86ca-6329ee026b07-kube-api-access-xvbdv\" (UniqueName: \"kubernetes.io/projected/abff79c3-d35b-48bf-86ca-6329ee026b07-kube-api-access-xvbdv\") pod \"abff79c3-d35b-48bf-86ca-6329ee026b07\" (UID: \"abff79c3-d35b-48bf-86ca-6329ee026b07\") "
Oct 30 19:59:39.400841 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:59:39.400821    4892 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/abff79c3-d35b-48bf-86ca-6329ee026b07-plugins-config-volume" pod "abff79c3-d35b-48bf-86ca-6329ee026b07" (UID: "abff79c3-d35b-48bf-86ca-6329ee026b07"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 19:59:39.400982 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:59:39.400953    4892 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/abff79c3-d35b-48bf-86ca-6329ee026b07-plugins-config-volume\") on node \"ip-10-0-132-82.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 19:59:39.401041 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:59:39.400982    4892 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qz6p5\" (UniqueName: \"kubernetes.io/projected/c2d43160-e748-4f65-81e1-0e9895b11593-kube-api-access-qz6p5\") on node \"ip-10-0-132-82.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 19:59:39.401041 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:59:39.400993    4892 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c2d43160-e748-4f65-81e1-0e9895b11593-plugins-config-volume\") on node \"ip-10-0-132-82.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 19:59:39.402702 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:59:39.402681    4892 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/abff79c3-d35b-48bf-86ca-6329ee026b07-kube-api-access-xvbdv" pod "abff79c3-d35b-48bf-86ca-6329ee026b07" (UID: "abff79c3-d35b-48bf-86ca-6329ee026b07"). InnerVolumeSpecName "kube-api-access-xvbdv". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:59:39.502080 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:59:39.502006    4892 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xvbdv\" (UniqueName: \"kubernetes.io/projected/abff79c3-d35b-48bf-86ca-6329ee026b07-kube-api-access-xvbdv\") on node \"ip-10-0-132-82.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 19:59:39.718684 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:59:39.718649    4892 generic.go:356] "Generic (PLEG): container finished" podID="c2d43160-e748-4f65-81e1-0e9895b11593" containerID="13ce3ff9e65fb78c33156c11db5f3bd4ae6096c17b535a5e9b6e25c43cc850f9" exitCode=0
Oct 30 19:59:39.719215 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:59:39.718697    4892 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-b-da4c62-5899dfc94d-2s5tq"
Oct 30 19:59:39.719215 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:59:39.718741    4892 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-b-da4c62-5899dfc94d-2s5tq" event={"ID":"c2d43160-e748-4f65-81e1-0e9895b11593","Type":"ContainerDied","Data":"13ce3ff9e65fb78c33156c11db5f3bd4ae6096c17b535a5e9b6e25c43cc850f9"}
Oct 30 19:59:39.719215 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:59:39.718793    4892 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-b-da4c62-5899dfc94d-2s5tq" event={"ID":"c2d43160-e748-4f65-81e1-0e9895b11593","Type":"ContainerDied","Data":"937dd8f89c00d77d92a2cbf5d9845d80dd95f34d127574b790e6d413ecd7dca5"}
Oct 30 19:59:39.719215 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:59:39.718823    4892 scope.go:122] "RemoveContainer" containerID="13ce3ff9e65fb78c33156c11db5f3bd4ae6096c17b535a5e9b6e25c43cc850f9"
Oct 30 19:59:39.720515 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:59:39.720490    4892 generic.go:356] "Generic (PLEG): container finished" podID="abff79c3-d35b-48bf-86ca-6329ee026b07" containerID="fc25cf446ce8f4bc7c6a1906171dee830df43635139fb6594740cc9c46cd30e4" exitCode=0
Oct 30 19:59:39.720630 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:59:39.720549    4892 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-b-da4c62-79568d7967-nmj4w" event={"ID":"abff79c3-d35b-48bf-86ca-6329ee026b07","Type":"ContainerDied","Data":"fc25cf446ce8f4bc7c6a1906171dee830df43635139fb6594740cc9c46cd30e4"}
Oct 30 19:59:39.720630 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:59:39.720595    4892 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-b-da4c62-79568d7967-nmj4w" event={"ID":"abff79c3-d35b-48bf-86ca-6329ee026b07","Type":"ContainerDied","Data":"b6efebd352b1c1220b4079ab92a63ba9d4b52a8755d566645cf9c6bf17432a3c"}
Oct 30 19:59:39.720713 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:59:39.720700    4892 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-b-da4c62-79568d7967-nmj4w"
Oct 30 19:59:39.732907 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:59:39.732889    4892 scope.go:122] "RemoveContainer" containerID="13ce3ff9e65fb78c33156c11db5f3bd4ae6096c17b535a5e9b6e25c43cc850f9"
Oct 30 19:59:39.733860 ip-10-0-132-82 kubenswrapper[4892]: E1030 19:59:39.733840    4892 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"13ce3ff9e65fb78c33156c11db5f3bd4ae6096c17b535a5e9b6e25c43cc850f9\": container with ID starting with 13ce3ff9e65fb78c33156c11db5f3bd4ae6096c17b535a5e9b6e25c43cc850f9 not found: ID does not exist" containerID="13ce3ff9e65fb78c33156c11db5f3bd4ae6096c17b535a5e9b6e25c43cc850f9"
Oct 30 19:59:39.733958 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:59:39.733864    4892 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"13ce3ff9e65fb78c33156c11db5f3bd4ae6096c17b535a5e9b6e25c43cc850f9"} err="failed to get container status \"13ce3ff9e65fb78c33156c11db5f3bd4ae6096c17b535a5e9b6e25c43cc850f9\": rpc error: code = NotFound desc = could not find container \"13ce3ff9e65fb78c33156c11db5f3bd4ae6096c17b535a5e9b6e25c43cc850f9\": container with ID starting with 13ce3ff9e65fb78c33156c11db5f3bd4ae6096c17b535a5e9b6e25c43cc850f9 not found: ID does not exist"
Oct 30 19:59:39.733958 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:59:39.733880    4892 scope.go:122] "RemoveContainer" containerID="fc25cf446ce8f4bc7c6a1906171dee830df43635139fb6594740cc9c46cd30e4"
Oct 30 19:59:39.745349 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:59:39.745310    4892 scope.go:122] "RemoveContainer" containerID="fc25cf446ce8f4bc7c6a1906171dee830df43635139fb6594740cc9c46cd30e4"
Oct 30 19:59:39.745659 ip-10-0-132-82 kubenswrapper[4892]: E1030 19:59:39.745636    4892 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"fc25cf446ce8f4bc7c6a1906171dee830df43635139fb6594740cc9c46cd30e4\": container with ID starting with fc25cf446ce8f4bc7c6a1906171dee830df43635139fb6594740cc9c46cd30e4 not found: ID does not exist" containerID="fc25cf446ce8f4bc7c6a1906171dee830df43635139fb6594740cc9c46cd30e4"
Oct 30 19:59:39.745735 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:59:39.745666    4892 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"fc25cf446ce8f4bc7c6a1906171dee830df43635139fb6594740cc9c46cd30e4"} err="failed to get container status \"fc25cf446ce8f4bc7c6a1906171dee830df43635139fb6594740cc9c46cd30e4\": rpc error: code = NotFound desc = could not find container \"fc25cf446ce8f4bc7c6a1906171dee830df43635139fb6594740cc9c46cd30e4\": container with ID starting with fc25cf446ce8f4bc7c6a1906171dee830df43635139fb6594740cc9c46cd30e4 not found: ID does not exist"
Oct 30 19:59:39.883841 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:59:39.883809    4892 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-da4c62-79568d7967-nmj4w"]
Oct 30 19:59:39.887280 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:59:39.887258    4892 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-da4c62-79568d7967-nmj4w"]
Oct 30 19:59:39.998019 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:59:39.997990    4892 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-da4c62-5899dfc94d-2s5tq"]
Oct 30 19:59:40.021981 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:59:40.021937    4892 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-da4c62-5899dfc94d-2s5tq"]
Oct 30 19:59:40.570805 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:59:40.570764    4892 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="abff79c3-d35b-48bf-86ca-6329ee026b07" path="/var/lib/kubelet/pods/abff79c3-d35b-48bf-86ca-6329ee026b07/volumes"
Oct 30 19:59:40.571294 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:59:40.571270    4892 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c2d43160-e748-4f65-81e1-0e9895b11593" path="/var/lib/kubelet/pods/c2d43160-e748-4f65-81e1-0e9895b11593/volumes"
Oct 30 19:59:44.189309 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:59:44.189271    4892 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-da4c62-mbvvh"]
Oct 30 19:59:44.238906 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:59:44.238873    4892 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-da4c62-mbvvh"]
Oct 30 19:59:44.570354 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:59:44.570271    4892 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="833f7eac-9bb0-4ba9-9192-158d25933978" path="/var/lib/kubelet/pods/833f7eac-9bb0-4ba9-9192-158d25933978/volumes"
Oct 30 19:59:45.861443 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:59:45.861348    4892 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-fsfvk"]
Oct 30 19:59:50.843579 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:59:50.843539    4892 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-b0c827-npkmz"]
Oct 30 19:59:50.879586 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:59:50.879549    4892 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-b0c827-npkmz"]
Oct 30 19:59:52.049990 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:59:52.044594    4892 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w0-2e5977-67qhn"]
Oct 30 19:59:52.091114 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:59:52.091082    4892 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w0-2e5977-67qhn"]
Oct 30 19:59:52.570289 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:59:52.570256    4892 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="135c03e8-7715-418e-96b8-6257bc7023d0" path="/var/lib/kubelet/pods/135c03e8-7715-418e-96b8-6257bc7023d0/volumes"
Oct 30 19:59:52.570680 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:59:52.570665    4892 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="6f18d694-a7ca-449e-8ba5-0b07d06cea98" path="/var/lib/kubelet/pods/6f18d694-a7ca-449e-8ba5-0b07d06cea98/volumes"
Oct 30 19:59:52.612428 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:59:52.612397    4892 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w6-5f55cd-2nmh7"]
Oct 30 19:59:52.612900 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:59:52.612888    4892 cpu_manager.go:414] "containerMap: removing container" podUID="f07835ed-2bb8-4dcc-8d53-9da5b4e2a14f" containerName="payload-pre-processing"
Oct 30 19:59:52.612949 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:59:52.612902    4892 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f07835ed-2bb8-4dcc-8d53-9da5b4e2a14f" containerName="payload-pre-processing"
Oct 30 19:59:52.612949 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:59:52.612917    4892 cpu_manager.go:414] "containerMap: removing container" podUID="135c03e8-7715-418e-96b8-6257bc7023d0" containerName="revoke-keys"
Oct 30 19:59:52.612949 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:59:52.612922    4892 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="135c03e8-7715-418e-96b8-6257bc7023d0" containerName="revoke-keys"
Oct 30 19:59:52.612949 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:59:52.612931    4892 cpu_manager.go:414] "containerMap: removing container" podUID="c2d43160-e748-4f65-81e1-0e9895b11593" containerName="payload-processing"
Oct 30 19:59:52.612949 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:59:52.612936    4892 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c2d43160-e748-4f65-81e1-0e9895b11593" containerName="payload-processing"
Oct 30 19:59:52.612949 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:59:52.612945    4892 cpu_manager.go:414] "containerMap: removing container" podUID="abff79c3-d35b-48bf-86ca-6329ee026b07" containerName="payload-pre-processing"
Oct 30 19:59:52.612949 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:59:52.612949    4892 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="abff79c3-d35b-48bf-86ca-6329ee026b07" containerName="payload-pre-processing"
Oct 30 19:59:52.613190 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:59:52.612957    4892 cpu_manager.go:414] "containerMap: removing container" podUID="81a5c6ce-725d-42b3-8bba-1f80fb5edfad" containerName="payload-pre-processing"
Oct 30 19:59:52.613190 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:59:52.612978    4892 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="81a5c6ce-725d-42b3-8bba-1f80fb5edfad" containerName="payload-pre-processing"
Oct 30 19:59:52.613190 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:59:52.612997    4892 cpu_manager.go:414] "containerMap: removing container" podUID="e0fa9dd2-6f90-485c-aaff-e343ccbe428a" containerName="payload-processing"
Oct 30 19:59:52.613190 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:59:52.613002    4892 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e0fa9dd2-6f90-485c-aaff-e343ccbe428a" containerName="payload-processing"
Oct 30 19:59:52.613190 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:59:52.613011    4892 cpu_manager.go:414] "containerMap: removing container" podUID="1ca3885d-5fb3-4c84-b1fc-2e9138885566" containerName="payload-processing"
Oct 30 19:59:52.613190 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:59:52.613016    4892 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1ca3885d-5fb3-4c84-b1fc-2e9138885566" containerName="payload-processing"
Oct 30 19:59:52.613190 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:59:52.613029    4892 cpu_manager.go:414] "containerMap: removing container" podUID="833f7eac-9bb0-4ba9-9192-158d25933978" containerName="revoke-keys"
Oct 30 19:59:52.613190 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:59:52.613033    4892 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="833f7eac-9bb0-4ba9-9192-158d25933978" containerName="revoke-keys"
Oct 30 19:59:52.613190 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:59:52.613117    4892 memory_manager.go:365] "RemoveStaleState removing state" podUID="135c03e8-7715-418e-96b8-6257bc7023d0" containerName="revoke-keys"
Oct 30 19:59:52.613190 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:59:52.613127    4892 memory_manager.go:365] "RemoveStaleState removing state" podUID="c2d43160-e748-4f65-81e1-0e9895b11593" containerName="payload-processing"
Oct 30 19:59:52.613190 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:59:52.613132    4892 memory_manager.go:365] "RemoveStaleState removing state" podUID="abff79c3-d35b-48bf-86ca-6329ee026b07" containerName="payload-pre-processing"
Oct 30 19:59:52.613190 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:59:52.613138    4892 memory_manager.go:365] "RemoveStaleState removing state" podUID="81a5c6ce-725d-42b3-8bba-1f80fb5edfad" containerName="payload-pre-processing"
Oct 30 19:59:52.613190 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:59:52.613143    4892 memory_manager.go:365] "RemoveStaleState removing state" podUID="e0fa9dd2-6f90-485c-aaff-e343ccbe428a" containerName="payload-processing"
Oct 30 19:59:52.613190 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:59:52.613150    4892 memory_manager.go:365] "RemoveStaleState removing state" podUID="1ca3885d-5fb3-4c84-b1fc-2e9138885566" containerName="payload-processing"
Oct 30 19:59:52.613190 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:59:52.613156    4892 memory_manager.go:365] "RemoveStaleState removing state" podUID="833f7eac-9bb0-4ba9-9192-158d25933978" containerName="revoke-keys"
Oct 30 19:59:52.613190 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:59:52.613166    4892 memory_manager.go:365] "RemoveStaleState removing state" podUID="f07835ed-2bb8-4dcc-8d53-9da5b4e2a14f" containerName="payload-pre-processing"
Oct 30 19:59:52.617430 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:59:52.617413    4892 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w6-5f55cd-2nmh7"
Oct 30 19:59:52.687265 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:59:52.687236    4892 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 19:59:52.756013 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:59:52.753620    4892 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-nlxp4\""
Oct 30 19:59:52.756013 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:59:52.753730    4892 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 30 19:59:52.756013 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:59:52.753816    4892 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 30 19:59:52.760883 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:59:52.760858    4892 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w6-5f55cd-2nmh7"]
Oct 30 19:59:52.770037 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:59:52.770005    4892 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tzv64\" (UniqueName: \"kubernetes.io/projected/a48896fd-ad37-43b3-af20-4bc3028c5460-kube-api-access-tzv64\") pod \"maas-api-revoke-keys-e2e-worker-w6-5f55cd-2nmh7\" (UID: \"a48896fd-ad37-43b3-af20-4bc3028c5460\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w6-5f55cd-2nmh7"
Oct 30 19:59:52.770185 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:59:52.770077    4892 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a48896fd-ad37-43b3-af20-4bc3028c5460-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-w6-5f55cd-2nmh7\" (UID: \"a48896fd-ad37-43b3-af20-4bc3028c5460\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w6-5f55cd-2nmh7"
Oct 30 19:59:52.770185 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:59:52.770162    4892 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a48896fd-ad37-43b3-af20-4bc3028c5460-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-w6-5f55cd-2nmh7\" (UID: \"a48896fd-ad37-43b3-af20-4bc3028c5460\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w6-5f55cd-2nmh7"
Oct 30 19:59:52.872401 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:59:52.872315    4892 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tzv64\" (UniqueName: \"kubernetes.io/projected/a48896fd-ad37-43b3-af20-4bc3028c5460-kube-api-access-tzv64\") pod \"maas-api-revoke-keys-e2e-worker-w6-5f55cd-2nmh7\" (UID: \"a48896fd-ad37-43b3-af20-4bc3028c5460\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w6-5f55cd-2nmh7"
Oct 30 19:59:52.872401 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:59:52.872396    4892 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a48896fd-ad37-43b3-af20-4bc3028c5460-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-w6-5f55cd-2nmh7\" (UID: \"a48896fd-ad37-43b3-af20-4bc3028c5460\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w6-5f55cd-2nmh7"
Oct 30 19:59:52.872588 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:59:52.872434    4892 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a48896fd-ad37-43b3-af20-4bc3028c5460-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-w6-5f55cd-2nmh7\" (UID: \"a48896fd-ad37-43b3-af20-4bc3028c5460\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w6-5f55cd-2nmh7"
Oct 30 19:59:52.873401 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:59:52.873378    4892 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a48896fd-ad37-43b3-af20-4bc3028c5460-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-w6-5f55cd-2nmh7\" (UID: \"a48896fd-ad37-43b3-af20-4bc3028c5460\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w6-5f55cd-2nmh7"
Oct 30 19:59:52.874923 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:59:52.874905    4892 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a48896fd-ad37-43b3-af20-4bc3028c5460-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-w6-5f55cd-2nmh7\" (UID: \"a48896fd-ad37-43b3-af20-4bc3028c5460\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w6-5f55cd-2nmh7"
Oct 30 19:59:52.946986 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:59:52.946654    4892 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tzv64\" (UniqueName: \"kubernetes.io/projected/a48896fd-ad37-43b3-af20-4bc3028c5460-kube-api-access-tzv64\") pod \"maas-api-revoke-keys-e2e-worker-w6-5f55cd-2nmh7\" (UID: \"a48896fd-ad37-43b3-af20-4bc3028c5460\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w6-5f55cd-2nmh7"
Oct 30 19:59:52.958176 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:59:52.958141    4892 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w6-5f55cd-2nmh7"
Oct 30 19:59:53.353364 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:59:53.353326    4892 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w6-5f55cd-2nmh7"]
Oct 30 19:59:53.354671 ip-10-0-132-82 kubenswrapper[4892]: W1030 19:59:53.354643    4892 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda48896fd_ad37_43b3_af20_4bc3028c5460.slice/crio-ab953c09467e0f411a2b39583785b1728cde1622de1fbf4846fdadba3c09d1b8 WatchSource:0}: Error finding container ab953c09467e0f411a2b39583785b1728cde1622de1fbf4846fdadba3c09d1b8: Status 404 returned error can't find the container with id ab953c09467e0f411a2b39583785b1728cde1622de1fbf4846fdadba3c09d1b8
Oct 30 19:59:53.986258 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:59:53.985942    4892 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w6-5f55cd-2nmh7" event={"ID":"a48896fd-ad37-43b3-af20-4bc3028c5460","Type":"ContainerStarted","Data":"e8c019a83307a40a60c50815101688a37f0a919049d2dee08ec8e5b291be8bd3"}
Oct 30 19:59:53.986258 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:59:53.986022    4892 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w6-5f55cd-2nmh7" event={"ID":"a48896fd-ad37-43b3-af20-4bc3028c5460","Type":"ContainerStarted","Data":"ab953c09467e0f411a2b39583785b1728cde1622de1fbf4846fdadba3c09d1b8"}
Oct 30 19:59:54.101168 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:59:54.101108    4892 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w6-5f55cd-2nmh7" podStartSLOduration=2.101090016 podStartE2EDuration="2.101090016s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:59:52 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:59:54.10041829 +0000 UTC m=+2641.670883889" watchObservedRunningTime="2026-10-30 19:59:54.101090016 +0000 UTC m=+2641.671555617"
Oct 30 19:59:54.635405 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:59:54.635358    4892 prober.go:122] "Probe failed" probeType="Liveness" pod="kube-system/kube-apiserver-proxy-ip-10-0-132-82.us-west-1.compute.internal" podUID="f9af2f5b3d8deb871a93a1e3f0011dc6" containerName="haproxy" probeResult="failure" output="Get \"https://172.20.0.1:6443/version\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:59:54.846223 ip-10-0-132-82 kubenswrapper[4892]: E1030 19:59:54.846184    4892 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-jrm87" containerName="extractor"
Oct 30 19:59:54.994896 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:59:54.994813    4892 generic.go:356] "Generic (PLEG): container finished" podID="a48896fd-ad37-43b3-af20-4bc3028c5460" containerID="e8c019a83307a40a60c50815101688a37f0a919049d2dee08ec8e5b291be8bd3" exitCode=0
Oct 30 19:59:54.995089 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:59:54.994900    4892 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w6-5f55cd-2nmh7" event={"ID":"a48896fd-ad37-43b3-af20-4bc3028c5460","Type":"ContainerDied","Data":"e8c019a83307a40a60c50815101688a37f0a919049d2dee08ec8e5b291be8bd3"}
Oct 30 19:59:56.077127 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:59:56.077092    4892 patch_prober.go:29] interesting pod/e2e-inf-a-132119-openshift-default-849b46588f-29f2x container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.214:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 30 19:59:56.077511 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:59:56.077160    4892 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-inf-a-132119-openshift-default-849b46588f-29f2x" podUID="c76fc24b-0fe2-490e-8967-9a5eb4e9ce12" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.214:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:59:56.172046 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:59:56.172020    4892 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w6-5f55cd-2nmh7"
Oct 30 19:59:56.215038 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:59:56.215006    4892 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a48896fd-ad37-43b3-af20-4bc3028c5460-kube-api-access-tzv64\" (UniqueName: \"kubernetes.io/projected/a48896fd-ad37-43b3-af20-4bc3028c5460-kube-api-access-tzv64\") pod \"a48896fd-ad37-43b3-af20-4bc3028c5460\" (UID: \"a48896fd-ad37-43b3-af20-4bc3028c5460\") "
Oct 30 19:59:56.215206 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:59:56.215083    4892 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a48896fd-ad37-43b3-af20-4bc3028c5460-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a48896fd-ad37-43b3-af20-4bc3028c5460-maas-api-openshift-ca\") pod \"a48896fd-ad37-43b3-af20-4bc3028c5460\" (UID: \"a48896fd-ad37-43b3-af20-4bc3028c5460\") "
Oct 30 19:59:56.215206 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:59:56.215197    4892 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/a48896fd-ad37-43b3-af20-4bc3028c5460-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a48896fd-ad37-43b3-af20-4bc3028c5460-maas-api-xks-ca\") pod \"a48896fd-ad37-43b3-af20-4bc3028c5460\" (UID: \"a48896fd-ad37-43b3-af20-4bc3028c5460\") "
Oct 30 19:59:56.215396 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:59:56.215360    4892 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a48896fd-ad37-43b3-af20-4bc3028c5460-maas-api-openshift-ca" pod "a48896fd-ad37-43b3-af20-4bc3028c5460" (UID: "a48896fd-ad37-43b3-af20-4bc3028c5460"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 19:59:56.215652 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:59:56.215627    4892 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a48896fd-ad37-43b3-af20-4bc3028c5460-maas-api-openshift-ca\") on node \"ip-10-0-132-82.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 19:59:56.217467 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:59:56.217442    4892 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a48896fd-ad37-43b3-af20-4bc3028c5460-kube-api-access-tzv64" pod "a48896fd-ad37-43b3-af20-4bc3028c5460" (UID: "a48896fd-ad37-43b3-af20-4bc3028c5460"). InnerVolumeSpecName "kube-api-access-tzv64". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:59:56.217467 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:59:56.217447    4892 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a48896fd-ad37-43b3-af20-4bc3028c5460-maas-api-xks-ca" pod "a48896fd-ad37-43b3-af20-4bc3028c5460" (UID: "a48896fd-ad37-43b3-af20-4bc3028c5460"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 19:59:56.317105 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:59:56.317022    4892 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tzv64\" (UniqueName: \"kubernetes.io/projected/a48896fd-ad37-43b3-af20-4bc3028c5460-kube-api-access-tzv64\") on node \"ip-10-0-132-82.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 19:59:56.317105 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:59:56.317062    4892 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a48896fd-ad37-43b3-af20-4bc3028c5460-maas-api-xks-ca\") on node \"ip-10-0-132-82.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 19:59:57.013546 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:59:57.013518    4892 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w6-5f55cd-2nmh7"
Oct 30 19:59:57.013794 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:59:57.013539    4892 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w6-5f55cd-2nmh7" event={"ID":"a48896fd-ad37-43b3-af20-4bc3028c5460","Type":"ContainerDied","Data":"ab953c09467e0f411a2b39583785b1728cde1622de1fbf4846fdadba3c09d1b8"}
Oct 30 19:59:57.013794 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:59:57.013585    4892 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="ab953c09467e0f411a2b39583785b1728cde1622de1fbf4846fdadba3c09d1b8"
Oct 30 19:59:57.015710 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:59:57.015678    4892 generic.go:356] "Generic (PLEG): container finished" podID="c76fc24b-0fe2-490e-8967-9a5eb4e9ce12" containerID="8104d1feef8b7a9b4f6b6cf7646ea8bfc8897bc7178a480aa99cb6ce22992041" exitCode=0
Oct 30 19:59:57.015866 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:59:57.015760    4892 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-132119-openshift-default-849b46588f-29f2x" event={"ID":"c76fc24b-0fe2-490e-8967-9a5eb4e9ce12","Type":"ContainerDied","Data":"8104d1feef8b7a9b4f6b6cf7646ea8bfc8897bc7178a480aa99cb6ce22992041"}
Oct 30 19:59:57.016240 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:59:57.016220    4892 scope.go:122] "RemoveContainer" containerID="8104d1feef8b7a9b4f6b6cf7646ea8bfc8897bc7178a480aa99cb6ce22992041"
Oct 30 19:59:57.017924 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:59:57.017891    4892 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892000Ki","pods":"250"}
Oct 30 19:59:57.018099 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:59:57.018007    4892 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892000Ki","pods":"250"}
Oct 30 19:59:57.018099 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:59:57.018064    4892 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892000Ki","pods":"250"}
Oct 30 19:59:57.309256 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:59:57.309175    4892 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w6-5f55cd/premium-w6-5f55cd-kserve-7f785c544b-pcvhb"]
Oct 30 19:59:57.328035 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:59:57.327999    4892 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w6-5f55cd/simulator-w6-5f55cd-kserve-5ff86b7447-h9tm7"]
Oct 30 19:59:57.790017 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:59:57.789986    4892 scope.go:122] "RemoveContainer" containerID="5531a1f3c92b4c7ca4bd40413771c96948f586cd8ca2a54db0009146b7562663"
Oct 30 19:59:58.044061 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:59:58.035082    4892 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-132119-openshift-default-849b46588f-29f2x" event={"ID":"c76fc24b-0fe2-490e-8967-9a5eb4e9ce12","Type":"ContainerStarted","Data":"25d1a04e483f98b8c00b859aaec861ddeae787e748fe9b510c5cbfc99be258c3"}
Oct 30 19:59:58.630040 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:59:58.620084    4892 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w1-d52df0-9q2ht"]
Oct 30 19:59:58.700443 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:59:58.700401    4892 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w1-d52df0-9q2ht"]
Oct 30 19:59:58.771294 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:59:58.771260    4892 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-a-132119-openshift-default-849b46588f-29f2x"
Oct 30 19:59:58.771456 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:59:58.771432    4892 patch_prober.go:29] interesting pod/e2e-inf-a-132119-openshift-default-849b46588f-29f2x container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.214:15021/healthz/ready\": dial tcp 10.132.0.214:15021: connect: connection refused" start-of-body=
Oct 30 19:59:58.771517 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:59:58.771483    4892 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-132119-openshift-default-849b46588f-29f2x" podUID="c76fc24b-0fe2-490e-8967-9a5eb4e9ce12" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.214:15021/healthz/ready\": dial tcp 10.132.0.214:15021: connect: connection refused"
Oct 30 19:59:59.776531 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:59:59.776494    4892 patch_prober.go:29] interesting pod/e2e-inf-a-132119-openshift-default-849b46588f-29f2x container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.214:15021/healthz/ready\": dial tcp 10.132.0.214:15021: connect: connection refused" start-of-body=
Oct 30 19:59:59.777215 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:59:59.777176    4892 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-132119-openshift-default-849b46588f-29f2x" podUID="c76fc24b-0fe2-490e-8967-9a5eb4e9ce12" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.214:15021/healthz/ready\": dial tcp 10.132.0.214:15021: connect: connection refused"
Oct 30 19:59:59.865294 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:59:59.865258    4892 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-dd9b8d79-dlbxt"]
Oct 30 19:59:59.908093 ip-10-0-132-82 kubenswrapper[4892]: I1030 19:59:59.908057    4892 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-dd9b8d79-dlbxt"]
Oct 30 20:00:00.179989 ip-10-0-132-82 kubenswrapper[4892]: I1030 20:00:00.176577    4892 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-6ac304-29889840-fhd4j"]
Oct 30 20:00:00.179989 ip-10-0-132-82 kubenswrapper[4892]: I1030 20:00:00.177394    4892 cpu_manager.go:414] "containerMap: removing container" podUID="a48896fd-ad37-43b3-af20-4bc3028c5460" containerName="revoke-keys"
Oct 30 20:00:00.179989 ip-10-0-132-82 kubenswrapper[4892]: I1030 20:00:00.177411    4892 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a48896fd-ad37-43b3-af20-4bc3028c5460" containerName="revoke-keys"
Oct 30 20:00:00.179989 ip-10-0-132-82 kubenswrapper[4892]: I1030 20:00:00.177537    4892 memory_manager.go:365] "RemoveStaleState removing state" podUID="a48896fd-ad37-43b3-af20-4bc3028c5460" containerName="revoke-keys"
Oct 30 20:00:00.183248 ip-10-0-132-82 kubenswrapper[4892]: I1030 20:00:00.183223    4892 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-b-1c5537-29889840-6qrmj"]
Oct 30 20:00:00.183438 ip-10-0-132-82 kubenswrapper[4892]: I1030 20:00:00.183413    4892 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-6ac304-29889840-fhd4j"
Oct 30 20:00:00.187919 ip-10-0-132-82 kubenswrapper[4892]: I1030 20:00:00.187894    4892 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-ada8d7-29889840-xldh2"]
Oct 30 20:00:00.188116 ip-10-0-132-82 kubenswrapper[4892]: I1030 20:00:00.188064    4892 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-b-1c5537-29889840-6qrmj"
Oct 30 20:00:00.192643 ip-10-0-132-82 kubenswrapper[4892]: I1030 20:00:00.192623    4892 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-c75ed4-29889840-7s578"]
Oct 30 20:00:00.192768 ip-10-0-132-82 kubenswrapper[4892]: I1030 20:00:00.192749    4892 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-ada8d7-29889840-xldh2"
Oct 30 20:00:00.196777 ip-10-0-132-82 kubenswrapper[4892]: I1030 20:00:00.196756    4892 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-2d9861-29889840-79j62"]
Oct 30 20:00:00.196884 ip-10-0-132-82 kubenswrapper[4892]: I1030 20:00:00.196857    4892 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-c75ed4-29889840-7s578"
Oct 30 20:00:00.201514 ip-10-0-132-82 kubenswrapper[4892]: I1030 20:00:00.201496    4892 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-cf9b6f-29889840-8k6mb"]
Oct 30 20:00:00.201652 ip-10-0-132-82 kubenswrapper[4892]: I1030 20:00:00.201614    4892 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-2d9861-29889840-79j62"
Oct 30 20:00:00.205850 ip-10-0-132-82 kubenswrapper[4892]: I1030 20:00:00.205829    4892 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-cf9b6f-29889840-8k6mb"
Oct 30 20:00:00.211100 ip-10-0-132-82 kubenswrapper[4892]: I1030 20:00:00.210911    4892 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-686689-29889840-l2mjk"]
Oct 30 20:00:00.215011 ip-10-0-132-82 kubenswrapper[4892]: I1030 20:00:00.214991    4892 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-686689-29889840-l2mjk"
Oct 30 20:00:00.222368 ip-10-0-132-82 kubenswrapper[4892]: I1030 20:00:00.222348    4892 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-132119-29889840-dhpf8"]
Oct 30 20:00:00.225388 ip-10-0-132-82 kubenswrapper[4892]: I1030 20:00:00.225359    4892 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/8152ed9d-b608-4a87-aded-8dda2d975174-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-b-w3-c75ed4-29889840-7s578\" (UID: \"8152ed9d-b608-4a87-aded-8dda2d975174\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-c75ed4-29889840-7s578"
Oct 30 20:00:00.225515 ip-10-0-132-82 kubenswrapper[4892]: I1030 20:00:00.225405    4892 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/82b75f10-16db-40c1-9696-f1e7f9917419-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w0-2d9861-29889840-79j62\" (UID: \"82b75f10-16db-40c1-9696-f1e7f9917419\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-2d9861-29889840-79j62"
Oct 30 20:00:00.225515 ip-10-0-132-82 kubenswrapper[4892]: I1030 20:00:00.225486    4892 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/35018750-1063-4fb6-928f-ce6bcd401f87-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w3-ada8d7-29889840-xldh2\" (UID: \"35018750-1063-4fb6-928f-ce6bcd401f87\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-ada8d7-29889840-xldh2"
Oct 30 20:00:00.225690 ip-10-0-132-82 kubenswrapper[4892]: I1030 20:00:00.225515    4892 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cbqqn\" (UniqueName: \"kubernetes.io/projected/25d4aedf-735a-42e6-ae08-4561c967fc35-kube-api-access-cbqqn\") pod \"maas-api-key-cleanup-e2e-shared-b-w0-6ac304-29889840-fhd4j\" (UID: \"25d4aedf-735a-42e6-ae08-4561c967fc35\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-6ac304-29889840-fhd4j"
Oct 30 20:00:00.225690 ip-10-0-132-82 kubenswrapper[4892]: I1030 20:00:00.225559    4892 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-q9qjb\" (UniqueName: \"kubernetes.io/projected/8152ed9d-b608-4a87-aded-8dda2d975174-kube-api-access-q9qjb\") pod \"maas-api-key-cleanup-e2e-shared-b-w3-c75ed4-29889840-7s578\" (UID: \"8152ed9d-b608-4a87-aded-8dda2d975174\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-c75ed4-29889840-7s578"
Oct 30 20:00:00.225690 ip-10-0-132-82 kubenswrapper[4892]: I1030 20:00:00.225577    4892 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4szjb\" (UniqueName: \"kubernetes.io/projected/35018750-1063-4fb6-928f-ce6bcd401f87-kube-api-access-4szjb\") pod \"maas-api-key-cleanup-e2e-worker-w3-ada8d7-29889840-xldh2\" (UID: \"35018750-1063-4fb6-928f-ce6bcd401f87\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-ada8d7-29889840-xldh2"
Oct 30 20:00:00.225690 ip-10-0-132-82 kubenswrapper[4892]: I1030 20:00:00.225629    4892 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/f6260396-890c-44ea-be8c-998dd2086ffa-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-api-b-1c5537-29889840-6qrmj\" (UID: \"f6260396-890c-44ea-be8c-998dd2086ffa\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-b-1c5537-29889840-6qrmj"
Oct 30 20:00:00.225690 ip-10-0-132-82 kubenswrapper[4892]: I1030 20:00:00.225656    4892 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4rthv\" (UniqueName: \"kubernetes.io/projected/f6260396-890c-44ea-be8c-998dd2086ffa-kube-api-access-4rthv\") pod \"maas-api-key-cleanup-e2e-api-b-1c5537-29889840-6qrmj\" (UID: \"f6260396-890c-44ea-be8c-998dd2086ffa\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-b-1c5537-29889840-6qrmj"
Oct 30 20:00:00.226015 ip-10-0-132-82 kubenswrapper[4892]: I1030 20:00:00.225694    4892 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/9df510ae-c995-4678-816a-9c4834778991-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-a-w3-cf9b6f-29889840-8k6mb\" (UID: \"9df510ae-c995-4678-816a-9c4834778991\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-cf9b6f-29889840-8k6mb"
Oct 30 20:00:00.226015 ip-10-0-132-82 kubenswrapper[4892]: I1030 20:00:00.225720    4892 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/25d4aedf-735a-42e6-ae08-4561c967fc35-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-b-w0-6ac304-29889840-fhd4j\" (UID: \"25d4aedf-735a-42e6-ae08-4561c967fc35\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-6ac304-29889840-fhd4j"
Oct 30 20:00:00.226015 ip-10-0-132-82 kubenswrapper[4892]: I1030 20:00:00.225787    4892 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wq78b\" (UniqueName: \"kubernetes.io/projected/82b75f10-16db-40c1-9696-f1e7f9917419-kube-api-access-wq78b\") pod \"maas-api-key-cleanup-e2e-worker-w0-2d9861-29889840-79j62\" (UID: \"82b75f10-16db-40c1-9696-f1e7f9917419\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-2d9861-29889840-79j62"
Oct 30 20:00:00.226015 ip-10-0-132-82 kubenswrapper[4892]: I1030 20:00:00.225804    4892 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9dtwd\" (UniqueName: \"kubernetes.io/projected/9df510ae-c995-4678-816a-9c4834778991-kube-api-access-9dtwd\") pod \"maas-api-key-cleanup-e2e-shared-a-w3-cf9b6f-29889840-8k6mb\" (UID: \"9df510ae-c995-4678-816a-9c4834778991\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-cf9b6f-29889840-8k6mb"
Oct 30 20:00:00.226570 ip-10-0-132-82 kubenswrapper[4892]: I1030 20:00:00.226555    4892 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-132119-29889840-dhpf8"
Oct 30 20:00:00.248487 ip-10-0-132-82 kubenswrapper[4892]: I1030 20:00:00.248460    4892 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 30 20:00:00.248779 ip-10-0-132-82 kubenswrapper[4892]: I1030 20:00:00.248688    4892 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 30 20:00:00.249468 ip-10-0-132-82 kubenswrapper[4892]: I1030 20:00:00.249451    4892 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-nlxp4\""
Oct 30 20:00:00.264408 ip-10-0-132-82 kubenswrapper[4892]: I1030 20:00:00.264367    4892 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-6ac304-29889840-fhd4j"]
Oct 30 20:00:00.282235 ip-10-0-132-82 kubenswrapper[4892]: I1030 20:00:00.281771    4892 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-b-1c5537-29889840-6qrmj"]
Oct 30 20:00:00.288944 ip-10-0-132-82 kubenswrapper[4892]: I1030 20:00:00.288918    4892 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-2d9861-29889840-79j62"]
Oct 30 20:00:00.300038 ip-10-0-132-82 kubenswrapper[4892]: I1030 20:00:00.300011    4892 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-cf9b6f-29889840-8k6mb"]
Oct 30 20:00:00.307060 ip-10-0-132-82 kubenswrapper[4892]: I1030 20:00:00.307034    4892 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-686689-29889840-l2mjk"]
Oct 30 20:00:00.312599 ip-10-0-132-82 kubenswrapper[4892]: I1030 20:00:00.312574    4892 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-132119-29889840-dhpf8"]
Oct 30 20:00:00.319195 ip-10-0-132-82 kubenswrapper[4892]: I1030 20:00:00.319165    4892 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-ada8d7-29889840-xldh2"]
Oct 30 20:00:00.326843 ip-10-0-132-82 kubenswrapper[4892]: I1030 20:00:00.326820    4892 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/9df510ae-c995-4678-816a-9c4834778991-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-a-w3-cf9b6f-29889840-8k6mb\" (UID: \"9df510ae-c995-4678-816a-9c4834778991\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-cf9b6f-29889840-8k6mb"
Oct 30 20:00:00.327019 ip-10-0-132-82 kubenswrapper[4892]: I1030 20:00:00.326999    4892 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/d16a477c-ad96-4363-b13d-b6d374f3f88d-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-inf-a-132119-29889840-dhpf8\" (UID: \"d16a477c-ad96-4363-b13d-b6d374f3f88d\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-132119-29889840-dhpf8"
Oct 30 20:00:00.327157 ip-10-0-132-82 kubenswrapper[4892]: I1030 20:00:00.327141    4892 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/25d4aedf-735a-42e6-ae08-4561c967fc35-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-b-w0-6ac304-29889840-fhd4j\" (UID: \"25d4aedf-735a-42e6-ae08-4561c967fc35\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-6ac304-29889840-fhd4j"
Oct 30 20:00:00.327306 ip-10-0-132-82 kubenswrapper[4892]: I1030 20:00:00.327292    4892 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/0b0b9f79-e0a9-41a1-b513-01ea5661f64d-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w5-686689-29889840-l2mjk\" (UID: \"0b0b9f79-e0a9-41a1-b513-01ea5661f64d\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-686689-29889840-l2mjk"
Oct 30 20:00:00.327430 ip-10-0-132-82 kubenswrapper[4892]: I1030 20:00:00.327417    4892 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wq78b\" (UniqueName: \"kubernetes.io/projected/82b75f10-16db-40c1-9696-f1e7f9917419-kube-api-access-wq78b\") pod \"maas-api-key-cleanup-e2e-worker-w0-2d9861-29889840-79j62\" (UID: \"82b75f10-16db-40c1-9696-f1e7f9917419\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-2d9861-29889840-79j62"
Oct 30 20:00:00.327529 ip-10-0-132-82 kubenswrapper[4892]: I1030 20:00:00.327517    4892 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9dtwd\" (UniqueName: \"kubernetes.io/projected/9df510ae-c995-4678-816a-9c4834778991-kube-api-access-9dtwd\") pod \"maas-api-key-cleanup-e2e-shared-a-w3-cf9b6f-29889840-8k6mb\" (UID: \"9df510ae-c995-4678-816a-9c4834778991\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-cf9b6f-29889840-8k6mb"
Oct 30 20:00:00.327641 ip-10-0-132-82 kubenswrapper[4892]: I1030 20:00:00.327628    4892 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/8152ed9d-b608-4a87-aded-8dda2d975174-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-b-w3-c75ed4-29889840-7s578\" (UID: \"8152ed9d-b608-4a87-aded-8dda2d975174\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-c75ed4-29889840-7s578"
Oct 30 20:00:00.327736 ip-10-0-132-82 kubenswrapper[4892]: I1030 20:00:00.327724    4892 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-d99hj\" (UniqueName: \"kubernetes.io/projected/d16a477c-ad96-4363-b13d-b6d374f3f88d-kube-api-access-d99hj\") pod \"maas-api-key-cleanup-e2e-inf-a-132119-29889840-dhpf8\" (UID: \"d16a477c-ad96-4363-b13d-b6d374f3f88d\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-132119-29889840-dhpf8"
Oct 30 20:00:00.327844 ip-10-0-132-82 kubenswrapper[4892]: I1030 20:00:00.327832    4892 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/82b75f10-16db-40c1-9696-f1e7f9917419-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w0-2d9861-29889840-79j62\" (UID: \"82b75f10-16db-40c1-9696-f1e7f9917419\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-2d9861-29889840-79j62"
Oct 30 20:00:00.337073 ip-10-0-132-82 kubenswrapper[4892]: I1030 20:00:00.337048    4892 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-plr9z\" (UniqueName: \"kubernetes.io/projected/0b0b9f79-e0a9-41a1-b513-01ea5661f64d-kube-api-access-plr9z\") pod \"maas-api-key-cleanup-e2e-worker-w5-686689-29889840-l2mjk\" (UID: \"0b0b9f79-e0a9-41a1-b513-01ea5661f64d\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-686689-29889840-l2mjk"
Oct 30 20:00:00.337235 ip-10-0-132-82 kubenswrapper[4892]: I1030 20:00:00.337217    4892 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/35018750-1063-4fb6-928f-ce6bcd401f87-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w3-ada8d7-29889840-xldh2\" (UID: \"35018750-1063-4fb6-928f-ce6bcd401f87\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-ada8d7-29889840-xldh2"
Oct 30 20:00:00.337347 ip-10-0-132-82 kubenswrapper[4892]: I1030 20:00:00.337334    4892 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cbqqn\" (UniqueName: \"kubernetes.io/projected/25d4aedf-735a-42e6-ae08-4561c967fc35-kube-api-access-cbqqn\") pod \"maas-api-key-cleanup-e2e-shared-b-w0-6ac304-29889840-fhd4j\" (UID: \"25d4aedf-735a-42e6-ae08-4561c967fc35\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-6ac304-29889840-fhd4j"
Oct 30 20:00:00.337490 ip-10-0-132-82 kubenswrapper[4892]: I1030 20:00:00.337473    4892 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-q9qjb\" (UniqueName: \"kubernetes.io/projected/8152ed9d-b608-4a87-aded-8dda2d975174-kube-api-access-q9qjb\") pod \"maas-api-key-cleanup-e2e-shared-b-w3-c75ed4-29889840-7s578\" (UID: \"8152ed9d-b608-4a87-aded-8dda2d975174\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-c75ed4-29889840-7s578"
Oct 30 20:00:00.355073 ip-10-0-132-82 kubenswrapper[4892]: I1030 20:00:00.355048    4892 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4szjb\" (UniqueName: \"kubernetes.io/projected/35018750-1063-4fb6-928f-ce6bcd401f87-kube-api-access-4szjb\") pod \"maas-api-key-cleanup-e2e-worker-w3-ada8d7-29889840-xldh2\" (UID: \"35018750-1063-4fb6-928f-ce6bcd401f87\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-ada8d7-29889840-xldh2"
Oct 30 20:00:00.355325 ip-10-0-132-82 kubenswrapper[4892]: I1030 20:00:00.355309    4892 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/f6260396-890c-44ea-be8c-998dd2086ffa-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-api-b-1c5537-29889840-6qrmj\" (UID: \"f6260396-890c-44ea-be8c-998dd2086ffa\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-b-1c5537-29889840-6qrmj"
Oct 30 20:00:00.360092 ip-10-0-132-82 kubenswrapper[4892]: I1030 20:00:00.360068    4892 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4rthv\" (UniqueName: \"kubernetes.io/projected/f6260396-890c-44ea-be8c-998dd2086ffa-kube-api-access-4rthv\") pod \"maas-api-key-cleanup-e2e-api-b-1c5537-29889840-6qrmj\" (UID: \"f6260396-890c-44ea-be8c-998dd2086ffa\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-b-1c5537-29889840-6qrmj"
Oct 30 20:00:00.360354 ip-10-0-132-82 kubenswrapper[4892]: I1030 20:00:00.345940    4892 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/25d4aedf-735a-42e6-ae08-4561c967fc35-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-b-w0-6ac304-29889840-fhd4j\" (UID: \"25d4aedf-735a-42e6-ae08-4561c967fc35\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-6ac304-29889840-fhd4j"
Oct 30 20:00:00.360457 ip-10-0-132-82 kubenswrapper[4892]: I1030 20:00:00.342927    4892 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/35018750-1063-4fb6-928f-ce6bcd401f87-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w3-ada8d7-29889840-xldh2\" (UID: \"35018750-1063-4fb6-928f-ce6bcd401f87\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-ada8d7-29889840-xldh2"
Oct 30 20:00:00.360781 ip-10-0-132-82 kubenswrapper[4892]: I1030 20:00:00.344671    4892 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/82b75f10-16db-40c1-9696-f1e7f9917419-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w0-2d9861-29889840-79j62\" (UID: \"82b75f10-16db-40c1-9696-f1e7f9917419\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-2d9861-29889840-79j62"
Oct 30 20:00:00.361535 ip-10-0-132-82 kubenswrapper[4892]: I1030 20:00:00.361513    4892 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/f6260396-890c-44ea-be8c-998dd2086ffa-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-api-b-1c5537-29889840-6qrmj\" (UID: \"f6260396-890c-44ea-be8c-998dd2086ffa\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-b-1c5537-29889840-6qrmj"
Oct 30 20:00:00.365022 ip-10-0-132-82 kubenswrapper[4892]: I1030 20:00:00.345308    4892 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/9df510ae-c995-4678-816a-9c4834778991-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-a-w3-cf9b6f-29889840-8k6mb\" (UID: \"9df510ae-c995-4678-816a-9c4834778991\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-cf9b6f-29889840-8k6mb"
Oct 30 20:00:00.366183 ip-10-0-132-82 kubenswrapper[4892]: I1030 20:00:00.366154    4892 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/8152ed9d-b608-4a87-aded-8dda2d975174-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-b-w3-c75ed4-29889840-7s578\" (UID: \"8152ed9d-b608-4a87-aded-8dda2d975174\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-c75ed4-29889840-7s578"
Oct 30 20:00:00.369562 ip-10-0-132-82 kubenswrapper[4892]: I1030 20:00:00.369543    4892 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-c75ed4-29889840-7s578"]
Oct 30 20:00:00.412959 ip-10-0-132-82 kubenswrapper[4892]: I1030 20:00:00.412922    4892 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-q9qjb\" (UniqueName: \"kubernetes.io/projected/8152ed9d-b608-4a87-aded-8dda2d975174-kube-api-access-q9qjb\") pod \"maas-api-key-cleanup-e2e-shared-b-w3-c75ed4-29889840-7s578\" (UID: \"8152ed9d-b608-4a87-aded-8dda2d975174\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-c75ed4-29889840-7s578"
Oct 30 20:00:00.414913 ip-10-0-132-82 kubenswrapper[4892]: I1030 20:00:00.414887    4892 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-cbqqn\" (UniqueName: \"kubernetes.io/projected/25d4aedf-735a-42e6-ae08-4561c967fc35-kube-api-access-cbqqn\") pod \"maas-api-key-cleanup-e2e-shared-b-w0-6ac304-29889840-fhd4j\" (UID: \"25d4aedf-735a-42e6-ae08-4561c967fc35\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-6ac304-29889840-fhd4j"
Oct 30 20:00:00.423795 ip-10-0-132-82 kubenswrapper[4892]: I1030 20:00:00.423774    4892 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4szjb\" (UniqueName: \"kubernetes.io/projected/35018750-1063-4fb6-928f-ce6bcd401f87-kube-api-access-4szjb\") pod \"maas-api-key-cleanup-e2e-worker-w3-ada8d7-29889840-xldh2\" (UID: \"35018750-1063-4fb6-928f-ce6bcd401f87\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-ada8d7-29889840-xldh2"
Oct 30 20:00:00.426555 ip-10-0-132-82 kubenswrapper[4892]: I1030 20:00:00.426498    4892 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9dtwd\" (UniqueName: \"kubernetes.io/projected/9df510ae-c995-4678-816a-9c4834778991-kube-api-access-9dtwd\") pod \"maas-api-key-cleanup-e2e-shared-a-w3-cf9b6f-29889840-8k6mb\" (UID: \"9df510ae-c995-4678-816a-9c4834778991\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-cf9b6f-29889840-8k6mb"
Oct 30 20:00:00.456541 ip-10-0-132-82 kubenswrapper[4892]: I1030 20:00:00.452916    4892 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4rthv\" (UniqueName: \"kubernetes.io/projected/f6260396-890c-44ea-be8c-998dd2086ffa-kube-api-access-4rthv\") pod \"maas-api-key-cleanup-e2e-api-b-1c5537-29889840-6qrmj\" (UID: \"f6260396-890c-44ea-be8c-998dd2086ffa\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-b-1c5537-29889840-6qrmj"
Oct 30 20:00:00.479988 ip-10-0-132-82 kubenswrapper[4892]: I1030 20:00:00.476644    4892 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wq78b\" (UniqueName: \"kubernetes.io/projected/82b75f10-16db-40c1-9696-f1e7f9917419-kube-api-access-wq78b\") pod \"maas-api-key-cleanup-e2e-worker-w0-2d9861-29889840-79j62\" (UID: \"82b75f10-16db-40c1-9696-f1e7f9917419\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-2d9861-29889840-79j62"
Oct 30 20:00:00.479988 ip-10-0-132-82 kubenswrapper[4892]: I1030 20:00:00.478377    4892 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/d16a477c-ad96-4363-b13d-b6d374f3f88d-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-inf-a-132119-29889840-dhpf8\" (UID: \"d16a477c-ad96-4363-b13d-b6d374f3f88d\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-132119-29889840-dhpf8"
Oct 30 20:00:00.479988 ip-10-0-132-82 kubenswrapper[4892]: I1030 20:00:00.478467    4892 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/0b0b9f79-e0a9-41a1-b513-01ea5661f64d-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w5-686689-29889840-l2mjk\" (UID: \"0b0b9f79-e0a9-41a1-b513-01ea5661f64d\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-686689-29889840-l2mjk"
Oct 30 20:00:00.479988 ip-10-0-132-82 kubenswrapper[4892]: I1030 20:00:00.478538    4892 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-d99hj\" (UniqueName: \"kubernetes.io/projected/d16a477c-ad96-4363-b13d-b6d374f3f88d-kube-api-access-d99hj\") pod \"maas-api-key-cleanup-e2e-inf-a-132119-29889840-dhpf8\" (UID: \"d16a477c-ad96-4363-b13d-b6d374f3f88d\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-132119-29889840-dhpf8"
Oct 30 20:00:00.479988 ip-10-0-132-82 kubenswrapper[4892]: I1030 20:00:00.478577    4892 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-plr9z\" (UniqueName: \"kubernetes.io/projected/0b0b9f79-e0a9-41a1-b513-01ea5661f64d-kube-api-access-plr9z\") pod \"maas-api-key-cleanup-e2e-worker-w5-686689-29889840-l2mjk\" (UID: \"0b0b9f79-e0a9-41a1-b513-01ea5661f64d\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-686689-29889840-l2mjk"
Oct 30 20:00:00.479988 ip-10-0-132-82 kubenswrapper[4892]: I1030 20:00:00.479841    4892 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/d16a477c-ad96-4363-b13d-b6d374f3f88d-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-inf-a-132119-29889840-dhpf8\" (UID: \"d16a477c-ad96-4363-b13d-b6d374f3f88d\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-132119-29889840-dhpf8"
Oct 30 20:00:00.481013 ip-10-0-132-82 kubenswrapper[4892]: I1030 20:00:00.480991    4892 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/0b0b9f79-e0a9-41a1-b513-01ea5661f64d-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w5-686689-29889840-l2mjk\" (UID: \"0b0b9f79-e0a9-41a1-b513-01ea5661f64d\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-686689-29889840-l2mjk"
Oct 30 20:00:00.501815 ip-10-0-132-82 kubenswrapper[4892]: I1030 20:00:00.501787    4892 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-6ac304-29889840-fhd4j"
Oct 30 20:00:00.511048 ip-10-0-132-82 kubenswrapper[4892]: I1030 20:00:00.511020    4892 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-b-1c5537-29889840-6qrmj"
Oct 30 20:00:00.519642 ip-10-0-132-82 kubenswrapper[4892]: I1030 20:00:00.519460    4892 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-ada8d7-29889840-xldh2"
Oct 30 20:00:00.623997 ip-10-0-132-82 kubenswrapper[4892]: I1030 20:00:00.621849    4892 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-2d9861-29889840-79j62"
Oct 30 20:00:00.638998 ip-10-0-132-82 kubenswrapper[4892]: I1030 20:00:00.636545    4892 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-c75ed4-29889840-7s578"
Oct 30 20:00:00.638998 ip-10-0-132-82 kubenswrapper[4892]: I1030 20:00:00.636653    4892 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-cf9b6f-29889840-8k6mb"
Oct 30 20:00:00.655107 ip-10-0-132-82 kubenswrapper[4892]: I1030 20:00:00.653740    4892 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-plr9z\" (UniqueName: \"kubernetes.io/projected/0b0b9f79-e0a9-41a1-b513-01ea5661f64d-kube-api-access-plr9z\") pod \"maas-api-key-cleanup-e2e-worker-w5-686689-29889840-l2mjk\" (UID: \"0b0b9f79-e0a9-41a1-b513-01ea5661f64d\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-686689-29889840-l2mjk"
Oct 30 20:00:00.657047 ip-10-0-132-82 kubenswrapper[4892]: I1030 20:00:00.656252    4892 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-d99hj\" (UniqueName: \"kubernetes.io/projected/d16a477c-ad96-4363-b13d-b6d374f3f88d-kube-api-access-d99hj\") pod \"maas-api-key-cleanup-e2e-inf-a-132119-29889840-dhpf8\" (UID: \"d16a477c-ad96-4363-b13d-b6d374f3f88d\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-132119-29889840-dhpf8"
Oct 30 20:00:00.669146 ip-10-0-132-82 kubenswrapper[4892]: I1030 20:00:00.669060    4892 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-132119-29889840-dhpf8"
Oct 30 20:00:00.683051 ip-10-0-132-82 kubenswrapper[4892]: I1030 20:00:00.670728    4892 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="8b7c4bca-c523-4094-a2d7-3bfee37be2e3" path="/var/lib/kubelet/pods/8b7c4bca-c523-4094-a2d7-3bfee37be2e3/volumes"
Oct 30 20:00:00.683051 ip-10-0-132-82 kubenswrapper[4892]: I1030 20:00:00.671767    4892 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d126159b-6468-4c98-808f-a4243e83f11b" path="/var/lib/kubelet/pods/d126159b-6468-4c98-808f-a4243e83f11b/volumes"
Oct 30 20:00:00.771162 ip-10-0-132-82 kubenswrapper[4892]: I1030 20:00:00.771125    4892 patch_prober.go:29] interesting pod/e2e-inf-a-132119-openshift-default-849b46588f-29f2x container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.214:15021/healthz/ready\": dial tcp 10.132.0.214:15021: connect: connection refused" start-of-body=
Oct 30 20:00:00.771254 ip-10-0-132-82 kubenswrapper[4892]: I1030 20:00:00.771202    4892 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-132119-openshift-default-849b46588f-29f2x" podUID="c76fc24b-0fe2-490e-8967-9a5eb4e9ce12" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.214:15021/healthz/ready\": dial tcp 10.132.0.214:15021: connect: connection refused"
Oct 30 20:00:00.916382 ip-10-0-132-82 kubenswrapper[4892]: I1030 20:00:00.916151    4892 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-686689-29889840-l2mjk"
Oct 30 20:00:01.083521 ip-10-0-132-82 kubenswrapper[4892]: I1030 20:00:01.083489    4892 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w0-6ac304-pcccr"]
Oct 30 20:00:01.088874 ip-10-0-132-82 kubenswrapper[4892]: I1030 20:00:01.088856    4892 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w0-6ac304-pcccr"
Oct 30 20:00:01.097551 ip-10-0-132-82 kubenswrapper[4892]: I1030 20:00:01.097527    4892 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-b-1c5537-29889840-6qrmj"]
Oct 30 20:00:01.097932 ip-10-0-132-82 kubenswrapper[4892]: W1030 20:00:01.097910    4892 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf6260396_890c_44ea_be8c_998dd2086ffa.slice/crio-96cb9bd84a95a3ae8ab368b8322ccc9f93f7c90e67681d04efb614527d89ec2c WatchSource:0}: Error finding container 96cb9bd84a95a3ae8ab368b8322ccc9f93f7c90e67681d04efb614527d89ec2c: Status 404 returned error can't find the container with id 96cb9bd84a95a3ae8ab368b8322ccc9f93f7c90e67681d04efb614527d89ec2c
Oct 30 20:00:01.169448 ip-10-0-132-82 kubenswrapper[4892]: I1030 20:00:01.159158    4892 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 20:00:01.178979 ip-10-0-132-82 kubenswrapper[4892]: I1030 20:00:01.178933    4892 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-ada8d7-29889840-xldh2"]
Oct 30 20:00:01.179118 ip-10-0-132-82 kubenswrapper[4892]: I1030 20:00:01.179060    4892 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w0-6ac304-pcccr"]
Oct 30 20:00:01.182075 ip-10-0-132-82 kubenswrapper[4892]: I1030 20:00:01.181688    4892 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4rjqf\" (UniqueName: \"kubernetes.io/projected/e5edd2c3-9676-48a1-9b30-8d8a98512582-kube-api-access-4rjqf\") pod \"maas-api-revoke-keys-e2e-shared-b-w0-6ac304-pcccr\" (UID: \"e5edd2c3-9676-48a1-9b30-8d8a98512582\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w0-6ac304-pcccr"
Oct 30 20:00:01.182075 ip-10-0-132-82 kubenswrapper[4892]: I1030 20:00:01.181760    4892 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e5edd2c3-9676-48a1-9b30-8d8a98512582-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w0-6ac304-pcccr\" (UID: \"e5edd2c3-9676-48a1-9b30-8d8a98512582\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w0-6ac304-pcccr"
Oct 30 20:00:01.182075 ip-10-0-132-82 kubenswrapper[4892]: I1030 20:00:01.181845    4892 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e5edd2c3-9676-48a1-9b30-8d8a98512582-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w0-6ac304-pcccr\" (UID: \"e5edd2c3-9676-48a1-9b30-8d8a98512582\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w0-6ac304-pcccr"
Oct 30 20:00:01.182933 ip-10-0-132-82 kubenswrapper[4892]: I1030 20:00:01.182901    4892 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-b-1c5537-29889840-6qrmj" event={"ID":"f6260396-890c-44ea-be8c-998dd2086ffa","Type":"ContainerStarted","Data":"96cb9bd84a95a3ae8ab368b8322ccc9f93f7c90e67681d04efb614527d89ec2c"}
Oct 30 20:00:01.283331 ip-10-0-132-82 kubenswrapper[4892]: I1030 20:00:01.283288    4892 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4rjqf\" (UniqueName: \"kubernetes.io/projected/e5edd2c3-9676-48a1-9b30-8d8a98512582-kube-api-access-4rjqf\") pod \"maas-api-revoke-keys-e2e-shared-b-w0-6ac304-pcccr\" (UID: \"e5edd2c3-9676-48a1-9b30-8d8a98512582\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w0-6ac304-pcccr"
Oct 30 20:00:01.283331 ip-10-0-132-82 kubenswrapper[4892]: I1030 20:00:01.283353    4892 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e5edd2c3-9676-48a1-9b30-8d8a98512582-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w0-6ac304-pcccr\" (UID: \"e5edd2c3-9676-48a1-9b30-8d8a98512582\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w0-6ac304-pcccr"
Oct 30 20:00:01.283564 ip-10-0-132-82 kubenswrapper[4892]: I1030 20:00:01.283385    4892 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e5edd2c3-9676-48a1-9b30-8d8a98512582-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w0-6ac304-pcccr\" (UID: \"e5edd2c3-9676-48a1-9b30-8d8a98512582\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w0-6ac304-pcccr"
Oct 30 20:00:01.283998 ip-10-0-132-82 kubenswrapper[4892]: I1030 20:00:01.283958    4892 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e5edd2c3-9676-48a1-9b30-8d8a98512582-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w0-6ac304-pcccr\" (UID: \"e5edd2c3-9676-48a1-9b30-8d8a98512582\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w0-6ac304-pcccr"
Oct 30 20:00:01.354934 ip-10-0-132-82 kubenswrapper[4892]: I1030 20:00:01.354864    4892 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e5edd2c3-9676-48a1-9b30-8d8a98512582-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w0-6ac304-pcccr\" (UID: \"e5edd2c3-9676-48a1-9b30-8d8a98512582\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w0-6ac304-pcccr"
Oct 30 20:00:01.356013 ip-10-0-132-82 kubenswrapper[4892]: I1030 20:00:01.355986    4892 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-6ac304-29889840-fhd4j"]
Oct 30 20:00:01.356660 ip-10-0-132-82 kubenswrapper[4892]: I1030 20:00:01.356637    4892 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4rjqf\" (UniqueName: \"kubernetes.io/projected/e5edd2c3-9676-48a1-9b30-8d8a98512582-kube-api-access-4rjqf\") pod \"maas-api-revoke-keys-e2e-shared-b-w0-6ac304-pcccr\" (UID: \"e5edd2c3-9676-48a1-9b30-8d8a98512582\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w0-6ac304-pcccr"
Oct 30 20:00:01.357540 ip-10-0-132-82 kubenswrapper[4892]: W1030 20:00:01.357513    4892 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod25d4aedf_735a_42e6_ae08_4561c967fc35.slice/crio-8ca2d3ca666329cfaace5e1d689aa149ff21ab3f2f28a278f028d86bf22e7338 WatchSource:0}: Error finding container 8ca2d3ca666329cfaace5e1d689aa149ff21ab3f2f28a278f028d86bf22e7338: Status 404 returned error can't find the container with id 8ca2d3ca666329cfaace5e1d689aa149ff21ab3f2f28a278f028d86bf22e7338
Oct 30 20:00:01.362081 ip-10-0-132-82 kubenswrapper[4892]: I1030 20:00:01.362060    4892 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-2d9861-29889840-79j62"]
Oct 30 20:00:01.371202 ip-10-0-132-82 kubenswrapper[4892]: W1030 20:00:01.371174    4892 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod82b75f10_16db_40c1_9696_f1e7f9917419.slice/crio-690ad5b700290187bdac89f0202ffbb6fc413182754b33cae1d048909b1e4a1d WatchSource:0}: Error finding container 690ad5b700290187bdac89f0202ffbb6fc413182754b33cae1d048909b1e4a1d: Status 404 returned error can't find the container with id 690ad5b700290187bdac89f0202ffbb6fc413182754b33cae1d048909b1e4a1d
Oct 30 20:00:01.376132 ip-10-0-132-82 kubenswrapper[4892]: I1030 20:00:01.376110    4892 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-686689-29889840-l2mjk"]
Oct 30 20:00:01.384635 ip-10-0-132-82 kubenswrapper[4892]: W1030 20:00:01.384604    4892 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0b0b9f79_e0a9_41a1_b513_01ea5661f64d.slice/crio-d05e464b89f5938f49369002fc5c8ca50342b826202ed5ef61cef2ab05904ad1 WatchSource:0}: Error finding container d05e464b89f5938f49369002fc5c8ca50342b826202ed5ef61cef2ab05904ad1: Status 404 returned error can't find the container with id d05e464b89f5938f49369002fc5c8ca50342b826202ed5ef61cef2ab05904ad1
Oct 30 20:00:01.393435 ip-10-0-132-82 kubenswrapper[4892]: I1030 20:00:01.393409    4892 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-cf9b6f-29889840-8k6mb"]
Oct 30 20:00:01.401329 ip-10-0-132-82 kubenswrapper[4892]: W1030 20:00:01.401303    4892 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9df510ae_c995_4678_816a_9c4834778991.slice/crio-201941280f0662cb615d76c79cbf45d160367299e67b8ecf92190988b79d7cba WatchSource:0}: Error finding container 201941280f0662cb615d76c79cbf45d160367299e67b8ecf92190988b79d7cba: Status 404 returned error can't find the container with id 201941280f0662cb615d76c79cbf45d160367299e67b8ecf92190988b79d7cba
Oct 30 20:00:01.404582 ip-10-0-132-82 kubenswrapper[4892]: I1030 20:00:01.404283    4892 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w0-6ac304-pcccr"
Oct 30 20:00:01.418498 ip-10-0-132-82 kubenswrapper[4892]: I1030 20:00:01.418473    4892 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-c75ed4-29889840-7s578"]
Oct 30 20:00:01.462988 ip-10-0-132-82 kubenswrapper[4892]: I1030 20:00:01.460918    4892 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-132119-29889840-dhpf8"]
Oct 30 20:00:01.560057 ip-10-0-132-82 kubenswrapper[4892]: W1030 20:00:01.560010    4892 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd16a477c_ad96_4363_b13d_b6d374f3f88d.slice/crio-5f067ece6d4879d86ea9bff3b58dc0b63c59fa98fbbdfad427f042e99afc802e WatchSource:0}: Error finding container 5f067ece6d4879d86ea9bff3b58dc0b63c59fa98fbbdfad427f042e99afc802e: Status 404 returned error can't find the container with id 5f067ece6d4879d86ea9bff3b58dc0b63c59fa98fbbdfad427f042e99afc802e
Oct 30 20:00:01.771264 ip-10-0-132-82 kubenswrapper[4892]: I1030 20:00:01.771238    4892 patch_prober.go:29] interesting pod/e2e-inf-a-132119-openshift-default-849b46588f-29f2x container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.214:15021/healthz/ready\": dial tcp 10.132.0.214:15021: connect: connection refused" start-of-body=
Oct 30 20:00:01.771369 ip-10-0-132-82 kubenswrapper[4892]: I1030 20:00:01.771283    4892 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-132119-openshift-default-849b46588f-29f2x" podUID="c76fc24b-0fe2-490e-8967-9a5eb4e9ce12" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.214:15021/healthz/ready\": dial tcp 10.132.0.214:15021: connect: connection refused"
Oct 30 20:00:01.968251 ip-10-0-132-82 kubenswrapper[4892]: I1030 20:00:01.968215    4892 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w0-6ac304-pcccr"]
Oct 30 20:00:02.192516 ip-10-0-132-82 kubenswrapper[4892]: I1030 20:00:02.192481    4892 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-cf9b6f-29889840-8k6mb" event={"ID":"9df510ae-c995-4678-816a-9c4834778991","Type":"ContainerStarted","Data":"4b853a33cf49f7bd8b51b13d652a340cbaabad4ca070afcae5b74eff868a0440"}
Oct 30 20:00:02.192651 ip-10-0-132-82 kubenswrapper[4892]: I1030 20:00:02.192544    4892 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-cf9b6f-29889840-8k6mb" event={"ID":"9df510ae-c995-4678-816a-9c4834778991","Type":"ContainerStarted","Data":"201941280f0662cb615d76c79cbf45d160367299e67b8ecf92190988b79d7cba"}
Oct 30 20:00:02.197483 ip-10-0-132-82 kubenswrapper[4892]: I1030 20:00:02.197448    4892 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-c75ed4-29889840-7s578" event={"ID":"8152ed9d-b608-4a87-aded-8dda2d975174","Type":"ContainerStarted","Data":"bce4df99cf6c5b27b2f216fee73b22ad4d1577335f04e91bd7c923d75c77acfe"}
Oct 30 20:00:02.197600 ip-10-0-132-82 kubenswrapper[4892]: I1030 20:00:02.197498    4892 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-c75ed4-29889840-7s578" event={"ID":"8152ed9d-b608-4a87-aded-8dda2d975174","Type":"ContainerStarted","Data":"d2e90f93ba7131e77d021830200df8aadbaa4661a849a8db6d827eca1f9a58c3"}
Oct 30 20:00:02.199110 ip-10-0-132-82 kubenswrapper[4892]: I1030 20:00:02.198909    4892 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-686689-29889840-l2mjk" event={"ID":"0b0b9f79-e0a9-41a1-b513-01ea5661f64d","Type":"ContainerStarted","Data":"061f932bd42963fa6662e82067091f68e02d84b9a616a914aac375c44fe93429"}
Oct 30 20:00:02.199110 ip-10-0-132-82 kubenswrapper[4892]: I1030 20:00:02.198958    4892 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-686689-29889840-l2mjk" event={"ID":"0b0b9f79-e0a9-41a1-b513-01ea5661f64d","Type":"ContainerStarted","Data":"d05e464b89f5938f49369002fc5c8ca50342b826202ed5ef61cef2ab05904ad1"}
Oct 30 20:00:02.209016 ip-10-0-132-82 kubenswrapper[4892]: I1030 20:00:02.208952    4892 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-2d9861-29889840-79j62" event={"ID":"82b75f10-16db-40c1-9696-f1e7f9917419","Type":"ContainerStarted","Data":"64bddd6fd07a41fee1b4b6177591b63f5f29a845409a20b8d75d196967662928"}
Oct 30 20:00:02.209186 ip-10-0-132-82 kubenswrapper[4892]: I1030 20:00:02.209167    4892 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-2d9861-29889840-79j62" event={"ID":"82b75f10-16db-40c1-9696-f1e7f9917419","Type":"ContainerStarted","Data":"690ad5b700290187bdac89f0202ffbb6fc413182754b33cae1d048909b1e4a1d"}
Oct 30 20:00:02.210386 ip-10-0-132-82 kubenswrapper[4892]: I1030 20:00:02.210337    4892 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w0-6ac304-pcccr" event={"ID":"e5edd2c3-9676-48a1-9b30-8d8a98512582","Type":"ContainerStarted","Data":"538a04b54e6c54ee2bc34c0b3f872fe6141567bf957261c4b048593cccc9b7bb"}
Oct 30 20:00:02.212190 ip-10-0-132-82 kubenswrapper[4892]: I1030 20:00:02.211877    4892 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-6ac304-29889840-fhd4j" event={"ID":"25d4aedf-735a-42e6-ae08-4561c967fc35","Type":"ContainerStarted","Data":"36de4fe16fcf502af6adffa8351c74b8fdf965db38d3a0198972fccef9a33019"}
Oct 30 20:00:02.212190 ip-10-0-132-82 kubenswrapper[4892]: I1030 20:00:02.211910    4892 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-6ac304-29889840-fhd4j" event={"ID":"25d4aedf-735a-42e6-ae08-4561c967fc35","Type":"ContainerStarted","Data":"8ca2d3ca666329cfaace5e1d689aa149ff21ab3f2f28a278f028d86bf22e7338"}
Oct 30 20:00:02.215069 ip-10-0-132-82 kubenswrapper[4892]: I1030 20:00:02.213875    4892 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-132119-29889840-dhpf8" event={"ID":"d16a477c-ad96-4363-b13d-b6d374f3f88d","Type":"ContainerStarted","Data":"6c36f8660059a6ee4bb79a48998d9f358363baed48a1f5bee4b61c36c32382e3"}
Oct 30 20:00:02.215069 ip-10-0-132-82 kubenswrapper[4892]: I1030 20:00:02.213920    4892 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-132119-29889840-dhpf8" event={"ID":"d16a477c-ad96-4363-b13d-b6d374f3f88d","Type":"ContainerStarted","Data":"5f067ece6d4879d86ea9bff3b58dc0b63c59fa98fbbdfad427f042e99afc802e"}
Oct 30 20:00:02.216256 ip-10-0-132-82 kubenswrapper[4892]: I1030 20:00:02.216234    4892 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-ada8d7-29889840-xldh2" event={"ID":"35018750-1063-4fb6-928f-ce6bcd401f87","Type":"ContainerStarted","Data":"d10bfe5efc03c322d25be6a0b3fbb1fb555758b5c7e6a31f26759377a01cb663"}
Oct 30 20:00:02.216360 ip-10-0-132-82 kubenswrapper[4892]: I1030 20:00:02.216276    4892 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-ada8d7-29889840-xldh2" event={"ID":"35018750-1063-4fb6-928f-ce6bcd401f87","Type":"ContainerStarted","Data":"0da9cc8c2cfd1bf0db28f2e7bf2de4e4dd90ac502bd11c1e17b912409f4e6ac7"}
Oct 30 20:00:02.218798 ip-10-0-132-82 kubenswrapper[4892]: I1030 20:00:02.218436    4892 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-b-1c5537-29889840-6qrmj" event={"ID":"f6260396-890c-44ea-be8c-998dd2086ffa","Type":"ContainerStarted","Data":"6ff12781af2267ff40fb3409e5e65849e3b6eed01164a272d07eb264de71cc99"}
Oct 30 20:00:02.319293 ip-10-0-132-82 kubenswrapper[4892]: I1030 20:00:02.319238    4892 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-cf9b6f-29889840-8k6mb" podStartSLOduration=2.319223527 podStartE2EDuration="2.319223527s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 20:00:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 20:00:02.316499957 +0000 UTC m=+2649.886965574" watchObservedRunningTime="2026-10-30 20:00:02.319223527 +0000 UTC m=+2649.889689126"
Oct 30 20:00:02.463577 ip-10-0-132-82 kubenswrapper[4892]: I1030 20:00:02.463529    4892 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-ada8d7-29889840-xldh2" podStartSLOduration=2.463509545 podStartE2EDuration="2.463509545s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 20:00:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 20:00:02.444199961 +0000 UTC m=+2650.014665565" watchObservedRunningTime="2026-10-30 20:00:02.463509545 +0000 UTC m=+2650.033975145"
Oct 30 20:00:02.560998 ip-10-0-132-82 kubenswrapper[4892]: I1030 20:00:02.558891    4892 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-6ac304-29889840-fhd4j" podStartSLOduration=2.5588709979999997 podStartE2EDuration="2.558870998s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 20:00:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 20:00:02.558010492 +0000 UTC m=+2650.128476091" watchObservedRunningTime="2026-10-30 20:00:02.558870998 +0000 UTC m=+2650.129336598"
Oct 30 20:00:02.676129 ip-10-0-132-82 kubenswrapper[4892]: I1030 20:00:02.676075    4892 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-686689-29889840-l2mjk" podStartSLOduration=2.676061127 podStartE2EDuration="2.676061127s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 20:00:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 20:00:02.673701844 +0000 UTC m=+2650.244167443" watchObservedRunningTime="2026-10-30 20:00:02.676061127 +0000 UTC m=+2650.246526747"
Oct 30 20:00:02.770579 ip-10-0-132-82 kubenswrapper[4892]: I1030 20:00:02.770552    4892 patch_prober.go:29] interesting pod/e2e-inf-a-132119-openshift-default-849b46588f-29f2x container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.214:15021/healthz/ready\": dial tcp 10.132.0.214:15021: connect: connection refused" start-of-body=
Oct 30 20:00:02.770699 ip-10-0-132-82 kubenswrapper[4892]: I1030 20:00:02.770605    4892 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-132119-openshift-default-849b46588f-29f2x" podUID="c76fc24b-0fe2-490e-8967-9a5eb4e9ce12" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.214:15021/healthz/ready\": dial tcp 10.132.0.214:15021: connect: connection refused"
Oct 30 20:00:02.956989 ip-10-0-132-82 kubenswrapper[4892]: I1030 20:00:02.955335    4892 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-132119-29889840-dhpf8" podStartSLOduration=2.955314172 podStartE2EDuration="2.955314172s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 20:00:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 20:00:02.926687644 +0000 UTC m=+2650.497153243" watchObservedRunningTime="2026-10-30 20:00:02.955314172 +0000 UTC m=+2650.525779773"
Oct 30 20:00:02.964803 ip-10-0-132-82 kubenswrapper[4892]: I1030 20:00:02.964757    4892 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-2d9861-29889840-79j62" podStartSLOduration=2.964746636 podStartE2EDuration="2.964746636s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 20:00:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 20:00:02.79686085 +0000 UTC m=+2650.367326458" watchObservedRunningTime="2026-10-30 20:00:02.964746636 +0000 UTC m=+2650.535212287"
Oct 30 20:00:03.064857 ip-10-0-132-82 kubenswrapper[4892]: I1030 20:00:03.064790    4892 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-b-1c5537-29889840-6qrmj" podStartSLOduration=3.064770903 podStartE2EDuration="3.064770903s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 20:00:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 20:00:03.052589331 +0000 UTC m=+2650.623054931" watchObservedRunningTime="2026-10-30 20:00:03.064770903 +0000 UTC m=+2650.635236504"
Oct 30 20:00:03.171299 ip-10-0-132-82 kubenswrapper[4892]: I1030 20:00:03.171232    4892 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-c75ed4-29889840-7s578" podStartSLOduration=3.171213961 podStartE2EDuration="3.171213961s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 20:00:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 20:00:03.166532513 +0000 UTC m=+2650.736998112" watchObservedRunningTime="2026-10-30 20:00:03.171213961 +0000 UTC m=+2650.741679562"
Oct 30 20:00:03.227744 ip-10-0-132-82 kubenswrapper[4892]: I1030 20:00:03.227706    4892 generic.go:356] "Generic (PLEG): container finished" podID="9df510ae-c995-4678-816a-9c4834778991" containerID="4b853a33cf49f7bd8b51b13d652a340cbaabad4ca070afcae5b74eff868a0440" exitCode=60
Oct 30 20:00:03.227953 ip-10-0-132-82 kubenswrapper[4892]: I1030 20:00:03.227793    4892 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-cf9b6f-29889840-8k6mb" event={"ID":"9df510ae-c995-4678-816a-9c4834778991","Type":"ContainerDied","Data":"4b853a33cf49f7bd8b51b13d652a340cbaabad4ca070afcae5b74eff868a0440"}
Oct 30 20:00:03.228476 ip-10-0-132-82 kubenswrapper[4892]: I1030 20:00:03.228454    4892 scope.go:122] "RemoveContainer" containerID="4b853a33cf49f7bd8b51b13d652a340cbaabad4ca070afcae5b74eff868a0440"
Oct 30 20:00:03.230013 ip-10-0-132-82 kubenswrapper[4892]: I1030 20:00:03.229958    4892 generic.go:356] "Generic (PLEG): container finished" podID="8152ed9d-b608-4a87-aded-8dda2d975174" containerID="bce4df99cf6c5b27b2f216fee73b22ad4d1577335f04e91bd7c923d75c77acfe" exitCode=60
Oct 30 20:00:03.230163 ip-10-0-132-82 kubenswrapper[4892]: I1030 20:00:03.230125    4892 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-c75ed4-29889840-7s578" event={"ID":"8152ed9d-b608-4a87-aded-8dda2d975174","Type":"ContainerDied","Data":"bce4df99cf6c5b27b2f216fee73b22ad4d1577335f04e91bd7c923d75c77acfe"}
Oct 30 20:00:03.230487 ip-10-0-132-82 kubenswrapper[4892]: I1030 20:00:03.230470    4892 scope.go:122] "RemoveContainer" containerID="bce4df99cf6c5b27b2f216fee73b22ad4d1577335f04e91bd7c923d75c77acfe"
Oct 30 20:00:03.232476 ip-10-0-132-82 kubenswrapper[4892]: I1030 20:00:03.232451    4892 generic.go:356] "Generic (PLEG): container finished" podID="0b0b9f79-e0a9-41a1-b513-01ea5661f64d" containerID="061f932bd42963fa6662e82067091f68e02d84b9a616a914aac375c44fe93429" exitCode=60
Oct 30 20:00:03.232592 ip-10-0-132-82 kubenswrapper[4892]: I1030 20:00:03.232552    4892 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-686689-29889840-l2mjk" event={"ID":"0b0b9f79-e0a9-41a1-b513-01ea5661f64d","Type":"ContainerDied","Data":"061f932bd42963fa6662e82067091f68e02d84b9a616a914aac375c44fe93429"}
Oct 30 20:00:03.233215 ip-10-0-132-82 kubenswrapper[4892]: I1030 20:00:03.233192    4892 scope.go:122] "RemoveContainer" containerID="061f932bd42963fa6662e82067091f68e02d84b9a616a914aac375c44fe93429"
Oct 30 20:00:03.234770 ip-10-0-132-82 kubenswrapper[4892]: I1030 20:00:03.234747    4892 generic.go:356] "Generic (PLEG): container finished" podID="82b75f10-16db-40c1-9696-f1e7f9917419" containerID="64bddd6fd07a41fee1b4b6177591b63f5f29a845409a20b8d75d196967662928" exitCode=60
Oct 30 20:00:03.234869 ip-10-0-132-82 kubenswrapper[4892]: I1030 20:00:03.234775    4892 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-2d9861-29889840-79j62" event={"ID":"82b75f10-16db-40c1-9696-f1e7f9917419","Type":"ContainerDied","Data":"64bddd6fd07a41fee1b4b6177591b63f5f29a845409a20b8d75d196967662928"}
Oct 30 20:00:03.235074 ip-10-0-132-82 kubenswrapper[4892]: I1030 20:00:03.235048    4892 scope.go:122] "RemoveContainer" containerID="64bddd6fd07a41fee1b4b6177591b63f5f29a845409a20b8d75d196967662928"
Oct 30 20:00:03.236919 ip-10-0-132-82 kubenswrapper[4892]: I1030 20:00:03.236896    4892 generic.go:356] "Generic (PLEG): container finished" podID="e5edd2c3-9676-48a1-9b30-8d8a98512582" containerID="47442b1f1e65b6d69a87050fedc25434e5fc3e9eaa41e51f8171a483d9ba9c08" exitCode=0
Oct 30 20:00:03.237037 ip-10-0-132-82 kubenswrapper[4892]: I1030 20:00:03.236994    4892 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w0-6ac304-pcccr" event={"ID":"e5edd2c3-9676-48a1-9b30-8d8a98512582","Type":"ContainerDied","Data":"47442b1f1e65b6d69a87050fedc25434e5fc3e9eaa41e51f8171a483d9ba9c08"}
Oct 30 20:00:03.238761 ip-10-0-132-82 kubenswrapper[4892]: I1030 20:00:03.238741    4892 generic.go:356] "Generic (PLEG): container finished" podID="25d4aedf-735a-42e6-ae08-4561c967fc35" containerID="36de4fe16fcf502af6adffa8351c74b8fdf965db38d3a0198972fccef9a33019" exitCode=60
Oct 30 20:00:03.238876 ip-10-0-132-82 kubenswrapper[4892]: I1030 20:00:03.238827    4892 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-6ac304-29889840-fhd4j" event={"ID":"25d4aedf-735a-42e6-ae08-4561c967fc35","Type":"ContainerDied","Data":"36de4fe16fcf502af6adffa8351c74b8fdf965db38d3a0198972fccef9a33019"}
Oct 30 20:00:03.239051 ip-10-0-132-82 kubenswrapper[4892]: I1030 20:00:03.239033    4892 scope.go:122] "RemoveContainer" containerID="36de4fe16fcf502af6adffa8351c74b8fdf965db38d3a0198972fccef9a33019"
Oct 30 20:00:03.240765 ip-10-0-132-82 kubenswrapper[4892]: I1030 20:00:03.240746    4892 generic.go:356] "Generic (PLEG): container finished" podID="d16a477c-ad96-4363-b13d-b6d374f3f88d" containerID="6c36f8660059a6ee4bb79a48998d9f358363baed48a1f5bee4b61c36c32382e3" exitCode=60
Oct 30 20:00:03.240956 ip-10-0-132-82 kubenswrapper[4892]: I1030 20:00:03.240933    4892 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-132119-29889840-dhpf8" event={"ID":"d16a477c-ad96-4363-b13d-b6d374f3f88d","Type":"ContainerDied","Data":"6c36f8660059a6ee4bb79a48998d9f358363baed48a1f5bee4b61c36c32382e3"}
Oct 30 20:00:03.241079 ip-10-0-132-82 kubenswrapper[4892]: I1030 20:00:03.241053    4892 scope.go:122] "RemoveContainer" containerID="6c36f8660059a6ee4bb79a48998d9f358363baed48a1f5bee4b61c36c32382e3"
Oct 30 20:00:03.253431 ip-10-0-132-82 kubenswrapper[4892]: I1030 20:00:03.253411    4892 generic.go:356] "Generic (PLEG): container finished" podID="35018750-1063-4fb6-928f-ce6bcd401f87" containerID="d10bfe5efc03c322d25be6a0b3fbb1fb555758b5c7e6a31f26759377a01cb663" exitCode=60
Oct 30 20:00:03.253700 ip-10-0-132-82 kubenswrapper[4892]: I1030 20:00:03.253670    4892 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-ada8d7-29889840-xldh2" event={"ID":"35018750-1063-4fb6-928f-ce6bcd401f87","Type":"ContainerDied","Data":"d10bfe5efc03c322d25be6a0b3fbb1fb555758b5c7e6a31f26759377a01cb663"}
Oct 30 20:00:03.255288 ip-10-0-132-82 kubenswrapper[4892]: I1030 20:00:03.254113    4892 scope.go:122] "RemoveContainer" containerID="d10bfe5efc03c322d25be6a0b3fbb1fb555758b5c7e6a31f26759377a01cb663"
Oct 30 20:00:03.256593 ip-10-0-132-82 kubenswrapper[4892]: I1030 20:00:03.256569    4892 generic.go:356] "Generic (PLEG): container finished" podID="f6260396-890c-44ea-be8c-998dd2086ffa" containerID="6ff12781af2267ff40fb3409e5e65849e3b6eed01164a272d07eb264de71cc99" exitCode=60
Oct 30 20:00:03.256705 ip-10-0-132-82 kubenswrapper[4892]: I1030 20:00:03.256659    4892 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-b-1c5537-29889840-6qrmj" event={"ID":"f6260396-890c-44ea-be8c-998dd2086ffa","Type":"ContainerDied","Data":"6ff12781af2267ff40fb3409e5e65849e3b6eed01164a272d07eb264de71cc99"}
Oct 30 20:00:03.256845 ip-10-0-132-82 kubenswrapper[4892]: I1030 20:00:03.256827    4892 scope.go:122] "RemoveContainer" containerID="6ff12781af2267ff40fb3409e5e65849e3b6eed01164a272d07eb264de71cc99"
Oct 30 20:00:03.752947 ip-10-0-132-82 kubenswrapper[4892]: I1030 20:00:03.752916    4892 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w6-5f55cd-688df7c466-t26tq"]
Oct 30 20:00:03.775243 ip-10-0-132-82 kubenswrapper[4892]: I1030 20:00:03.775222    4892 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-a-132119-openshift-default-849b46588f-29f2x"
Oct 30 20:00:03.775504 ip-10-0-132-82 kubenswrapper[4892]: I1030 20:00:03.775483    4892 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-a-132119-openshift-default-849b46588f-29f2x"
Oct 30 20:00:03.776272 ip-10-0-132-82 kubenswrapper[4892]: I1030 20:00:03.776254    4892 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-a-132119-openshift-default-849b46588f-29f2x"
Oct 30 20:00:03.859993 ip-10-0-132-82 kubenswrapper[4892]: I1030 20:00:03.859948    4892 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w6-5f55cd-85cb4d6b67-9qrfs"]
Oct 30 20:00:04.277268 ip-10-0-132-82 kubenswrapper[4892]: I1030 20:00:04.277232    4892 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-6ac304-29889840-fhd4j" event={"ID":"25d4aedf-735a-42e6-ae08-4561c967fc35","Type":"ContainerStarted","Data":"6504efa1b5cb74f3f5fd3a151f2d401baa624be2926dc67a9b9b5ee14a7b5667"}
Oct 30 20:00:04.280096 ip-10-0-132-82 kubenswrapper[4892]: I1030 20:00:04.280059    4892 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-132119-29889840-dhpf8" event={"ID":"d16a477c-ad96-4363-b13d-b6d374f3f88d","Type":"ContainerStarted","Data":"8e3128732b35cf64c18addaa4a4bf68fa805cddf2e389136c5c96b68a9d82cf1"}
Oct 30 20:00:04.282074 ip-10-0-132-82 kubenswrapper[4892]: I1030 20:00:04.282048    4892 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-ada8d7-29889840-xldh2" event={"ID":"35018750-1063-4fb6-928f-ce6bcd401f87","Type":"ContainerStarted","Data":"9543375f0cc7e181b0fcdd4110a5c18a0ed681347cded35b833f8cc1154d6bbc"}
Oct 30 20:00:04.283901 ip-10-0-132-82 kubenswrapper[4892]: I1030 20:00:04.283879    4892 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-b-1c5537-29889840-6qrmj" event={"ID":"f6260396-890c-44ea-be8c-998dd2086ffa","Type":"ContainerStarted","Data":"eb45c7d92510d4cc281510421063331b6b57544c377370befcd0bb5e38984272"}
Oct 30 20:00:04.285744 ip-10-0-132-82 kubenswrapper[4892]: I1030 20:00:04.285725    4892 generic.go:356] "Generic (PLEG): container finished" podID="9df510ae-c995-4678-816a-9c4834778991" containerID="b25696256c49c627f70a0c87690749c4b46e7c19b9e955dcfd8b5e66b117685d" exitCode=60
Oct 30 20:00:04.285850 ip-10-0-132-82 kubenswrapper[4892]: I1030 20:00:04.285782    4892 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-cf9b6f-29889840-8k6mb" event={"ID":"9df510ae-c995-4678-816a-9c4834778991","Type":"ContainerDied","Data":"b25696256c49c627f70a0c87690749c4b46e7c19b9e955dcfd8b5e66b117685d"}
Oct 30 20:00:04.285850 ip-10-0-132-82 kubenswrapper[4892]: I1030 20:00:04.285815    4892 scope.go:122] "RemoveContainer" containerID="4b853a33cf49f7bd8b51b13d652a340cbaabad4ca070afcae5b74eff868a0440"
Oct 30 20:00:04.286054 ip-10-0-132-82 kubenswrapper[4892]: I1030 20:00:04.286036    4892 scope.go:122] "RemoveContainer" containerID="b25696256c49c627f70a0c87690749c4b46e7c19b9e955dcfd8b5e66b117685d"
Oct 30 20:00:04.286306 ip-10-0-132-82 kubenswrapper[4892]: E1030 20:00:04.286284    4892 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-a-w3-cf9b6f-29889840-8k6mb_odh-ai-gateway-infra(9df510ae-c995-4678-816a-9c4834778991)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-cf9b6f-29889840-8k6mb" podUID="9df510ae-c995-4678-816a-9c4834778991"
Oct 30 20:00:04.287694 ip-10-0-132-82 kubenswrapper[4892]: I1030 20:00:04.287672    4892 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-c75ed4-29889840-7s578" event={"ID":"8152ed9d-b608-4a87-aded-8dda2d975174","Type":"ContainerStarted","Data":"8f2e18fda9b1dfff0c3a988200a5ffc3cd947fbfb5d96afb08d42195e1fbb32f"}
Oct 30 20:00:04.289525 ip-10-0-132-82 kubenswrapper[4892]: I1030 20:00:04.289504    4892 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-686689-29889840-l2mjk" event={"ID":"0b0b9f79-e0a9-41a1-b513-01ea5661f64d","Type":"ContainerStarted","Data":"5a03d21d9c4839f536751bb171a61b0ba0ee1f8a963d6f3120838a3ce21a623e"}
Oct 30 20:00:04.291585 ip-10-0-132-82 kubenswrapper[4892]: I1030 20:00:04.291562    4892 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-2d9861-29889840-79j62" event={"ID":"82b75f10-16db-40c1-9696-f1e7f9917419","Type":"ContainerStarted","Data":"407af9f51a2672047a92dc943183d990891432c0b35f9bb3457bdeae6421d72b"}
Oct 30 20:00:04.648798 ip-10-0-132-82 kubenswrapper[4892]: I1030 20:00:04.648768    4892 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w5-686689/distinct-2-w5-686689-kserve-5ff578b4cd-rkc6s"]
Oct 30 20:00:04.897894 ip-10-0-132-82 kubenswrapper[4892]: I1030 20:00:04.897787    4892 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w0-6ac304-pcccr"
Oct 30 20:00:04.974114 ip-10-0-132-82 kubenswrapper[4892]: I1030 20:00:04.974074    4892 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/e5edd2c3-9676-48a1-9b30-8d8a98512582-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e5edd2c3-9676-48a1-9b30-8d8a98512582-maas-api-xks-ca\") pod \"e5edd2c3-9676-48a1-9b30-8d8a98512582\" (UID: \"e5edd2c3-9676-48a1-9b30-8d8a98512582\") "
Oct 30 20:00:04.974331 ip-10-0-132-82 kubenswrapper[4892]: I1030 20:00:04.974291    4892 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e5edd2c3-9676-48a1-9b30-8d8a98512582-kube-api-access-4rjqf\" (UniqueName: \"kubernetes.io/projected/e5edd2c3-9676-48a1-9b30-8d8a98512582-kube-api-access-4rjqf\") pod \"e5edd2c3-9676-48a1-9b30-8d8a98512582\" (UID: \"e5edd2c3-9676-48a1-9b30-8d8a98512582\") "
Oct 30 20:00:04.974407 ip-10-0-132-82 kubenswrapper[4892]: I1030 20:00:04.974329    4892 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e5edd2c3-9676-48a1-9b30-8d8a98512582-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e5edd2c3-9676-48a1-9b30-8d8a98512582-maas-api-openshift-ca\") pod \"e5edd2c3-9676-48a1-9b30-8d8a98512582\" (UID: \"e5edd2c3-9676-48a1-9b30-8d8a98512582\") "
Oct 30 20:00:04.974923 ip-10-0-132-82 kubenswrapper[4892]: I1030 20:00:04.974889    4892 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e5edd2c3-9676-48a1-9b30-8d8a98512582-maas-api-openshift-ca" pod "e5edd2c3-9676-48a1-9b30-8d8a98512582" (UID: "e5edd2c3-9676-48a1-9b30-8d8a98512582"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 20:00:04.976487 ip-10-0-132-82 kubenswrapper[4892]: I1030 20:00:04.976459    4892 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e5edd2c3-9676-48a1-9b30-8d8a98512582-maas-api-xks-ca" pod "e5edd2c3-9676-48a1-9b30-8d8a98512582" (UID: "e5edd2c3-9676-48a1-9b30-8d8a98512582"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 20:00:04.976996 ip-10-0-132-82 kubenswrapper[4892]: I1030 20:00:04.976942    4892 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e5edd2c3-9676-48a1-9b30-8d8a98512582-kube-api-access-4rjqf" pod "e5edd2c3-9676-48a1-9b30-8d8a98512582" (UID: "e5edd2c3-9676-48a1-9b30-8d8a98512582"). InnerVolumeSpecName "kube-api-access-4rjqf". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 20:00:05.076297 ip-10-0-132-82 kubenswrapper[4892]: I1030 20:00:05.076240    4892 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4rjqf\" (UniqueName: \"kubernetes.io/projected/e5edd2c3-9676-48a1-9b30-8d8a98512582-kube-api-access-4rjqf\") on node \"ip-10-0-132-82.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:00:05.076496 ip-10-0-132-82 kubenswrapper[4892]: I1030 20:00:05.076316    4892 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e5edd2c3-9676-48a1-9b30-8d8a98512582-maas-api-openshift-ca\") on node \"ip-10-0-132-82.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:00:05.076496 ip-10-0-132-82 kubenswrapper[4892]: I1030 20:00:05.076335    4892 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e5edd2c3-9676-48a1-9b30-8d8a98512582-maas-api-xks-ca\") on node \"ip-10-0-132-82.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:00:05.227711 ip-10-0-132-82 kubenswrapper[4892]: I1030 20:00:05.227623    4892 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-mt-dd9b8d79/e2e-lifecycle-model-dd9b8d79-kserve-7659b4b7d9-r2flr"]
Oct 30 20:00:05.322242 ip-10-0-132-82 kubenswrapper[4892]: I1030 20:00:05.321614    4892 scope.go:122] "RemoveContainer" containerID="b25696256c49c627f70a0c87690749c4b46e7c19b9e955dcfd8b5e66b117685d"
Oct 30 20:00:05.322242 ip-10-0-132-82 kubenswrapper[4892]: E1030 20:00:05.321948    4892 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-a-w3-cf9b6f-29889840-8k6mb_odh-ai-gateway-infra(9df510ae-c995-4678-816a-9c4834778991)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-cf9b6f-29889840-8k6mb" podUID="9df510ae-c995-4678-816a-9c4834778991"
Oct 30 20:00:05.325070 ip-10-0-132-82 kubenswrapper[4892]: I1030 20:00:05.325038    4892 generic.go:356] "Generic (PLEG): container finished" podID="8152ed9d-b608-4a87-aded-8dda2d975174" containerID="8f2e18fda9b1dfff0c3a988200a5ffc3cd947fbfb5d96afb08d42195e1fbb32f" exitCode=60
Oct 30 20:00:05.325224 ip-10-0-132-82 kubenswrapper[4892]: I1030 20:00:05.325185    4892 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-c75ed4-29889840-7s578" event={"ID":"8152ed9d-b608-4a87-aded-8dda2d975174","Type":"ContainerDied","Data":"8f2e18fda9b1dfff0c3a988200a5ffc3cd947fbfb5d96afb08d42195e1fbb32f"}
Oct 30 20:00:05.325310 ip-10-0-132-82 kubenswrapper[4892]: I1030 20:00:05.325265    4892 scope.go:122] "RemoveContainer" containerID="bce4df99cf6c5b27b2f216fee73b22ad4d1577335f04e91bd7c923d75c77acfe"
Oct 30 20:00:05.325435 ip-10-0-132-82 kubenswrapper[4892]: I1030 20:00:05.325415    4892 scope.go:122] "RemoveContainer" containerID="8f2e18fda9b1dfff0c3a988200a5ffc3cd947fbfb5d96afb08d42195e1fbb32f"
Oct 30 20:00:05.325784 ip-10-0-132-82 kubenswrapper[4892]: E1030 20:00:05.325758    4892 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-b-w3-c75ed4-29889840-7s578_odh-ai-gateway-infra(8152ed9d-b608-4a87-aded-8dda2d975174)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-c75ed4-29889840-7s578" podUID="8152ed9d-b608-4a87-aded-8dda2d975174"
Oct 30 20:00:05.329396 ip-10-0-132-82 kubenswrapper[4892]: I1030 20:00:05.328156    4892 generic.go:356] "Generic (PLEG): container finished" podID="0b0b9f79-e0a9-41a1-b513-01ea5661f64d" containerID="5a03d21d9c4839f536751bb171a61b0ba0ee1f8a963d6f3120838a3ce21a623e" exitCode=60
Oct 30 20:00:05.329396 ip-10-0-132-82 kubenswrapper[4892]: I1030 20:00:05.328247    4892 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-686689-29889840-l2mjk" event={"ID":"0b0b9f79-e0a9-41a1-b513-01ea5661f64d","Type":"ContainerDied","Data":"5a03d21d9c4839f536751bb171a61b0ba0ee1f8a963d6f3120838a3ce21a623e"}
Oct 30 20:00:05.329396 ip-10-0-132-82 kubenswrapper[4892]: I1030 20:00:05.328540    4892 scope.go:122] "RemoveContainer" containerID="5a03d21d9c4839f536751bb171a61b0ba0ee1f8a963d6f3120838a3ce21a623e"
Oct 30 20:00:05.329396 ip-10-0-132-82 kubenswrapper[4892]: E1030 20:00:05.328786    4892 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w5-686689-29889840-l2mjk_odh-ai-gateway-infra(0b0b9f79-e0a9-41a1-b513-01ea5661f64d)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-686689-29889840-l2mjk" podUID="0b0b9f79-e0a9-41a1-b513-01ea5661f64d"
Oct 30 20:00:05.330651 ip-10-0-132-82 kubenswrapper[4892]: I1030 20:00:05.330613    4892 generic.go:356] "Generic (PLEG): container finished" podID="82b75f10-16db-40c1-9696-f1e7f9917419" containerID="407af9f51a2672047a92dc943183d990891432c0b35f9bb3457bdeae6421d72b" exitCode=60
Oct 30 20:00:05.330937 ip-10-0-132-82 kubenswrapper[4892]: I1030 20:00:05.330695    4892 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-2d9861-29889840-79j62" event={"ID":"82b75f10-16db-40c1-9696-f1e7f9917419","Type":"ContainerDied","Data":"407af9f51a2672047a92dc943183d990891432c0b35f9bb3457bdeae6421d72b"}
Oct 30 20:00:05.331059 ip-10-0-132-82 kubenswrapper[4892]: I1030 20:00:05.330939    4892 scope.go:122] "RemoveContainer" containerID="407af9f51a2672047a92dc943183d990891432c0b35f9bb3457bdeae6421d72b"
Oct 30 20:00:05.331578 ip-10-0-132-82 kubenswrapper[4892]: E1030 20:00:05.331265    4892 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w0-2d9861-29889840-79j62_odh-ai-gateway-infra(82b75f10-16db-40c1-9696-f1e7f9917419)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-2d9861-29889840-79j62" podUID="82b75f10-16db-40c1-9696-f1e7f9917419"
Oct 30 20:00:05.333699 ip-10-0-132-82 kubenswrapper[4892]: I1030 20:00:05.333017    4892 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w0-6ac304-pcccr" event={"ID":"e5edd2c3-9676-48a1-9b30-8d8a98512582","Type":"ContainerDied","Data":"538a04b54e6c54ee2bc34c0b3f872fe6141567bf957261c4b048593cccc9b7bb"}
Oct 30 20:00:05.333699 ip-10-0-132-82 kubenswrapper[4892]: I1030 20:00:05.333062    4892 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="538a04b54e6c54ee2bc34c0b3f872fe6141567bf957261c4b048593cccc9b7bb"
Oct 30 20:00:05.334026 ip-10-0-132-82 kubenswrapper[4892]: I1030 20:00:05.334012    4892 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w0-6ac304-pcccr"
Oct 30 20:00:05.343077 ip-10-0-132-82 kubenswrapper[4892]: I1030 20:00:05.337660    4892 generic.go:356] "Generic (PLEG): container finished" podID="25d4aedf-735a-42e6-ae08-4561c967fc35" containerID="6504efa1b5cb74f3f5fd3a151f2d401baa624be2926dc67a9b9b5ee14a7b5667" exitCode=60
Oct 30 20:00:05.343077 ip-10-0-132-82 kubenswrapper[4892]: I1030 20:00:05.337730    4892 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-6ac304-29889840-fhd4j" event={"ID":"25d4aedf-735a-42e6-ae08-4561c967fc35","Type":"ContainerDied","Data":"6504efa1b5cb74f3f5fd3a151f2d401baa624be2926dc67a9b9b5ee14a7b5667"}
Oct 30 20:00:05.343077 ip-10-0-132-82 kubenswrapper[4892]: I1030 20:00:05.338043    4892 scope.go:122] "RemoveContainer" containerID="6504efa1b5cb74f3f5fd3a151f2d401baa624be2926dc67a9b9b5ee14a7b5667"
Oct 30 20:00:05.344288 ip-10-0-132-82 kubenswrapper[4892]: E1030 20:00:05.343788    4892 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-b-w0-6ac304-29889840-fhd4j_odh-ai-gateway-infra(25d4aedf-735a-42e6-ae08-4561c967fc35)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-6ac304-29889840-fhd4j" podUID="25d4aedf-735a-42e6-ae08-4561c967fc35"
Oct 30 20:00:05.346796 ip-10-0-132-82 kubenswrapper[4892]: I1030 20:00:05.346209    4892 generic.go:356] "Generic (PLEG): container finished" podID="d16a477c-ad96-4363-b13d-b6d374f3f88d" containerID="8e3128732b35cf64c18addaa4a4bf68fa805cddf2e389136c5c96b68a9d82cf1" exitCode=60
Oct 30 20:00:05.346796 ip-10-0-132-82 kubenswrapper[4892]: I1030 20:00:05.346278    4892 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-132119-29889840-dhpf8" event={"ID":"d16a477c-ad96-4363-b13d-b6d374f3f88d","Type":"ContainerDied","Data":"8e3128732b35cf64c18addaa4a4bf68fa805cddf2e389136c5c96b68a9d82cf1"}
Oct 30 20:00:05.346796 ip-10-0-132-82 kubenswrapper[4892]: I1030 20:00:05.346544    4892 scope.go:122] "RemoveContainer" containerID="8e3128732b35cf64c18addaa4a4bf68fa805cddf2e389136c5c96b68a9d82cf1"
Oct 30 20:00:05.346796 ip-10-0-132-82 kubenswrapper[4892]: E1030 20:00:05.346753    4892 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-inf-a-132119-29889840-dhpf8_odh-ai-gateway-infra(d16a477c-ad96-4363-b13d-b6d374f3f88d)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-132119-29889840-dhpf8" podUID="d16a477c-ad96-4363-b13d-b6d374f3f88d"
Oct 30 20:00:05.349837 ip-10-0-132-82 kubenswrapper[4892]: I1030 20:00:05.349807    4892 generic.go:356] "Generic (PLEG): container finished" podID="35018750-1063-4fb6-928f-ce6bcd401f87" containerID="9543375f0cc7e181b0fcdd4110a5c18a0ed681347cded35b833f8cc1154d6bbc" exitCode=60
Oct 30 20:00:05.349956 ip-10-0-132-82 kubenswrapper[4892]: I1030 20:00:05.349940    4892 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-ada8d7-29889840-xldh2" event={"ID":"35018750-1063-4fb6-928f-ce6bcd401f87","Type":"ContainerDied","Data":"9543375f0cc7e181b0fcdd4110a5c18a0ed681347cded35b833f8cc1154d6bbc"}
Oct 30 20:00:05.350345 ip-10-0-132-82 kubenswrapper[4892]: I1030 20:00:05.350326    4892 scope.go:122] "RemoveContainer" containerID="9543375f0cc7e181b0fcdd4110a5c18a0ed681347cded35b833f8cc1154d6bbc"
Oct 30 20:00:05.350640 ip-10-0-132-82 kubenswrapper[4892]: E1030 20:00:05.350615    4892 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w3-ada8d7-29889840-xldh2_odh-ai-gateway-infra(35018750-1063-4fb6-928f-ce6bcd401f87)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-ada8d7-29889840-xldh2" podUID="35018750-1063-4fb6-928f-ce6bcd401f87"
Oct 30 20:00:05.351474 ip-10-0-132-82 kubenswrapper[4892]: I1030 20:00:05.351459    4892 scope.go:122] "RemoveContainer" containerID="061f932bd42963fa6662e82067091f68e02d84b9a616a914aac375c44fe93429"
Oct 30 20:00:05.353789 ip-10-0-132-82 kubenswrapper[4892]: I1030 20:00:05.353770    4892 generic.go:356] "Generic (PLEG): container finished" podID="f6260396-890c-44ea-be8c-998dd2086ffa" containerID="eb45c7d92510d4cc281510421063331b6b57544c377370befcd0bb5e38984272" exitCode=60
Oct 30 20:00:05.354054 ip-10-0-132-82 kubenswrapper[4892]: I1030 20:00:05.354034    4892 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-b-1c5537-29889840-6qrmj" event={"ID":"f6260396-890c-44ea-be8c-998dd2086ffa","Type":"ContainerDied","Data":"eb45c7d92510d4cc281510421063331b6b57544c377370befcd0bb5e38984272"}
Oct 30 20:00:05.354440 ip-10-0-132-82 kubenswrapper[4892]: I1030 20:00:05.354422    4892 scope.go:122] "RemoveContainer" containerID="eb45c7d92510d4cc281510421063331b6b57544c377370befcd0bb5e38984272"
Oct 30 20:00:05.354608 ip-10-0-132-82 kubenswrapper[4892]: E1030 20:00:05.354592    4892 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-api-b-1c5537-29889840-6qrmj_odh-ai-gateway-infra(f6260396-890c-44ea-be8c-998dd2086ffa)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-b-1c5537-29889840-6qrmj" podUID="f6260396-890c-44ea-be8c-998dd2086ffa"
Oct 30 20:00:05.367536 ip-10-0-132-82 kubenswrapper[4892]: I1030 20:00:05.367518    4892 scope.go:122] "RemoveContainer" containerID="64bddd6fd07a41fee1b4b6177591b63f5f29a845409a20b8d75d196967662928"
Oct 30 20:00:05.385292 ip-10-0-132-82 kubenswrapper[4892]: I1030 20:00:05.385263    4892 scope.go:122] "RemoveContainer" containerID="36de4fe16fcf502af6adffa8351c74b8fdf965db38d3a0198972fccef9a33019"
Oct 30 20:00:05.405079 ip-10-0-132-82 kubenswrapper[4892]: I1030 20:00:05.405051    4892 scope.go:122] "RemoveContainer" containerID="6c36f8660059a6ee4bb79a48998d9f358363baed48a1f5bee4b61c36c32382e3"
Oct 30 20:00:05.420473 ip-10-0-132-82 kubenswrapper[4892]: I1030 20:00:05.420450    4892 scope.go:122] "RemoveContainer" containerID="d10bfe5efc03c322d25be6a0b3fbb1fb555758b5c7e6a31f26759377a01cb663"
Oct 30 20:00:05.437252 ip-10-0-132-82 kubenswrapper[4892]: I1030 20:00:05.437228    4892 scope.go:122] "RemoveContainer" containerID="6ff12781af2267ff40fb3409e5e65849e3b6eed01164a272d07eb264de71cc99"
Oct 30 20:00:06.369691 ip-10-0-132-82 kubenswrapper[4892]: I1030 20:00:06.369633    4892 scope.go:122] "RemoveContainer" containerID="6504efa1b5cb74f3f5fd3a151f2d401baa624be2926dc67a9b9b5ee14a7b5667"
Oct 30 20:00:06.370624 ip-10-0-132-82 kubenswrapper[4892]: E1030 20:00:06.370592    4892 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-b-w0-6ac304-29889840-fhd4j_odh-ai-gateway-infra(25d4aedf-735a-42e6-ae08-4561c967fc35)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-6ac304-29889840-fhd4j" podUID="25d4aedf-735a-42e6-ae08-4561c967fc35"
Oct 30 20:00:06.380208 ip-10-0-132-82 kubenswrapper[4892]: I1030 20:00:06.379267    4892 scope.go:122] "RemoveContainer" containerID="8e3128732b35cf64c18addaa4a4bf68fa805cddf2e389136c5c96b68a9d82cf1"
Oct 30 20:00:06.380208 ip-10-0-132-82 kubenswrapper[4892]: E1030 20:00:06.379562    4892 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-inf-a-132119-29889840-dhpf8_odh-ai-gateway-infra(d16a477c-ad96-4363-b13d-b6d374f3f88d)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-132119-29889840-dhpf8" podUID="d16a477c-ad96-4363-b13d-b6d374f3f88d"
Oct 30 20:00:06.382137 ip-10-0-132-82 kubenswrapper[4892]: I1030 20:00:06.382114    4892 scope.go:122] "RemoveContainer" containerID="9543375f0cc7e181b0fcdd4110a5c18a0ed681347cded35b833f8cc1154d6bbc"
Oct 30 20:00:06.382393 ip-10-0-132-82 kubenswrapper[4892]: E1030 20:00:06.382365    4892 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w3-ada8d7-29889840-xldh2_odh-ai-gateway-infra(35018750-1063-4fb6-928f-ce6bcd401f87)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-ada8d7-29889840-xldh2" podUID="35018750-1063-4fb6-928f-ce6bcd401f87"
Oct 30 20:00:06.384416 ip-10-0-132-82 kubenswrapper[4892]: I1030 20:00:06.384395    4892 scope.go:122] "RemoveContainer" containerID="eb45c7d92510d4cc281510421063331b6b57544c377370befcd0bb5e38984272"
Oct 30 20:00:06.384609 ip-10-0-132-82 kubenswrapper[4892]: E1030 20:00:06.384588    4892 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-api-b-1c5537-29889840-6qrmj_odh-ai-gateway-infra(f6260396-890c-44ea-be8c-998dd2086ffa)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-b-1c5537-29889840-6qrmj" podUID="f6260396-890c-44ea-be8c-998dd2086ffa"
Oct 30 20:00:06.386419 ip-10-0-132-82 kubenswrapper[4892]: I1030 20:00:06.386396    4892 scope.go:122] "RemoveContainer" containerID="8f2e18fda9b1dfff0c3a988200a5ffc3cd947fbfb5d96afb08d42195e1fbb32f"
Oct 30 20:00:06.386687 ip-10-0-132-82 kubenswrapper[4892]: E1030 20:00:06.386656    4892 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-b-w3-c75ed4-29889840-7s578_odh-ai-gateway-infra(8152ed9d-b608-4a87-aded-8dda2d975174)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-c75ed4-29889840-7s578" podUID="8152ed9d-b608-4a87-aded-8dda2d975174"
Oct 30 20:00:06.388303 ip-10-0-132-82 kubenswrapper[4892]: I1030 20:00:06.388273    4892 scope.go:122] "RemoveContainer" containerID="5a03d21d9c4839f536751bb171a61b0ba0ee1f8a963d6f3120838a3ce21a623e"
Oct 30 20:00:06.388518 ip-10-0-132-82 kubenswrapper[4892]: E1030 20:00:06.388490    4892 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w5-686689-29889840-l2mjk_odh-ai-gateway-infra(0b0b9f79-e0a9-41a1-b513-01ea5661f64d)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-686689-29889840-l2mjk" podUID="0b0b9f79-e0a9-41a1-b513-01ea5661f64d"
Oct 30 20:00:06.390305 ip-10-0-132-82 kubenswrapper[4892]: I1030 20:00:06.390286    4892 scope.go:122] "RemoveContainer" containerID="407af9f51a2672047a92dc943183d990891432c0b35f9bb3457bdeae6421d72b"
Oct 30 20:00:06.390519 ip-10-0-132-82 kubenswrapper[4892]: E1030 20:00:06.390496    4892 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w0-2d9861-29889840-79j62_odh-ai-gateway-infra(82b75f10-16db-40c1-9696-f1e7f9917419)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-2d9861-29889840-79j62" podUID="82b75f10-16db-40c1-9696-f1e7f9917419"
Oct 30 20:00:06.750310 ip-10-0-132-82 kubenswrapper[4892]: I1030 20:00:06.750277    4892 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-d52df0-openshift-default-6d5f887b45-985gh"]
Oct 30 20:00:06.750537 ip-10-0-132-82 kubenswrapper[4892]: I1030 20:00:06.750510    4892 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w1-d52df0-openshift-default-6d5f887b45-985gh" podUID="e453311b-d7b1-4819-a57a-d1de4886de26" containerName="istio-proxy" containerID="cri-o://cb0272a78a5802f04e51979e7d05a3ba3bf4351cfe7d1f4180da73935a42deb6" gracePeriod=30
Oct 30 20:00:08.753610 ip-10-0-132-82 kubenswrapper[4892]: I1030 20:00:08.753563    4892 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-worker-w6-5f55cd-688df7c466-t26tq" podUID="b31aad4b-7f5c-46c1-9c46-e39ea2ea5803" containerName="payload-processing" containerID="cri-o://07f691a8d9b079f12d2fd30161803246153cf95492383e4e9879b569a667dd9e" gracePeriod=25
Oct 30 20:00:08.861060 ip-10-0-132-82 kubenswrapper[4892]: I1030 20:00:08.861017    4892 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-worker-w6-5f55cd-85cb4d6b67-9qrfs" podUID="4286dc2a-9c31-454f-891d-31d896d7a66b" containerName="payload-pre-processing" containerID="cri-o://6ac8f7e45ada17a45ce0acac47bd7dc40934edee4d643c7161eeee66cb5817be" gracePeriod=25
Oct 30 20:00:12.141308 ip-10-0-132-82 kubenswrapper[4892]: I1030 20:00:12.141283    4892 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-d52df0-openshift-default-6d5f887b45-985gh"
Oct 30 20:00:12.294637 ip-10-0-132-82 kubenswrapper[4892]: I1030 20:00:12.294598    4892 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e453311b-d7b1-4819-a57a-d1de4886de26-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e453311b-d7b1-4819-a57a-d1de4886de26-workload-certs\") pod \"e453311b-d7b1-4819-a57a-d1de4886de26\" (UID: \"e453311b-d7b1-4819-a57a-d1de4886de26\") "
Oct 30 20:00:12.294832 ip-10-0-132-82 kubenswrapper[4892]: I1030 20:00:12.294659    4892 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e453311b-d7b1-4819-a57a-d1de4886de26-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e453311b-d7b1-4819-a57a-d1de4886de26-istio-envoy\") pod \"e453311b-d7b1-4819-a57a-d1de4886de26\" (UID: \"e453311b-d7b1-4819-a57a-d1de4886de26\") "
Oct 30 20:00:12.294832 ip-10-0-132-82 kubenswrapper[4892]: I1030 20:00:12.294701    4892 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e453311b-d7b1-4819-a57a-d1de4886de26-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e453311b-d7b1-4819-a57a-d1de4886de26-istiod-ca-cert\") pod \"e453311b-d7b1-4819-a57a-d1de4886de26\" (UID: \"e453311b-d7b1-4819-a57a-d1de4886de26\") "
Oct 30 20:00:12.294832 ip-10-0-132-82 kubenswrapper[4892]: I1030 20:00:12.294749    4892 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e453311b-d7b1-4819-a57a-d1de4886de26-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e453311b-d7b1-4819-a57a-d1de4886de26-credential-socket\") pod \"e453311b-d7b1-4819-a57a-d1de4886de26\" (UID: \"e453311b-d7b1-4819-a57a-d1de4886de26\") "
Oct 30 20:00:12.294832 ip-10-0-132-82 kubenswrapper[4892]: I1030 20:00:12.294791    4892 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/e453311b-d7b1-4819-a57a-d1de4886de26-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e453311b-d7b1-4819-a57a-d1de4886de26-istio-podinfo\") pod \"e453311b-d7b1-4819-a57a-d1de4886de26\" (UID: \"e453311b-d7b1-4819-a57a-d1de4886de26\") "
Oct 30 20:00:12.295097 ip-10-0-132-82 kubenswrapper[4892]: I1030 20:00:12.294833    4892 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e453311b-d7b1-4819-a57a-d1de4886de26-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e453311b-d7b1-4819-a57a-d1de4886de26-workload-socket\") pod \"e453311b-d7b1-4819-a57a-d1de4886de26\" (UID: \"e453311b-d7b1-4819-a57a-d1de4886de26\") "
Oct 30 20:00:12.295097 ip-10-0-132-82 kubenswrapper[4892]: I1030 20:00:12.294877    4892 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e453311b-d7b1-4819-a57a-d1de4886de26-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e453311b-d7b1-4819-a57a-d1de4886de26-istio-data\") pod \"e453311b-d7b1-4819-a57a-d1de4886de26\" (UID: \"e453311b-d7b1-4819-a57a-d1de4886de26\") "
Oct 30 20:00:12.295097 ip-10-0-132-82 kubenswrapper[4892]: I1030 20:00:12.294918    4892 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e453311b-d7b1-4819-a57a-d1de4886de26-workload-certs" pod "e453311b-d7b1-4819-a57a-d1de4886de26" (UID: "e453311b-d7b1-4819-a57a-d1de4886de26"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 20:00:12.295097 ip-10-0-132-82 kubenswrapper[4892]: I1030 20:00:12.294942    4892 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e453311b-d7b1-4819-a57a-d1de4886de26-istio-token\" (UniqueName: \"kubernetes.io/projected/e453311b-d7b1-4819-a57a-d1de4886de26-istio-token\") pod \"e453311b-d7b1-4819-a57a-d1de4886de26\" (UID: \"e453311b-d7b1-4819-a57a-d1de4886de26\") "
Oct 30 20:00:12.295097 ip-10-0-132-82 kubenswrapper[4892]: I1030 20:00:12.295074    4892 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e453311b-d7b1-4819-a57a-d1de4886de26-kube-api-access-bkvrn\" (UniqueName: \"kubernetes.io/projected/e453311b-d7b1-4819-a57a-d1de4886de26-kube-api-access-bkvrn\") pod \"e453311b-d7b1-4819-a57a-d1de4886de26\" (UID: \"e453311b-d7b1-4819-a57a-d1de4886de26\") "
Oct 30 20:00:12.295372 ip-10-0-132-82 kubenswrapper[4892]: I1030 20:00:12.295122    4892 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e453311b-d7b1-4819-a57a-d1de4886de26-istiod-ca-cert" pod "e453311b-d7b1-4819-a57a-d1de4886de26" (UID: "e453311b-d7b1-4819-a57a-d1de4886de26"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 20:00:12.295372 ip-10-0-132-82 kubenswrapper[4892]: I1030 20:00:12.295189    4892 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e453311b-d7b1-4819-a57a-d1de4886de26-credential-socket" pod "e453311b-d7b1-4819-a57a-d1de4886de26" (UID: "e453311b-d7b1-4819-a57a-d1de4886de26"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 20:00:12.295483 ip-10-0-132-82 kubenswrapper[4892]: I1030 20:00:12.295371    4892 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e453311b-d7b1-4819-a57a-d1de4886de26-istio-data" pod "e453311b-d7b1-4819-a57a-d1de4886de26" (UID: "e453311b-d7b1-4819-a57a-d1de4886de26"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 20:00:12.295483 ip-10-0-132-82 kubenswrapper[4892]: I1030 20:00:12.295391    4892 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e453311b-d7b1-4819-a57a-d1de4886de26-workload-socket" pod "e453311b-d7b1-4819-a57a-d1de4886de26" (UID: "e453311b-d7b1-4819-a57a-d1de4886de26"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 20:00:12.295656 ip-10-0-132-82 kubenswrapper[4892]: I1030 20:00:12.295637    4892 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e453311b-d7b1-4819-a57a-d1de4886de26-istio-data\") on node \"ip-10-0-132-82.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:00:12.295733 ip-10-0-132-82 kubenswrapper[4892]: I1030 20:00:12.295658    4892 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e453311b-d7b1-4819-a57a-d1de4886de26-workload-certs\") on node \"ip-10-0-132-82.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:00:12.295733 ip-10-0-132-82 kubenswrapper[4892]: I1030 20:00:12.295669    4892 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e453311b-d7b1-4819-a57a-d1de4886de26-istiod-ca-cert\") on node \"ip-10-0-132-82.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:00:12.295733 ip-10-0-132-82 kubenswrapper[4892]: I1030 20:00:12.295679    4892 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e453311b-d7b1-4819-a57a-d1de4886de26-credential-socket\") on node \"ip-10-0-132-82.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:00:12.295733 ip-10-0-132-82 kubenswrapper[4892]: I1030 20:00:12.295688    4892 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e453311b-d7b1-4819-a57a-d1de4886de26-workload-socket\") on node \"ip-10-0-132-82.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:00:12.298204 ip-10-0-132-82 kubenswrapper[4892]: I1030 20:00:12.298177    4892 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e453311b-d7b1-4819-a57a-d1de4886de26-kube-api-access-bkvrn" pod "e453311b-d7b1-4819-a57a-d1de4886de26" (UID: "e453311b-d7b1-4819-a57a-d1de4886de26"). InnerVolumeSpecName "kube-api-access-bkvrn". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 20:00:12.298411 ip-10-0-132-82 kubenswrapper[4892]: I1030 20:00:12.298381    4892 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e453311b-d7b1-4819-a57a-d1de4886de26-istio-token" pod "e453311b-d7b1-4819-a57a-d1de4886de26" (UID: "e453311b-d7b1-4819-a57a-d1de4886de26"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 20:00:12.299357 ip-10-0-132-82 kubenswrapper[4892]: I1030 20:00:12.299285    4892 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/e453311b-d7b1-4819-a57a-d1de4886de26-istio-podinfo" pod "e453311b-d7b1-4819-a57a-d1de4886de26" (UID: "e453311b-d7b1-4819-a57a-d1de4886de26"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 20:00:12.301476 ip-10-0-132-82 kubenswrapper[4892]: I1030 20:00:12.301439    4892 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e453311b-d7b1-4819-a57a-d1de4886de26-istio-envoy" pod "e453311b-d7b1-4819-a57a-d1de4886de26" (UID: "e453311b-d7b1-4819-a57a-d1de4886de26"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 20:00:12.334327 ip-10-0-132-82 kubenswrapper[4892]: I1030 20:00:12.334260    4892 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w6-5f55cd/premium-w6-5f55cd-kserve-7f785c544b-pcvhb" podUID="191593a2-e3a9-4ce4-a1c6-c089723b0237" containerName="main" containerID="cri-o://1704c766f16e7a726d7dcfdc4866656f49d41dcd941c72047c420a8145a5e9f2" gracePeriod=45
Oct 30 20:00:12.351569 ip-10-0-132-82 kubenswrapper[4892]: I1030 20:00:12.351535    4892 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w6-5f55cd/simulator-w6-5f55cd-kserve-5ff86b7447-h9tm7" podUID="3cb47a99-8be1-4756-b31a-43b58cc6b8dc" containerName="main" containerID="cri-o://7fb9393a625a7900385890a2e7393fa67154902bd3f995959de80d728f5fe573" gracePeriod=45
Oct 30 20:00:12.397540 ip-10-0-132-82 kubenswrapper[4892]: I1030 20:00:12.397511    4892 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e453311b-d7b1-4819-a57a-d1de4886de26-istio-podinfo\") on node \"ip-10-0-132-82.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:00:12.397624 ip-10-0-132-82 kubenswrapper[4892]: I1030 20:00:12.397548    4892 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e453311b-d7b1-4819-a57a-d1de4886de26-istio-token\") on node \"ip-10-0-132-82.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:00:12.397624 ip-10-0-132-82 kubenswrapper[4892]: I1030 20:00:12.397564    4892 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bkvrn\" (UniqueName: \"kubernetes.io/projected/e453311b-d7b1-4819-a57a-d1de4886de26-kube-api-access-bkvrn\") on node \"ip-10-0-132-82.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:00:12.397624 ip-10-0-132-82 kubenswrapper[4892]: I1030 20:00:12.397579    4892 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e453311b-d7b1-4819-a57a-d1de4886de26-istio-envoy\") on node \"ip-10-0-132-82.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:00:12.471490 ip-10-0-132-82 kubenswrapper[4892]: I1030 20:00:12.471457    4892 generic.go:356] "Generic (PLEG): container finished" podID="e453311b-d7b1-4819-a57a-d1de4886de26" containerID="cb0272a78a5802f04e51979e7d05a3ba3bf4351cfe7d1f4180da73935a42deb6" exitCode=0
Oct 30 20:00:12.471602 ip-10-0-132-82 kubenswrapper[4892]: I1030 20:00:12.471522    4892 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-d52df0-openshift-default-6d5f887b45-985gh"
Oct 30 20:00:12.471602 ip-10-0-132-82 kubenswrapper[4892]: I1030 20:00:12.471533    4892 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-d52df0-openshift-default-6d5f887b45-985gh" event={"ID":"e453311b-d7b1-4819-a57a-d1de4886de26","Type":"ContainerDied","Data":"cb0272a78a5802f04e51979e7d05a3ba3bf4351cfe7d1f4180da73935a42deb6"}
Oct 30 20:00:12.471714 ip-10-0-132-82 kubenswrapper[4892]: I1030 20:00:12.471619    4892 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-d52df0-openshift-default-6d5f887b45-985gh" event={"ID":"e453311b-d7b1-4819-a57a-d1de4886de26","Type":"ContainerDied","Data":"0c8b344fc0851738484e86eefa65a91e967c0b555e45d5247d8c8346ad6f4af9"}
Oct 30 20:00:12.471714 ip-10-0-132-82 kubenswrapper[4892]: I1030 20:00:12.471650    4892 scope.go:122] "RemoveContainer" containerID="cb0272a78a5802f04e51979e7d05a3ba3bf4351cfe7d1f4180da73935a42deb6"
Oct 30 20:00:12.474552 ip-10-0-132-82 kubenswrapper[4892]: I1030 20:00:12.474451    4892 generic.go:356] "Generic (PLEG): container finished" podID="191593a2-e3a9-4ce4-a1c6-c089723b0237" containerID="1704c766f16e7a726d7dcfdc4866656f49d41dcd941c72047c420a8145a5e9f2" exitCode=0
Oct 30 20:00:12.474552 ip-10-0-132-82 kubenswrapper[4892]: I1030 20:00:12.474480    4892 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w6-5f55cd/premium-w6-5f55cd-kserve-7f785c544b-pcvhb" event={"ID":"191593a2-e3a9-4ce4-a1c6-c089723b0237","Type":"ContainerDied","Data":"1704c766f16e7a726d7dcfdc4866656f49d41dcd941c72047c420a8145a5e9f2"}
Oct 30 20:00:12.482409 ip-10-0-132-82 kubenswrapper[4892]: I1030 20:00:12.481481    4892 generic.go:356] "Generic (PLEG): container finished" podID="3cb47a99-8be1-4756-b31a-43b58cc6b8dc" containerID="7fb9393a625a7900385890a2e7393fa67154902bd3f995959de80d728f5fe573" exitCode=0
Oct 30 20:00:12.482409 ip-10-0-132-82 kubenswrapper[4892]: I1030 20:00:12.481544    4892 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w6-5f55cd/simulator-w6-5f55cd-kserve-5ff86b7447-h9tm7" event={"ID":"3cb47a99-8be1-4756-b31a-43b58cc6b8dc","Type":"ContainerDied","Data":"7fb9393a625a7900385890a2e7393fa67154902bd3f995959de80d728f5fe573"}
Oct 30 20:00:12.489505 ip-10-0-132-82 kubenswrapper[4892]: I1030 20:00:12.489486    4892 scope.go:122] "RemoveContainer" containerID="cb0272a78a5802f04e51979e7d05a3ba3bf4351cfe7d1f4180da73935a42deb6"
Oct 30 20:00:12.489816 ip-10-0-132-82 kubenswrapper[4892]: E1030 20:00:12.489796    4892 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"cb0272a78a5802f04e51979e7d05a3ba3bf4351cfe7d1f4180da73935a42deb6\": container with ID starting with cb0272a78a5802f04e51979e7d05a3ba3bf4351cfe7d1f4180da73935a42deb6 not found: ID does not exist" containerID="cb0272a78a5802f04e51979e7d05a3ba3bf4351cfe7d1f4180da73935a42deb6"
Oct 30 20:00:12.489866 ip-10-0-132-82 kubenswrapper[4892]: I1030 20:00:12.489825    4892 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"cb0272a78a5802f04e51979e7d05a3ba3bf4351cfe7d1f4180da73935a42deb6"} err="failed to get container status \"cb0272a78a5802f04e51979e7d05a3ba3bf4351cfe7d1f4180da73935a42deb6\": rpc error: code = NotFound desc = could not find container \"cb0272a78a5802f04e51979e7d05a3ba3bf4351cfe7d1f4180da73935a42deb6\": container with ID starting with cb0272a78a5802f04e51979e7d05a3ba3bf4351cfe7d1f4180da73935a42deb6 not found: ID does not exist"
Oct 30 20:00:12.634686 ip-10-0-132-82 kubenswrapper[4892]: I1030 20:00:12.634651    4892 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w6-5f55cd/simulator-w6-5f55cd-kserve-5ff86b7447-h9tm7"
Oct 30 20:00:12.642239 ip-10-0-132-82 kubenswrapper[4892]: I1030 20:00:12.642213    4892 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-d52df0-openshift-default-6d5f887b45-985gh"]
Oct 30 20:00:12.648632 ip-10-0-132-82 kubenswrapper[4892]: I1030 20:00:12.648567    4892 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w1-d52df0-openshift-default-6d5f887b45-985gh"]
Oct 30 20:00:12.665712 ip-10-0-132-82 kubenswrapper[4892]: I1030 20:00:12.665683    4892 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w6-5f55cd/premium-w6-5f55cd-kserve-7f785c544b-pcvhb"
Oct 30 20:00:12.802790 ip-10-0-132-82 kubenswrapper[4892]: I1030 20:00:12.802757    4892 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3cb47a99-8be1-4756-b31a-43b58cc6b8dc-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3cb47a99-8be1-4756-b31a-43b58cc6b8dc-tmp-dir\") pod \"3cb47a99-8be1-4756-b31a-43b58cc6b8dc\" (UID: \"3cb47a99-8be1-4756-b31a-43b58cc6b8dc\") "
Oct 30 20:00:12.802941 ip-10-0-132-82 kubenswrapper[4892]: I1030 20:00:12.802807    4892 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/3cb47a99-8be1-4756-b31a-43b58cc6b8dc-tls-certs\" (UniqueName: \"kubernetes.io/secret/3cb47a99-8be1-4756-b31a-43b58cc6b8dc-tls-certs\") pod \"3cb47a99-8be1-4756-b31a-43b58cc6b8dc\" (UID: \"3cb47a99-8be1-4756-b31a-43b58cc6b8dc\") "
Oct 30 20:00:12.802941 ip-10-0-132-82 kubenswrapper[4892]: I1030 20:00:12.802881    4892 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/191593a2-e3a9-4ce4-a1c6-c089723b0237-tls-certs\" (UniqueName: \"kubernetes.io/secret/191593a2-e3a9-4ce4-a1c6-c089723b0237-tls-certs\") pod \"191593a2-e3a9-4ce4-a1c6-c089723b0237\" (UID: \"191593a2-e3a9-4ce4-a1c6-c089723b0237\") "
Oct 30 20:00:12.803034 ip-10-0-132-82 kubenswrapper[4892]: I1030 20:00:12.802941    4892 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/191593a2-e3a9-4ce4-a1c6-c089723b0237-kube-api-access-g5bj7\" (UniqueName: \"kubernetes.io/projected/191593a2-e3a9-4ce4-a1c6-c089723b0237-kube-api-access-g5bj7\") pod \"191593a2-e3a9-4ce4-a1c6-c089723b0237\" (UID: \"191593a2-e3a9-4ce4-a1c6-c089723b0237\") "
Oct 30 20:00:12.803034 ip-10-0-132-82 kubenswrapper[4892]: I1030 20:00:12.803008    4892 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3cb47a99-8be1-4756-b31a-43b58cc6b8dc-kube-api-access-5bw6x\" (UniqueName: \"kubernetes.io/projected/3cb47a99-8be1-4756-b31a-43b58cc6b8dc-kube-api-access-5bw6x\") pod \"3cb47a99-8be1-4756-b31a-43b58cc6b8dc\" (UID: \"3cb47a99-8be1-4756-b31a-43b58cc6b8dc\") "
Oct 30 20:00:12.803117 ip-10-0-132-82 kubenswrapper[4892]: I1030 20:00:12.803050    4892 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/191593a2-e3a9-4ce4-a1c6-c089723b0237-dshm\" (UniqueName: \"kubernetes.io/empty-dir/191593a2-e3a9-4ce4-a1c6-c089723b0237-dshm\") pod \"191593a2-e3a9-4ce4-a1c6-c089723b0237\" (UID: \"191593a2-e3a9-4ce4-a1c6-c089723b0237\") "
Oct 30 20:00:12.803117 ip-10-0-132-82 kubenswrapper[4892]: I1030 20:00:12.803079    4892 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3cb47a99-8be1-4756-b31a-43b58cc6b8dc-dshm\" (UniqueName: \"kubernetes.io/empty-dir/3cb47a99-8be1-4756-b31a-43b58cc6b8dc-dshm\") pod \"3cb47a99-8be1-4756-b31a-43b58cc6b8dc\" (UID: \"3cb47a99-8be1-4756-b31a-43b58cc6b8dc\") "
Oct 30 20:00:12.803117 ip-10-0-132-82 kubenswrapper[4892]: I1030 20:00:12.803088    4892 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3cb47a99-8be1-4756-b31a-43b58cc6b8dc-tmp-dir" pod "3cb47a99-8be1-4756-b31a-43b58cc6b8dc" (UID: "3cb47a99-8be1-4756-b31a-43b58cc6b8dc"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 20:00:12.803221 ip-10-0-132-82 kubenswrapper[4892]: I1030 20:00:12.803166    4892 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3cb47a99-8be1-4756-b31a-43b58cc6b8dc-home\" (UniqueName: \"kubernetes.io/empty-dir/3cb47a99-8be1-4756-b31a-43b58cc6b8dc-home\") pod \"3cb47a99-8be1-4756-b31a-43b58cc6b8dc\" (UID: \"3cb47a99-8be1-4756-b31a-43b58cc6b8dc\") "
Oct 30 20:00:12.803297 ip-10-0-132-82 kubenswrapper[4892]: I1030 20:00:12.803280    4892 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/191593a2-e3a9-4ce4-a1c6-c089723b0237-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/191593a2-e3a9-4ce4-a1c6-c089723b0237-tmp-dir\") pod \"191593a2-e3a9-4ce4-a1c6-c089723b0237\" (UID: \"191593a2-e3a9-4ce4-a1c6-c089723b0237\") "
Oct 30 20:00:12.803425 ip-10-0-132-82 kubenswrapper[4892]: I1030 20:00:12.803345    4892 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/191593a2-e3a9-4ce4-a1c6-c089723b0237-home\" (UniqueName: \"kubernetes.io/empty-dir/191593a2-e3a9-4ce4-a1c6-c089723b0237-home\") pod \"191593a2-e3a9-4ce4-a1c6-c089723b0237\" (UID: \"191593a2-e3a9-4ce4-a1c6-c089723b0237\") "
Oct 30 20:00:12.803712 ip-10-0-132-82 kubenswrapper[4892]: I1030 20:00:12.803689    4892 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3cb47a99-8be1-4756-b31a-43b58cc6b8dc-home" pod "3cb47a99-8be1-4756-b31a-43b58cc6b8dc" (UID: "3cb47a99-8be1-4756-b31a-43b58cc6b8dc"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 20:00:12.803847 ip-10-0-132-82 kubenswrapper[4892]: I1030 20:00:12.803759    4892 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/191593a2-e3a9-4ce4-a1c6-c089723b0237-home" pod "191593a2-e3a9-4ce4-a1c6-c089723b0237" (UID: "191593a2-e3a9-4ce4-a1c6-c089723b0237"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 20:00:12.803847 ip-10-0-132-82 kubenswrapper[4892]: I1030 20:00:12.803785    4892 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/191593a2-e3a9-4ce4-a1c6-c089723b0237-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/191593a2-e3a9-4ce4-a1c6-c089723b0237-model-cache\") pod \"191593a2-e3a9-4ce4-a1c6-c089723b0237\" (UID: \"191593a2-e3a9-4ce4-a1c6-c089723b0237\") "
Oct 30 20:00:12.803847 ip-10-0-132-82 kubenswrapper[4892]: I1030 20:00:12.803832    4892 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3cb47a99-8be1-4756-b31a-43b58cc6b8dc-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/3cb47a99-8be1-4756-b31a-43b58cc6b8dc-model-cache\") pod \"3cb47a99-8be1-4756-b31a-43b58cc6b8dc\" (UID: \"3cb47a99-8be1-4756-b31a-43b58cc6b8dc\") "
Oct 30 20:00:12.804071 ip-10-0-132-82 kubenswrapper[4892]: I1030 20:00:12.804003    4892 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/191593a2-e3a9-4ce4-a1c6-c089723b0237-tmp-dir" pod "191593a2-e3a9-4ce4-a1c6-c089723b0237" (UID: "191593a2-e3a9-4ce4-a1c6-c089723b0237"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 20:00:12.804212 ip-10-0-132-82 kubenswrapper[4892]: I1030 20:00:12.804186    4892 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/191593a2-e3a9-4ce4-a1c6-c089723b0237-model-cache" pod "191593a2-e3a9-4ce4-a1c6-c089723b0237" (UID: "191593a2-e3a9-4ce4-a1c6-c089723b0237"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 20:00:12.804492 ip-10-0-132-82 kubenswrapper[4892]: I1030 20:00:12.804412    4892 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3cb47a99-8be1-4756-b31a-43b58cc6b8dc-model-cache" pod "3cb47a99-8be1-4756-b31a-43b58cc6b8dc" (UID: "3cb47a99-8be1-4756-b31a-43b58cc6b8dc"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 20:00:12.804604 ip-10-0-132-82 kubenswrapper[4892]: I1030 20:00:12.804502    4892 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/3cb47a99-8be1-4756-b31a-43b58cc6b8dc-home\") on node \"ip-10-0-132-82.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:00:12.804604 ip-10-0-132-82 kubenswrapper[4892]: I1030 20:00:12.804523    4892 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/191593a2-e3a9-4ce4-a1c6-c089723b0237-tmp-dir\") on node \"ip-10-0-132-82.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:00:12.804604 ip-10-0-132-82 kubenswrapper[4892]: I1030 20:00:12.804536    4892 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/191593a2-e3a9-4ce4-a1c6-c089723b0237-home\") on node \"ip-10-0-132-82.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:00:12.804604 ip-10-0-132-82 kubenswrapper[4892]: I1030 20:00:12.804549    4892 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/191593a2-e3a9-4ce4-a1c6-c089723b0237-model-cache\") on node \"ip-10-0-132-82.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:00:12.804604 ip-10-0-132-82 kubenswrapper[4892]: I1030 20:00:12.804564    4892 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3cb47a99-8be1-4756-b31a-43b58cc6b8dc-tmp-dir\") on node \"ip-10-0-132-82.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:00:12.810628 ip-10-0-132-82 kubenswrapper[4892]: I1030 20:00:12.810601    4892 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/191593a2-e3a9-4ce4-a1c6-c089723b0237-kube-api-access-g5bj7" pod "191593a2-e3a9-4ce4-a1c6-c089723b0237" (UID: "191593a2-e3a9-4ce4-a1c6-c089723b0237"). InnerVolumeSpecName "kube-api-access-g5bj7". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 20:00:12.810747 ip-10-0-132-82 kubenswrapper[4892]: I1030 20:00:12.810685    4892 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/191593a2-e3a9-4ce4-a1c6-c089723b0237-dshm" pod "191593a2-e3a9-4ce4-a1c6-c089723b0237" (UID: "191593a2-e3a9-4ce4-a1c6-c089723b0237"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 20:00:12.810912 ip-10-0-132-82 kubenswrapper[4892]: I1030 20:00:12.810881    4892 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/191593a2-e3a9-4ce4-a1c6-c089723b0237-tls-certs" pod "191593a2-e3a9-4ce4-a1c6-c089723b0237" (UID: "191593a2-e3a9-4ce4-a1c6-c089723b0237"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 20:00:12.811201 ip-10-0-132-82 kubenswrapper[4892]: I1030 20:00:12.811168    4892 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/3cb47a99-8be1-4756-b31a-43b58cc6b8dc-tls-certs" pod "3cb47a99-8be1-4756-b31a-43b58cc6b8dc" (UID: "3cb47a99-8be1-4756-b31a-43b58cc6b8dc"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 20:00:12.811201 ip-10-0-132-82 kubenswrapper[4892]: I1030 20:00:12.811177    4892 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3cb47a99-8be1-4756-b31a-43b58cc6b8dc-dshm" pod "3cb47a99-8be1-4756-b31a-43b58cc6b8dc" (UID: "3cb47a99-8be1-4756-b31a-43b58cc6b8dc"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 20:00:12.813043 ip-10-0-132-82 kubenswrapper[4892]: I1030 20:00:12.812950    4892 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3cb47a99-8be1-4756-b31a-43b58cc6b8dc-kube-api-access-5bw6x" pod "3cb47a99-8be1-4756-b31a-43b58cc6b8dc" (UID: "3cb47a99-8be1-4756-b31a-43b58cc6b8dc"). InnerVolumeSpecName "kube-api-access-5bw6x". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 20:00:12.906209 ip-10-0-132-82 kubenswrapper[4892]: I1030 20:00:12.906022    4892 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/3cb47a99-8be1-4756-b31a-43b58cc6b8dc-tls-certs\") on node \"ip-10-0-132-82.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:00:12.906209 ip-10-0-132-82 kubenswrapper[4892]: I1030 20:00:12.906061    4892 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/191593a2-e3a9-4ce4-a1c6-c089723b0237-tls-certs\") on node \"ip-10-0-132-82.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:00:12.906209 ip-10-0-132-82 kubenswrapper[4892]: I1030 20:00:12.906077    4892 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-g5bj7\" (UniqueName: \"kubernetes.io/projected/191593a2-e3a9-4ce4-a1c6-c089723b0237-kube-api-access-g5bj7\") on node \"ip-10-0-132-82.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:00:12.906209 ip-10-0-132-82 kubenswrapper[4892]: I1030 20:00:12.906094    4892 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5bw6x\" (UniqueName: \"kubernetes.io/projected/3cb47a99-8be1-4756-b31a-43b58cc6b8dc-kube-api-access-5bw6x\") on node \"ip-10-0-132-82.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:00:12.906209 ip-10-0-132-82 kubenswrapper[4892]: I1030 20:00:12.906108    4892 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/191593a2-e3a9-4ce4-a1c6-c089723b0237-dshm\") on node \"ip-10-0-132-82.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:00:12.906209 ip-10-0-132-82 kubenswrapper[4892]: I1030 20:00:12.906121    4892 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/3cb47a99-8be1-4756-b31a-43b58cc6b8dc-dshm\") on node \"ip-10-0-132-82.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:00:12.906209 ip-10-0-132-82 kubenswrapper[4892]: I1030 20:00:12.906132    4892 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/3cb47a99-8be1-4756-b31a-43b58cc6b8dc-model-cache\") on node \"ip-10-0-132-82.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:00:13.492827 ip-10-0-132-82 kubenswrapper[4892]: I1030 20:00:13.492797    4892 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w6-5f55cd/premium-w6-5f55cd-kserve-7f785c544b-pcvhb"
Oct 30 20:00:13.493401 ip-10-0-132-82 kubenswrapper[4892]: I1030 20:00:13.492799    4892 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w6-5f55cd/premium-w6-5f55cd-kserve-7f785c544b-pcvhb" event={"ID":"191593a2-e3a9-4ce4-a1c6-c089723b0237","Type":"ContainerDied","Data":"5a6384cc7c027b634daa052bc2016391b33e529904c98c29118b9d39683ea050"}
Oct 30 20:00:13.493401 ip-10-0-132-82 kubenswrapper[4892]: I1030 20:00:13.492889    4892 scope.go:122] "RemoveContainer" containerID="1704c766f16e7a726d7dcfdc4866656f49d41dcd941c72047c420a8145a5e9f2"
Oct 30 20:00:13.495402 ip-10-0-132-82 kubenswrapper[4892]: I1030 20:00:13.495378    4892 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w6-5f55cd/simulator-w6-5f55cd-kserve-5ff86b7447-h9tm7"
Oct 30 20:00:13.495537 ip-10-0-132-82 kubenswrapper[4892]: I1030 20:00:13.495379    4892 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w6-5f55cd/simulator-w6-5f55cd-kserve-5ff86b7447-h9tm7" event={"ID":"3cb47a99-8be1-4756-b31a-43b58cc6b8dc","Type":"ContainerDied","Data":"2e98566bd9e10fbf0e8b578d31f14ac5c2f80e0d7ca7f2f1ea90d2804cd2503c"}
Oct 30 20:00:13.506952 ip-10-0-132-82 kubenswrapper[4892]: I1030 20:00:13.506937    4892 scope.go:122] "RemoveContainer" containerID="7fb9393a625a7900385890a2e7393fa67154902bd3f995959de80d728f5fe573"
Oct 30 20:00:13.658266 ip-10-0-132-82 kubenswrapper[4892]: I1030 20:00:13.658235    4892 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w6-5f55cd/premium-w6-5f55cd-kserve-7f785c544b-pcvhb"]
Oct 30 20:00:13.664847 ip-10-0-132-82 kubenswrapper[4892]: I1030 20:00:13.664824    4892 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w6-5f55cd/premium-w6-5f55cd-kserve-7f785c544b-pcvhb"]
Oct 30 20:00:13.772557 ip-10-0-132-82 kubenswrapper[4892]: I1030 20:00:13.772481    4892 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w6-5f55cd/simulator-w6-5f55cd-kserve-5ff86b7447-h9tm7"]
Oct 30 20:00:13.776170 ip-10-0-132-82 kubenswrapper[4892]: I1030 20:00:13.776148    4892 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w6-5f55cd/simulator-w6-5f55cd-kserve-5ff86b7447-h9tm7"]
Oct 30 20:00:14.294708 ip-10-0-132-82 kubenswrapper[4892]: I1030 20:00:14.294672    4892 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w0-6ac304-846999954d-tq2ft"]
Oct 30 20:00:14.400308 ip-10-0-132-82 kubenswrapper[4892]: I1030 20:00:14.400276    4892 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w0-6ac304-79859bfff5-zx2hb"]
Oct 30 20:00:14.569740 ip-10-0-132-82 kubenswrapper[4892]: I1030 20:00:14.569658    4892 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="191593a2-e3a9-4ce4-a1c6-c089723b0237" path="/var/lib/kubelet/pods/191593a2-e3a9-4ce4-a1c6-c089723b0237/volumes"
Oct 30 20:00:14.570267 ip-10-0-132-82 kubenswrapper[4892]: I1030 20:00:14.570245    4892 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="3cb47a99-8be1-4756-b31a-43b58cc6b8dc" path="/var/lib/kubelet/pods/3cb47a99-8be1-4756-b31a-43b58cc6b8dc/volumes"
Oct 30 20:00:14.570727 ip-10-0-132-82 kubenswrapper[4892]: I1030 20:00:14.570706    4892 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e453311b-d7b1-4819-a57a-d1de4886de26" path="/var/lib/kubelet/pods/e453311b-d7b1-4819-a57a-d1de4886de26/volumes"
Oct 30 20:00:15.562203 ip-10-0-132-82 kubenswrapper[4892]: I1030 20:00:15.562174    4892 scope.go:122] "RemoveContainer" containerID="eb45c7d92510d4cc281510421063331b6b57544c377370befcd0bb5e38984272"
Oct 30 20:00:15.562510 ip-10-0-132-82 kubenswrapper[4892]: I1030 20:00:15.562337    4892 scope.go:122] "RemoveContainer" containerID="6504efa1b5cb74f3f5fd3a151f2d401baa624be2926dc67a9b9b5ee14a7b5667"
Oct 30 20:00:16.525987 ip-10-0-132-82 kubenswrapper[4892]: I1030 20:00:16.525927    4892 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-b-1c5537-29889840-6qrmj" event={"ID":"f6260396-890c-44ea-be8c-998dd2086ffa","Type":"ContainerStarted","Data":"0ab8f258fe6c982e2bb75984394f145c1f5bc4bda4cd533af11f27b88609ede8"}
Oct 30 20:00:16.528241 ip-10-0-132-82 kubenswrapper[4892]: I1030 20:00:16.528216    4892 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-6ac304-29889840-fhd4j" event={"ID":"25d4aedf-735a-42e6-ae08-4561c967fc35","Type":"ContainerStarted","Data":"bed3d188e35539b21550fe9cd503df49975e446ac3baeb2774660ea14ba9bb0b"}
Oct 30 20:00:16.563033 ip-10-0-132-82 kubenswrapper[4892]: I1030 20:00:16.563012    4892 scope.go:122] "RemoveContainer" containerID="b25696256c49c627f70a0c87690749c4b46e7c19b9e955dcfd8b5e66b117685d"
Oct 30 20:00:17.271335 ip-10-0-132-82 kubenswrapper[4892]: I1030 20:00:17.271302    4892 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-worker-w6-5f55cd-85cb4d6b67-9qrfs" podUID="4286dc2a-9c31-454f-891d-31d896d7a66b" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 30 20:00:17.271517 ip-10-0-132-82 kubenswrapper[4892]: I1030 20:00:17.271302    4892 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-worker-w6-5f55cd-688df7c466-t26tq" podUID="b31aad4b-7f5c-46c1-9c46-e39ea2ea5803" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 30 20:00:17.541344 ip-10-0-132-82 kubenswrapper[4892]: I1030 20:00:17.541308    4892 generic.go:356] "Generic (PLEG): container finished" podID="25d4aedf-735a-42e6-ae08-4561c967fc35" containerID="bed3d188e35539b21550fe9cd503df49975e446ac3baeb2774660ea14ba9bb0b" exitCode=60
Oct 30 20:00:17.541828 ip-10-0-132-82 kubenswrapper[4892]: I1030 20:00:17.541372    4892 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-6ac304-29889840-fhd4j" event={"ID":"25d4aedf-735a-42e6-ae08-4561c967fc35","Type":"ContainerDied","Data":"bed3d188e35539b21550fe9cd503df49975e446ac3baeb2774660ea14ba9bb0b"}
Oct 30 20:00:17.541828 ip-10-0-132-82 kubenswrapper[4892]: I1030 20:00:17.541437    4892 scope.go:122] "RemoveContainer" containerID="6504efa1b5cb74f3f5fd3a151f2d401baa624be2926dc67a9b9b5ee14a7b5667"
Oct 30 20:00:17.541828 ip-10-0-132-82 kubenswrapper[4892]: I1030 20:00:17.541818    4892 scope.go:122] "RemoveContainer" containerID="bed3d188e35539b21550fe9cd503df49975e446ac3baeb2774660ea14ba9bb0b"
Oct 30 20:00:17.542253 ip-10-0-132-82 kubenswrapper[4892]: E1030 20:00:17.542154    4892 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-b-w0-6ac304-29889840-fhd4j_odh-ai-gateway-infra(25d4aedf-735a-42e6-ae08-4561c967fc35)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-6ac304-29889840-fhd4j" podUID="25d4aedf-735a-42e6-ae08-4561c967fc35"
Oct 30 20:00:17.544558 ip-10-0-132-82 kubenswrapper[4892]: I1030 20:00:17.544533    4892 generic.go:356] "Generic (PLEG): container finished" podID="f6260396-890c-44ea-be8c-998dd2086ffa" containerID="0ab8f258fe6c982e2bb75984394f145c1f5bc4bda4cd533af11f27b88609ede8" exitCode=60
Oct 30 20:00:17.544683 ip-10-0-132-82 kubenswrapper[4892]: I1030 20:00:17.544566    4892 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-b-1c5537-29889840-6qrmj" event={"ID":"f6260396-890c-44ea-be8c-998dd2086ffa","Type":"ContainerDied","Data":"0ab8f258fe6c982e2bb75984394f145c1f5bc4bda4cd533af11f27b88609ede8"}
Oct 30 20:00:17.545013 ip-10-0-132-82 kubenswrapper[4892]: I1030 20:00:17.544990    4892 scope.go:122] "RemoveContainer" containerID="0ab8f258fe6c982e2bb75984394f145c1f5bc4bda4cd533af11f27b88609ede8"
Oct 30 20:00:17.545334 ip-10-0-132-82 kubenswrapper[4892]: E1030 20:00:17.545310    4892 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-api-b-1c5537-29889840-6qrmj_odh-ai-gateway-infra(f6260396-890c-44ea-be8c-998dd2086ffa)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-b-1c5537-29889840-6qrmj" podUID="f6260396-890c-44ea-be8c-998dd2086ffa"
Oct 30 20:00:17.547045 ip-10-0-132-82 kubenswrapper[4892]: I1030 20:00:17.547023    4892 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-cf9b6f-29889840-8k6mb" event={"ID":"9df510ae-c995-4678-816a-9c4834778991","Type":"ContainerStarted","Data":"7fd36bfe6dd50b93eb0758b8b0bc1bfacbd4ddcdaabec4261f990dff59972a2f"}
Oct 30 20:00:17.547406 ip-10-0-132-82 kubenswrapper[4892]: I1030 20:00:17.547390    4892 scope.go:122] "RemoveContainer" containerID="7fd36bfe6dd50b93eb0758b8b0bc1bfacbd4ddcdaabec4261f990dff59972a2f"
Oct 30 20:00:17.547681 ip-10-0-132-82 kubenswrapper[4892]: E1030 20:00:17.547660    4892 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-a-w3-cf9b6f-29889840-8k6mb_odh-ai-gateway-infra(9df510ae-c995-4678-816a-9c4834778991)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-cf9b6f-29889840-8k6mb" podUID="9df510ae-c995-4678-816a-9c4834778991"
Oct 30 20:00:17.562916 ip-10-0-132-82 kubenswrapper[4892]: I1030 20:00:17.562894    4892 scope.go:122] "RemoveContainer" containerID="8f2e18fda9b1dfff0c3a988200a5ffc3cd947fbfb5d96afb08d42195e1fbb32f"
Oct 30 20:00:17.563054 ip-10-0-132-82 kubenswrapper[4892]: I1030 20:00:17.562987    4892 scope.go:122] "RemoveContainer" containerID="5a03d21d9c4839f536751bb171a61b0ba0ee1f8a963d6f3120838a3ce21a623e"
Oct 30 20:00:17.563341 ip-10-0-132-82 kubenswrapper[4892]: I1030 20:00:17.563320    4892 scope.go:122] "RemoveContainer" containerID="9543375f0cc7e181b0fcdd4110a5c18a0ed681347cded35b833f8cc1154d6bbc"
Oct 30 20:00:17.563984 ip-10-0-132-82 kubenswrapper[4892]: I1030 20:00:17.563508    4892 scope.go:122] "RemoveContainer" containerID="407af9f51a2672047a92dc943183d990891432c0b35f9bb3457bdeae6421d72b"
Oct 30 20:00:17.563984 ip-10-0-132-82 kubenswrapper[4892]: I1030 20:00:17.563647    4892 scope.go:122] "RemoveContainer" containerID="8e3128732b35cf64c18addaa4a4bf68fa805cddf2e389136c5c96b68a9d82cf1"
Oct 30 20:00:17.609043 ip-10-0-132-82 kubenswrapper[4892]: I1030 20:00:17.608951    4892 scope.go:122] "RemoveContainer" containerID="eb45c7d92510d4cc281510421063331b6b57544c377370befcd0bb5e38984272"
Oct 30 20:00:17.668603 ip-10-0-132-82 kubenswrapper[4892]: I1030 20:00:17.668581    4892 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-b-1c5537-29889840-6qrmj"]
Oct 30 20:00:17.784939 ip-10-0-132-82 kubenswrapper[4892]: I1030 20:00:17.784850    4892 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-6ac304-29889840-fhd4j"]
Oct 30 20:00:18.579127 ip-10-0-132-82 kubenswrapper[4892]: I1030 20:00:18.579095    4892 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-686689-29889840-l2mjk" event={"ID":"0b0b9f79-e0a9-41a1-b513-01ea5661f64d","Type":"ContainerStarted","Data":"dede5f77a2988363346a4b2ddecca528d3d145df64acf7739027053c46c78001"}
Oct 30 20:00:18.579634 ip-10-0-132-82 kubenswrapper[4892]: I1030 20:00:18.579609    4892 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-2d9861-29889840-79j62" event={"ID":"82b75f10-16db-40c1-9696-f1e7f9917419","Type":"ContainerStarted","Data":"d153ca52e4bef4921aecf64bf745d0709d0ecc3d11bad89701c1626745aff774"}
Oct 30 20:00:18.579788 ip-10-0-132-82 kubenswrapper[4892]: I1030 20:00:18.579772    4892 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-132119-29889840-dhpf8" event={"ID":"d16a477c-ad96-4363-b13d-b6d374f3f88d","Type":"ContainerStarted","Data":"e8dbb29bc0bd55a5da1e44d40a0795a67ee54ba33779371f263abe5ddd529ad2"}
Oct 30 20:00:18.579908 ip-10-0-132-82 kubenswrapper[4892]: I1030 20:00:18.579894    4892 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-ada8d7-29889840-xldh2" event={"ID":"35018750-1063-4fb6-928f-ce6bcd401f87","Type":"ContainerStarted","Data":"2c871a60e44b312e5946a37677af97cb215c9bab0c88ad4e332c408a13a5068c"}
Oct 30 20:00:18.580052 ip-10-0-132-82 kubenswrapper[4892]: I1030 20:00:18.580037    4892 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-cf9b6f-29889840-8k6mb" event={"ID":"9df510ae-c995-4678-816a-9c4834778991","Type":"ContainerDied","Data":"7fd36bfe6dd50b93eb0758b8b0bc1bfacbd4ddcdaabec4261f990dff59972a2f"}
Oct 30 20:00:18.580172 ip-10-0-132-82 kubenswrapper[4892]: I1030 20:00:18.580159    4892 scope.go:122] "RemoveContainer" containerID="b25696256c49c627f70a0c87690749c4b46e7c19b9e955dcfd8b5e66b117685d"
Oct 30 20:00:18.580359 ip-10-0-132-82 kubenswrapper[4892]: I1030 20:00:18.579213    4892 generic.go:356] "Generic (PLEG): container finished" podID="9df510ae-c995-4678-816a-9c4834778991" containerID="7fd36bfe6dd50b93eb0758b8b0bc1bfacbd4ddcdaabec4261f990dff59972a2f" exitCode=60
Oct 30 20:00:18.580903 ip-10-0-132-82 kubenswrapper[4892]: I1030 20:00:18.580882    4892 scope.go:122] "RemoveContainer" containerID="7fd36bfe6dd50b93eb0758b8b0bc1bfacbd4ddcdaabec4261f990dff59972a2f"
Oct 30 20:00:18.581169 ip-10-0-132-82 kubenswrapper[4892]: E1030 20:00:18.581132    4892 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-a-w3-cf9b6f-29889840-8k6mb_odh-ai-gateway-infra(9df510ae-c995-4678-816a-9c4834778991)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-cf9b6f-29889840-8k6mb" podUID="9df510ae-c995-4678-816a-9c4834778991"
Oct 30 20:00:18.582603 ip-10-0-132-82 kubenswrapper[4892]: I1030 20:00:18.582576    4892 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-c75ed4-29889840-7s578" event={"ID":"8152ed9d-b608-4a87-aded-8dda2d975174","Type":"ContainerStarted","Data":"2aec635d950d0ef0bf65c6c0436980ebbbf91d78ec13ed07e1007cb1c61cb2da"}
Oct 30 20:00:18.898392 ip-10-0-132-82 kubenswrapper[4892]: I1030 20:00:18.898360    4892 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-cf9b6f-29889840-8k6mb"]
Oct 30 20:00:18.964174 ip-10-0-132-82 kubenswrapper[4892]: I1030 20:00:18.964149    4892 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-6ac304-29889840-fhd4j"
Oct 30 20:00:18.968158 ip-10-0-132-82 kubenswrapper[4892]: I1030 20:00:18.968140    4892 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-b-1c5537-29889840-6qrmj"
Oct 30 20:00:19.102244 ip-10-0-132-82 kubenswrapper[4892]: I1030 20:00:19.102171    4892 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/25d4aedf-735a-42e6-ae08-4561c967fc35-kube-api-access-cbqqn\" (UniqueName: \"kubernetes.io/projected/25d4aedf-735a-42e6-ae08-4561c967fc35-kube-api-access-cbqqn\") pod \"25d4aedf-735a-42e6-ae08-4561c967fc35\" (UID: \"25d4aedf-735a-42e6-ae08-4561c967fc35\") "
Oct 30 20:00:19.102408 ip-10-0-132-82 kubenswrapper[4892]: I1030 20:00:19.102253    4892 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f6260396-890c-44ea-be8c-998dd2086ffa-kube-api-access-4rthv\" (UniqueName: \"kubernetes.io/projected/f6260396-890c-44ea-be8c-998dd2086ffa-kube-api-access-4rthv\") pod \"f6260396-890c-44ea-be8c-998dd2086ffa\" (UID: \"f6260396-890c-44ea-be8c-998dd2086ffa\") "
Oct 30 20:00:19.102408 ip-10-0-132-82 kubenswrapper[4892]: I1030 20:00:19.102274    4892 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f6260396-890c-44ea-be8c-998dd2086ffa-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/f6260396-890c-44ea-be8c-998dd2086ffa-maas-api-service-ca\") pod \"f6260396-890c-44ea-be8c-998dd2086ffa\" (UID: \"f6260396-890c-44ea-be8c-998dd2086ffa\") "
Oct 30 20:00:19.102408 ip-10-0-132-82 kubenswrapper[4892]: I1030 20:00:19.102329    4892 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/25d4aedf-735a-42e6-ae08-4561c967fc35-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/25d4aedf-735a-42e6-ae08-4561c967fc35-maas-api-service-ca\") pod \"25d4aedf-735a-42e6-ae08-4561c967fc35\" (UID: \"25d4aedf-735a-42e6-ae08-4561c967fc35\") "
Oct 30 20:00:19.102685 ip-10-0-132-82 kubenswrapper[4892]: I1030 20:00:19.102663    4892 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f6260396-890c-44ea-be8c-998dd2086ffa-maas-api-service-ca" pod "f6260396-890c-44ea-be8c-998dd2086ffa" (UID: "f6260396-890c-44ea-be8c-998dd2086ffa"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 20:00:19.102771 ip-10-0-132-82 kubenswrapper[4892]: I1030 20:00:19.102696    4892 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/25d4aedf-735a-42e6-ae08-4561c967fc35-maas-api-service-ca" pod "25d4aedf-735a-42e6-ae08-4561c967fc35" (UID: "25d4aedf-735a-42e6-ae08-4561c967fc35"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 20:00:19.104462 ip-10-0-132-82 kubenswrapper[4892]: I1030 20:00:19.104446    4892 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/25d4aedf-735a-42e6-ae08-4561c967fc35-kube-api-access-cbqqn" pod "25d4aedf-735a-42e6-ae08-4561c967fc35" (UID: "25d4aedf-735a-42e6-ae08-4561c967fc35"). InnerVolumeSpecName "kube-api-access-cbqqn". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 20:00:19.104539 ip-10-0-132-82 kubenswrapper[4892]: I1030 20:00:19.104520    4892 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f6260396-890c-44ea-be8c-998dd2086ffa-kube-api-access-4rthv" pod "f6260396-890c-44ea-be8c-998dd2086ffa" (UID: "f6260396-890c-44ea-be8c-998dd2086ffa"). InnerVolumeSpecName "kube-api-access-4rthv". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 20:00:19.203395 ip-10-0-132-82 kubenswrapper[4892]: I1030 20:00:19.203370    4892 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/f6260396-890c-44ea-be8c-998dd2086ffa-maas-api-service-ca\") on node \"ip-10-0-132-82.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:00:19.203395 ip-10-0-132-82 kubenswrapper[4892]: I1030 20:00:19.203394    4892 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/25d4aedf-735a-42e6-ae08-4561c967fc35-maas-api-service-ca\") on node \"ip-10-0-132-82.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:00:19.203566 ip-10-0-132-82 kubenswrapper[4892]: I1030 20:00:19.203404    4892 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-cbqqn\" (UniqueName: \"kubernetes.io/projected/25d4aedf-735a-42e6-ae08-4561c967fc35-kube-api-access-cbqqn\") on node \"ip-10-0-132-82.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:00:19.203566 ip-10-0-132-82 kubenswrapper[4892]: I1030 20:00:19.203414    4892 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4rthv\" (UniqueName: \"kubernetes.io/projected/f6260396-890c-44ea-be8c-998dd2086ffa-kube-api-access-4rthv\") on node \"ip-10-0-132-82.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:00:19.295375 ip-10-0-132-82 kubenswrapper[4892]: I1030 20:00:19.295339    4892 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-shared-b-w0-6ac304-846999954d-tq2ft" podUID="12863020-d0ea-419d-aba2-816627ab0431" containerName="payload-processing" containerID="cri-o://52a6070c0003df14be74d3b52ca40a184ef6a0d76edc4de151480c1d25789884" gracePeriod=25
Oct 30 20:00:19.401032 ip-10-0-132-82 kubenswrapper[4892]: I1030 20:00:19.400999    4892 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w0-6ac304-79859bfff5-zx2hb" podUID="f1d57e1b-e316-48ac-a1a5-a9520a9e068e" containerName="payload-pre-processing" containerID="cri-o://45e7e4c2c487385d9b7c19a9b2ffafa7dea4d76bc721247e55757afa40a29171" gracePeriod=25
Oct 30 20:00:19.563782 ip-10-0-132-82 kubenswrapper[4892]: I1030 20:00:19.563743    4892 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w0-6ac304-846999954d-tq2ft"
Oct 30 20:00:19.601502 ip-10-0-132-82 kubenswrapper[4892]: I1030 20:00:19.601478    4892 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-b-1c5537-29889840-6qrmj"
Oct 30 20:00:19.601937 ip-10-0-132-82 kubenswrapper[4892]: I1030 20:00:19.601490    4892 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-b-1c5537-29889840-6qrmj" event={"ID":"f6260396-890c-44ea-be8c-998dd2086ffa","Type":"ContainerDied","Data":"96cb9bd84a95a3ae8ab368b8322ccc9f93f7c90e67681d04efb614527d89ec2c"}
Oct 30 20:00:19.601937 ip-10-0-132-82 kubenswrapper[4892]: I1030 20:00:19.601568    4892 scope.go:122] "RemoveContainer" containerID="0ab8f258fe6c982e2bb75984394f145c1f5bc4bda4cd533af11f27b88609ede8"
Oct 30 20:00:19.609680 ip-10-0-132-82 kubenswrapper[4892]: I1030 20:00:19.609651    4892 generic.go:356] "Generic (PLEG): container finished" podID="8152ed9d-b608-4a87-aded-8dda2d975174" containerID="2aec635d950d0ef0bf65c6c0436980ebbbf91d78ec13ed07e1007cb1c61cb2da" exitCode=60
Oct 30 20:00:19.609805 ip-10-0-132-82 kubenswrapper[4892]: I1030 20:00:19.609786    4892 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-c75ed4-29889840-7s578" event={"ID":"8152ed9d-b608-4a87-aded-8dda2d975174","Type":"ContainerDied","Data":"2aec635d950d0ef0bf65c6c0436980ebbbf91d78ec13ed07e1007cb1c61cb2da"}
Oct 30 20:00:19.610218 ip-10-0-132-82 kubenswrapper[4892]: I1030 20:00:19.610197    4892 scope.go:122] "RemoveContainer" containerID="2aec635d950d0ef0bf65c6c0436980ebbbf91d78ec13ed07e1007cb1c61cb2da"
Oct 30 20:00:19.610925 ip-10-0-132-82 kubenswrapper[4892]: E1030 20:00:19.610546    4892 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-b-w3-c75ed4-29889840-7s578_odh-ai-gateway-infra(8152ed9d-b608-4a87-aded-8dda2d975174)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-c75ed4-29889840-7s578" podUID="8152ed9d-b608-4a87-aded-8dda2d975174"
Oct 30 20:00:19.617064 ip-10-0-132-82 kubenswrapper[4892]: I1030 20:00:19.616263    4892 generic.go:356] "Generic (PLEG): container finished" podID="0b0b9f79-e0a9-41a1-b513-01ea5661f64d" containerID="dede5f77a2988363346a4b2ddecca528d3d145df64acf7739027053c46c78001" exitCode=60
Oct 30 20:00:19.617064 ip-10-0-132-82 kubenswrapper[4892]: I1030 20:00:19.616356    4892 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-686689-29889840-l2mjk" event={"ID":"0b0b9f79-e0a9-41a1-b513-01ea5661f64d","Type":"ContainerDied","Data":"dede5f77a2988363346a4b2ddecca528d3d145df64acf7739027053c46c78001"}
Oct 30 20:00:19.617064 ip-10-0-132-82 kubenswrapper[4892]: I1030 20:00:19.616685    4892 scope.go:122] "RemoveContainer" containerID="dede5f77a2988363346a4b2ddecca528d3d145df64acf7739027053c46c78001"
Oct 30 20:00:19.617064 ip-10-0-132-82 kubenswrapper[4892]: E1030 20:00:19.616995    4892 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w5-686689-29889840-l2mjk_odh-ai-gateway-infra(0b0b9f79-e0a9-41a1-b513-01ea5661f64d)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-686689-29889840-l2mjk" podUID="0b0b9f79-e0a9-41a1-b513-01ea5661f64d"
Oct 30 20:00:19.618017 ip-10-0-132-82 kubenswrapper[4892]: I1030 20:00:19.617920    4892 scope.go:122] "RemoveContainer" containerID="8f2e18fda9b1dfff0c3a988200a5ffc3cd947fbfb5d96afb08d42195e1fbb32f"
Oct 30 20:00:19.622669 ip-10-0-132-82 kubenswrapper[4892]: I1030 20:00:19.622071    4892 generic.go:356] "Generic (PLEG): container finished" podID="82b75f10-16db-40c1-9696-f1e7f9917419" containerID="d153ca52e4bef4921aecf64bf745d0709d0ecc3d11bad89701c1626745aff774" exitCode=60
Oct 30 20:00:19.622669 ip-10-0-132-82 kubenswrapper[4892]: I1030 20:00:19.622158    4892 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-2d9861-29889840-79j62" event={"ID":"82b75f10-16db-40c1-9696-f1e7f9917419","Type":"ContainerDied","Data":"d153ca52e4bef4921aecf64bf745d0709d0ecc3d11bad89701c1626745aff774"}
Oct 30 20:00:19.622669 ip-10-0-132-82 kubenswrapper[4892]: I1030 20:00:19.622474    4892 scope.go:122] "RemoveContainer" containerID="d153ca52e4bef4921aecf64bf745d0709d0ecc3d11bad89701c1626745aff774"
Oct 30 20:00:19.623443 ip-10-0-132-82 kubenswrapper[4892]: E1030 20:00:19.623384    4892 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w0-2d9861-29889840-79j62_odh-ai-gateway-infra(82b75f10-16db-40c1-9696-f1e7f9917419)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-2d9861-29889840-79j62" podUID="82b75f10-16db-40c1-9696-f1e7f9917419"
Oct 30 20:00:19.625023 ip-10-0-132-82 kubenswrapper[4892]: I1030 20:00:19.624997    4892 generic.go:356] "Generic (PLEG): container finished" podID="f1d57e1b-e316-48ac-a1a5-a9520a9e068e" containerID="45e7e4c2c487385d9b7c19a9b2ffafa7dea4d76bc721247e55757afa40a29171" exitCode=0
Oct 30 20:00:19.625136 ip-10-0-132-82 kubenswrapper[4892]: I1030 20:00:19.625093    4892 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w0-6ac304-79859bfff5-zx2hb" event={"ID":"f1d57e1b-e316-48ac-a1a5-a9520a9e068e","Type":"ContainerDied","Data":"45e7e4c2c487385d9b7c19a9b2ffafa7dea4d76bc721247e55757afa40a29171"}
Oct 30 20:00:19.627798 ip-10-0-132-82 kubenswrapper[4892]: I1030 20:00:19.627398    4892 generic.go:356] "Generic (PLEG): container finished" podID="12863020-d0ea-419d-aba2-816627ab0431" containerID="52a6070c0003df14be74d3b52ca40a184ef6a0d76edc4de151480c1d25789884" exitCode=0
Oct 30 20:00:19.627798 ip-10-0-132-82 kubenswrapper[4892]: I1030 20:00:19.627519    4892 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w0-6ac304-846999954d-tq2ft" event={"ID":"12863020-d0ea-419d-aba2-816627ab0431","Type":"ContainerDied","Data":"52a6070c0003df14be74d3b52ca40a184ef6a0d76edc4de151480c1d25789884"}
Oct 30 20:00:19.627798 ip-10-0-132-82 kubenswrapper[4892]: I1030 20:00:19.627557    4892 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w0-6ac304-846999954d-tq2ft" event={"ID":"12863020-d0ea-419d-aba2-816627ab0431","Type":"ContainerDied","Data":"e67a3656d34f7514aec2aef0e041e24f35c31ff4ca2c95f2c91ff0849610a2db"}
Oct 30 20:00:19.627798 ip-10-0-132-82 kubenswrapper[4892]: I1030 20:00:19.627619    4892 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w0-6ac304-846999954d-tq2ft"
Oct 30 20:00:19.630683 ip-10-0-132-82 kubenswrapper[4892]: I1030 20:00:19.630645    4892 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-6ac304-29889840-fhd4j" event={"ID":"25d4aedf-735a-42e6-ae08-4561c967fc35","Type":"ContainerDied","Data":"8ca2d3ca666329cfaace5e1d689aa149ff21ab3f2f28a278f028d86bf22e7338"}
Oct 30 20:00:19.630786 ip-10-0-132-82 kubenswrapper[4892]: I1030 20:00:19.630739    4892 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-6ac304-29889840-fhd4j"
Oct 30 20:00:19.662890 ip-10-0-132-82 kubenswrapper[4892]: I1030 20:00:19.660820    4892 scope.go:122] "RemoveContainer" containerID="5a03d21d9c4839f536751bb171a61b0ba0ee1f8a963d6f3120838a3ce21a623e"
Oct 30 20:00:19.674873 ip-10-0-132-82 kubenswrapper[4892]: I1030 20:00:19.674747    4892 generic.go:356] "Generic (PLEG): container finished" podID="d16a477c-ad96-4363-b13d-b6d374f3f88d" containerID="e8dbb29bc0bd55a5da1e44d40a0795a67ee54ba33779371f263abe5ddd529ad2" exitCode=60
Oct 30 20:00:19.674873 ip-10-0-132-82 kubenswrapper[4892]: I1030 20:00:19.674816    4892 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-132119-29889840-dhpf8" event={"ID":"d16a477c-ad96-4363-b13d-b6d374f3f88d","Type":"ContainerDied","Data":"e8dbb29bc0bd55a5da1e44d40a0795a67ee54ba33779371f263abe5ddd529ad2"}
Oct 30 20:00:19.675907 ip-10-0-132-82 kubenswrapper[4892]: I1030 20:00:19.675295    4892 scope.go:122] "RemoveContainer" containerID="e8dbb29bc0bd55a5da1e44d40a0795a67ee54ba33779371f263abe5ddd529ad2"
Oct 30 20:00:19.675907 ip-10-0-132-82 kubenswrapper[4892]: E1030 20:00:19.675624    4892 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-inf-a-132119-29889840-dhpf8_odh-ai-gateway-infra(d16a477c-ad96-4363-b13d-b6d374f3f88d)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-132119-29889840-dhpf8" podUID="d16a477c-ad96-4363-b13d-b6d374f3f88d"
Oct 30 20:00:19.678192 ip-10-0-132-82 kubenswrapper[4892]: I1030 20:00:19.677292    4892 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w5-686689/distinct-2-w5-686689-kserve-5ff578b4cd-rkc6s" podUID="a8829732-ac6f-4238-b8ce-fc781084668b" containerName="main" containerID="cri-o://1dbd29f80b8f4f21ed34a016fec351a1060e931fb51709446a65f40332fd4fb6" gracePeriod=45
Oct 30 20:00:19.688027 ip-10-0-132-82 kubenswrapper[4892]: I1030 20:00:19.685306    4892 generic.go:356] "Generic (PLEG): container finished" podID="35018750-1063-4fb6-928f-ce6bcd401f87" containerID="2c871a60e44b312e5946a37677af97cb215c9bab0c88ad4e332c408a13a5068c" exitCode=60
Oct 30 20:00:19.688027 ip-10-0-132-82 kubenswrapper[4892]: I1030 20:00:19.685359    4892 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-ada8d7-29889840-xldh2" event={"ID":"35018750-1063-4fb6-928f-ce6bcd401f87","Type":"ContainerDied","Data":"2c871a60e44b312e5946a37677af97cb215c9bab0c88ad4e332c408a13a5068c"}
Oct 30 20:00:19.688027 ip-10-0-132-82 kubenswrapper[4892]: I1030 20:00:19.687349    4892 scope.go:122] "RemoveContainer" containerID="2c871a60e44b312e5946a37677af97cb215c9bab0c88ad4e332c408a13a5068c"
Oct 30 20:00:19.688027 ip-10-0-132-82 kubenswrapper[4892]: E1030 20:00:19.687725    4892 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w3-ada8d7-29889840-xldh2_odh-ai-gateway-infra(35018750-1063-4fb6-928f-ce6bcd401f87)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-ada8d7-29889840-xldh2" podUID="35018750-1063-4fb6-928f-ce6bcd401f87"
Oct 30 20:00:19.693866 ip-10-0-132-82 kubenswrapper[4892]: I1030 20:00:19.693847    4892 scope.go:122] "RemoveContainer" containerID="407af9f51a2672047a92dc943183d990891432c0b35f9bb3457bdeae6421d72b"
Oct 30 20:00:19.709927 ip-10-0-132-82 kubenswrapper[4892]: I1030 20:00:19.708592    4892 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/12863020-d0ea-419d-aba2-816627ab0431-kube-api-access-cs7ph\" (UniqueName: \"kubernetes.io/projected/12863020-d0ea-419d-aba2-816627ab0431-kube-api-access-cs7ph\") pod \"12863020-d0ea-419d-aba2-816627ab0431\" (UID: \"12863020-d0ea-419d-aba2-816627ab0431\") "
Oct 30 20:00:19.709927 ip-10-0-132-82 kubenswrapper[4892]: I1030 20:00:19.708699    4892 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/12863020-d0ea-419d-aba2-816627ab0431-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/12863020-d0ea-419d-aba2-816627ab0431-plugins-config-volume\") pod \"12863020-d0ea-419d-aba2-816627ab0431\" (UID: \"12863020-d0ea-419d-aba2-816627ab0431\") "
Oct 30 20:00:19.711239 ip-10-0-132-82 kubenswrapper[4892]: I1030 20:00:19.711096    4892 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/12863020-d0ea-419d-aba2-816627ab0431-plugins-config-volume" pod "12863020-d0ea-419d-aba2-816627ab0431" (UID: "12863020-d0ea-419d-aba2-816627ab0431"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 20:00:19.712248 ip-10-0-132-82 kubenswrapper[4892]: I1030 20:00:19.712060    4892 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/12863020-d0ea-419d-aba2-816627ab0431-kube-api-access-cs7ph" pod "12863020-d0ea-419d-aba2-816627ab0431" (UID: "12863020-d0ea-419d-aba2-816627ab0431"). InnerVolumeSpecName "kube-api-access-cs7ph". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 20:00:19.775545 ip-10-0-132-82 kubenswrapper[4892]: I1030 20:00:19.775484    4892 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-686689/distinct-2-w5-686689-kserve-5ff578b4cd-rkc6s" podUID="a8829732-ac6f-4238-b8ce-fc781084668b" containerName="main" probeResult="failure" output="Get \"https://10.132.0.87:8000/ready\": dial tcp 10.132.0.87:8000: connect: connection refused"
Oct 30 20:00:19.810609 ip-10-0-132-82 kubenswrapper[4892]: I1030 20:00:19.810360    4892 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/12863020-d0ea-419d-aba2-816627ab0431-plugins-config-volume\") on node \"ip-10-0-132-82.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:00:19.810609 ip-10-0-132-82 kubenswrapper[4892]: I1030 20:00:19.810393    4892 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-cs7ph\" (UniqueName: \"kubernetes.io/projected/12863020-d0ea-419d-aba2-816627ab0431-kube-api-access-cs7ph\") on node \"ip-10-0-132-82.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:00:19.812817 ip-10-0-132-82 kubenswrapper[4892]: I1030 20:00:19.812486    4892 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w0-6ac304-79859bfff5-zx2hb"
Oct 30 20:00:19.812817 ip-10-0-132-82 kubenswrapper[4892]: I1030 20:00:19.812621    4892 scope.go:122] "RemoveContainer" containerID="52a6070c0003df14be74d3b52ca40a184ef6a0d76edc4de151480c1d25789884"
Oct 30 20:00:19.823164 ip-10-0-132-82 kubenswrapper[4892]: I1030 20:00:19.823140    4892 scope.go:122] "RemoveContainer" containerID="52a6070c0003df14be74d3b52ca40a184ef6a0d76edc4de151480c1d25789884"
Oct 30 20:00:19.823480 ip-10-0-132-82 kubenswrapper[4892]: E1030 20:00:19.823456    4892 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"52a6070c0003df14be74d3b52ca40a184ef6a0d76edc4de151480c1d25789884\": container with ID starting with 52a6070c0003df14be74d3b52ca40a184ef6a0d76edc4de151480c1d25789884 not found: ID does not exist" containerID="52a6070c0003df14be74d3b52ca40a184ef6a0d76edc4de151480c1d25789884"
Oct 30 20:00:19.823522 ip-10-0-132-82 kubenswrapper[4892]: I1030 20:00:19.823492    4892 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"52a6070c0003df14be74d3b52ca40a184ef6a0d76edc4de151480c1d25789884"} err="failed to get container status \"52a6070c0003df14be74d3b52ca40a184ef6a0d76edc4de151480c1d25789884\": rpc error: code = NotFound desc = could not find container \"52a6070c0003df14be74d3b52ca40a184ef6a0d76edc4de151480c1d25789884\": container with ID starting with 52a6070c0003df14be74d3b52ca40a184ef6a0d76edc4de151480c1d25789884 not found: ID does not exist"
Oct 30 20:00:19.823522 ip-10-0-132-82 kubenswrapper[4892]: I1030 20:00:19.823514    4892 scope.go:122] "RemoveContainer" containerID="bed3d188e35539b21550fe9cd503df49975e446ac3baeb2774660ea14ba9bb0b"
Oct 30 20:00:19.856582 ip-10-0-132-82 kubenswrapper[4892]: I1030 20:00:19.856557    4892 scope.go:122] "RemoveContainer" containerID="8e3128732b35cf64c18addaa4a4bf68fa805cddf2e389136c5c96b68a9d82cf1"
Oct 30 20:00:19.859655 ip-10-0-132-82 kubenswrapper[4892]: I1030 20:00:19.859635    4892 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-cf9b6f-29889840-8k6mb"
Oct 30 20:00:19.879053 ip-10-0-132-82 kubenswrapper[4892]: I1030 20:00:19.879030    4892 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-b-1c5537-29889840-6qrmj"]
Oct 30 20:00:19.879260 ip-10-0-132-82 kubenswrapper[4892]: I1030 20:00:19.878990    4892 scope.go:122] "RemoveContainer" containerID="9543375f0cc7e181b0fcdd4110a5c18a0ed681347cded35b833f8cc1154d6bbc"
Oct 30 20:00:19.883375 ip-10-0-132-82 kubenswrapper[4892]: I1030 20:00:19.883354    4892 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-b-1c5537-29889840-6qrmj"]
Oct 30 20:00:19.911700 ip-10-0-132-82 kubenswrapper[4892]: I1030 20:00:19.911641    4892 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f1d57e1b-e316-48ac-a1a5-a9520a9e068e-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f1d57e1b-e316-48ac-a1a5-a9520a9e068e-plugins-config-volume\") pod \"f1d57e1b-e316-48ac-a1a5-a9520a9e068e\" (UID: \"f1d57e1b-e316-48ac-a1a5-a9520a9e068e\") "
Oct 30 20:00:19.911826 ip-10-0-132-82 kubenswrapper[4892]: I1030 20:00:19.911728    4892 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f1d57e1b-e316-48ac-a1a5-a9520a9e068e-kube-api-access-4qqhq\" (UniqueName: \"kubernetes.io/projected/f1d57e1b-e316-48ac-a1a5-a9520a9e068e-kube-api-access-4qqhq\") pod \"f1d57e1b-e316-48ac-a1a5-a9520a9e068e\" (UID: \"f1d57e1b-e316-48ac-a1a5-a9520a9e068e\") "
Oct 30 20:00:19.912077 ip-10-0-132-82 kubenswrapper[4892]: I1030 20:00:19.912052    4892 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f1d57e1b-e316-48ac-a1a5-a9520a9e068e-plugins-config-volume" pod "f1d57e1b-e316-48ac-a1a5-a9520a9e068e" (UID: "f1d57e1b-e316-48ac-a1a5-a9520a9e068e"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 20:00:19.912323 ip-10-0-132-82 kubenswrapper[4892]: I1030 20:00:19.912303    4892 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f1d57e1b-e316-48ac-a1a5-a9520a9e068e-plugins-config-volume\") on node \"ip-10-0-132-82.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:00:19.914545 ip-10-0-132-82 kubenswrapper[4892]: I1030 20:00:19.914521    4892 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f1d57e1b-e316-48ac-a1a5-a9520a9e068e-kube-api-access-4qqhq" pod "f1d57e1b-e316-48ac-a1a5-a9520a9e068e" (UID: "f1d57e1b-e316-48ac-a1a5-a9520a9e068e"). InnerVolumeSpecName "kube-api-access-4qqhq". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 20:00:19.943906 ip-10-0-132-82 kubenswrapper[4892]: I1030 20:00:19.943881    4892 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-686689-29889840-l2mjk"]
Oct 30 20:00:19.950644 ip-10-0-132-82 kubenswrapper[4892]: I1030 20:00:19.950622    4892 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w5-686689/distinct-2-w5-686689-kserve-5ff578b4cd-rkc6s"
Oct 30 20:00:20.013451 ip-10-0-132-82 kubenswrapper[4892]: I1030 20:00:20.013415    4892 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/9df510ae-c995-4678-816a-9c4834778991-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/9df510ae-c995-4678-816a-9c4834778991-maas-api-service-ca\") pod \"9df510ae-c995-4678-816a-9c4834778991\" (UID: \"9df510ae-c995-4678-816a-9c4834778991\") "
Oct 30 20:00:20.013649 ip-10-0-132-82 kubenswrapper[4892]: I1030 20:00:20.013508    4892 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9df510ae-c995-4678-816a-9c4834778991-kube-api-access-9dtwd\" (UniqueName: \"kubernetes.io/projected/9df510ae-c995-4678-816a-9c4834778991-kube-api-access-9dtwd\") pod \"9df510ae-c995-4678-816a-9c4834778991\" (UID: \"9df510ae-c995-4678-816a-9c4834778991\") "
Oct 30 20:00:20.013763 ip-10-0-132-82 kubenswrapper[4892]: I1030 20:00:20.013738    4892 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9df510ae-c995-4678-816a-9c4834778991-maas-api-service-ca" pod "9df510ae-c995-4678-816a-9c4834778991" (UID: "9df510ae-c995-4678-816a-9c4834778991"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 20:00:20.014104 ip-10-0-132-82 kubenswrapper[4892]: I1030 20:00:20.014082    4892 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4qqhq\" (UniqueName: \"kubernetes.io/projected/f1d57e1b-e316-48ac-a1a5-a9520a9e068e-kube-api-access-4qqhq\") on node \"ip-10-0-132-82.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:00:20.014206 ip-10-0-132-82 kubenswrapper[4892]: I1030 20:00:20.014109    4892 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/9df510ae-c995-4678-816a-9c4834778991-maas-api-service-ca\") on node \"ip-10-0-132-82.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:00:20.015765 ip-10-0-132-82 kubenswrapper[4892]: I1030 20:00:20.015732    4892 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9df510ae-c995-4678-816a-9c4834778991-kube-api-access-9dtwd" pod "9df510ae-c995-4678-816a-9c4834778991" (UID: "9df510ae-c995-4678-816a-9c4834778991"). InnerVolumeSpecName "kube-api-access-9dtwd". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 20:00:20.063987 ip-10-0-132-82 kubenswrapper[4892]: I1030 20:00:20.063397    4892 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-2d9861-29889840-79j62"]
Oct 30 20:00:20.114993 ip-10-0-132-82 kubenswrapper[4892]: I1030 20:00:20.114471    4892 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a8829732-ac6f-4238-b8ce-fc781084668b-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a8829732-ac6f-4238-b8ce-fc781084668b-tmp-dir\") pod \"a8829732-ac6f-4238-b8ce-fc781084668b\" (UID: \"a8829732-ac6f-4238-b8ce-fc781084668b\") "
Oct 30 20:00:20.114993 ip-10-0-132-82 kubenswrapper[4892]: I1030 20:00:20.114573    4892 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a8829732-ac6f-4238-b8ce-fc781084668b-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a8829732-ac6f-4238-b8ce-fc781084668b-model-cache\") pod \"a8829732-ac6f-4238-b8ce-fc781084668b\" (UID: \"a8829732-ac6f-4238-b8ce-fc781084668b\") "
Oct 30 20:00:20.114993 ip-10-0-132-82 kubenswrapper[4892]: I1030 20:00:20.114639    4892 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a8829732-ac6f-4238-b8ce-fc781084668b-kube-api-access-98wsx\" (UniqueName: \"kubernetes.io/projected/a8829732-ac6f-4238-b8ce-fc781084668b-kube-api-access-98wsx\") pod \"a8829732-ac6f-4238-b8ce-fc781084668b\" (UID: \"a8829732-ac6f-4238-b8ce-fc781084668b\") "
Oct 30 20:00:20.114993 ip-10-0-132-82 kubenswrapper[4892]: I1030 20:00:20.114775    4892 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/a8829732-ac6f-4238-b8ce-fc781084668b-tls-certs\" (UniqueName: \"kubernetes.io/secret/a8829732-ac6f-4238-b8ce-fc781084668b-tls-certs\") pod \"a8829732-ac6f-4238-b8ce-fc781084668b\" (UID: \"a8829732-ac6f-4238-b8ce-fc781084668b\") "
Oct 30 20:00:20.114993 ip-10-0-132-82 kubenswrapper[4892]: I1030 20:00:20.114785    4892 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a8829732-ac6f-4238-b8ce-fc781084668b-tmp-dir" pod "a8829732-ac6f-4238-b8ce-fc781084668b" (UID: "a8829732-ac6f-4238-b8ce-fc781084668b"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 20:00:20.114993 ip-10-0-132-82 kubenswrapper[4892]: I1030 20:00:20.114796    4892 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a8829732-ac6f-4238-b8ce-fc781084668b-home\" (UniqueName: \"kubernetes.io/empty-dir/a8829732-ac6f-4238-b8ce-fc781084668b-home\") pod \"a8829732-ac6f-4238-b8ce-fc781084668b\" (UID: \"a8829732-ac6f-4238-b8ce-fc781084668b\") "
Oct 30 20:00:20.114993 ip-10-0-132-82 kubenswrapper[4892]: I1030 20:00:20.114866    4892 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a8829732-ac6f-4238-b8ce-fc781084668b-dshm\" (UniqueName: \"kubernetes.io/empty-dir/a8829732-ac6f-4238-b8ce-fc781084668b-dshm\") pod \"a8829732-ac6f-4238-b8ce-fc781084668b\" (UID: \"a8829732-ac6f-4238-b8ce-fc781084668b\") "
Oct 30 20:00:20.115467 ip-10-0-132-82 kubenswrapper[4892]: I1030 20:00:20.115272    4892 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a8829732-ac6f-4238-b8ce-fc781084668b-model-cache" pod "a8829732-ac6f-4238-b8ce-fc781084668b" (UID: "a8829732-ac6f-4238-b8ce-fc781084668b"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 20:00:20.115467 ip-10-0-132-82 kubenswrapper[4892]: I1030 20:00:20.115453    4892 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a8829732-ac6f-4238-b8ce-fc781084668b-tmp-dir\") on node \"ip-10-0-132-82.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:00:20.115542 ip-10-0-132-82 kubenswrapper[4892]: I1030 20:00:20.115477    4892 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9dtwd\" (UniqueName: \"kubernetes.io/projected/9df510ae-c995-4678-816a-9c4834778991-kube-api-access-9dtwd\") on node \"ip-10-0-132-82.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:00:20.115542 ip-10-0-132-82 kubenswrapper[4892]: I1030 20:00:20.115488    4892 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a8829732-ac6f-4238-b8ce-fc781084668b-model-cache\") on node \"ip-10-0-132-82.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:00:20.115924 ip-10-0-132-82 kubenswrapper[4892]: I1030 20:00:20.115659    4892 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a8829732-ac6f-4238-b8ce-fc781084668b-home" pod "a8829732-ac6f-4238-b8ce-fc781084668b" (UID: "a8829732-ac6f-4238-b8ce-fc781084668b"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 20:00:20.117899 ip-10-0-132-82 kubenswrapper[4892]: I1030 20:00:20.117875    4892 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a8829732-ac6f-4238-b8ce-fc781084668b-dshm" pod "a8829732-ac6f-4238-b8ce-fc781084668b" (UID: "a8829732-ac6f-4238-b8ce-fc781084668b"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 20:00:20.117899 ip-10-0-132-82 kubenswrapper[4892]: I1030 20:00:20.117925    4892 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a8829732-ac6f-4238-b8ce-fc781084668b-kube-api-access-98wsx" pod "a8829732-ac6f-4238-b8ce-fc781084668b" (UID: "a8829732-ac6f-4238-b8ce-fc781084668b"). InnerVolumeSpecName "kube-api-access-98wsx". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 20:00:20.117899 ip-10-0-132-82 kubenswrapper[4892]: I1030 20:00:20.118105    4892 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a8829732-ac6f-4238-b8ce-fc781084668b-tls-certs" pod "a8829732-ac6f-4238-b8ce-fc781084668b" (UID: "a8829732-ac6f-4238-b8ce-fc781084668b"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 20:00:20.173064 ip-10-0-132-82 kubenswrapper[4892]: I1030 20:00:20.172989    4892 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-132119-29889840-dhpf8"]
Oct 30 20:00:20.217147 ip-10-0-132-82 kubenswrapper[4892]: I1030 20:00:20.217106    4892 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a8829732-ac6f-4238-b8ce-fc781084668b-tls-certs\") on node \"ip-10-0-132-82.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:00:20.217147 ip-10-0-132-82 kubenswrapper[4892]: I1030 20:00:20.217142    4892 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a8829732-ac6f-4238-b8ce-fc781084668b-home\") on node \"ip-10-0-132-82.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:00:20.217147 ip-10-0-132-82 kubenswrapper[4892]: I1030 20:00:20.217150    4892 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a8829732-ac6f-4238-b8ce-fc781084668b-dshm\") on node \"ip-10-0-132-82.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:00:20.217147 ip-10-0-132-82 kubenswrapper[4892]: I1030 20:00:20.217160    4892 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-98wsx\" (UniqueName: \"kubernetes.io/projected/a8829732-ac6f-4238-b8ce-fc781084668b-kube-api-access-98wsx\") on node \"ip-10-0-132-82.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:00:20.285046 ip-10-0-132-82 kubenswrapper[4892]: I1030 20:00:20.285008    4892 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-mt-dd9b8d79/e2e-lifecycle-model-dd9b8d79-kserve-7659b4b7d9-r2flr" podUID="e469e8c3-e158-4b45-9135-6da275fc897f" containerName="main" containerID="cri-o://a8265c669ccf9c1b2da8388e1843b62adc4cb67ef19134165d5034efe5354ee0" gracePeriod=45
Oct 30 20:00:20.409389 ip-10-0-132-82 kubenswrapper[4892]: I1030 20:00:20.409358    4892 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-c75ed4-29889840-7s578"]
Oct 30 20:00:20.480723 ip-10-0-132-82 kubenswrapper[4892]: I1030 20:00:20.479379    4892 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-6ac304-29889840-fhd4j"]
Oct 30 20:00:20.483197 ip-10-0-132-82 kubenswrapper[4892]: I1030 20:00:20.483170    4892 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-6ac304-29889840-fhd4j"]
Oct 30 20:00:20.522945 ip-10-0-132-82 kubenswrapper[4892]: I1030 20:00:20.522922    4892 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-ada8d7-29889840-xldh2"]
Oct 30 20:00:20.545405 ip-10-0-132-82 kubenswrapper[4892]: I1030 20:00:20.545383    4892 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-mt-dd9b8d79/e2e-lifecycle-model-dd9b8d79-kserve-7659b4b7d9-r2flr"
Oct 30 20:00:20.572269 ip-10-0-132-82 kubenswrapper[4892]: I1030 20:00:20.572231    4892 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="25d4aedf-735a-42e6-ae08-4561c967fc35" path="/var/lib/kubelet/pods/25d4aedf-735a-42e6-ae08-4561c967fc35/volumes"
Oct 30 20:00:20.572758 ip-10-0-132-82 kubenswrapper[4892]: I1030 20:00:20.572737    4892 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f6260396-890c-44ea-be8c-998dd2086ffa" path="/var/lib/kubelet/pods/f6260396-890c-44ea-be8c-998dd2086ffa/volumes"
Oct 30 20:00:20.693766 ip-10-0-132-82 kubenswrapper[4892]: I1030 20:00:20.693735    4892 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-cf9b6f-29889840-8k6mb" event={"ID":"9df510ae-c995-4678-816a-9c4834778991","Type":"ContainerDied","Data":"201941280f0662cb615d76c79cbf45d160367299e67b8ecf92190988b79d7cba"}
Oct 30 20:00:20.694234 ip-10-0-132-82 kubenswrapper[4892]: I1030 20:00:20.693789    4892 scope.go:122] "RemoveContainer" containerID="7fd36bfe6dd50b93eb0758b8b0bc1bfacbd4ddcdaabec4261f990dff59972a2f"
Oct 30 20:00:20.694234 ip-10-0-132-82 kubenswrapper[4892]: I1030 20:00:20.693745    4892 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-cf9b6f-29889840-8k6mb"
Oct 30 20:00:20.701335 ip-10-0-132-82 kubenswrapper[4892]: I1030 20:00:20.701313    4892 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w0-6ac304-79859bfff5-zx2hb"
Oct 30 20:00:20.701453 ip-10-0-132-82 kubenswrapper[4892]: I1030 20:00:20.701312    4892 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w0-6ac304-79859bfff5-zx2hb" event={"ID":"f1d57e1b-e316-48ac-a1a5-a9520a9e068e","Type":"ContainerDied","Data":"855f82b370d62be0b08f4ede7f7d9b77357e518e814795500de05e18d379a3e4"}
Oct 30 20:00:20.704308 ip-10-0-132-82 kubenswrapper[4892]: I1030 20:00:20.704279    4892 generic.go:356] "Generic (PLEG): container finished" podID="a8829732-ac6f-4238-b8ce-fc781084668b" containerID="1dbd29f80b8f4f21ed34a016fec351a1060e931fb51709446a65f40332fd4fb6" exitCode=0
Oct 30 20:00:20.704452 ip-10-0-132-82 kubenswrapper[4892]: I1030 20:00:20.704309    4892 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w5-686689/distinct-2-w5-686689-kserve-5ff578b4cd-rkc6s"
Oct 30 20:00:20.704452 ip-10-0-132-82 kubenswrapper[4892]: I1030 20:00:20.704336    4892 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w5-686689/distinct-2-w5-686689-kserve-5ff578b4cd-rkc6s" event={"ID":"a8829732-ac6f-4238-b8ce-fc781084668b","Type":"ContainerDied","Data":"1dbd29f80b8f4f21ed34a016fec351a1060e931fb51709446a65f40332fd4fb6"}
Oct 30 20:00:20.704452 ip-10-0-132-82 kubenswrapper[4892]: I1030 20:00:20.704365    4892 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w5-686689/distinct-2-w5-686689-kserve-5ff578b4cd-rkc6s" event={"ID":"a8829732-ac6f-4238-b8ce-fc781084668b","Type":"ContainerDied","Data":"e864449f59a7d28459aa2ba16d05d1ba99d97fc377a1190547d98610a4c76b78"}
Oct 30 20:00:20.708280 ip-10-0-132-82 kubenswrapper[4892]: I1030 20:00:20.708050    4892 scope.go:122] "RemoveContainer" containerID="45e7e4c2c487385d9b7c19a9b2ffafa7dea4d76bc721247e55757afa40a29171"
Oct 30 20:00:20.718529 ip-10-0-132-82 kubenswrapper[4892]: I1030 20:00:20.718227    4892 generic.go:356] "Generic (PLEG): container finished" podID="e469e8c3-e158-4b45-9135-6da275fc897f" containerID="a8265c669ccf9c1b2da8388e1843b62adc4cb67ef19134165d5034efe5354ee0" exitCode=0
Oct 30 20:00:20.718529 ip-10-0-132-82 kubenswrapper[4892]: I1030 20:00:20.718334    4892 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-mt-dd9b8d79/e2e-lifecycle-model-dd9b8d79-kserve-7659b4b7d9-r2flr" event={"ID":"e469e8c3-e158-4b45-9135-6da275fc897f","Type":"ContainerDied","Data":"a8265c669ccf9c1b2da8388e1843b62adc4cb67ef19134165d5034efe5354ee0"}
Oct 30 20:00:20.718529 ip-10-0-132-82 kubenswrapper[4892]: I1030 20:00:20.718359    4892 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-mt-dd9b8d79/e2e-lifecycle-model-dd9b8d79-kserve-7659b4b7d9-r2flr" event={"ID":"e469e8c3-e158-4b45-9135-6da275fc897f","Type":"ContainerDied","Data":"ea8d1e5a49bc366dde5a9a98f2088dcaad9c4510e7ba6ec89fe6da1946901a35"}
Oct 30 20:00:20.718529 ip-10-0-132-82 kubenswrapper[4892]: I1030 20:00:20.718419    4892 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-mt-dd9b8d79/e2e-lifecycle-model-dd9b8d79-kserve-7659b4b7d9-r2flr"
Oct 30 20:00:20.724430 ip-10-0-132-82 kubenswrapper[4892]: I1030 20:00:20.724404    4892 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/e469e8c3-e158-4b45-9135-6da275fc897f-tls-certs\" (UniqueName: \"kubernetes.io/secret/e469e8c3-e158-4b45-9135-6da275fc897f-tls-certs\") pod \"e469e8c3-e158-4b45-9135-6da275fc897f\" (UID: \"e469e8c3-e158-4b45-9135-6da275fc897f\") "
Oct 30 20:00:20.724601 ip-10-0-132-82 kubenswrapper[4892]: I1030 20:00:20.724565    4892 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e469e8c3-e158-4b45-9135-6da275fc897f-kube-api-access-m4ps7\" (UniqueName: \"kubernetes.io/projected/e469e8c3-e158-4b45-9135-6da275fc897f-kube-api-access-m4ps7\") pod \"e469e8c3-e158-4b45-9135-6da275fc897f\" (UID: \"e469e8c3-e158-4b45-9135-6da275fc897f\") "
Oct 30 20:00:20.724681 ip-10-0-132-82 kubenswrapper[4892]: I1030 20:00:20.724641    4892 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e469e8c3-e158-4b45-9135-6da275fc897f-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e469e8c3-e158-4b45-9135-6da275fc897f-tmp-dir\") pod \"e469e8c3-e158-4b45-9135-6da275fc897f\" (UID: \"e469e8c3-e158-4b45-9135-6da275fc897f\") "
Oct 30 20:00:20.724745 ip-10-0-132-82 kubenswrapper[4892]: I1030 20:00:20.724679    4892 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e469e8c3-e158-4b45-9135-6da275fc897f-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e469e8c3-e158-4b45-9135-6da275fc897f-model-cache\") pod \"e469e8c3-e158-4b45-9135-6da275fc897f\" (UID: \"e469e8c3-e158-4b45-9135-6da275fc897f\") "
Oct 30 20:00:20.724745 ip-10-0-132-82 kubenswrapper[4892]: I1030 20:00:20.724730    4892 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e469e8c3-e158-4b45-9135-6da275fc897f-home\" (UniqueName: \"kubernetes.io/empty-dir/e469e8c3-e158-4b45-9135-6da275fc897f-home\") pod \"e469e8c3-e158-4b45-9135-6da275fc897f\" (UID: \"e469e8c3-e158-4b45-9135-6da275fc897f\") "
Oct 30 20:00:20.724986 ip-10-0-132-82 kubenswrapper[4892]: I1030 20:00:20.724771    4892 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e469e8c3-e158-4b45-9135-6da275fc897f-dshm\" (UniqueName: \"kubernetes.io/empty-dir/e469e8c3-e158-4b45-9135-6da275fc897f-dshm\") pod \"e469e8c3-e158-4b45-9135-6da275fc897f\" (UID: \"e469e8c3-e158-4b45-9135-6da275fc897f\") "
Oct 30 20:00:20.725281 ip-10-0-132-82 kubenswrapper[4892]: I1030 20:00:20.725259    4892 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e469e8c3-e158-4b45-9135-6da275fc897f-tmp-dir" pod "e469e8c3-e158-4b45-9135-6da275fc897f" (UID: "e469e8c3-e158-4b45-9135-6da275fc897f"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 20:00:20.725548 ip-10-0-132-82 kubenswrapper[4892]: I1030 20:00:20.725524    4892 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e469e8c3-e158-4b45-9135-6da275fc897f-model-cache" pod "e469e8c3-e158-4b45-9135-6da275fc897f" (UID: "e469e8c3-e158-4b45-9135-6da275fc897f"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 20:00:20.725844 ip-10-0-132-82 kubenswrapper[4892]: I1030 20:00:20.725673    4892 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e469e8c3-e158-4b45-9135-6da275fc897f-home" pod "e469e8c3-e158-4b45-9135-6da275fc897f" (UID: "e469e8c3-e158-4b45-9135-6da275fc897f"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 20:00:20.725995 ip-10-0-132-82 kubenswrapper[4892]: I1030 20:00:20.725939    4892 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e469e8c3-e158-4b45-9135-6da275fc897f-tmp-dir\") on node \"ip-10-0-132-82.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:00:20.726079 ip-10-0-132-82 kubenswrapper[4892]: I1030 20:00:20.726068    4892 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e469e8c3-e158-4b45-9135-6da275fc897f-model-cache\") on node \"ip-10-0-132-82.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:00:20.726136 ip-10-0-132-82 kubenswrapper[4892]: I1030 20:00:20.726086    4892 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e469e8c3-e158-4b45-9135-6da275fc897f-home\") on node \"ip-10-0-132-82.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:00:20.749222 ip-10-0-132-82 kubenswrapper[4892]: I1030 20:00:20.749182    4892 scope.go:122] "RemoveContainer" containerID="1dbd29f80b8f4f21ed34a016fec351a1060e931fb51709446a65f40332fd4fb6"
Oct 30 20:00:20.754486 ip-10-0-132-82 kubenswrapper[4892]: I1030 20:00:20.754451    4892 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e469e8c3-e158-4b45-9135-6da275fc897f-tls-certs" pod "e469e8c3-e158-4b45-9135-6da275fc897f" (UID: "e469e8c3-e158-4b45-9135-6da275fc897f"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 20:00:20.754706 ip-10-0-132-82 kubenswrapper[4892]: I1030 20:00:20.754683    4892 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e469e8c3-e158-4b45-9135-6da275fc897f-kube-api-access-m4ps7" pod "e469e8c3-e158-4b45-9135-6da275fc897f" (UID: "e469e8c3-e158-4b45-9135-6da275fc897f"). InnerVolumeSpecName "kube-api-access-m4ps7". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 20:00:20.775353 ip-10-0-132-82 kubenswrapper[4892]: I1030 20:00:20.775322    4892 scope.go:122] "RemoveContainer" containerID="1dbd29f80b8f4f21ed34a016fec351a1060e931fb51709446a65f40332fd4fb6"
Oct 30 20:00:20.775818 ip-10-0-132-82 kubenswrapper[4892]: E1030 20:00:20.775787    4892 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1dbd29f80b8f4f21ed34a016fec351a1060e931fb51709446a65f40332fd4fb6\": container with ID starting with 1dbd29f80b8f4f21ed34a016fec351a1060e931fb51709446a65f40332fd4fb6 not found: ID does not exist" containerID="1dbd29f80b8f4f21ed34a016fec351a1060e931fb51709446a65f40332fd4fb6"
Oct 30 20:00:20.775926 ip-10-0-132-82 kubenswrapper[4892]: I1030 20:00:20.775831    4892 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1dbd29f80b8f4f21ed34a016fec351a1060e931fb51709446a65f40332fd4fb6"} err="failed to get container status \"1dbd29f80b8f4f21ed34a016fec351a1060e931fb51709446a65f40332fd4fb6\": rpc error: code = NotFound desc = could not find container \"1dbd29f80b8f4f21ed34a016fec351a1060e931fb51709446a65f40332fd4fb6\": container with ID starting with 1dbd29f80b8f4f21ed34a016fec351a1060e931fb51709446a65f40332fd4fb6 not found: ID does not exist"
Oct 30 20:00:20.775926 ip-10-0-132-82 kubenswrapper[4892]: I1030 20:00:20.775856    4892 scope.go:122] "RemoveContainer" containerID="a8265c669ccf9c1b2da8388e1843b62adc4cb67ef19134165d5034efe5354ee0"
Oct 30 20:00:20.778597 ip-10-0-132-82 kubenswrapper[4892]: I1030 20:00:20.778563    4892 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e469e8c3-e158-4b45-9135-6da275fc897f-dshm" pod "e469e8c3-e158-4b45-9135-6da275fc897f" (UID: "e469e8c3-e158-4b45-9135-6da275fc897f"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 20:00:20.795362 ip-10-0-132-82 kubenswrapper[4892]: I1030 20:00:20.795332    4892 scope.go:122] "RemoveContainer" containerID="a8265c669ccf9c1b2da8388e1843b62adc4cb67ef19134165d5034efe5354ee0"
Oct 30 20:00:20.795811 ip-10-0-132-82 kubenswrapper[4892]: E1030 20:00:20.795781    4892 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a8265c669ccf9c1b2da8388e1843b62adc4cb67ef19134165d5034efe5354ee0\": container with ID starting with a8265c669ccf9c1b2da8388e1843b62adc4cb67ef19134165d5034efe5354ee0 not found: ID does not exist" containerID="a8265c669ccf9c1b2da8388e1843b62adc4cb67ef19134165d5034efe5354ee0"
Oct 30 20:00:20.795914 ip-10-0-132-82 kubenswrapper[4892]: I1030 20:00:20.795821    4892 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a8265c669ccf9c1b2da8388e1843b62adc4cb67ef19134165d5034efe5354ee0"} err="failed to get container status \"a8265c669ccf9c1b2da8388e1843b62adc4cb67ef19134165d5034efe5354ee0\": rpc error: code = NotFound desc = could not find container \"a8265c669ccf9c1b2da8388e1843b62adc4cb67ef19134165d5034efe5354ee0\": container with ID starting with a8265c669ccf9c1b2da8388e1843b62adc4cb67ef19134165d5034efe5354ee0 not found: ID does not exist"
Oct 30 20:00:20.829417 ip-10-0-132-82 kubenswrapper[4892]: I1030 20:00:20.827351    4892 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e469e8c3-e158-4b45-9135-6da275fc897f-tls-certs\") on node \"ip-10-0-132-82.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:00:20.829417 ip-10-0-132-82 kubenswrapper[4892]: I1030 20:00:20.827379    4892 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-m4ps7\" (UniqueName: \"kubernetes.io/projected/e469e8c3-e158-4b45-9135-6da275fc897f-kube-api-access-m4ps7\") on node \"ip-10-0-132-82.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:00:20.829417 ip-10-0-132-82 kubenswrapper[4892]: I1030 20:00:20.827395    4892 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e469e8c3-e158-4b45-9135-6da275fc897f-dshm\") on node \"ip-10-0-132-82.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:00:20.898839 ip-10-0-132-82 kubenswrapper[4892]: I1030 20:00:20.898813    4892 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-c75ed4-29889840-7s578"
Oct 30 20:00:21.005492 ip-10-0-132-82 kubenswrapper[4892]: I1030 20:00:21.005467    4892 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-2d9861-29889840-79j62"
Oct 30 20:00:21.009156 ip-10-0-132-82 kubenswrapper[4892]: I1030 20:00:21.009130    4892 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-686689-29889840-l2mjk"
Oct 30 20:00:21.012948 ip-10-0-132-82 kubenswrapper[4892]: I1030 20:00:21.012924    4892 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-132119-29889840-dhpf8"
Oct 30 20:00:21.030135 ip-10-0-132-82 kubenswrapper[4892]: I1030 20:00:21.029913    4892 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/8152ed9d-b608-4a87-aded-8dda2d975174-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/8152ed9d-b608-4a87-aded-8dda2d975174-maas-api-service-ca\") pod \"8152ed9d-b608-4a87-aded-8dda2d975174\" (UID: \"8152ed9d-b608-4a87-aded-8dda2d975174\") "
Oct 30 20:00:21.030135 ip-10-0-132-82 kubenswrapper[4892]: I1030 20:00:21.030004    4892 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8152ed9d-b608-4a87-aded-8dda2d975174-kube-api-access-q9qjb\" (UniqueName: \"kubernetes.io/projected/8152ed9d-b608-4a87-aded-8dda2d975174-kube-api-access-q9qjb\") pod \"8152ed9d-b608-4a87-aded-8dda2d975174\" (UID: \"8152ed9d-b608-4a87-aded-8dda2d975174\") "
Oct 30 20:00:21.030423 ip-10-0-132-82 kubenswrapper[4892]: I1030 20:00:21.030395    4892 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8152ed9d-b608-4a87-aded-8dda2d975174-maas-api-service-ca" pod "8152ed9d-b608-4a87-aded-8dda2d975174" (UID: "8152ed9d-b608-4a87-aded-8dda2d975174"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 20:00:21.030579 ip-10-0-132-82 kubenswrapper[4892]: I1030 20:00:21.030557    4892 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/8152ed9d-b608-4a87-aded-8dda2d975174-maas-api-service-ca\") on node \"ip-10-0-132-82.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:00:21.032594 ip-10-0-132-82 kubenswrapper[4892]: I1030 20:00:21.032573    4892 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8152ed9d-b608-4a87-aded-8dda2d975174-kube-api-access-q9qjb" pod "8152ed9d-b608-4a87-aded-8dda2d975174" (UID: "8152ed9d-b608-4a87-aded-8dda2d975174"). InnerVolumeSpecName "kube-api-access-q9qjb". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 20:00:21.034112 ip-10-0-132-82 kubenswrapper[4892]: I1030 20:00:21.034093    4892 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-ada8d7-29889840-xldh2"
Oct 30 20:00:21.114386 ip-10-0-132-82 kubenswrapper[4892]: I1030 20:00:21.114345    4892 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-cf9b6f-29889840-8k6mb"]
Oct 30 20:00:21.119672 ip-10-0-132-82 kubenswrapper[4892]: I1030 20:00:21.119644    4892 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-cf9b6f-29889840-8k6mb"]
Oct 30 20:00:21.131189 ip-10-0-132-82 kubenswrapper[4892]: I1030 20:00:21.131157    4892 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d16a477c-ad96-4363-b13d-b6d374f3f88d-kube-api-access-d99hj\" (UniqueName: \"kubernetes.io/projected/d16a477c-ad96-4363-b13d-b6d374f3f88d-kube-api-access-d99hj\") pod \"d16a477c-ad96-4363-b13d-b6d374f3f88d\" (UID: \"d16a477c-ad96-4363-b13d-b6d374f3f88d\") "
Oct 30 20:00:21.131330 ip-10-0-132-82 kubenswrapper[4892]: I1030 20:00:21.131230    4892 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/35018750-1063-4fb6-928f-ce6bcd401f87-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/35018750-1063-4fb6-928f-ce6bcd401f87-maas-api-service-ca\") pod \"35018750-1063-4fb6-928f-ce6bcd401f87\" (UID: \"35018750-1063-4fb6-928f-ce6bcd401f87\") "
Oct 30 20:00:21.131382 ip-10-0-132-82 kubenswrapper[4892]: I1030 20:00:21.131361    4892 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d16a477c-ad96-4363-b13d-b6d374f3f88d-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/d16a477c-ad96-4363-b13d-b6d374f3f88d-maas-api-service-ca\") pod \"d16a477c-ad96-4363-b13d-b6d374f3f88d\" (UID: \"d16a477c-ad96-4363-b13d-b6d374f3f88d\") "
Oct 30 20:00:21.131494 ip-10-0-132-82 kubenswrapper[4892]: I1030 20:00:21.131476    4892 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/82b75f10-16db-40c1-9696-f1e7f9917419-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/82b75f10-16db-40c1-9696-f1e7f9917419-maas-api-service-ca\") pod \"82b75f10-16db-40c1-9696-f1e7f9917419\" (UID: \"82b75f10-16db-40c1-9696-f1e7f9917419\") "
Oct 30 20:00:21.131569 ip-10-0-132-82 kubenswrapper[4892]: I1030 20:00:21.131526    4892 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0b0b9f79-e0a9-41a1-b513-01ea5661f64d-kube-api-access-plr9z\" (UniqueName: \"kubernetes.io/projected/0b0b9f79-e0a9-41a1-b513-01ea5661f64d-kube-api-access-plr9z\") pod \"0b0b9f79-e0a9-41a1-b513-01ea5661f64d\" (UID: \"0b0b9f79-e0a9-41a1-b513-01ea5661f64d\") "
Oct 30 20:00:21.131668 ip-10-0-132-82 kubenswrapper[4892]: I1030 20:00:21.131640    4892 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/35018750-1063-4fb6-928f-ce6bcd401f87-maas-api-service-ca" pod "35018750-1063-4fb6-928f-ce6bcd401f87" (UID: "35018750-1063-4fb6-928f-ce6bcd401f87"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 20:00:21.131668 ip-10-0-132-82 kubenswrapper[4892]: I1030 20:00:21.131657    4892 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/0b0b9f79-e0a9-41a1-b513-01ea5661f64d-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/0b0b9f79-e0a9-41a1-b513-01ea5661f64d-maas-api-service-ca\") pod \"0b0b9f79-e0a9-41a1-b513-01ea5661f64d\" (UID: \"0b0b9f79-e0a9-41a1-b513-01ea5661f64d\") "
Oct 30 20:00:21.131819 ip-10-0-132-82 kubenswrapper[4892]: I1030 20:00:21.131693    4892 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/35018750-1063-4fb6-928f-ce6bcd401f87-kube-api-access-4szjb\" (UniqueName: \"kubernetes.io/projected/35018750-1063-4fb6-928f-ce6bcd401f87-kube-api-access-4szjb\") pod \"35018750-1063-4fb6-928f-ce6bcd401f87\" (UID: \"35018750-1063-4fb6-928f-ce6bcd401f87\") "
Oct 30 20:00:21.131819 ip-10-0-132-82 kubenswrapper[4892]: I1030 20:00:21.131739    4892 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/82b75f10-16db-40c1-9696-f1e7f9917419-kube-api-access-wq78b\" (UniqueName: \"kubernetes.io/projected/82b75f10-16db-40c1-9696-f1e7f9917419-kube-api-access-wq78b\") pod \"82b75f10-16db-40c1-9696-f1e7f9917419\" (UID: \"82b75f10-16db-40c1-9696-f1e7f9917419\") "
Oct 30 20:00:21.131819 ip-10-0-132-82 kubenswrapper[4892]: I1030 20:00:21.131801    4892 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d16a477c-ad96-4363-b13d-b6d374f3f88d-maas-api-service-ca" pod "d16a477c-ad96-4363-b13d-b6d374f3f88d" (UID: "d16a477c-ad96-4363-b13d-b6d374f3f88d"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 20:00:21.132101 ip-10-0-132-82 kubenswrapper[4892]: I1030 20:00:21.132068    4892 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0b0b9f79-e0a9-41a1-b513-01ea5661f64d-maas-api-service-ca" pod "0b0b9f79-e0a9-41a1-b513-01ea5661f64d" (UID: "0b0b9f79-e0a9-41a1-b513-01ea5661f64d"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 20:00:21.132265 ip-10-0-132-82 kubenswrapper[4892]: I1030 20:00:21.132231    4892 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/82b75f10-16db-40c1-9696-f1e7f9917419-maas-api-service-ca" pod "82b75f10-16db-40c1-9696-f1e7f9917419" (UID: "82b75f10-16db-40c1-9696-f1e7f9917419"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 20:00:21.132435 ip-10-0-132-82 kubenswrapper[4892]: I1030 20:00:21.132392    4892 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/0b0b9f79-e0a9-41a1-b513-01ea5661f64d-maas-api-service-ca\") on node \"ip-10-0-132-82.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:00:21.132435 ip-10-0-132-82 kubenswrapper[4892]: I1030 20:00:21.132418    4892 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/35018750-1063-4fb6-928f-ce6bcd401f87-maas-api-service-ca\") on node \"ip-10-0-132-82.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:00:21.132553 ip-10-0-132-82 kubenswrapper[4892]: I1030 20:00:21.132434    4892 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-q9qjb\" (UniqueName: \"kubernetes.io/projected/8152ed9d-b608-4a87-aded-8dda2d975174-kube-api-access-q9qjb\") on node \"ip-10-0-132-82.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:00:21.132553 ip-10-0-132-82 kubenswrapper[4892]: I1030 20:00:21.132453    4892 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/d16a477c-ad96-4363-b13d-b6d374f3f88d-maas-api-service-ca\") on node \"ip-10-0-132-82.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:00:21.132553 ip-10-0-132-82 kubenswrapper[4892]: I1030 20:00:21.132467    4892 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/82b75f10-16db-40c1-9696-f1e7f9917419-maas-api-service-ca\") on node \"ip-10-0-132-82.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:00:21.134054 ip-10-0-132-82 kubenswrapper[4892]: I1030 20:00:21.134026    4892 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/35018750-1063-4fb6-928f-ce6bcd401f87-kube-api-access-4szjb" pod "35018750-1063-4fb6-928f-ce6bcd401f87" (UID: "35018750-1063-4fb6-928f-ce6bcd401f87"). InnerVolumeSpecName "kube-api-access-4szjb". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 20:00:21.134257 ip-10-0-132-82 kubenswrapper[4892]: I1030 20:00:21.134231    4892 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d16a477c-ad96-4363-b13d-b6d374f3f88d-kube-api-access-d99hj" pod "d16a477c-ad96-4363-b13d-b6d374f3f88d" (UID: "d16a477c-ad96-4363-b13d-b6d374f3f88d"). InnerVolumeSpecName "kube-api-access-d99hj". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 20:00:21.134257 ip-10-0-132-82 kubenswrapper[4892]: I1030 20:00:21.134243    4892 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0b0b9f79-e0a9-41a1-b513-01ea5661f64d-kube-api-access-plr9z" pod "0b0b9f79-e0a9-41a1-b513-01ea5661f64d" (UID: "0b0b9f79-e0a9-41a1-b513-01ea5661f64d"). InnerVolumeSpecName "kube-api-access-plr9z". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 20:00:21.134578 ip-10-0-132-82 kubenswrapper[4892]: I1030 20:00:21.134557    4892 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/82b75f10-16db-40c1-9696-f1e7f9917419-kube-api-access-wq78b" pod "82b75f10-16db-40c1-9696-f1e7f9917419" (UID: "82b75f10-16db-40c1-9696-f1e7f9917419"). InnerVolumeSpecName "kube-api-access-wq78b". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 20:00:21.233910 ip-10-0-132-82 kubenswrapper[4892]: I1030 20:00:21.233876    4892 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-d99hj\" (UniqueName: \"kubernetes.io/projected/d16a477c-ad96-4363-b13d-b6d374f3f88d-kube-api-access-d99hj\") on node \"ip-10-0-132-82.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:00:21.233910 ip-10-0-132-82 kubenswrapper[4892]: I1030 20:00:21.233904    4892 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-plr9z\" (UniqueName: \"kubernetes.io/projected/0b0b9f79-e0a9-41a1-b513-01ea5661f64d-kube-api-access-plr9z\") on node \"ip-10-0-132-82.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:00:21.233910 ip-10-0-132-82 kubenswrapper[4892]: I1030 20:00:21.233914    4892 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4szjb\" (UniqueName: \"kubernetes.io/projected/35018750-1063-4fb6-928f-ce6bcd401f87-kube-api-access-4szjb\") on node \"ip-10-0-132-82.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:00:21.234134 ip-10-0-132-82 kubenswrapper[4892]: I1030 20:00:21.233924    4892 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wq78b\" (UniqueName: \"kubernetes.io/projected/82b75f10-16db-40c1-9696-f1e7f9917419-kube-api-access-wq78b\") on node \"ip-10-0-132-82.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:00:21.480179 ip-10-0-132-82 kubenswrapper[4892]: I1030 20:00:21.478828    4892 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-mt-dd9b8d79/e2e-lifecycle-model-dd9b8d79-kserve-7659b4b7d9-r2flr" podUID="e469e8c3-e158-4b45-9135-6da275fc897f" containerName="main" probeResult="failure" output="Get \"https://10.132.0.180:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 20:00:21.657085 ip-10-0-132-82 kubenswrapper[4892]: I1030 20:00:21.657052    4892 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w5-686689/distinct-2-w5-686689-kserve-5ff578b4cd-rkc6s"]
Oct 30 20:00:21.662669 ip-10-0-132-82 kubenswrapper[4892]: I1030 20:00:21.662641    4892 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w5-686689/distinct-2-w5-686689-kserve-5ff578b4cd-rkc6s"]
Oct 30 20:00:21.733198 ip-10-0-132-82 kubenswrapper[4892]: I1030 20:00:21.733090    4892 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-c75ed4-29889840-7s578"
Oct 30 20:00:21.733655 ip-10-0-132-82 kubenswrapper[4892]: I1030 20:00:21.733126    4892 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-c75ed4-29889840-7s578" event={"ID":"8152ed9d-b608-4a87-aded-8dda2d975174","Type":"ContainerDied","Data":"d2e90f93ba7131e77d021830200df8aadbaa4661a849a8db6d827eca1f9a58c3"}
Oct 30 20:00:21.733655 ip-10-0-132-82 kubenswrapper[4892]: I1030 20:00:21.733491    4892 scope.go:122] "RemoveContainer" containerID="2aec635d950d0ef0bf65c6c0436980ebbbf91d78ec13ed07e1007cb1c61cb2da"
Oct 30 20:00:21.735374 ip-10-0-132-82 kubenswrapper[4892]: I1030 20:00:21.735300    4892 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-686689-29889840-l2mjk" event={"ID":"0b0b9f79-e0a9-41a1-b513-01ea5661f64d","Type":"ContainerDied","Data":"d05e464b89f5938f49369002fc5c8ca50342b826202ed5ef61cef2ab05904ad1"}
Oct 30 20:00:21.735610 ip-10-0-132-82 kubenswrapper[4892]: I1030 20:00:21.735577    4892 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-686689-29889840-l2mjk"
Oct 30 20:00:21.743280 ip-10-0-132-82 kubenswrapper[4892]: I1030 20:00:21.743261    4892 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-2d9861-29889840-79j62"
Oct 30 20:00:21.743449 ip-10-0-132-82 kubenswrapper[4892]: I1030 20:00:21.743429    4892 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-2d9861-29889840-79j62" event={"ID":"82b75f10-16db-40c1-9696-f1e7f9917419","Type":"ContainerDied","Data":"690ad5b700290187bdac89f0202ffbb6fc413182754b33cae1d048909b1e4a1d"}
Oct 30 20:00:21.747798 ip-10-0-132-82 kubenswrapper[4892]: I1030 20:00:21.747777    4892 scope.go:122] "RemoveContainer" containerID="dede5f77a2988363346a4b2ddecca528d3d145df64acf7739027053c46c78001"
Oct 30 20:00:21.748405 ip-10-0-132-82 kubenswrapper[4892]: I1030 20:00:21.748386    4892 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-132119-29889840-dhpf8"
Oct 30 20:00:21.748499 ip-10-0-132-82 kubenswrapper[4892]: I1030 20:00:21.748387    4892 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-132119-29889840-dhpf8" event={"ID":"d16a477c-ad96-4363-b13d-b6d374f3f88d","Type":"ContainerDied","Data":"5f067ece6d4879d86ea9bff3b58dc0b63c59fa98fbbdfad427f042e99afc802e"}
Oct 30 20:00:21.750032 ip-10-0-132-82 kubenswrapper[4892]: I1030 20:00:21.750010    4892 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-ada8d7-29889840-xldh2" event={"ID":"35018750-1063-4fb6-928f-ce6bcd401f87","Type":"ContainerDied","Data":"0da9cc8c2cfd1bf0db28f2e7bf2de4e4dd90ac502bd11c1e17b912409f4e6ac7"}
Oct 30 20:00:21.750284 ip-10-0-132-82 kubenswrapper[4892]: I1030 20:00:21.750263    4892 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-ada8d7-29889840-xldh2"
Oct 30 20:00:21.760656 ip-10-0-132-82 kubenswrapper[4892]: I1030 20:00:21.760589    4892 scope.go:122] "RemoveContainer" containerID="d153ca52e4bef4921aecf64bf745d0709d0ecc3d11bad89701c1626745aff774"
Oct 30 20:00:21.770771 ip-10-0-132-82 kubenswrapper[4892]: I1030 20:00:21.770753    4892 scope.go:122] "RemoveContainer" containerID="e8dbb29bc0bd55a5da1e44d40a0795a67ee54ba33779371f263abe5ddd529ad2"
Oct 30 20:00:21.781352 ip-10-0-132-82 kubenswrapper[4892]: I1030 20:00:21.781328    4892 scope.go:122] "RemoveContainer" containerID="2c871a60e44b312e5946a37677af97cb215c9bab0c88ad4e332c408a13a5068c"
Oct 30 20:00:21.786424 ip-10-0-132-82 kubenswrapper[4892]: I1030 20:00:21.786404    4892 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-f8a9d8-676dfc58bc-5jxrg"]
Oct 30 20:00:21.787028 ip-10-0-132-82 kubenswrapper[4892]: I1030 20:00:21.787012    4892 cpu_manager.go:414] "containerMap: removing container" podUID="35018750-1063-4fb6-928f-ce6bcd401f87" containerName="cleanup"
Oct 30 20:00:21.787072 ip-10-0-132-82 kubenswrapper[4892]: I1030 20:00:21.787030    4892 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="35018750-1063-4fb6-928f-ce6bcd401f87" containerName="cleanup"
Oct 30 20:00:21.787072 ip-10-0-132-82 kubenswrapper[4892]: I1030 20:00:21.787059    4892 cpu_manager.go:414] "containerMap: removing container" podUID="82b75f10-16db-40c1-9696-f1e7f9917419" containerName="cleanup"
Oct 30 20:00:21.787072 ip-10-0-132-82 kubenswrapper[4892]: I1030 20:00:21.787064    4892 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="82b75f10-16db-40c1-9696-f1e7f9917419" containerName="cleanup"
Oct 30 20:00:21.787175 ip-10-0-132-82 kubenswrapper[4892]: I1030 20:00:21.787073    4892 cpu_manager.go:414] "containerMap: removing container" podUID="a8829732-ac6f-4238-b8ce-fc781084668b" containerName="main"
Oct 30 20:00:21.787175 ip-10-0-132-82 kubenswrapper[4892]: I1030 20:00:21.787078    4892 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a8829732-ac6f-4238-b8ce-fc781084668b" containerName="main"
Oct 30 20:00:21.787175 ip-10-0-132-82 kubenswrapper[4892]: I1030 20:00:21.787087    4892 cpu_manager.go:414] "containerMap: removing container" podUID="e469e8c3-e158-4b45-9135-6da275fc897f" containerName="main"
Oct 30 20:00:21.787175 ip-10-0-132-82 kubenswrapper[4892]: I1030 20:00:21.787091    4892 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e469e8c3-e158-4b45-9135-6da275fc897f" containerName="main"
Oct 30 20:00:21.787175 ip-10-0-132-82 kubenswrapper[4892]: I1030 20:00:21.787100    4892 cpu_manager.go:414] "containerMap: removing container" podUID="d16a477c-ad96-4363-b13d-b6d374f3f88d" containerName="cleanup"
Oct 30 20:00:21.787175 ip-10-0-132-82 kubenswrapper[4892]: I1030 20:00:21.787104    4892 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d16a477c-ad96-4363-b13d-b6d374f3f88d" containerName="cleanup"
Oct 30 20:00:21.787175 ip-10-0-132-82 kubenswrapper[4892]: I1030 20:00:21.787111    4892 cpu_manager.go:414] "containerMap: removing container" podUID="25d4aedf-735a-42e6-ae08-4561c967fc35" containerName="cleanup"
Oct 30 20:00:21.787175 ip-10-0-132-82 kubenswrapper[4892]: I1030 20:00:21.787116    4892 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="25d4aedf-735a-42e6-ae08-4561c967fc35" containerName="cleanup"
Oct 30 20:00:21.787175 ip-10-0-132-82 kubenswrapper[4892]: I1030 20:00:21.787123    4892 cpu_manager.go:414] "containerMap: removing container" podUID="8152ed9d-b608-4a87-aded-8dda2d975174" containerName="cleanup"
Oct 30 20:00:21.787175 ip-10-0-132-82 kubenswrapper[4892]: I1030 20:00:21.787127    4892 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8152ed9d-b608-4a87-aded-8dda2d975174" containerName="cleanup"
Oct 30 20:00:21.787175 ip-10-0-132-82 kubenswrapper[4892]: I1030 20:00:21.787133    4892 cpu_manager.go:414] "containerMap: removing container" podUID="0b0b9f79-e0a9-41a1-b513-01ea5661f64d" containerName="cleanup"
Oct 30 20:00:21.787175 ip-10-0-132-82 kubenswrapper[4892]: I1030 20:00:21.787137    4892 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0b0b9f79-e0a9-41a1-b513-01ea5661f64d" containerName="cleanup"
Oct 30 20:00:21.787175 ip-10-0-132-82 kubenswrapper[4892]: I1030 20:00:21.787142    4892 cpu_manager.go:414] "containerMap: removing container" podUID="d16a477c-ad96-4363-b13d-b6d374f3f88d" containerName="cleanup"
Oct 30 20:00:21.787175 ip-10-0-132-82 kubenswrapper[4892]: I1030 20:00:21.787147    4892 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d16a477c-ad96-4363-b13d-b6d374f3f88d" containerName="cleanup"
Oct 30 20:00:21.787175 ip-10-0-132-82 kubenswrapper[4892]: I1030 20:00:21.787156    4892 cpu_manager.go:414] "containerMap: removing container" podUID="3cb47a99-8be1-4756-b31a-43b58cc6b8dc" containerName="main"
Oct 30 20:00:21.787175 ip-10-0-132-82 kubenswrapper[4892]: I1030 20:00:21.787160    4892 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3cb47a99-8be1-4756-b31a-43b58cc6b8dc" containerName="main"
Oct 30 20:00:21.787175 ip-10-0-132-82 kubenswrapper[4892]: I1030 20:00:21.787166    4892 cpu_manager.go:414] "containerMap: removing container" podUID="e453311b-d7b1-4819-a57a-d1de4886de26" containerName="istio-proxy"
Oct 30 20:00:21.787175 ip-10-0-132-82 kubenswrapper[4892]: I1030 20:00:21.787172    4892 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e453311b-d7b1-4819-a57a-d1de4886de26" containerName="istio-proxy"
Oct 30 20:00:21.787702 ip-10-0-132-82 kubenswrapper[4892]: I1030 20:00:21.787184    4892 cpu_manager.go:414] "containerMap: removing container" podUID="f6260396-890c-44ea-be8c-998dd2086ffa" containerName="cleanup"
Oct 30 20:00:21.787702 ip-10-0-132-82 kubenswrapper[4892]: I1030 20:00:21.787189    4892 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f6260396-890c-44ea-be8c-998dd2086ffa" containerName="cleanup"
Oct 30 20:00:21.787702 ip-10-0-132-82 kubenswrapper[4892]: I1030 20:00:21.787195    4892 cpu_manager.go:414] "containerMap: removing container" podUID="191593a2-e3a9-4ce4-a1c6-c089723b0237" containerName="main"
Oct 30 20:00:21.787702 ip-10-0-132-82 kubenswrapper[4892]: I1030 20:00:21.787199    4892 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="191593a2-e3a9-4ce4-a1c6-c089723b0237" containerName="main"
Oct 30 20:00:21.787702 ip-10-0-132-82 kubenswrapper[4892]: I1030 20:00:21.787210    4892 cpu_manager.go:414] "containerMap: removing container" podUID="8152ed9d-b608-4a87-aded-8dda2d975174" containerName="cleanup"
Oct 30 20:00:21.787702 ip-10-0-132-82 kubenswrapper[4892]: I1030 20:00:21.787214    4892 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8152ed9d-b608-4a87-aded-8dda2d975174" containerName="cleanup"
Oct 30 20:00:21.787702 ip-10-0-132-82 kubenswrapper[4892]: I1030 20:00:21.787220    4892 cpu_manager.go:414] "containerMap: removing container" podUID="9df510ae-c995-4678-816a-9c4834778991" containerName="cleanup"
Oct 30 20:00:21.787702 ip-10-0-132-82 kubenswrapper[4892]: I1030 20:00:21.787224    4892 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9df510ae-c995-4678-816a-9c4834778991" containerName="cleanup"
Oct 30 20:00:21.787702 ip-10-0-132-82 kubenswrapper[4892]: I1030 20:00:21.787229    4892 cpu_manager.go:414] "containerMap: removing container" podUID="e5edd2c3-9676-48a1-9b30-8d8a98512582" containerName="revoke-keys"
Oct 30 20:00:21.787702 ip-10-0-132-82 kubenswrapper[4892]: I1030 20:00:21.787233    4892 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e5edd2c3-9676-48a1-9b30-8d8a98512582" containerName="revoke-keys"
Oct 30 20:00:21.787702 ip-10-0-132-82 kubenswrapper[4892]: I1030 20:00:21.787239    4892 cpu_manager.go:414] "containerMap: removing container" podUID="f6260396-890c-44ea-be8c-998dd2086ffa" containerName="cleanup"
Oct 30 20:00:21.787702 ip-10-0-132-82 kubenswrapper[4892]: I1030 20:00:21.787244    4892 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f6260396-890c-44ea-be8c-998dd2086ffa" containerName="cleanup"
Oct 30 20:00:21.787702 ip-10-0-132-82 kubenswrapper[4892]: I1030 20:00:21.787250    4892 cpu_manager.go:414] "containerMap: removing container" podUID="82b75f10-16db-40c1-9696-f1e7f9917419" containerName="cleanup"
Oct 30 20:00:21.787702 ip-10-0-132-82 kubenswrapper[4892]: I1030 20:00:21.787254    4892 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="82b75f10-16db-40c1-9696-f1e7f9917419" containerName="cleanup"
Oct 30 20:00:21.787702 ip-10-0-132-82 kubenswrapper[4892]: I1030 20:00:21.787261    4892 cpu_manager.go:414] "containerMap: removing container" podUID="12863020-d0ea-419d-aba2-816627ab0431" containerName="payload-processing"
Oct 30 20:00:21.787702 ip-10-0-132-82 kubenswrapper[4892]: I1030 20:00:21.787265    4892 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="12863020-d0ea-419d-aba2-816627ab0431" containerName="payload-processing"
Oct 30 20:00:21.787702 ip-10-0-132-82 kubenswrapper[4892]: I1030 20:00:21.787278    4892 cpu_manager.go:414] "containerMap: removing container" podUID="35018750-1063-4fb6-928f-ce6bcd401f87" containerName="cleanup"
Oct 30 20:00:21.787702 ip-10-0-132-82 kubenswrapper[4892]: I1030 20:00:21.787282    4892 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="35018750-1063-4fb6-928f-ce6bcd401f87" containerName="cleanup"
Oct 30 20:00:21.787702 ip-10-0-132-82 kubenswrapper[4892]: I1030 20:00:21.787289    4892 cpu_manager.go:414] "containerMap: removing container" podUID="8152ed9d-b608-4a87-aded-8dda2d975174" containerName="cleanup"
Oct 30 20:00:21.787702 ip-10-0-132-82 kubenswrapper[4892]: I1030 20:00:21.787293    4892 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8152ed9d-b608-4a87-aded-8dda2d975174" containerName="cleanup"
Oct 30 20:00:21.787702 ip-10-0-132-82 kubenswrapper[4892]: I1030 20:00:21.787305    4892 cpu_manager.go:414] "containerMap: removing container" podUID="f1d57e1b-e316-48ac-a1a5-a9520a9e068e" containerName="payload-pre-processing"
Oct 30 20:00:21.787702 ip-10-0-132-82 kubenswrapper[4892]: I1030 20:00:21.787310    4892 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f1d57e1b-e316-48ac-a1a5-a9520a9e068e" containerName="payload-pre-processing"
Oct 30 20:00:21.787702 ip-10-0-132-82 kubenswrapper[4892]: I1030 20:00:21.787321    4892 cpu_manager.go:414] "containerMap: removing container" podUID="25d4aedf-735a-42e6-ae08-4561c967fc35" containerName="cleanup"
Oct 30 20:00:21.787702 ip-10-0-132-82 kubenswrapper[4892]: I1030 20:00:21.787325    4892 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="25d4aedf-735a-42e6-ae08-4561c967fc35" containerName="cleanup"
Oct 30 20:00:21.787702 ip-10-0-132-82 kubenswrapper[4892]: I1030 20:00:21.787332    4892 cpu_manager.go:414] "containerMap: removing container" podUID="9df510ae-c995-4678-816a-9c4834778991" containerName="cleanup"
Oct 30 20:00:21.787702 ip-10-0-132-82 kubenswrapper[4892]: I1030 20:00:21.787336    4892 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9df510ae-c995-4678-816a-9c4834778991" containerName="cleanup"
Oct 30 20:00:21.787702 ip-10-0-132-82 kubenswrapper[4892]: I1030 20:00:21.787343    4892 cpu_manager.go:414] "containerMap: removing container" podUID="9df510ae-c995-4678-816a-9c4834778991" containerName="cleanup"
Oct 30 20:00:21.787702 ip-10-0-132-82 kubenswrapper[4892]: I1030 20:00:21.787347    4892 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9df510ae-c995-4678-816a-9c4834778991" containerName="cleanup"
Oct 30 20:00:21.787702 ip-10-0-132-82 kubenswrapper[4892]: I1030 20:00:21.787354    4892 cpu_manager.go:414] "containerMap: removing container" podUID="0b0b9f79-e0a9-41a1-b513-01ea5661f64d" containerName="cleanup"
Oct 30 20:00:21.787702 ip-10-0-132-82 kubenswrapper[4892]: I1030 20:00:21.787358    4892 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0b0b9f79-e0a9-41a1-b513-01ea5661f64d" containerName="cleanup"
Oct 30 20:00:21.787702 ip-10-0-132-82 kubenswrapper[4892]: I1030 20:00:21.787431    4892 memory_manager.go:365] "RemoveStaleState removing state" podUID="f1d57e1b-e316-48ac-a1a5-a9520a9e068e" containerName="payload-pre-processing"
Oct 30 20:00:21.787702 ip-10-0-132-82 kubenswrapper[4892]: I1030 20:00:21.787440    4892 memory_manager.go:365] "RemoveStaleState removing state" podUID="25d4aedf-735a-42e6-ae08-4561c967fc35" containerName="cleanup"
Oct 30 20:00:21.787702 ip-10-0-132-82 kubenswrapper[4892]: I1030 20:00:21.787447    4892 memory_manager.go:365] "RemoveStaleState removing state" podUID="9df510ae-c995-4678-816a-9c4834778991" containerName="cleanup"
Oct 30 20:00:21.787702 ip-10-0-132-82 kubenswrapper[4892]: I1030 20:00:21.787452    4892 memory_manager.go:365] "RemoveStaleState removing state" podUID="9df510ae-c995-4678-816a-9c4834778991" containerName="cleanup"
Oct 30 20:00:21.787702 ip-10-0-132-82 kubenswrapper[4892]: I1030 20:00:21.787457    4892 memory_manager.go:365] "RemoveStaleState removing state" podUID="0b0b9f79-e0a9-41a1-b513-01ea5661f64d" containerName="cleanup"
Oct 30 20:00:21.787702 ip-10-0-132-82 kubenswrapper[4892]: I1030 20:00:21.787462    4892 memory_manager.go:365] "RemoveStaleState removing state" podUID="f6260396-890c-44ea-be8c-998dd2086ffa" containerName="cleanup"
Oct 30 20:00:21.787702 ip-10-0-132-82 kubenswrapper[4892]: I1030 20:00:21.787466    4892 memory_manager.go:365] "RemoveStaleState removing state" podUID="35018750-1063-4fb6-928f-ce6bcd401f87" containerName="cleanup"
Oct 30 20:00:21.787702 ip-10-0-132-82 kubenswrapper[4892]: I1030 20:00:21.787471    4892 memory_manager.go:365] "RemoveStaleState removing state" podUID="82b75f10-16db-40c1-9696-f1e7f9917419" containerName="cleanup"
Oct 30 20:00:21.787702 ip-10-0-132-82 kubenswrapper[4892]: I1030 20:00:21.787477    4892 memory_manager.go:365] "RemoveStaleState removing state" podUID="a8829732-ac6f-4238-b8ce-fc781084668b" containerName="main"
Oct 30 20:00:21.787702 ip-10-0-132-82 kubenswrapper[4892]: I1030 20:00:21.787482    4892 memory_manager.go:365] "RemoveStaleState removing state" podUID="e469e8c3-e158-4b45-9135-6da275fc897f" containerName="main"
Oct 30 20:00:21.787702 ip-10-0-132-82 kubenswrapper[4892]: I1030 20:00:21.787487    4892 memory_manager.go:365] "RemoveStaleState removing state" podUID="d16a477c-ad96-4363-b13d-b6d374f3f88d" containerName="cleanup"
Oct 30 20:00:21.787702 ip-10-0-132-82 kubenswrapper[4892]: I1030 20:00:21.787491    4892 memory_manager.go:365] "RemoveStaleState removing state" podUID="25d4aedf-735a-42e6-ae08-4561c967fc35" containerName="cleanup"
Oct 30 20:00:21.787702 ip-10-0-132-82 kubenswrapper[4892]: I1030 20:00:21.787496    4892 memory_manager.go:365] "RemoveStaleState removing state" podUID="8152ed9d-b608-4a87-aded-8dda2d975174" containerName="cleanup"
Oct 30 20:00:21.787702 ip-10-0-132-82 kubenswrapper[4892]: I1030 20:00:21.787501    4892 memory_manager.go:365] "RemoveStaleState removing state" podUID="0b0b9f79-e0a9-41a1-b513-01ea5661f64d" containerName="cleanup"
Oct 30 20:00:21.787702 ip-10-0-132-82 kubenswrapper[4892]: I1030 20:00:21.787506    4892 memory_manager.go:365] "RemoveStaleState removing state" podUID="3cb47a99-8be1-4756-b31a-43b58cc6b8dc" containerName="main"
Oct 30 20:00:21.787702 ip-10-0-132-82 kubenswrapper[4892]: I1030 20:00:21.787511    4892 memory_manager.go:365] "RemoveStaleState removing state" podUID="d16a477c-ad96-4363-b13d-b6d374f3f88d" containerName="cleanup"
Oct 30 20:00:21.787702 ip-10-0-132-82 kubenswrapper[4892]: I1030 20:00:21.787516    4892 memory_manager.go:365] "RemoveStaleState removing state" podUID="e453311b-d7b1-4819-a57a-d1de4886de26" containerName="istio-proxy"
Oct 30 20:00:21.787702 ip-10-0-132-82 kubenswrapper[4892]: I1030 20:00:21.787521    4892 memory_manager.go:365] "RemoveStaleState removing state" podUID="f6260396-890c-44ea-be8c-998dd2086ffa" containerName="cleanup"
Oct 30 20:00:21.787702 ip-10-0-132-82 kubenswrapper[4892]: I1030 20:00:21.787525    4892 memory_manager.go:365] "RemoveStaleState removing state" podUID="191593a2-e3a9-4ce4-a1c6-c089723b0237" containerName="main"
Oct 30 20:00:21.787702 ip-10-0-132-82 kubenswrapper[4892]: I1030 20:00:21.787532    4892 memory_manager.go:365] "RemoveStaleState removing state" podUID="8152ed9d-b608-4a87-aded-8dda2d975174" containerName="cleanup"
Oct 30 20:00:21.787702 ip-10-0-132-82 kubenswrapper[4892]: I1030 20:00:21.787538    4892 memory_manager.go:365] "RemoveStaleState removing state" podUID="9df510ae-c995-4678-816a-9c4834778991" containerName="cleanup"
Oct 30 20:00:21.787702 ip-10-0-132-82 kubenswrapper[4892]: I1030 20:00:21.787543    4892 memory_manager.go:365] "RemoveStaleState removing state" podUID="e5edd2c3-9676-48a1-9b30-8d8a98512582" containerName="revoke-keys"
Oct 30 20:00:21.787702 ip-10-0-132-82 kubenswrapper[4892]: I1030 20:00:21.787549    4892 memory_manager.go:365] "RemoveStaleState removing state" podUID="82b75f10-16db-40c1-9696-f1e7f9917419" containerName="cleanup"
Oct 30 20:00:21.787702 ip-10-0-132-82 kubenswrapper[4892]: I1030 20:00:21.787555    4892 memory_manager.go:365] "RemoveStaleState removing state" podUID="12863020-d0ea-419d-aba2-816627ab0431" containerName="payload-processing"
Oct 30 20:00:21.787702 ip-10-0-132-82 kubenswrapper[4892]: I1030 20:00:21.787562    4892 memory_manager.go:365] "RemoveStaleState removing state" podUID="35018750-1063-4fb6-928f-ce6bcd401f87" containerName="cleanup"
Oct 30 20:00:21.792451 ip-10-0-132-82 kubenswrapper[4892]: I1030 20:00:21.792430    4892 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-f8a9d8-676dfc58bc-5jxrg"
Oct 30 20:00:21.850992 ip-10-0-132-82 kubenswrapper[4892]: I1030 20:00:21.840091    4892 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w0-6ac304-79859bfff5-zx2hb"]
Oct 30 20:00:21.850992 ip-10-0-132-82 kubenswrapper[4892]: I1030 20:00:21.844833    4892 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-inf-b-f8a9d8\""
Oct 30 20:00:21.850992 ip-10-0-132-82 kubenswrapper[4892]: I1030 20:00:21.844935    4892 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-inf-b-f8a9d8-dockercfg-rp74z\""
Oct 30 20:00:21.905415 ip-10-0-132-82 kubenswrapper[4892]: I1030 20:00:21.905381    4892 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w0-6ac304-79859bfff5-zx2hb"]
Oct 30 20:00:21.905686 ip-10-0-132-82 kubenswrapper[4892]: I1030 20:00:21.905669    4892 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-f8a9d8-676dfc58bc-5jxrg"]
Oct 30 20:00:21.946857 ip-10-0-132-82 kubenswrapper[4892]: I1030 20:00:21.946822    4892 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-f8a9d8-d9dccd5b6-h6wsr"]
Oct 30 20:00:21.947691 ip-10-0-132-82 kubenswrapper[4892]: I1030 20:00:21.947667    4892 cpu_manager.go:414] "containerMap: removing container" podUID="f6260396-890c-44ea-be8c-998dd2086ffa" containerName="cleanup"
Oct 30 20:00:21.947874 ip-10-0-132-82 kubenswrapper[4892]: I1030 20:00:21.947858    4892 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f6260396-890c-44ea-be8c-998dd2086ffa" containerName="cleanup"
Oct 30 20:00:21.948010 ip-10-0-132-82 kubenswrapper[4892]: I1030 20:00:21.947998    4892 cpu_manager.go:414] "containerMap: removing container" podUID="35018750-1063-4fb6-928f-ce6bcd401f87" containerName="cleanup"
Oct 30 20:00:21.948094 ip-10-0-132-82 kubenswrapper[4892]: I1030 20:00:21.948085    4892 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="35018750-1063-4fb6-928f-ce6bcd401f87" containerName="cleanup"
Oct 30 20:00:21.948272 ip-10-0-132-82 kubenswrapper[4892]: I1030 20:00:21.948213    4892 cpu_manager.go:414] "containerMap: removing container" podUID="82b75f10-16db-40c1-9696-f1e7f9917419" containerName="cleanup"
Oct 30 20:00:21.948375 ip-10-0-132-82 kubenswrapper[4892]: I1030 20:00:21.948363    4892 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="82b75f10-16db-40c1-9696-f1e7f9917419" containerName="cleanup"
Oct 30 20:00:21.948454 ip-10-0-132-82 kubenswrapper[4892]: I1030 20:00:21.948445    4892 cpu_manager.go:414] "containerMap: removing container" podUID="25d4aedf-735a-42e6-ae08-4561c967fc35" containerName="cleanup"
Oct 30 20:00:21.948547 ip-10-0-132-82 kubenswrapper[4892]: I1030 20:00:21.948536    4892 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="25d4aedf-735a-42e6-ae08-4561c967fc35" containerName="cleanup"
Oct 30 20:00:21.948906 ip-10-0-132-82 kubenswrapper[4892]: I1030 20:00:21.948881    4892 cpu_manager.go:414] "containerMap: removing container" podUID="0b0b9f79-e0a9-41a1-b513-01ea5661f64d" containerName="cleanup"
Oct 30 20:00:21.948906 ip-10-0-132-82 kubenswrapper[4892]: I1030 20:00:21.948905    4892 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0b0b9f79-e0a9-41a1-b513-01ea5661f64d" containerName="cleanup"
Oct 30 20:00:21.949119 ip-10-0-132-82 kubenswrapper[4892]: I1030 20:00:21.948940    4892 cpu_manager.go:414] "containerMap: removing container" podUID="d16a477c-ad96-4363-b13d-b6d374f3f88d" containerName="cleanup"
Oct 30 20:00:21.949119 ip-10-0-132-82 kubenswrapper[4892]: I1030 20:00:21.948949    4892 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d16a477c-ad96-4363-b13d-b6d374f3f88d" containerName="cleanup"
Oct 30 20:00:21.949119 ip-10-0-132-82 kubenswrapper[4892]: I1030 20:00:21.949116    4892 memory_manager.go:365] "RemoveStaleState removing state" podUID="d16a477c-ad96-4363-b13d-b6d374f3f88d" containerName="cleanup"
Oct 30 20:00:21.949296 ip-10-0-132-82 kubenswrapper[4892]: I1030 20:00:21.949133    4892 memory_manager.go:365] "RemoveStaleState removing state" podUID="0b0b9f79-e0a9-41a1-b513-01ea5661f64d" containerName="cleanup"
Oct 30 20:00:21.949296 ip-10-0-132-82 kubenswrapper[4892]: I1030 20:00:21.949145    4892 memory_manager.go:365] "RemoveStaleState removing state" podUID="f6260396-890c-44ea-be8c-998dd2086ffa" containerName="cleanup"
Oct 30 20:00:21.949296 ip-10-0-132-82 kubenswrapper[4892]: I1030 20:00:21.949156    4892 memory_manager.go:365] "RemoveStaleState removing state" podUID="8152ed9d-b608-4a87-aded-8dda2d975174" containerName="cleanup"
Oct 30 20:00:21.949296 ip-10-0-132-82 kubenswrapper[4892]: I1030 20:00:21.949164    4892 memory_manager.go:365] "RemoveStaleState removing state" podUID="82b75f10-16db-40c1-9696-f1e7f9917419" containerName="cleanup"
Oct 30 20:00:21.949296 ip-10-0-132-82 kubenswrapper[4892]: I1030 20:00:21.949180    4892 memory_manager.go:365] "RemoveStaleState removing state" podUID="25d4aedf-735a-42e6-ae08-4561c967fc35" containerName="cleanup"
Oct 30 20:00:21.949296 ip-10-0-132-82 kubenswrapper[4892]: I1030 20:00:21.949191    4892 memory_manager.go:365] "RemoveStaleState removing state" podUID="35018750-1063-4fb6-928f-ce6bcd401f87" containerName="cleanup"
Oct 30 20:00:21.952005 ip-10-0-132-82 kubenswrapper[4892]: I1030 20:00:21.951919    4892 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5f415c4a-4dda-40e3-bdae-8ad67ad15bdf-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-b-f8a9d8-676dfc58bc-5jxrg\" (UID: \"5f415c4a-4dda-40e3-bdae-8ad67ad15bdf\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-f8a9d8-676dfc58bc-5jxrg"
Oct 30 20:00:21.952125 ip-10-0-132-82 kubenswrapper[4892]: I1030 20:00:21.952003    4892 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vrdbm\" (UniqueName: \"kubernetes.io/projected/5f415c4a-4dda-40e3-bdae-8ad67ad15bdf-kube-api-access-vrdbm\") pod \"payload-pre-processing-e2e-inf-b-f8a9d8-676dfc58bc-5jxrg\" (UID: \"5f415c4a-4dda-40e3-bdae-8ad67ad15bdf\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-f8a9d8-676dfc58bc-5jxrg"
Oct 30 20:00:21.956153 ip-10-0-132-82 kubenswrapper[4892]: I1030 20:00:21.956123    4892 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-b-f8a9d8-d9dccd5b6-h6wsr"
Oct 30 20:00:21.968286 ip-10-0-132-82 kubenswrapper[4892]: I1030 20:00:21.963787    4892 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w0-6ac304-846999954d-tq2ft"]
Oct 30 20:00:21.971061 ip-10-0-132-82 kubenswrapper[4892]: I1030 20:00:21.970820    4892 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w0-6ac304-846999954d-tq2ft"]
Oct 30 20:00:22.010197 ip-10-0-132-82 kubenswrapper[4892]: I1030 20:00:22.010139    4892 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-f8a9d8-d9dccd5b6-h6wsr"]
Oct 30 20:00:22.055054 ip-10-0-132-82 kubenswrapper[4892]: I1030 20:00:22.055027    4892 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/92e73440-f470-44d9-983e-2d38334d834c-plugins-config-volume\") pod \"payload-processing-e2e-inf-b-f8a9d8-d9dccd5b6-h6wsr\" (UID: \"92e73440-f470-44d9-983e-2d38334d834c\") " pod="openshift-ingress/payload-processing-e2e-inf-b-f8a9d8-d9dccd5b6-h6wsr"
Oct 30 20:00:22.055241 ip-10-0-132-82 kubenswrapper[4892]: I1030 20:00:22.055072    4892 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4cnpw\" (UniqueName: \"kubernetes.io/projected/92e73440-f470-44d9-983e-2d38334d834c-kube-api-access-4cnpw\") pod \"payload-processing-e2e-inf-b-f8a9d8-d9dccd5b6-h6wsr\" (UID: \"92e73440-f470-44d9-983e-2d38334d834c\") " pod="openshift-ingress/payload-processing-e2e-inf-b-f8a9d8-d9dccd5b6-h6wsr"
Oct 30 20:00:22.055241 ip-10-0-132-82 kubenswrapper[4892]: I1030 20:00:22.055146    4892 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5f415c4a-4dda-40e3-bdae-8ad67ad15bdf-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-b-f8a9d8-676dfc58bc-5jxrg\" (UID: \"5f415c4a-4dda-40e3-bdae-8ad67ad15bdf\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-f8a9d8-676dfc58bc-5jxrg"
Oct 30 20:00:22.055241 ip-10-0-132-82 kubenswrapper[4892]: I1030 20:00:22.055168    4892 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vrdbm\" (UniqueName: \"kubernetes.io/projected/5f415c4a-4dda-40e3-bdae-8ad67ad15bdf-kube-api-access-vrdbm\") pod \"payload-pre-processing-e2e-inf-b-f8a9d8-676dfc58bc-5jxrg\" (UID: \"5f415c4a-4dda-40e3-bdae-8ad67ad15bdf\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-f8a9d8-676dfc58bc-5jxrg"
Oct 30 20:00:22.056014 ip-10-0-132-82 kubenswrapper[4892]: I1030 20:00:22.055988    4892 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5f415c4a-4dda-40e3-bdae-8ad67ad15bdf-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-b-f8a9d8-676dfc58bc-5jxrg\" (UID: \"5f415c4a-4dda-40e3-bdae-8ad67ad15bdf\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-f8a9d8-676dfc58bc-5jxrg"
Oct 30 20:00:22.112887 ip-10-0-132-82 kubenswrapper[4892]: I1030 20:00:22.112860    4892 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vrdbm\" (UniqueName: \"kubernetes.io/projected/5f415c4a-4dda-40e3-bdae-8ad67ad15bdf-kube-api-access-vrdbm\") pod \"payload-pre-processing-e2e-inf-b-f8a9d8-676dfc58bc-5jxrg\" (UID: \"5f415c4a-4dda-40e3-bdae-8ad67ad15bdf\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-f8a9d8-676dfc58bc-5jxrg"
Oct 30 20:00:22.156465 ip-10-0-132-82 kubenswrapper[4892]: I1030 20:00:22.156436    4892 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/92e73440-f470-44d9-983e-2d38334d834c-plugins-config-volume\") pod \"payload-processing-e2e-inf-b-f8a9d8-d9dccd5b6-h6wsr\" (UID: \"92e73440-f470-44d9-983e-2d38334d834c\") " pod="openshift-ingress/payload-processing-e2e-inf-b-f8a9d8-d9dccd5b6-h6wsr"
Oct 30 20:00:22.156616 ip-10-0-132-82 kubenswrapper[4892]: I1030 20:00:22.156485    4892 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4cnpw\" (UniqueName: \"kubernetes.io/projected/92e73440-f470-44d9-983e-2d38334d834c-kube-api-access-4cnpw\") pod \"payload-processing-e2e-inf-b-f8a9d8-d9dccd5b6-h6wsr\" (UID: \"92e73440-f470-44d9-983e-2d38334d834c\") " pod="openshift-ingress/payload-processing-e2e-inf-b-f8a9d8-d9dccd5b6-h6wsr"
Oct 30 20:00:22.157213 ip-10-0-132-82 kubenswrapper[4892]: I1030 20:00:22.157192    4892 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/92e73440-f470-44d9-983e-2d38334d834c-plugins-config-volume\") pod \"payload-processing-e2e-inf-b-f8a9d8-d9dccd5b6-h6wsr\" (UID: \"92e73440-f470-44d9-983e-2d38334d834c\") " pod="openshift-ingress/payload-processing-e2e-inf-b-f8a9d8-d9dccd5b6-h6wsr"
Oct 30 20:00:22.214251 ip-10-0-132-82 kubenswrapper[4892]: I1030 20:00:22.214224    4892 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4cnpw\" (UniqueName: \"kubernetes.io/projected/92e73440-f470-44d9-983e-2d38334d834c-kube-api-access-4cnpw\") pod \"payload-processing-e2e-inf-b-f8a9d8-d9dccd5b6-h6wsr\" (UID: \"92e73440-f470-44d9-983e-2d38334d834c\") " pod="openshift-ingress/payload-processing-e2e-inf-b-f8a9d8-d9dccd5b6-h6wsr"
Oct 30 20:00:22.286076 ip-10-0-132-82 kubenswrapper[4892]: I1030 20:00:22.286003    4892 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-b-f8a9d8-d9dccd5b6-h6wsr"
Oct 30 20:00:22.392044 ip-10-0-132-82 kubenswrapper[4892]: I1030 20:00:22.391102    4892 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-2d9861-29889840-79j62"]
Oct 30 20:00:22.395600 ip-10-0-132-82 kubenswrapper[4892]: I1030 20:00:22.395382    4892 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-2d9861-29889840-79j62"]
Oct 30 20:00:22.406838 ip-10-0-132-82 kubenswrapper[4892]: I1030 20:00:22.406815    4892 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-f8a9d8-676dfc58bc-5jxrg"
Oct 30 20:00:22.560712 ip-10-0-132-82 kubenswrapper[4892]: I1030 20:00:22.560633    4892 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-132119-29889840-dhpf8"]
Oct 30 20:00:22.571496 ip-10-0-132-82 kubenswrapper[4892]: I1030 20:00:22.571451    4892 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="12863020-d0ea-419d-aba2-816627ab0431" path="/var/lib/kubelet/pods/12863020-d0ea-419d-aba2-816627ab0431/volumes"
Oct 30 20:00:22.571950 ip-10-0-132-82 kubenswrapper[4892]: I1030 20:00:22.571928    4892 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="82b75f10-16db-40c1-9696-f1e7f9917419" path="/var/lib/kubelet/pods/82b75f10-16db-40c1-9696-f1e7f9917419/volumes"
Oct 30 20:00:22.572401 ip-10-0-132-82 kubenswrapper[4892]: I1030 20:00:22.572381    4892 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="9df510ae-c995-4678-816a-9c4834778991" path="/var/lib/kubelet/pods/9df510ae-c995-4678-816a-9c4834778991/volumes"
Oct 30 20:00:22.573177 ip-10-0-132-82 kubenswrapper[4892]: I1030 20:00:22.573156    4892 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a8829732-ac6f-4238-b8ce-fc781084668b" path="/var/lib/kubelet/pods/a8829732-ac6f-4238-b8ce-fc781084668b/volumes"
Oct 30 20:00:22.573603 ip-10-0-132-82 kubenswrapper[4892]: I1030 20:00:22.573580    4892 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f1d57e1b-e316-48ac-a1a5-a9520a9e068e" path="/var/lib/kubelet/pods/f1d57e1b-e316-48ac-a1a5-a9520a9e068e/volumes"
Oct 30 20:00:22.574003 ip-10-0-132-82 kubenswrapper[4892]: I1030 20:00:22.573959    4892 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-132119-29889840-dhpf8"]
Oct 30 20:00:22.714309 ip-10-0-132-82 kubenswrapper[4892]: W1030 20:00:22.714267    4892 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod92e73440_f470_44d9_983e_2d38334d834c.slice/crio-edf42999566353b1bbee45376492e373060a4e2db778c978b33af2c74757dbd7 WatchSource:0}: Error finding container edf42999566353b1bbee45376492e373060a4e2db778c978b33af2c74757dbd7: Status 404 returned error can't find the container with id edf42999566353b1bbee45376492e373060a4e2db778c978b33af2c74757dbd7
Oct 30 20:00:22.715523 ip-10-0-132-82 kubenswrapper[4892]: I1030 20:00:22.715495    4892 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-f8a9d8-d9dccd5b6-h6wsr"]
Oct 30 20:00:22.747880 ip-10-0-132-82 kubenswrapper[4892]: I1030 20:00:22.747850    4892 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-c75ed4-29889840-7s578"]
Oct 30 20:00:22.752942 ip-10-0-132-82 kubenswrapper[4892]: I1030 20:00:22.752916    4892 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-c75ed4-29889840-7s578"]
Oct 30 20:00:22.768899 ip-10-0-132-82 kubenswrapper[4892]: I1030 20:00:22.768875    4892 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-b-f8a9d8-d9dccd5b6-h6wsr" event={"ID":"92e73440-f470-44d9-983e-2d38334d834c","Type":"ContainerStarted","Data":"edf42999566353b1bbee45376492e373060a4e2db778c978b33af2c74757dbd7"}
Oct 30 20:00:22.799142 ip-10-0-132-82 kubenswrapper[4892]: I1030 20:00:22.799116    4892 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-f8a9d8-676dfc58bc-5jxrg"]
Oct 30 20:00:22.799615 ip-10-0-132-82 kubenswrapper[4892]: W1030 20:00:22.799590    4892 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5f415c4a_4dda_40e3_bdae_8ad67ad15bdf.slice/crio-d61ca2f9b84c2a345f2a91fb0a090622cfa74a58a5427b23164c516321f03de8 WatchSource:0}: Error finding container d61ca2f9b84c2a345f2a91fb0a090622cfa74a58a5427b23164c516321f03de8: Status 404 returned error can't find the container with id d61ca2f9b84c2a345f2a91fb0a090622cfa74a58a5427b23164c516321f03de8
Oct 30 20:00:22.929343 ip-10-0-132-82 kubenswrapper[4892]: I1030 20:00:22.929306    4892 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-ada8d7-29889840-xldh2"]
Oct 30 20:00:22.934493 ip-10-0-132-82 kubenswrapper[4892]: I1030 20:00:22.934467    4892 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-ada8d7-29889840-xldh2"]
Oct 30 20:00:23.155506 ip-10-0-132-82 kubenswrapper[4892]: I1030 20:00:23.155465    4892 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-mt-dd9b8d79/e2e-lifecycle-model-dd9b8d79-kserve-7659b4b7d9-r2flr"]
Oct 30 20:00:23.160066 ip-10-0-132-82 kubenswrapper[4892]: I1030 20:00:23.160041    4892 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-mt-dd9b8d79/e2e-lifecycle-model-dd9b8d79-kserve-7659b4b7d9-r2flr"]
Oct 30 20:00:23.468064 ip-10-0-132-82 kubenswrapper[4892]: I1030 20:00:23.467983    4892 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-686689-29889840-l2mjk"]
Oct 30 20:00:23.468986 ip-10-0-132-82 kubenswrapper[4892]: I1030 20:00:23.468939    4892 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-686689-29889840-l2mjk"]
Oct 30 20:00:23.782897 ip-10-0-132-82 kubenswrapper[4892]: I1030 20:00:23.782805    4892 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-f8a9d8-676dfc58bc-5jxrg" event={"ID":"5f415c4a-4dda-40e3-bdae-8ad67ad15bdf","Type":"ContainerStarted","Data":"e42503fda96d54472d8152bbe0731f0a2c600231e7d4f52c4f092ad4f4aeaf00"}
Oct 30 20:00:23.782897 ip-10-0-132-82 kubenswrapper[4892]: I1030 20:00:23.782862    4892 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-f8a9d8-676dfc58bc-5jxrg" event={"ID":"5f415c4a-4dda-40e3-bdae-8ad67ad15bdf","Type":"ContainerStarted","Data":"d61ca2f9b84c2a345f2a91fb0a090622cfa74a58a5427b23164c516321f03de8"}
Oct 30 20:00:23.783387 ip-10-0-132-82 kubenswrapper[4892]: I1030 20:00:23.782904    4892 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-f8a9d8-676dfc58bc-5jxrg"
Oct 30 20:00:23.784636 ip-10-0-132-82 kubenswrapper[4892]: I1030 20:00:23.784612    4892 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-b-f8a9d8-d9dccd5b6-h6wsr" event={"ID":"92e73440-f470-44d9-983e-2d38334d834c","Type":"ContainerStarted","Data":"611900ee659f729ed841e28409759972fefe52ddbbad56aad81db74a646c56dd"}
Oct 30 20:00:23.784749 ip-10-0-132-82 kubenswrapper[4892]: I1030 20:00:23.784661    4892 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-inf-b-f8a9d8-d9dccd5b6-h6wsr"
Oct 30 20:00:23.898237 ip-10-0-132-82 kubenswrapper[4892]: I1030 20:00:23.898189    4892 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-f8a9d8-676dfc58bc-5jxrg" podStartSLOduration=2.898175766 podStartE2EDuration="2.898175766s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 20:00:21 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 20:00:23.895611744 +0000 UTC m=+2671.466077342" watchObservedRunningTime="2026-10-30 20:00:23.898175766 +0000 UTC m=+2671.468641364"
Oct 30 20:00:24.021001 ip-10-0-132-82 kubenswrapper[4892]: I1030 20:00:24.018429    4892 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-inf-b-f8a9d8-d9dccd5b6-h6wsr" podStartSLOduration=3.018398503 podStartE2EDuration="3.018398503s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 20:00:21 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 20:00:24.016397301 +0000 UTC m=+2671.586862902" watchObservedRunningTime="2026-10-30 20:00:24.018398503 +0000 UTC m=+2671.588864139"
Oct 30 20:00:24.571642 ip-10-0-132-82 kubenswrapper[4892]: I1030 20:00:24.571607    4892 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0b0b9f79-e0a9-41a1-b513-01ea5661f64d" path="/var/lib/kubelet/pods/0b0b9f79-e0a9-41a1-b513-01ea5661f64d/volumes"
Oct 30 20:00:24.572072 ip-10-0-132-82 kubenswrapper[4892]: I1030 20:00:24.572055    4892 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="35018750-1063-4fb6-928f-ce6bcd401f87" path="/var/lib/kubelet/pods/35018750-1063-4fb6-928f-ce6bcd401f87/volumes"
Oct 30 20:00:24.572429 ip-10-0-132-82 kubenswrapper[4892]: I1030 20:00:24.572413    4892 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="8152ed9d-b608-4a87-aded-8dda2d975174" path="/var/lib/kubelet/pods/8152ed9d-b608-4a87-aded-8dda2d975174/volumes"
Oct 30 20:00:24.573079 ip-10-0-132-82 kubenswrapper[4892]: I1030 20:00:24.573062    4892 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d16a477c-ad96-4363-b13d-b6d374f3f88d" path="/var/lib/kubelet/pods/d16a477c-ad96-4363-b13d-b6d374f3f88d/volumes"
Oct 30 20:00:24.573464 ip-10-0-132-82 kubenswrapper[4892]: I1030 20:00:24.573438    4892 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e469e8c3-e158-4b45-9135-6da275fc897f" path="/var/lib/kubelet/pods/e469e8c3-e158-4b45-9135-6da275fc897f/volumes"
Oct 30 20:00:24.848427 ip-10-0-132-82 kubenswrapper[4892]: E1030 20:00:24.848308    4892 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-jrm87" containerName="extractor"
Oct 30 20:00:26.429579 ip-10-0-132-82 kubenswrapper[4892]: I1030 20:00:26.429545    4892 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-1c5537-8xk7g"]
Oct 30 20:00:26.477260 ip-10-0-132-82 kubenswrapper[4892]: I1030 20:00:26.477229    4892 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-1c5537-8xk7g"]
Oct 30 20:00:26.569346 ip-10-0-132-82 kubenswrapper[4892]: I1030 20:00:26.569318    4892 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="6cd3a5c6-0c31-49f6-ba5b-e1e02d756f1a" path="/var/lib/kubelet/pods/6cd3a5c6-0c31-49f6-ba5b-e1e02d756f1a/volumes"
Oct 30 20:00:27.271134 ip-10-0-132-82 kubenswrapper[4892]: I1030 20:00:27.271099    4892 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-worker-w6-5f55cd-688df7c466-t26tq" podUID="b31aad4b-7f5c-46c1-9c46-e39ea2ea5803" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 30 20:00:27.271321 ip-10-0-132-82 kubenswrapper[4892]: I1030 20:00:27.271162    4892 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-worker-w6-5f55cd-85cb4d6b67-9qrfs" podUID="4286dc2a-9c31-454f-891d-31d896d7a66b" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 30 20:00:29.014060 ip-10-0-132-82 kubenswrapper[4892]: I1030 20:00:29.013993    4892 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w6-5f55cd-688df7c466-t26tq"
Oct 30 20:00:29.055557 ip-10-0-132-82 kubenswrapper[4892]: I1030 20:00:29.055527    4892 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b31aad4b-7f5c-46c1-9c46-e39ea2ea5803-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b31aad4b-7f5c-46c1-9c46-e39ea2ea5803-plugins-config-volume\") pod \"b31aad4b-7f5c-46c1-9c46-e39ea2ea5803\" (UID: \"b31aad4b-7f5c-46c1-9c46-e39ea2ea5803\") "
Oct 30 20:00:29.055708 ip-10-0-132-82 kubenswrapper[4892]: I1030 20:00:29.055616    4892 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b31aad4b-7f5c-46c1-9c46-e39ea2ea5803-kube-api-access-r6ksk\" (UniqueName: \"kubernetes.io/projected/b31aad4b-7f5c-46c1-9c46-e39ea2ea5803-kube-api-access-r6ksk\") pod \"b31aad4b-7f5c-46c1-9c46-e39ea2ea5803\" (UID: \"b31aad4b-7f5c-46c1-9c46-e39ea2ea5803\") "
Oct 30 20:00:29.056040 ip-10-0-132-82 kubenswrapper[4892]: I1030 20:00:29.055984    4892 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b31aad4b-7f5c-46c1-9c46-e39ea2ea5803-plugins-config-volume" pod "b31aad4b-7f5c-46c1-9c46-e39ea2ea5803" (UID: "b31aad4b-7f5c-46c1-9c46-e39ea2ea5803"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 20:00:29.056165 ip-10-0-132-82 kubenswrapper[4892]: I1030 20:00:29.056148    4892 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b31aad4b-7f5c-46c1-9c46-e39ea2ea5803-plugins-config-volume\") on node \"ip-10-0-132-82.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:00:29.058196 ip-10-0-132-82 kubenswrapper[4892]: I1030 20:00:29.058161    4892 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b31aad4b-7f5c-46c1-9c46-e39ea2ea5803-kube-api-access-r6ksk" pod "b31aad4b-7f5c-46c1-9c46-e39ea2ea5803" (UID: "b31aad4b-7f5c-46c1-9c46-e39ea2ea5803"). InnerVolumeSpecName "kube-api-access-r6ksk". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 20:00:29.109931 ip-10-0-132-82 kubenswrapper[4892]: I1030 20:00:29.109909    4892 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w6-5f55cd-85cb4d6b67-9qrfs"
Oct 30 20:00:29.157046 ip-10-0-132-82 kubenswrapper[4892]: I1030 20:00:29.157017    4892 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4286dc2a-9c31-454f-891d-31d896d7a66b-kube-api-access-tf5gc\" (UniqueName: \"kubernetes.io/projected/4286dc2a-9c31-454f-891d-31d896d7a66b-kube-api-access-tf5gc\") pod \"4286dc2a-9c31-454f-891d-31d896d7a66b\" (UID: \"4286dc2a-9c31-454f-891d-31d896d7a66b\") "
Oct 30 20:00:29.157197 ip-10-0-132-82 kubenswrapper[4892]: I1030 20:00:29.157080    4892 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/4286dc2a-9c31-454f-891d-31d896d7a66b-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4286dc2a-9c31-454f-891d-31d896d7a66b-plugins-config-volume\") pod \"4286dc2a-9c31-454f-891d-31d896d7a66b\" (UID: \"4286dc2a-9c31-454f-891d-31d896d7a66b\") "
Oct 30 20:00:29.157442 ip-10-0-132-82 kubenswrapper[4892]: I1030 20:00:29.157422    4892 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-r6ksk\" (UniqueName: \"kubernetes.io/projected/b31aad4b-7f5c-46c1-9c46-e39ea2ea5803-kube-api-access-r6ksk\") on node \"ip-10-0-132-82.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:00:29.157442 ip-10-0-132-82 kubenswrapper[4892]: I1030 20:00:29.157433    4892 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4286dc2a-9c31-454f-891d-31d896d7a66b-plugins-config-volume" pod "4286dc2a-9c31-454f-891d-31d896d7a66b" (UID: "4286dc2a-9c31-454f-891d-31d896d7a66b"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 20:00:29.159274 ip-10-0-132-82 kubenswrapper[4892]: I1030 20:00:29.159256    4892 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4286dc2a-9c31-454f-891d-31d896d7a66b-kube-api-access-tf5gc" pod "4286dc2a-9c31-454f-891d-31d896d7a66b" (UID: "4286dc2a-9c31-454f-891d-31d896d7a66b"). InnerVolumeSpecName "kube-api-access-tf5gc". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 20:00:29.258878 ip-10-0-132-82 kubenswrapper[4892]: I1030 20:00:29.258813    4892 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4286dc2a-9c31-454f-891d-31d896d7a66b-plugins-config-volume\") on node \"ip-10-0-132-82.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:00:29.258878 ip-10-0-132-82 kubenswrapper[4892]: I1030 20:00:29.258862    4892 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tf5gc\" (UniqueName: \"kubernetes.io/projected/4286dc2a-9c31-454f-891d-31d896d7a66b-kube-api-access-tf5gc\") on node \"ip-10-0-132-82.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:00:29.832114 ip-10-0-132-82 kubenswrapper[4892]: I1030 20:00:29.832085    4892 generic.go:356] "Generic (PLEG): container finished" podID="b31aad4b-7f5c-46c1-9c46-e39ea2ea5803" containerID="07f691a8d9b079f12d2fd30161803246153cf95492383e4e9879b569a667dd9e" exitCode=0
Oct 30 20:00:29.832274 ip-10-0-132-82 kubenswrapper[4892]: I1030 20:00:29.832132    4892 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w6-5f55cd-688df7c466-t26tq"
Oct 30 20:00:29.832274 ip-10-0-132-82 kubenswrapper[4892]: I1030 20:00:29.832169    4892 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w6-5f55cd-688df7c466-t26tq" event={"ID":"b31aad4b-7f5c-46c1-9c46-e39ea2ea5803","Type":"ContainerDied","Data":"07f691a8d9b079f12d2fd30161803246153cf95492383e4e9879b569a667dd9e"}
Oct 30 20:00:29.832274 ip-10-0-132-82 kubenswrapper[4892]: I1030 20:00:29.832221    4892 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w6-5f55cd-688df7c466-t26tq" event={"ID":"b31aad4b-7f5c-46c1-9c46-e39ea2ea5803","Type":"ContainerDied","Data":"3d65a5ff773b5255657dd74a7ec70775db634a7c3770f41b284fa644b47206fe"}
Oct 30 20:00:29.832274 ip-10-0-132-82 kubenswrapper[4892]: I1030 20:00:29.832250    4892 scope.go:122] "RemoveContainer" containerID="07f691a8d9b079f12d2fd30161803246153cf95492383e4e9879b569a667dd9e"
Oct 30 20:00:29.833930 ip-10-0-132-82 kubenswrapper[4892]: I1030 20:00:29.833905    4892 generic.go:356] "Generic (PLEG): container finished" podID="4286dc2a-9c31-454f-891d-31d896d7a66b" containerID="6ac8f7e45ada17a45ce0acac47bd7dc40934edee4d643c7161eeee66cb5817be" exitCode=0
Oct 30 20:00:29.834057 ip-10-0-132-82 kubenswrapper[4892]: I1030 20:00:29.833959    4892 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w6-5f55cd-85cb4d6b67-9qrfs"
Oct 30 20:00:29.834057 ip-10-0-132-82 kubenswrapper[4892]: I1030 20:00:29.833997    4892 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w6-5f55cd-85cb4d6b67-9qrfs" event={"ID":"4286dc2a-9c31-454f-891d-31d896d7a66b","Type":"ContainerDied","Data":"6ac8f7e45ada17a45ce0acac47bd7dc40934edee4d643c7161eeee66cb5817be"}
Oct 30 20:00:29.834057 ip-10-0-132-82 kubenswrapper[4892]: I1030 20:00:29.834042    4892 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w6-5f55cd-85cb4d6b67-9qrfs" event={"ID":"4286dc2a-9c31-454f-891d-31d896d7a66b","Type":"ContainerDied","Data":"bbfe9c659dd9e36206fe23a6f7202c8cd93eedd4c9f431539a77989a88aa290a"}
Oct 30 20:00:29.843101 ip-10-0-132-82 kubenswrapper[4892]: I1030 20:00:29.843082    4892 scope.go:122] "RemoveContainer" containerID="07f691a8d9b079f12d2fd30161803246153cf95492383e4e9879b569a667dd9e"
Oct 30 20:00:29.843407 ip-10-0-132-82 kubenswrapper[4892]: E1030 20:00:29.843389    4892 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"07f691a8d9b079f12d2fd30161803246153cf95492383e4e9879b569a667dd9e\": container with ID starting with 07f691a8d9b079f12d2fd30161803246153cf95492383e4e9879b569a667dd9e not found: ID does not exist" containerID="07f691a8d9b079f12d2fd30161803246153cf95492383e4e9879b569a667dd9e"
Oct 30 20:00:29.843465 ip-10-0-132-82 kubenswrapper[4892]: I1030 20:00:29.843414    4892 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"07f691a8d9b079f12d2fd30161803246153cf95492383e4e9879b569a667dd9e"} err="failed to get container status \"07f691a8d9b079f12d2fd30161803246153cf95492383e4e9879b569a667dd9e\": rpc error: code = NotFound desc = could not find container \"07f691a8d9b079f12d2fd30161803246153cf95492383e4e9879b569a667dd9e\": container with ID starting with 07f691a8d9b079f12d2fd30161803246153cf95492383e4e9879b569a667dd9e not found: ID does not exist"
Oct 30 20:00:29.843465 ip-10-0-132-82 kubenswrapper[4892]: I1030 20:00:29.843431    4892 scope.go:122] "RemoveContainer" containerID="6ac8f7e45ada17a45ce0acac47bd7dc40934edee4d643c7161eeee66cb5817be"
Oct 30 20:00:29.852898 ip-10-0-132-82 kubenswrapper[4892]: I1030 20:00:29.852883    4892 scope.go:122] "RemoveContainer" containerID="6ac8f7e45ada17a45ce0acac47bd7dc40934edee4d643c7161eeee66cb5817be"
Oct 30 20:00:29.853170 ip-10-0-132-82 kubenswrapper[4892]: E1030 20:00:29.853153    4892 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6ac8f7e45ada17a45ce0acac47bd7dc40934edee4d643c7161eeee66cb5817be\": container with ID starting with 6ac8f7e45ada17a45ce0acac47bd7dc40934edee4d643c7161eeee66cb5817be not found: ID does not exist" containerID="6ac8f7e45ada17a45ce0acac47bd7dc40934edee4d643c7161eeee66cb5817be"
Oct 30 20:00:29.853241 ip-10-0-132-82 kubenswrapper[4892]: I1030 20:00:29.853174    4892 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6ac8f7e45ada17a45ce0acac47bd7dc40934edee4d643c7161eeee66cb5817be"} err="failed to get container status \"6ac8f7e45ada17a45ce0acac47bd7dc40934edee4d643c7161eeee66cb5817be\": rpc error: code = NotFound desc = could not find container \"6ac8f7e45ada17a45ce0acac47bd7dc40934edee4d643c7161eeee66cb5817be\": container with ID starting with 6ac8f7e45ada17a45ce0acac47bd7dc40934edee4d643c7161eeee66cb5817be not found: ID does not exist"
Oct 30 20:00:30.002313 ip-10-0-132-82 kubenswrapper[4892]: I1030 20:00:30.002272    4892 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w6-5f55cd-85cb4d6b67-9qrfs"]
Oct 30 20:00:30.005550 ip-10-0-132-82 kubenswrapper[4892]: I1030 20:00:30.005527    4892 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w6-5f55cd-85cb4d6b67-9qrfs"]
Oct 30 20:00:30.130858 ip-10-0-132-82 kubenswrapper[4892]: I1030 20:00:30.130780    4892 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w6-5f55cd-688df7c466-t26tq"]
Oct 30 20:00:30.135437 ip-10-0-132-82 kubenswrapper[4892]: I1030 20:00:30.135412    4892 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w6-5f55cd-688df7c466-t26tq"]
Oct 30 20:00:30.572766 ip-10-0-132-82 kubenswrapper[4892]: I1030 20:00:30.572731    4892 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="4286dc2a-9c31-454f-891d-31d896d7a66b" path="/var/lib/kubelet/pods/4286dc2a-9c31-454f-891d-31d896d7a66b/volumes"
Oct 30 20:00:30.573308 ip-10-0-132-82 kubenswrapper[4892]: I1030 20:00:30.573283    4892 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b31aad4b-7f5c-46c1-9c46-e39ea2ea5803" path="/var/lib/kubelet/pods/b31aad4b-7f5c-46c1-9c46-e39ea2ea5803/volumes"
Oct 30 20:00:34.795451 ip-10-0-132-82 kubenswrapper[4892]: I1030 20:00:34.795419    4892 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-f8a9d8-676dfc58bc-5jxrg"
Oct 30 20:00:34.795905 ip-10-0-132-82 kubenswrapper[4892]: I1030 20:00:34.795512    4892 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-inf-b-f8a9d8-d9dccd5b6-h6wsr"
Oct 30 20:00:34.906909 ip-10-0-132-82 kubenswrapper[4892]: I1030 20:00:34.906875    4892 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-cf9b6f-m26d6"]
Oct 30 20:00:34.907388 ip-10-0-132-82 kubenswrapper[4892]: I1030 20:00:34.907374    4892 cpu_manager.go:414] "containerMap: removing container" podUID="4286dc2a-9c31-454f-891d-31d896d7a66b" containerName="payload-pre-processing"
Oct 30 20:00:34.907452 ip-10-0-132-82 kubenswrapper[4892]: I1030 20:00:34.907389    4892 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4286dc2a-9c31-454f-891d-31d896d7a66b" containerName="payload-pre-processing"
Oct 30 20:00:34.907452 ip-10-0-132-82 kubenswrapper[4892]: I1030 20:00:34.907409    4892 cpu_manager.go:414] "containerMap: removing container" podUID="b31aad4b-7f5c-46c1-9c46-e39ea2ea5803" containerName="payload-processing"
Oct 30 20:00:34.907452 ip-10-0-132-82 kubenswrapper[4892]: I1030 20:00:34.907414    4892 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b31aad4b-7f5c-46c1-9c46-e39ea2ea5803" containerName="payload-processing"
Oct 30 20:00:34.907575 ip-10-0-132-82 kubenswrapper[4892]: I1030 20:00:34.907499    4892 memory_manager.go:365] "RemoveStaleState removing state" podUID="4286dc2a-9c31-454f-891d-31d896d7a66b" containerName="payload-pre-processing"
Oct 30 20:00:34.907575 ip-10-0-132-82 kubenswrapper[4892]: I1030 20:00:34.907510    4892 memory_manager.go:365] "RemoveStaleState removing state" podUID="b31aad4b-7f5c-46c1-9c46-e39ea2ea5803" containerName="payload-processing"
Oct 30 20:00:34.912388 ip-10-0-132-82 kubenswrapper[4892]: I1030 20:00:34.912360    4892 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-cf9b6f-m26d6"
Oct 30 20:00:34.916037 ip-10-0-132-82 kubenswrapper[4892]: I1030 20:00:34.915999    4892 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/aa6d7d96-304e-404f-b62e-68b3fa0a0266-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w3-cf9b6f-m26d6\" (UID: \"aa6d7d96-304e-404f-b62e-68b3fa0a0266\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-cf9b6f-m26d6"
Oct 30 20:00:34.916203 ip-10-0-132-82 kubenswrapper[4892]: I1030 20:00:34.916179    4892 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-g7c99\" (UniqueName: \"kubernetes.io/projected/aa6d7d96-304e-404f-b62e-68b3fa0a0266-kube-api-access-g7c99\") pod \"maas-api-revoke-keys-e2e-shared-a-w3-cf9b6f-m26d6\" (UID: \"aa6d7d96-304e-404f-b62e-68b3fa0a0266\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-cf9b6f-m26d6"
Oct 30 20:00:34.916429 ip-10-0-132-82 kubenswrapper[4892]: I1030 20:00:34.916405    4892 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/aa6d7d96-304e-404f-b62e-68b3fa0a0266-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w3-cf9b6f-m26d6\" (UID: \"aa6d7d96-304e-404f-b62e-68b3fa0a0266\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-cf9b6f-m26d6"
Oct 30 20:00:34.957113 ip-10-0-132-82 kubenswrapper[4892]: I1030 20:00:34.956666    4892 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-cf9b6f-m26d6"]
Oct 30 20:00:34.965420 ip-10-0-132-82 kubenswrapper[4892]: I1030 20:00:34.965393    4892 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 30 20:00:34.968231 ip-10-0-132-82 kubenswrapper[4892]: I1030 20:00:34.968206    4892 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 30 20:00:35.003671 ip-10-0-132-82 kubenswrapper[4892]: I1030 20:00:35.003639    4892 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 20:00:35.003847 ip-10-0-132-82 kubenswrapper[4892]: I1030 20:00:35.003705    4892 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-nlxp4\""
Oct 30 20:00:35.017891 ip-10-0-132-82 kubenswrapper[4892]: I1030 20:00:35.017850    4892 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/aa6d7d96-304e-404f-b62e-68b3fa0a0266-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w3-cf9b6f-m26d6\" (UID: \"aa6d7d96-304e-404f-b62e-68b3fa0a0266\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-cf9b6f-m26d6"
Oct 30 20:00:35.018101 ip-10-0-132-82 kubenswrapper[4892]: I1030 20:00:35.018035    4892 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-g7c99\" (UniqueName: \"kubernetes.io/projected/aa6d7d96-304e-404f-b62e-68b3fa0a0266-kube-api-access-g7c99\") pod \"maas-api-revoke-keys-e2e-shared-a-w3-cf9b6f-m26d6\" (UID: \"aa6d7d96-304e-404f-b62e-68b3fa0a0266\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-cf9b6f-m26d6"
Oct 30 20:00:35.018101 ip-10-0-132-82 kubenswrapper[4892]: I1030 20:00:35.018089    4892 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/aa6d7d96-304e-404f-b62e-68b3fa0a0266-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w3-cf9b6f-m26d6\" (UID: \"aa6d7d96-304e-404f-b62e-68b3fa0a0266\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-cf9b6f-m26d6"
Oct 30 20:00:35.018864 ip-10-0-132-82 kubenswrapper[4892]: I1030 20:00:35.018835    4892 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/aa6d7d96-304e-404f-b62e-68b3fa0a0266-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w3-cf9b6f-m26d6\" (UID: \"aa6d7d96-304e-404f-b62e-68b3fa0a0266\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-cf9b6f-m26d6"
Oct 30 20:00:35.021617 ip-10-0-132-82 kubenswrapper[4892]: I1030 20:00:35.021191    4892 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/aa6d7d96-304e-404f-b62e-68b3fa0a0266-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w3-cf9b6f-m26d6\" (UID: \"aa6d7d96-304e-404f-b62e-68b3fa0a0266\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-cf9b6f-m26d6"
Oct 30 20:00:35.084935 ip-10-0-132-82 kubenswrapper[4892]: I1030 20:00:35.084686    4892 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-g7c99\" (UniqueName: \"kubernetes.io/projected/aa6d7d96-304e-404f-b62e-68b3fa0a0266-kube-api-access-g7c99\") pod \"maas-api-revoke-keys-e2e-shared-a-w3-cf9b6f-m26d6\" (UID: \"aa6d7d96-304e-404f-b62e-68b3fa0a0266\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-cf9b6f-m26d6"
Oct 30 20:00:35.229536 ip-10-0-132-82 kubenswrapper[4892]: I1030 20:00:35.229498    4892 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-cf9b6f-m26d6"
Oct 30 20:00:35.613892 ip-10-0-132-82 kubenswrapper[4892]: I1030 20:00:35.613871    4892 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-cf9b6f-m26d6"]
Oct 30 20:00:35.615607 ip-10-0-132-82 kubenswrapper[4892]: W1030 20:00:35.615577    4892 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podaa6d7d96_304e_404f_b62e_68b3fa0a0266.slice/crio-7cf949669fb143935fb6c1e8b1a502e8eff9d6c809af3ecb78d57b7a32a46661 WatchSource:0}: Error finding container 7cf949669fb143935fb6c1e8b1a502e8eff9d6c809af3ecb78d57b7a32a46661: Status 404 returned error can't find the container with id 7cf949669fb143935fb6c1e8b1a502e8eff9d6c809af3ecb78d57b7a32a46661
Oct 30 20:00:35.887849 ip-10-0-132-82 kubenswrapper[4892]: I1030 20:00:35.887817    4892 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-cf9b6f-m26d6" event={"ID":"aa6d7d96-304e-404f-b62e-68b3fa0a0266","Type":"ContainerStarted","Data":"7cf949669fb143935fb6c1e8b1a502e8eff9d6c809af3ecb78d57b7a32a46661"}
Oct 30 20:00:36.898653 ip-10-0-132-82 kubenswrapper[4892]: I1030 20:00:36.898623    4892 generic.go:356] "Generic (PLEG): container finished" podID="aa6d7d96-304e-404f-b62e-68b3fa0a0266" containerID="8aa9d2b80f9c0d20e6c7010ceb122d6fbf82d8b5b3b80b94d78a44f21316e326" exitCode=0
Oct 30 20:00:36.899049 ip-10-0-132-82 kubenswrapper[4892]: I1030 20:00:36.898663    4892 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-cf9b6f-m26d6" event={"ID":"aa6d7d96-304e-404f-b62e-68b3fa0a0266","Type":"ContainerDied","Data":"8aa9d2b80f9c0d20e6c7010ceb122d6fbf82d8b5b3b80b94d78a44f21316e326"}
Oct 30 20:00:38.053162 ip-10-0-132-82 kubenswrapper[4892]: I1030 20:00:38.053137    4892 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-cf9b6f-m26d6"
Oct 30 20:00:38.153499 ip-10-0-132-82 kubenswrapper[4892]: I1030 20:00:38.153471    4892 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/aa6d7d96-304e-404f-b62e-68b3fa0a0266-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/aa6d7d96-304e-404f-b62e-68b3fa0a0266-maas-api-openshift-ca\") pod \"aa6d7d96-304e-404f-b62e-68b3fa0a0266\" (UID: \"aa6d7d96-304e-404f-b62e-68b3fa0a0266\") "
Oct 30 20:00:38.153650 ip-10-0-132-82 kubenswrapper[4892]: I1030 20:00:38.153531    4892 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/aa6d7d96-304e-404f-b62e-68b3fa0a0266-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/aa6d7d96-304e-404f-b62e-68b3fa0a0266-maas-api-xks-ca\") pod \"aa6d7d96-304e-404f-b62e-68b3fa0a0266\" (UID: \"aa6d7d96-304e-404f-b62e-68b3fa0a0266\") "
Oct 30 20:00:38.153650 ip-10-0-132-82 kubenswrapper[4892]: I1030 20:00:38.153573    4892 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/aa6d7d96-304e-404f-b62e-68b3fa0a0266-kube-api-access-g7c99\" (UniqueName: \"kubernetes.io/projected/aa6d7d96-304e-404f-b62e-68b3fa0a0266-kube-api-access-g7c99\") pod \"aa6d7d96-304e-404f-b62e-68b3fa0a0266\" (UID: \"aa6d7d96-304e-404f-b62e-68b3fa0a0266\") "
Oct 30 20:00:38.153854 ip-10-0-132-82 kubenswrapper[4892]: I1030 20:00:38.153826    4892 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/aa6d7d96-304e-404f-b62e-68b3fa0a0266-maas-api-openshift-ca" pod "aa6d7d96-304e-404f-b62e-68b3fa0a0266" (UID: "aa6d7d96-304e-404f-b62e-68b3fa0a0266"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 20:00:38.155727 ip-10-0-132-82 kubenswrapper[4892]: I1030 20:00:38.155707    4892 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/aa6d7d96-304e-404f-b62e-68b3fa0a0266-kube-api-access-g7c99" pod "aa6d7d96-304e-404f-b62e-68b3fa0a0266" (UID: "aa6d7d96-304e-404f-b62e-68b3fa0a0266"). InnerVolumeSpecName "kube-api-access-g7c99". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 20:00:38.155909 ip-10-0-132-82 kubenswrapper[4892]: I1030 20:00:38.155890    4892 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/aa6d7d96-304e-404f-b62e-68b3fa0a0266-maas-api-xks-ca" pod "aa6d7d96-304e-404f-b62e-68b3fa0a0266" (UID: "aa6d7d96-304e-404f-b62e-68b3fa0a0266"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 20:00:38.254740 ip-10-0-132-82 kubenswrapper[4892]: I1030 20:00:38.254707    4892 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-g7c99\" (UniqueName: \"kubernetes.io/projected/aa6d7d96-304e-404f-b62e-68b3fa0a0266-kube-api-access-g7c99\") on node \"ip-10-0-132-82.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:00:38.254740 ip-10-0-132-82 kubenswrapper[4892]: I1030 20:00:38.254736    4892 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/aa6d7d96-304e-404f-b62e-68b3fa0a0266-maas-api-openshift-ca\") on node \"ip-10-0-132-82.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:00:38.254902 ip-10-0-132-82 kubenswrapper[4892]: I1030 20:00:38.254751    4892 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/aa6d7d96-304e-404f-b62e-68b3fa0a0266-maas-api-xks-ca\") on node \"ip-10-0-132-82.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:00:38.917204 ip-10-0-132-82 kubenswrapper[4892]: I1030 20:00:38.917176    4892 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-cf9b6f-m26d6"
Oct 30 20:00:38.917361 ip-10-0-132-82 kubenswrapper[4892]: I1030 20:00:38.917209    4892 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-cf9b6f-m26d6" event={"ID":"aa6d7d96-304e-404f-b62e-68b3fa0a0266","Type":"ContainerDied","Data":"7cf949669fb143935fb6c1e8b1a502e8eff9d6c809af3ecb78d57b7a32a46661"}
Oct 30 20:00:38.917361 ip-10-0-132-82 kubenswrapper[4892]: I1030 20:00:38.917256    4892 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="7cf949669fb143935fb6c1e8b1a502e8eff9d6c809af3ecb78d57b7a32a46661"
Oct 30 20:00:45.962306 ip-10-0-132-82 kubenswrapper[4892]: I1030 20:00:45.962272    4892 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-fa928509-4w4hc"]
Oct 30 20:00:45.962790 ip-10-0-132-82 kubenswrapper[4892]: I1030 20:00:45.962774    4892 cpu_manager.go:414] "containerMap: removing container" podUID="aa6d7d96-304e-404f-b62e-68b3fa0a0266" containerName="revoke-keys"
Oct 30 20:00:45.962839 ip-10-0-132-82 kubenswrapper[4892]: I1030 20:00:45.962792    4892 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="aa6d7d96-304e-404f-b62e-68b3fa0a0266" containerName="revoke-keys"
Oct 30 20:00:45.962905 ip-10-0-132-82 kubenswrapper[4892]: I1030 20:00:45.962896    4892 memory_manager.go:365] "RemoveStaleState removing state" podUID="aa6d7d96-304e-404f-b62e-68b3fa0a0266" containerName="revoke-keys"
Oct 30 20:00:45.966135 ip-10-0-132-82 kubenswrapper[4892]: I1030 20:00:45.966116    4892 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-fa928509-4w4hc"
Oct 30 20:00:46.043619 ip-10-0-132-82 kubenswrapper[4892]: I1030 20:00:46.043584    4892 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a4f83474-cd19-4c19-b6d5-2914706168e8-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-fa928509-4w4hc\" (UID: \"a4f83474-cd19-4c19-b6d5-2914706168e8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fa928509-4w4hc"
Oct 30 20:00:46.043832 ip-10-0-132-82 kubenswrapper[4892]: I1030 20:00:46.043632    4892 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a4f83474-cd19-4c19-b6d5-2914706168e8-maas-api-xks-ca\") pod \"maas-api-revoke-sub-fa928509-4w4hc\" (UID: \"a4f83474-cd19-4c19-b6d5-2914706168e8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fa928509-4w4hc"
Oct 30 20:00:46.043832 ip-10-0-132-82 kubenswrapper[4892]: I1030 20:00:46.043657    4892 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zf6gl\" (UniqueName: \"kubernetes.io/projected/a4f83474-cd19-4c19-b6d5-2914706168e8-kube-api-access-zf6gl\") pod \"maas-api-revoke-sub-fa928509-4w4hc\" (UID: \"a4f83474-cd19-4c19-b6d5-2914706168e8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fa928509-4w4hc"
Oct 30 20:00:46.071931 ip-10-0-132-82 kubenswrapper[4892]: I1030 20:00:46.071900    4892 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 20:00:46.072153 ip-10-0-132-82 kubenswrapper[4892]: I1030 20:00:46.072126    4892 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-nlxp4\""
Oct 30 20:00:46.072153 ip-10-0-132-82 kubenswrapper[4892]: I1030 20:00:46.072056    4892 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 30 20:00:46.072422 ip-10-0-132-82 kubenswrapper[4892]: I1030 20:00:46.071944    4892 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 30 20:00:46.077485 ip-10-0-132-82 kubenswrapper[4892]: I1030 20:00:46.077463    4892 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-fa928509-4w4hc"]
Oct 30 20:00:46.084666 ip-10-0-132-82 kubenswrapper[4892]: I1030 20:00:46.084642    4892 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b372b6b1-jhvr9"]
Oct 30 20:00:46.089587 ip-10-0-132-82 kubenswrapper[4892]: I1030 20:00:46.089564    4892 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b372b6b1-jhvr9"
Oct 30 20:00:46.128575 ip-10-0-132-82 kubenswrapper[4892]: I1030 20:00:46.128551    4892 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b372b6b1-jhvr9"]
Oct 30 20:00:46.144637 ip-10-0-132-82 kubenswrapper[4892]: I1030 20:00:46.144609    4892 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a4f83474-cd19-4c19-b6d5-2914706168e8-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-fa928509-4w4hc\" (UID: \"a4f83474-cd19-4c19-b6d5-2914706168e8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fa928509-4w4hc"
Oct 30 20:00:46.144773 ip-10-0-132-82 kubenswrapper[4892]: I1030 20:00:46.144655    4892 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a4f83474-cd19-4c19-b6d5-2914706168e8-maas-api-xks-ca\") pod \"maas-api-revoke-sub-fa928509-4w4hc\" (UID: \"a4f83474-cd19-4c19-b6d5-2914706168e8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fa928509-4w4hc"
Oct 30 20:00:46.144773 ip-10-0-132-82 kubenswrapper[4892]: I1030 20:00:46.144681    4892 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zf6gl\" (UniqueName: \"kubernetes.io/projected/a4f83474-cd19-4c19-b6d5-2914706168e8-kube-api-access-zf6gl\") pod \"maas-api-revoke-sub-fa928509-4w4hc\" (UID: \"a4f83474-cd19-4c19-b6d5-2914706168e8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fa928509-4w4hc"
Oct 30 20:00:46.144773 ip-10-0-132-82 kubenswrapper[4892]: I1030 20:00:46.144712    4892 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/05dbc393-f688-4634-ab42-8c00bf839520-maas-api-xks-ca\") pod \"maas-api-revoke-sub-b372b6b1-jhvr9\" (UID: \"05dbc393-f688-4634-ab42-8c00bf839520\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b372b6b1-jhvr9"
Oct 30 20:00:46.144943 ip-10-0-132-82 kubenswrapper[4892]: I1030 20:00:46.144777    4892 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/05dbc393-f688-4634-ab42-8c00bf839520-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-b372b6b1-jhvr9\" (UID: \"05dbc393-f688-4634-ab42-8c00bf839520\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b372b6b1-jhvr9"
Oct 30 20:00:46.144943 ip-10-0-132-82 kubenswrapper[4892]: I1030 20:00:46.144884    4892 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6hfnx\" (UniqueName: \"kubernetes.io/projected/05dbc393-f688-4634-ab42-8c00bf839520-kube-api-access-6hfnx\") pod \"maas-api-revoke-sub-b372b6b1-jhvr9\" (UID: \"05dbc393-f688-4634-ab42-8c00bf839520\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b372b6b1-jhvr9"
Oct 30 20:00:46.145347 ip-10-0-132-82 kubenswrapper[4892]: I1030 20:00:46.145318    4892 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a4f83474-cd19-4c19-b6d5-2914706168e8-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-fa928509-4w4hc\" (UID: \"a4f83474-cd19-4c19-b6d5-2914706168e8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fa928509-4w4hc"
Oct 30 20:00:46.147182 ip-10-0-132-82 kubenswrapper[4892]: I1030 20:00:46.147163    4892 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a4f83474-cd19-4c19-b6d5-2914706168e8-maas-api-xks-ca\") pod \"maas-api-revoke-sub-fa928509-4w4hc\" (UID: \"a4f83474-cd19-4c19-b6d5-2914706168e8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fa928509-4w4hc"
Oct 30 20:00:46.213798 ip-10-0-132-82 kubenswrapper[4892]: I1030 20:00:46.213734    4892 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zf6gl\" (UniqueName: \"kubernetes.io/projected/a4f83474-cd19-4c19-b6d5-2914706168e8-kube-api-access-zf6gl\") pod \"maas-api-revoke-sub-fa928509-4w4hc\" (UID: \"a4f83474-cd19-4c19-b6d5-2914706168e8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fa928509-4w4hc"
Oct 30 20:00:46.222100 ip-10-0-132-82 kubenswrapper[4892]: I1030 20:00:46.222073    4892 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-de9ce181-rtrn7"]
Oct 30 20:00:46.226591 ip-10-0-132-82 kubenswrapper[4892]: I1030 20:00:46.226574    4892 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-de9ce181-rtrn7"
Oct 30 20:00:46.245626 ip-10-0-132-82 kubenswrapper[4892]: I1030 20:00:46.245602    4892 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6hfnx\" (UniqueName: \"kubernetes.io/projected/05dbc393-f688-4634-ab42-8c00bf839520-kube-api-access-6hfnx\") pod \"maas-api-revoke-sub-b372b6b1-jhvr9\" (UID: \"05dbc393-f688-4634-ab42-8c00bf839520\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b372b6b1-jhvr9"
Oct 30 20:00:46.245757 ip-10-0-132-82 kubenswrapper[4892]: I1030 20:00:46.245687    4892 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/03e83bd1-55e2-4aa5-956b-7964b3529ad4-maas-api-xks-ca\") pod \"maas-api-revoke-sub-de9ce181-rtrn7\" (UID: \"03e83bd1-55e2-4aa5-956b-7964b3529ad4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-de9ce181-rtrn7"
Oct 30 20:00:46.245757 ip-10-0-132-82 kubenswrapper[4892]: I1030 20:00:46.245733    4892 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-h8w5n\" (UniqueName: \"kubernetes.io/projected/03e83bd1-55e2-4aa5-956b-7964b3529ad4-kube-api-access-h8w5n\") pod \"maas-api-revoke-sub-de9ce181-rtrn7\" (UID: \"03e83bd1-55e2-4aa5-956b-7964b3529ad4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-de9ce181-rtrn7"
Oct 30 20:00:46.245911 ip-10-0-132-82 kubenswrapper[4892]: I1030 20:00:46.245888    4892 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/05dbc393-f688-4634-ab42-8c00bf839520-maas-api-xks-ca\") pod \"maas-api-revoke-sub-b372b6b1-jhvr9\" (UID: \"05dbc393-f688-4634-ab42-8c00bf839520\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b372b6b1-jhvr9"
Oct 30 20:00:46.246021 ip-10-0-132-82 kubenswrapper[4892]: I1030 20:00:46.246004    4892 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/05dbc393-f688-4634-ab42-8c00bf839520-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-b372b6b1-jhvr9\" (UID: \"05dbc393-f688-4634-ab42-8c00bf839520\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b372b6b1-jhvr9"
Oct 30 20:00:46.246081 ip-10-0-132-82 kubenswrapper[4892]: I1030 20:00:46.246041    4892 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/03e83bd1-55e2-4aa5-956b-7964b3529ad4-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-de9ce181-rtrn7\" (UID: \"03e83bd1-55e2-4aa5-956b-7964b3529ad4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-de9ce181-rtrn7"
Oct 30 20:00:46.246565 ip-10-0-132-82 kubenswrapper[4892]: I1030 20:00:46.246534    4892 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/05dbc393-f688-4634-ab42-8c00bf839520-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-b372b6b1-jhvr9\" (UID: \"05dbc393-f688-4634-ab42-8c00bf839520\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b372b6b1-jhvr9"
Oct 30 20:00:46.248224 ip-10-0-132-82 kubenswrapper[4892]: I1030 20:00:46.248201    4892 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/05dbc393-f688-4634-ab42-8c00bf839520-maas-api-xks-ca\") pod \"maas-api-revoke-sub-b372b6b1-jhvr9\" (UID: \"05dbc393-f688-4634-ab42-8c00bf839520\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b372b6b1-jhvr9"
Oct 30 20:00:46.282017 ip-10-0-132-82 kubenswrapper[4892]: I1030 20:00:46.281989    4892 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-fa928509-4w4hc"
Oct 30 20:00:46.303560 ip-10-0-132-82 kubenswrapper[4892]: I1030 20:00:46.303535    4892 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-584d0bf7-wx8bn"]
Oct 30 20:00:46.308688 ip-10-0-132-82 kubenswrapper[4892]: I1030 20:00:46.308667    4892 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-584d0bf7-wx8bn"
Oct 30 20:00:46.317902 ip-10-0-132-82 kubenswrapper[4892]: I1030 20:00:46.317875    4892 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6hfnx\" (UniqueName: \"kubernetes.io/projected/05dbc393-f688-4634-ab42-8c00bf839520-kube-api-access-6hfnx\") pod \"maas-api-revoke-sub-b372b6b1-jhvr9\" (UID: \"05dbc393-f688-4634-ab42-8c00bf839520\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b372b6b1-jhvr9"
Oct 30 20:00:46.325631 ip-10-0-132-82 kubenswrapper[4892]: I1030 20:00:46.325607    4892 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-de9ce181-rtrn7"]
Oct 30 20:00:46.346903 ip-10-0-132-82 kubenswrapper[4892]: I1030 20:00:46.346853    4892 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/956121ad-97c4-4c74-8934-3367fbff6c24-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-584d0bf7-wx8bn\" (UID: \"956121ad-97c4-4c74-8934-3367fbff6c24\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-584d0bf7-wx8bn"
Oct 30 20:00:46.346903 ip-10-0-132-82 kubenswrapper[4892]: I1030 20:00:46.346902    4892 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/956121ad-97c4-4c74-8934-3367fbff6c24-maas-api-xks-ca\") pod \"maas-api-revoke-sub-584d0bf7-wx8bn\" (UID: \"956121ad-97c4-4c74-8934-3367fbff6c24\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-584d0bf7-wx8bn"
Oct 30 20:00:46.347158 ip-10-0-132-82 kubenswrapper[4892]: I1030 20:00:46.346987    4892 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/03e83bd1-55e2-4aa5-956b-7964b3529ad4-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-de9ce181-rtrn7\" (UID: \"03e83bd1-55e2-4aa5-956b-7964b3529ad4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-de9ce181-rtrn7"
Oct 30 20:00:46.347158 ip-10-0-132-82 kubenswrapper[4892]: I1030 20:00:46.347015    4892 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5w8vx\" (UniqueName: \"kubernetes.io/projected/956121ad-97c4-4c74-8934-3367fbff6c24-kube-api-access-5w8vx\") pod \"maas-api-revoke-sub-584d0bf7-wx8bn\" (UID: \"956121ad-97c4-4c74-8934-3367fbff6c24\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-584d0bf7-wx8bn"
Oct 30 20:00:46.347158 ip-10-0-132-82 kubenswrapper[4892]: I1030 20:00:46.347112    4892 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/03e83bd1-55e2-4aa5-956b-7964b3529ad4-maas-api-xks-ca\") pod \"maas-api-revoke-sub-de9ce181-rtrn7\" (UID: \"03e83bd1-55e2-4aa5-956b-7964b3529ad4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-de9ce181-rtrn7"
Oct 30 20:00:46.347158 ip-10-0-132-82 kubenswrapper[4892]: I1030 20:00:46.347154    4892 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-h8w5n\" (UniqueName: \"kubernetes.io/projected/03e83bd1-55e2-4aa5-956b-7964b3529ad4-kube-api-access-h8w5n\") pod \"maas-api-revoke-sub-de9ce181-rtrn7\" (UID: \"03e83bd1-55e2-4aa5-956b-7964b3529ad4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-de9ce181-rtrn7"
Oct 30 20:00:46.348420 ip-10-0-132-82 kubenswrapper[4892]: I1030 20:00:46.348331    4892 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/03e83bd1-55e2-4aa5-956b-7964b3529ad4-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-de9ce181-rtrn7\" (UID: \"03e83bd1-55e2-4aa5-956b-7964b3529ad4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-de9ce181-rtrn7"
Oct 30 20:00:46.351389 ip-10-0-132-82 kubenswrapper[4892]: I1030 20:00:46.351342    4892 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/03e83bd1-55e2-4aa5-956b-7964b3529ad4-maas-api-xks-ca\") pod \"maas-api-revoke-sub-de9ce181-rtrn7\" (UID: \"03e83bd1-55e2-4aa5-956b-7964b3529ad4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-de9ce181-rtrn7"
Oct 30 20:00:46.382323 ip-10-0-132-82 kubenswrapper[4892]: I1030 20:00:46.380028    4892 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-584d0bf7-wx8bn"]
Oct 30 20:00:46.403690 ip-10-0-132-82 kubenswrapper[4892]: I1030 20:00:46.403482    4892 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b372b6b1-jhvr9"
Oct 30 20:00:46.416879 ip-10-0-132-82 kubenswrapper[4892]: I1030 20:00:46.416853    4892 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-h8w5n\" (UniqueName: \"kubernetes.io/projected/03e83bd1-55e2-4aa5-956b-7964b3529ad4-kube-api-access-h8w5n\") pod \"maas-api-revoke-sub-de9ce181-rtrn7\" (UID: \"03e83bd1-55e2-4aa5-956b-7964b3529ad4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-de9ce181-rtrn7"
Oct 30 20:00:46.449983 ip-10-0-132-82 kubenswrapper[4892]: I1030 20:00:46.448126    4892 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/956121ad-97c4-4c74-8934-3367fbff6c24-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-584d0bf7-wx8bn\" (UID: \"956121ad-97c4-4c74-8934-3367fbff6c24\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-584d0bf7-wx8bn"
Oct 30 20:00:46.449983 ip-10-0-132-82 kubenswrapper[4892]: I1030 20:00:46.448159    4892 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/956121ad-97c4-4c74-8934-3367fbff6c24-maas-api-xks-ca\") pod \"maas-api-revoke-sub-584d0bf7-wx8bn\" (UID: \"956121ad-97c4-4c74-8934-3367fbff6c24\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-584d0bf7-wx8bn"
Oct 30 20:00:46.449983 ip-10-0-132-82 kubenswrapper[4892]: I1030 20:00:46.448206    4892 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5w8vx\" (UniqueName: \"kubernetes.io/projected/956121ad-97c4-4c74-8934-3367fbff6c24-kube-api-access-5w8vx\") pod \"maas-api-revoke-sub-584d0bf7-wx8bn\" (UID: \"956121ad-97c4-4c74-8934-3367fbff6c24\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-584d0bf7-wx8bn"
Oct 30 20:00:46.449983 ip-10-0-132-82 kubenswrapper[4892]: I1030 20:00:46.448932    4892 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/956121ad-97c4-4c74-8934-3367fbff6c24-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-584d0bf7-wx8bn\" (UID: \"956121ad-97c4-4c74-8934-3367fbff6c24\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-584d0bf7-wx8bn"
Oct 30 20:00:46.452002 ip-10-0-132-82 kubenswrapper[4892]: I1030 20:00:46.450866    4892 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/956121ad-97c4-4c74-8934-3367fbff6c24-maas-api-xks-ca\") pod \"maas-api-revoke-sub-584d0bf7-wx8bn\" (UID: \"956121ad-97c4-4c74-8934-3367fbff6c24\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-584d0bf7-wx8bn"
Oct 30 20:00:46.515493 ip-10-0-132-82 kubenswrapper[4892]: I1030 20:00:46.515458    4892 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5w8vx\" (UniqueName: \"kubernetes.io/projected/956121ad-97c4-4c74-8934-3367fbff6c24-kube-api-access-5w8vx\") pod \"maas-api-revoke-sub-584d0bf7-wx8bn\" (UID: \"956121ad-97c4-4c74-8934-3367fbff6c24\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-584d0bf7-wx8bn"
Oct 30 20:00:46.540312 ip-10-0-132-82 kubenswrapper[4892]: I1030 20:00:46.540285    4892 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-de9ce181-rtrn7"
Oct 30 20:00:46.641537 ip-10-0-132-82 kubenswrapper[4892]: I1030 20:00:46.641268    4892 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-584d0bf7-wx8bn"
Oct 30 20:00:46.655496 ip-10-0-132-82 kubenswrapper[4892]: I1030 20:00:46.655466    4892 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w3-cf9b6f-7bb7c4ccf4-zc2s9"]
Oct 30 20:00:46.692238 ip-10-0-132-82 kubenswrapper[4892]: W1030 20:00:46.692196    4892 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda4f83474_cd19_4c19_b6d5_2914706168e8.slice/crio-7312d18cb3ee052350fbe8db104d3fb52882abea92d91ebba251d3bdfff9652e WatchSource:0}: Error finding container 7312d18cb3ee052350fbe8db104d3fb52882abea92d91ebba251d3bdfff9652e: Status 404 returned error can't find the container with id 7312d18cb3ee052350fbe8db104d3fb52882abea92d91ebba251d3bdfff9652e
Oct 30 20:00:46.694508 ip-10-0-132-82 kubenswrapper[4892]: I1030 20:00:46.694471    4892 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-fa928509-4w4hc"]
Oct 30 20:00:46.852009 ip-10-0-132-82 kubenswrapper[4892]: I1030 20:00:46.850066    4892 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w3-cf9b6f-59d848ddcf-jxlzh"]
Oct 30 20:00:46.856338 ip-10-0-132-82 kubenswrapper[4892]: I1030 20:00:46.856305    4892 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b372b6b1-jhvr9"]
Oct 30 20:00:46.965032 ip-10-0-132-82 kubenswrapper[4892]: W1030 20:00:46.931293    4892 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod05dbc393_f688_4634_ab42_8c00bf839520.slice/crio-94bd73bf9fcf841fa1f322cffa7f0c91c9b6ebf9cdacf59b7c18a057cf190397 WatchSource:0}: Error finding container 94bd73bf9fcf841fa1f322cffa7f0c91c9b6ebf9cdacf59b7c18a057cf190397: Status 404 returned error can't find the container with id 94bd73bf9fcf841fa1f322cffa7f0c91c9b6ebf9cdacf59b7c18a057cf190397
Oct 30 20:00:46.993682 ip-10-0-132-82 kubenswrapper[4892]: I1030 20:00:46.993644    4892 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b372b6b1-jhvr9" event={"ID":"05dbc393-f688-4634-ab42-8c00bf839520","Type":"ContainerStarted","Data":"94bd73bf9fcf841fa1f322cffa7f0c91c9b6ebf9cdacf59b7c18a057cf190397"}
Oct 30 20:00:46.995486 ip-10-0-132-82 kubenswrapper[4892]: I1030 20:00:46.995171    4892 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-fa928509-4w4hc" event={"ID":"a4f83474-cd19-4c19-b6d5-2914706168e8","Type":"ContainerStarted","Data":"7312d18cb3ee052350fbe8db104d3fb52882abea92d91ebba251d3bdfff9652e"}
Oct 30 20:00:47.050788 ip-10-0-132-82 kubenswrapper[4892]: I1030 20:00:47.050761    4892 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-de9ce181-rtrn7"]
Oct 30 20:00:47.052423 ip-10-0-132-82 kubenswrapper[4892]: W1030 20:00:47.052384    4892 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod03e83bd1_55e2_4aa5_956b_7964b3529ad4.slice/crio-e30d527635c75f3ce21792264c46e63e279056dda4dc57c87a92bfb4399dcd9a WatchSource:0}: Error finding container e30d527635c75f3ce21792264c46e63e279056dda4dc57c87a92bfb4399dcd9a: Status 404 returned error can't find the container with id e30d527635c75f3ce21792264c46e63e279056dda4dc57c87a92bfb4399dcd9a
Oct 30 20:00:47.095019 ip-10-0-132-82 kubenswrapper[4892]: I1030 20:00:47.094989    4892 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-584d0bf7-wx8bn"]
Oct 30 20:00:47.101586 ip-10-0-132-82 kubenswrapper[4892]: W1030 20:00:47.101555    4892 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod956121ad_97c4_4c74_8934_3367fbff6c24.slice/crio-8e8886030988aed2573f9d004b2ffdaf62c2c26d27803ba25e6ab0da1ebb5833 WatchSource:0}: Error finding container 8e8886030988aed2573f9d004b2ffdaf62c2c26d27803ba25e6ab0da1ebb5833: Status 404 returned error can't find the container with id 8e8886030988aed2573f9d004b2ffdaf62c2c26d27803ba25e6ab0da1ebb5833
Oct 30 20:00:47.707542 ip-10-0-132-82 kubenswrapper[4892]: I1030 20:00:47.707511    4892 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-fsfvk"]
Oct 30 20:00:48.006928 ip-10-0-132-82 kubenswrapper[4892]: I1030 20:00:48.006832    4892 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-de9ce181-rtrn7" event={"ID":"03e83bd1-55e2-4aa5-956b-7964b3529ad4","Type":"ContainerStarted","Data":"db0d3018ceb00f3dea046d1269b7ac0fa7ba6998159960f6b6b146602e027de0"}
Oct 30 20:00:48.006928 ip-10-0-132-82 kubenswrapper[4892]: I1030 20:00:48.006898    4892 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-de9ce181-rtrn7" event={"ID":"03e83bd1-55e2-4aa5-956b-7964b3529ad4","Type":"ContainerStarted","Data":"e30d527635c75f3ce21792264c46e63e279056dda4dc57c87a92bfb4399dcd9a"}
Oct 30 20:00:48.008911 ip-10-0-132-82 kubenswrapper[4892]: I1030 20:00:48.008880    4892 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b372b6b1-jhvr9" event={"ID":"05dbc393-f688-4634-ab42-8c00bf839520","Type":"ContainerStarted","Data":"258af30f9920c6703e06161d0a4019f9c452a0bdc78f4d27359a7a2444b2694d"}
Oct 30 20:00:48.010715 ip-10-0-132-82 kubenswrapper[4892]: I1030 20:00:48.010682    4892 generic.go:356] "Generic (PLEG): container finished" podID="a4f83474-cd19-4c19-b6d5-2914706168e8" containerID="51025d0f9bd9e146de4ea0bf0049f9b1569843e327823e5309f32b43e48842fb" exitCode=0
Oct 30 20:00:48.010851 ip-10-0-132-82 kubenswrapper[4892]: I1030 20:00:48.010757    4892 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-fa928509-4w4hc" event={"ID":"a4f83474-cd19-4c19-b6d5-2914706168e8","Type":"ContainerDied","Data":"51025d0f9bd9e146de4ea0bf0049f9b1569843e327823e5309f32b43e48842fb"}
Oct 30 20:00:48.012639 ip-10-0-132-82 kubenswrapper[4892]: I1030 20:00:48.012614    4892 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-584d0bf7-wx8bn" event={"ID":"956121ad-97c4-4c74-8934-3367fbff6c24","Type":"ContainerStarted","Data":"ddf751afe27a96f71b7695b975e5daf54bafd08d5680891f8a516c52dc068ed5"}
Oct 30 20:00:48.012813 ip-10-0-132-82 kubenswrapper[4892]: I1030 20:00:48.012789    4892 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-584d0bf7-wx8bn" event={"ID":"956121ad-97c4-4c74-8934-3367fbff6c24","Type":"ContainerStarted","Data":"8e8886030988aed2573f9d004b2ffdaf62c2c26d27803ba25e6ab0da1ebb5833"}
Oct 30 20:00:48.123184 ip-10-0-132-82 kubenswrapper[4892]: I1030 20:00:48.123116    4892 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-de9ce181-rtrn7" podStartSLOduration=2.123098793 podStartE2EDuration="2.123098793s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 20:00:46 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 20:00:48.122387614 +0000 UTC m=+2695.692853241" watchObservedRunningTime="2026-10-30 20:00:48.123098793 +0000 UTC m=+2695.693564393"
Oct 30 20:00:48.244607 ip-10-0-132-82 kubenswrapper[4892]: I1030 20:00:48.244541    4892 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-584d0bf7-wx8bn" podStartSLOduration=2.244521261 podStartE2EDuration="2.244521261s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 20:00:46 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 20:00:48.242535881 +0000 UTC m=+2695.813001479" watchObservedRunningTime="2026-10-30 20:00:48.244521261 +0000 UTC m=+2695.814986860"
Oct 30 20:00:48.511236 ip-10-0-132-82 kubenswrapper[4892]: I1030 20:00:48.511184    4892 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b372b6b1-jhvr9" podStartSLOduration=3.5111705029999998 podStartE2EDuration="3.511170503s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 20:00:45 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 20:00:48.508519467 +0000 UTC m=+2696.078985079" watchObservedRunningTime="2026-10-30 20:00:48.511170503 +0000 UTC m=+2696.081636102"
Oct 30 20:00:48.799176 ip-10-0-132-82 kubenswrapper[4892]: I1030 20:00:48.799102    4892 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w0-6ac304-pcccr"]
Oct 30 20:00:48.804145 ip-10-0-132-82 kubenswrapper[4892]: I1030 20:00:48.804120    4892 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w0-6ac304-pcccr"]
Oct 30 20:00:49.027371 ip-10-0-132-82 kubenswrapper[4892]: I1030 20:00:49.027337    4892 generic.go:356] "Generic (PLEG): container finished" podID="03e83bd1-55e2-4aa5-956b-7964b3529ad4" containerID="db0d3018ceb00f3dea046d1269b7ac0fa7ba6998159960f6b6b146602e027de0" exitCode=0
Oct 30 20:00:49.027824 ip-10-0-132-82 kubenswrapper[4892]: I1030 20:00:49.027375    4892 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-de9ce181-rtrn7" event={"ID":"03e83bd1-55e2-4aa5-956b-7964b3529ad4","Type":"ContainerDied","Data":"db0d3018ceb00f3dea046d1269b7ac0fa7ba6998159960f6b6b146602e027de0"}
Oct 30 20:00:49.029078 ip-10-0-132-82 kubenswrapper[4892]: I1030 20:00:49.029055    4892 generic.go:356] "Generic (PLEG): container finished" podID="05dbc393-f688-4634-ab42-8c00bf839520" containerID="258af30f9920c6703e06161d0a4019f9c452a0bdc78f4d27359a7a2444b2694d" exitCode=0
Oct 30 20:00:49.029219 ip-10-0-132-82 kubenswrapper[4892]: I1030 20:00:49.029144    4892 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b372b6b1-jhvr9" event={"ID":"05dbc393-f688-4634-ab42-8c00bf839520","Type":"ContainerDied","Data":"258af30f9920c6703e06161d0a4019f9c452a0bdc78f4d27359a7a2444b2694d"}
Oct 30 20:00:49.030668 ip-10-0-132-82 kubenswrapper[4892]: I1030 20:00:49.030651    4892 generic.go:356] "Generic (PLEG): container finished" podID="956121ad-97c4-4c74-8934-3367fbff6c24" containerID="ddf751afe27a96f71b7695b975e5daf54bafd08d5680891f8a516c52dc068ed5" exitCode=0
Oct 30 20:00:49.030739 ip-10-0-132-82 kubenswrapper[4892]: I1030 20:00:49.030723    4892 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-584d0bf7-wx8bn" event={"ID":"956121ad-97c4-4c74-8934-3367fbff6c24","Type":"ContainerDied","Data":"ddf751afe27a96f71b7695b975e5daf54bafd08d5680891f8a516c52dc068ed5"}
Oct 30 20:00:49.168097 ip-10-0-132-82 kubenswrapper[4892]: I1030 20:00:49.168070    4892 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-fa928509-4w4hc"
Oct 30 20:00:49.264209 ip-10-0-132-82 kubenswrapper[4892]: I1030 20:00:49.264185    4892 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/a4f83474-cd19-4c19-b6d5-2914706168e8-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a4f83474-cd19-4c19-b6d5-2914706168e8-maas-api-xks-ca\") pod \"a4f83474-cd19-4c19-b6d5-2914706168e8\" (UID: \"a4f83474-cd19-4c19-b6d5-2914706168e8\") "
Oct 30 20:00:49.264357 ip-10-0-132-82 kubenswrapper[4892]: I1030 20:00:49.264315    4892 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a4f83474-cd19-4c19-b6d5-2914706168e8-kube-api-access-zf6gl\" (UniqueName: \"kubernetes.io/projected/a4f83474-cd19-4c19-b6d5-2914706168e8-kube-api-access-zf6gl\") pod \"a4f83474-cd19-4c19-b6d5-2914706168e8\" (UID: \"a4f83474-cd19-4c19-b6d5-2914706168e8\") "
Oct 30 20:00:49.264357 ip-10-0-132-82 kubenswrapper[4892]: I1030 20:00:49.264351    4892 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a4f83474-cd19-4c19-b6d5-2914706168e8-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a4f83474-cd19-4c19-b6d5-2914706168e8-maas-api-openshift-ca\") pod \"a4f83474-cd19-4c19-b6d5-2914706168e8\" (UID: \"a4f83474-cd19-4c19-b6d5-2914706168e8\") "
Oct 30 20:00:49.264709 ip-10-0-132-82 kubenswrapper[4892]: I1030 20:00:49.264683    4892 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a4f83474-cd19-4c19-b6d5-2914706168e8-maas-api-openshift-ca" pod "a4f83474-cd19-4c19-b6d5-2914706168e8" (UID: "a4f83474-cd19-4c19-b6d5-2914706168e8"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 20:00:49.266365 ip-10-0-132-82 kubenswrapper[4892]: I1030 20:00:49.266339    4892 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a4f83474-cd19-4c19-b6d5-2914706168e8-maas-api-xks-ca" pod "a4f83474-cd19-4c19-b6d5-2914706168e8" (UID: "a4f83474-cd19-4c19-b6d5-2914706168e8"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 20:00:49.266765 ip-10-0-132-82 kubenswrapper[4892]: I1030 20:00:49.266699    4892 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a4f83474-cd19-4c19-b6d5-2914706168e8-kube-api-access-zf6gl" pod "a4f83474-cd19-4c19-b6d5-2914706168e8" (UID: "a4f83474-cd19-4c19-b6d5-2914706168e8"). InnerVolumeSpecName "kube-api-access-zf6gl". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 20:00:49.366100 ip-10-0-132-82 kubenswrapper[4892]: I1030 20:00:49.366044    4892 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a4f83474-cd19-4c19-b6d5-2914706168e8-maas-api-xks-ca\") on node \"ip-10-0-132-82.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:00:49.366100 ip-10-0-132-82 kubenswrapper[4892]: I1030 20:00:49.366066    4892 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zf6gl\" (UniqueName: \"kubernetes.io/projected/a4f83474-cd19-4c19-b6d5-2914706168e8-kube-api-access-zf6gl\") on node \"ip-10-0-132-82.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:00:49.366100 ip-10-0-132-82 kubenswrapper[4892]: I1030 20:00:49.366076    4892 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a4f83474-cd19-4c19-b6d5-2914706168e8-maas-api-openshift-ca\") on node \"ip-10-0-132-82.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:00:50.039303 ip-10-0-132-82 kubenswrapper[4892]: I1030 20:00:50.039275    4892 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-fa928509-4w4hc"
Oct 30 20:00:50.039805 ip-10-0-132-82 kubenswrapper[4892]: I1030 20:00:50.039312    4892 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-fa928509-4w4hc" event={"ID":"a4f83474-cd19-4c19-b6d5-2914706168e8","Type":"ContainerDied","Data":"7312d18cb3ee052350fbe8db104d3fb52882abea92d91ebba251d3bdfff9652e"}
Oct 30 20:00:50.039805 ip-10-0-132-82 kubenswrapper[4892]: I1030 20:00:50.039391    4892 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="7312d18cb3ee052350fbe8db104d3fb52882abea92d91ebba251d3bdfff9652e"
Oct 30 20:00:50.203150 ip-10-0-132-82 kubenswrapper[4892]: I1030 20:00:50.203121    4892 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-de9ce181-rtrn7"
Oct 30 20:00:50.225052 ip-10-0-132-82 kubenswrapper[4892]: I1030 20:00:50.225029    4892 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-584d0bf7-wx8bn"
Oct 30 20:00:50.249462 ip-10-0-132-82 kubenswrapper[4892]: I1030 20:00:50.249440    4892 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b372b6b1-jhvr9"
Oct 30 20:00:50.378410 ip-10-0-132-82 kubenswrapper[4892]: I1030 20:00:50.378337    4892 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/956121ad-97c4-4c74-8934-3367fbff6c24-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/956121ad-97c4-4c74-8934-3367fbff6c24-maas-api-openshift-ca\") pod \"956121ad-97c4-4c74-8934-3367fbff6c24\" (UID: \"956121ad-97c4-4c74-8934-3367fbff6c24\") "
Oct 30 20:00:50.378631 ip-10-0-132-82 kubenswrapper[4892]: I1030 20:00:50.378425    4892 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/956121ad-97c4-4c74-8934-3367fbff6c24-kube-api-access-5w8vx\" (UniqueName: \"kubernetes.io/projected/956121ad-97c4-4c74-8934-3367fbff6c24-kube-api-access-5w8vx\") pod \"956121ad-97c4-4c74-8934-3367fbff6c24\" (UID: \"956121ad-97c4-4c74-8934-3367fbff6c24\") "
Oct 30 20:00:50.378631 ip-10-0-132-82 kubenswrapper[4892]: I1030 20:00:50.378469    4892 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/03e83bd1-55e2-4aa5-956b-7964b3529ad4-kube-api-access-h8w5n\" (UniqueName: \"kubernetes.io/projected/03e83bd1-55e2-4aa5-956b-7964b3529ad4-kube-api-access-h8w5n\") pod \"03e83bd1-55e2-4aa5-956b-7964b3529ad4\" (UID: \"03e83bd1-55e2-4aa5-956b-7964b3529ad4\") "
Oct 30 20:00:50.378631 ip-10-0-132-82 kubenswrapper[4892]: I1030 20:00:50.378499    4892 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/05dbc393-f688-4634-ab42-8c00bf839520-kube-api-access-6hfnx\" (UniqueName: \"kubernetes.io/projected/05dbc393-f688-4634-ab42-8c00bf839520-kube-api-access-6hfnx\") pod \"05dbc393-f688-4634-ab42-8c00bf839520\" (UID: \"05dbc393-f688-4634-ab42-8c00bf839520\") "
Oct 30 20:00:50.378631 ip-10-0-132-82 kubenswrapper[4892]: I1030 20:00:50.378546    4892 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/03e83bd1-55e2-4aa5-956b-7964b3529ad4-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/03e83bd1-55e2-4aa5-956b-7964b3529ad4-maas-api-xks-ca\") pod \"03e83bd1-55e2-4aa5-956b-7964b3529ad4\" (UID: \"03e83bd1-55e2-4aa5-956b-7964b3529ad4\") "
Oct 30 20:00:50.378631 ip-10-0-132-82 kubenswrapper[4892]: I1030 20:00:50.378615    4892 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/05dbc393-f688-4634-ab42-8c00bf839520-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/05dbc393-f688-4634-ab42-8c00bf839520-maas-api-openshift-ca\") pod \"05dbc393-f688-4634-ab42-8c00bf839520\" (UID: \"05dbc393-f688-4634-ab42-8c00bf839520\") "
Oct 30 20:00:50.378955 ip-10-0-132-82 kubenswrapper[4892]: I1030 20:00:50.378783    4892 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/956121ad-97c4-4c74-8934-3367fbff6c24-maas-api-openshift-ca" pod "956121ad-97c4-4c74-8934-3367fbff6c24" (UID: "956121ad-97c4-4c74-8934-3367fbff6c24"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 20:00:50.379063 ip-10-0-132-82 kubenswrapper[4892]: I1030 20:00:50.379033    4892 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/05dbc393-f688-4634-ab42-8c00bf839520-maas-api-openshift-ca" pod "05dbc393-f688-4634-ab42-8c00bf839520" (UID: "05dbc393-f688-4634-ab42-8c00bf839520"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 20:00:50.379133 ip-10-0-132-82 kubenswrapper[4892]: I1030 20:00:50.378645    4892 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/956121ad-97c4-4c74-8934-3367fbff6c24-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/956121ad-97c4-4c74-8934-3367fbff6c24-maas-api-xks-ca\") pod \"956121ad-97c4-4c74-8934-3367fbff6c24\" (UID: \"956121ad-97c4-4c74-8934-3367fbff6c24\") "
Oct 30 20:00:50.379133 ip-10-0-132-82 kubenswrapper[4892]: I1030 20:00:50.379122    4892 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/05dbc393-f688-4634-ab42-8c00bf839520-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/05dbc393-f688-4634-ab42-8c00bf839520-maas-api-xks-ca\") pod \"05dbc393-f688-4634-ab42-8c00bf839520\" (UID: \"05dbc393-f688-4634-ab42-8c00bf839520\") "
Oct 30 20:00:50.379244 ip-10-0-132-82 kubenswrapper[4892]: I1030 20:00:50.379176    4892 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/03e83bd1-55e2-4aa5-956b-7964b3529ad4-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/03e83bd1-55e2-4aa5-956b-7964b3529ad4-maas-api-openshift-ca\") pod \"03e83bd1-55e2-4aa5-956b-7964b3529ad4\" (UID: \"03e83bd1-55e2-4aa5-956b-7964b3529ad4\") "
Oct 30 20:00:50.379778 ip-10-0-132-82 kubenswrapper[4892]: I1030 20:00:50.379754    4892 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/03e83bd1-55e2-4aa5-956b-7964b3529ad4-maas-api-openshift-ca" pod "03e83bd1-55e2-4aa5-956b-7964b3529ad4" (UID: "03e83bd1-55e2-4aa5-956b-7964b3529ad4"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 20:00:50.380155 ip-10-0-132-82 kubenswrapper[4892]: I1030 20:00:50.379993    4892 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/05dbc393-f688-4634-ab42-8c00bf839520-maas-api-openshift-ca\") on node \"ip-10-0-132-82.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:00:50.380155 ip-10-0-132-82 kubenswrapper[4892]: I1030 20:00:50.380122    4892 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/956121ad-97c4-4c74-8934-3367fbff6c24-maas-api-openshift-ca\") on node \"ip-10-0-132-82.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:00:50.381466 ip-10-0-132-82 kubenswrapper[4892]: I1030 20:00:50.381439    4892 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/956121ad-97c4-4c74-8934-3367fbff6c24-kube-api-access-5w8vx" pod "956121ad-97c4-4c74-8934-3367fbff6c24" (UID: "956121ad-97c4-4c74-8934-3367fbff6c24"). InnerVolumeSpecName "kube-api-access-5w8vx". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 20:00:50.381584 ip-10-0-132-82 kubenswrapper[4892]: I1030 20:00:50.381478    4892 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/05dbc393-f688-4634-ab42-8c00bf839520-kube-api-access-6hfnx" pod "05dbc393-f688-4634-ab42-8c00bf839520" (UID: "05dbc393-f688-4634-ab42-8c00bf839520"). InnerVolumeSpecName "kube-api-access-6hfnx". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 20:00:50.381724 ip-10-0-132-82 kubenswrapper[4892]: I1030 20:00:50.381697    4892 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/05dbc393-f688-4634-ab42-8c00bf839520-maas-api-xks-ca" pod "05dbc393-f688-4634-ab42-8c00bf839520" (UID: "05dbc393-f688-4634-ab42-8c00bf839520"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 20:00:50.381860 ip-10-0-132-82 kubenswrapper[4892]: I1030 20:00:50.381838    4892 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/03e83bd1-55e2-4aa5-956b-7964b3529ad4-maas-api-xks-ca" pod "03e83bd1-55e2-4aa5-956b-7964b3529ad4" (UID: "03e83bd1-55e2-4aa5-956b-7964b3529ad4"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 20:00:50.382096 ip-10-0-132-82 kubenswrapper[4892]: I1030 20:00:50.382074    4892 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/956121ad-97c4-4c74-8934-3367fbff6c24-maas-api-xks-ca" pod "956121ad-97c4-4c74-8934-3367fbff6c24" (UID: "956121ad-97c4-4c74-8934-3367fbff6c24"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 20:00:50.382191 ip-10-0-132-82 kubenswrapper[4892]: I1030 20:00:50.382172    4892 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/03e83bd1-55e2-4aa5-956b-7964b3529ad4-kube-api-access-h8w5n" pod "03e83bd1-55e2-4aa5-956b-7964b3529ad4" (UID: "03e83bd1-55e2-4aa5-956b-7964b3529ad4"). InnerVolumeSpecName "kube-api-access-h8w5n". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 20:00:50.480727 ip-10-0-132-82 kubenswrapper[4892]: I1030 20:00:50.480697    4892 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5w8vx\" (UniqueName: \"kubernetes.io/projected/956121ad-97c4-4c74-8934-3367fbff6c24-kube-api-access-5w8vx\") on node \"ip-10-0-132-82.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:00:50.480727 ip-10-0-132-82 kubenswrapper[4892]: I1030 20:00:50.480726    4892 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-h8w5n\" (UniqueName: \"kubernetes.io/projected/03e83bd1-55e2-4aa5-956b-7964b3529ad4-kube-api-access-h8w5n\") on node \"ip-10-0-132-82.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:00:50.480909 ip-10-0-132-82 kubenswrapper[4892]: I1030 20:00:50.480735    4892 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6hfnx\" (UniqueName: \"kubernetes.io/projected/05dbc393-f688-4634-ab42-8c00bf839520-kube-api-access-6hfnx\") on node \"ip-10-0-132-82.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:00:50.480909 ip-10-0-132-82 kubenswrapper[4892]: I1030 20:00:50.480745    4892 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/03e83bd1-55e2-4aa5-956b-7964b3529ad4-maas-api-xks-ca\") on node \"ip-10-0-132-82.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:00:50.480909 ip-10-0-132-82 kubenswrapper[4892]: I1030 20:00:50.480754    4892 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/956121ad-97c4-4c74-8934-3367fbff6c24-maas-api-xks-ca\") on node \"ip-10-0-132-82.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:00:50.480909 ip-10-0-132-82 kubenswrapper[4892]: I1030 20:00:50.480763    4892 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/05dbc393-f688-4634-ab42-8c00bf839520-maas-api-xks-ca\") on node \"ip-10-0-132-82.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:00:50.480909 ip-10-0-132-82 kubenswrapper[4892]: I1030 20:00:50.480772    4892 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/03e83bd1-55e2-4aa5-956b-7964b3529ad4-maas-api-openshift-ca\") on node \"ip-10-0-132-82.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:00:50.570892 ip-10-0-132-82 kubenswrapper[4892]: I1030 20:00:50.570859    4892 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e5edd2c3-9676-48a1-9b30-8d8a98512582" path="/var/lib/kubelet/pods/e5edd2c3-9676-48a1-9b30-8d8a98512582/volumes"
Oct 30 20:00:51.049576 ip-10-0-132-82 kubenswrapper[4892]: I1030 20:00:51.049541    4892 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-de9ce181-rtrn7"
Oct 30 20:00:51.050032 ip-10-0-132-82 kubenswrapper[4892]: I1030 20:00:51.049549    4892 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-de9ce181-rtrn7" event={"ID":"03e83bd1-55e2-4aa5-956b-7964b3529ad4","Type":"ContainerDied","Data":"e30d527635c75f3ce21792264c46e63e279056dda4dc57c87a92bfb4399dcd9a"}
Oct 30 20:00:51.050032 ip-10-0-132-82 kubenswrapper[4892]: I1030 20:00:51.049638    4892 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="e30d527635c75f3ce21792264c46e63e279056dda4dc57c87a92bfb4399dcd9a"
Oct 30 20:00:51.051017 ip-10-0-132-82 kubenswrapper[4892]: I1030 20:00:51.050994    4892 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b372b6b1-jhvr9"
Oct 30 20:00:51.051017 ip-10-0-132-82 kubenswrapper[4892]: I1030 20:00:51.051000    4892 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b372b6b1-jhvr9" event={"ID":"05dbc393-f688-4634-ab42-8c00bf839520","Type":"ContainerDied","Data":"94bd73bf9fcf841fa1f322cffa7f0c91c9b6ebf9cdacf59b7c18a057cf190397"}
Oct 30 20:00:51.051194 ip-10-0-132-82 kubenswrapper[4892]: I1030 20:00:51.051045    4892 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="94bd73bf9fcf841fa1f322cffa7f0c91c9b6ebf9cdacf59b7c18a057cf190397"
Oct 30 20:00:51.052744 ip-10-0-132-82 kubenswrapper[4892]: I1030 20:00:51.052719    4892 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-584d0bf7-wx8bn" event={"ID":"956121ad-97c4-4c74-8934-3367fbff6c24","Type":"ContainerDied","Data":"8e8886030988aed2573f9d004b2ffdaf62c2c26d27803ba25e6ab0da1ebb5833"}
Oct 30 20:00:51.052898 ip-10-0-132-82 kubenswrapper[4892]: I1030 20:00:51.052762    4892 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="8e8886030988aed2573f9d004b2ffdaf62c2c26d27803ba25e6ab0da1ebb5833"
Oct 30 20:00:51.052898 ip-10-0-132-82 kubenswrapper[4892]: I1030 20:00:51.052724    4892 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-584d0bf7-wx8bn"
Oct 30 20:00:51.656424 ip-10-0-132-82 kubenswrapper[4892]: I1030 20:00:51.656383    4892 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-shared-a-w3-cf9b6f-7bb7c4ccf4-zc2s9" podUID="44646b7f-870a-49c1-bd3d-8e3e7817d6a3" containerName="payload-processing" containerID="cri-o://4d233e5567a3b6abc12cf1ec1841e234250403e061929b81c33958ee9a4b79fc" gracePeriod=25
Oct 30 20:00:51.851207 ip-10-0-132-82 kubenswrapper[4892]: I1030 20:00:51.851168    4892 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-cf9b6f-59d848ddcf-jxlzh" podUID="4bd85267-2235-4bc6-80cd-473a48b0098c" containerName="payload-pre-processing" containerID="cri-o://b14502fe4af0f3ad1bdf46b8cc4c5b7c04040ee3b14e2eb804d39f95a2741824" gracePeriod=25
Oct 30 20:00:51.979188 ip-10-0-132-82 kubenswrapper[4892]: I1030 20:00:51.979165    4892 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w3-cf9b6f-7bb7c4ccf4-zc2s9"
Oct 30 20:00:52.065933 ip-10-0-132-82 kubenswrapper[4892]: I1030 20:00:52.065898    4892 generic.go:356] "Generic (PLEG): container finished" podID="4bd85267-2235-4bc6-80cd-473a48b0098c" containerID="b14502fe4af0f3ad1bdf46b8cc4c5b7c04040ee3b14e2eb804d39f95a2741824" exitCode=0
Oct 30 20:00:52.066363 ip-10-0-132-82 kubenswrapper[4892]: I1030 20:00:52.065987    4892 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-cf9b6f-59d848ddcf-jxlzh" event={"ID":"4bd85267-2235-4bc6-80cd-473a48b0098c","Type":"ContainerDied","Data":"b14502fe4af0f3ad1bdf46b8cc4c5b7c04040ee3b14e2eb804d39f95a2741824"}
Oct 30 20:00:52.067756 ip-10-0-132-82 kubenswrapper[4892]: I1030 20:00:52.067710    4892 generic.go:356] "Generic (PLEG): container finished" podID="44646b7f-870a-49c1-bd3d-8e3e7817d6a3" containerID="4d233e5567a3b6abc12cf1ec1841e234250403e061929b81c33958ee9a4b79fc" exitCode=0
Oct 30 20:00:52.068007 ip-10-0-132-82 kubenswrapper[4892]: I1030 20:00:52.067779    4892 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w3-cf9b6f-7bb7c4ccf4-zc2s9"
Oct 30 20:00:52.068100 ip-10-0-132-82 kubenswrapper[4892]: I1030 20:00:52.068005    4892 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w3-cf9b6f-7bb7c4ccf4-zc2s9" event={"ID":"44646b7f-870a-49c1-bd3d-8e3e7817d6a3","Type":"ContainerDied","Data":"4d233e5567a3b6abc12cf1ec1841e234250403e061929b81c33958ee9a4b79fc"}
Oct 30 20:00:52.068100 ip-10-0-132-82 kubenswrapper[4892]: I1030 20:00:52.068066    4892 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w3-cf9b6f-7bb7c4ccf4-zc2s9" event={"ID":"44646b7f-870a-49c1-bd3d-8e3e7817d6a3","Type":"ContainerDied","Data":"7d97dd2f5c482b3d6d85d7e9a11c2b16df4ac9483c52b0dc9fbca1000e1fa67f"}
Oct 30 20:00:52.068100 ip-10-0-132-82 kubenswrapper[4892]: I1030 20:00:52.068098    4892 scope.go:122] "RemoveContainer" containerID="4d233e5567a3b6abc12cf1ec1841e234250403e061929b81c33958ee9a4b79fc"
Oct 30 20:00:52.081176 ip-10-0-132-82 kubenswrapper[4892]: I1030 20:00:52.081155    4892 scope.go:122] "RemoveContainer" containerID="4d233e5567a3b6abc12cf1ec1841e234250403e061929b81c33958ee9a4b79fc"
Oct 30 20:00:52.081513 ip-10-0-132-82 kubenswrapper[4892]: E1030 20:00:52.081489    4892 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4d233e5567a3b6abc12cf1ec1841e234250403e061929b81c33958ee9a4b79fc\": container with ID starting with 4d233e5567a3b6abc12cf1ec1841e234250403e061929b81c33958ee9a4b79fc not found: ID does not exist" containerID="4d233e5567a3b6abc12cf1ec1841e234250403e061929b81c33958ee9a4b79fc"
Oct 30 20:00:52.081609 ip-10-0-132-82 kubenswrapper[4892]: I1030 20:00:52.081527    4892 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4d233e5567a3b6abc12cf1ec1841e234250403e061929b81c33958ee9a4b79fc"} err="failed to get container status \"4d233e5567a3b6abc12cf1ec1841e234250403e061929b81c33958ee9a4b79fc\": rpc error: code = NotFound desc = could not find container \"4d233e5567a3b6abc12cf1ec1841e234250403e061929b81c33958ee9a4b79fc\": container with ID starting with 4d233e5567a3b6abc12cf1ec1841e234250403e061929b81c33958ee9a4b79fc not found: ID does not exist"
Oct 30 20:00:52.095289 ip-10-0-132-82 kubenswrapper[4892]: I1030 20:00:52.095265    4892 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/44646b7f-870a-49c1-bd3d-8e3e7817d6a3-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/44646b7f-870a-49c1-bd3d-8e3e7817d6a3-plugins-config-volume\") pod \"44646b7f-870a-49c1-bd3d-8e3e7817d6a3\" (UID: \"44646b7f-870a-49c1-bd3d-8e3e7817d6a3\") "
Oct 30 20:00:52.095388 ip-10-0-132-82 kubenswrapper[4892]: I1030 20:00:52.095299    4892 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/44646b7f-870a-49c1-bd3d-8e3e7817d6a3-kube-api-access-mpw2q\" (UniqueName: \"kubernetes.io/projected/44646b7f-870a-49c1-bd3d-8e3e7817d6a3-kube-api-access-mpw2q\") pod \"44646b7f-870a-49c1-bd3d-8e3e7817d6a3\" (UID: \"44646b7f-870a-49c1-bd3d-8e3e7817d6a3\") "
Oct 30 20:00:52.095663 ip-10-0-132-82 kubenswrapper[4892]: I1030 20:00:52.095639    4892 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/44646b7f-870a-49c1-bd3d-8e3e7817d6a3-plugins-config-volume" pod "44646b7f-870a-49c1-bd3d-8e3e7817d6a3" (UID: "44646b7f-870a-49c1-bd3d-8e3e7817d6a3"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 20:00:52.097417 ip-10-0-132-82 kubenswrapper[4892]: I1030 20:00:52.097397    4892 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/44646b7f-870a-49c1-bd3d-8e3e7817d6a3-kube-api-access-mpw2q" pod "44646b7f-870a-49c1-bd3d-8e3e7817d6a3" (UID: "44646b7f-870a-49c1-bd3d-8e3e7817d6a3"). InnerVolumeSpecName "kube-api-access-mpw2q". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 20:00:52.112925 ip-10-0-132-82 kubenswrapper[4892]: I1030 20:00:52.112904    4892 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-cf9b6f-59d848ddcf-jxlzh"
Oct 30 20:00:52.197103 ip-10-0-132-82 kubenswrapper[4892]: I1030 20:00:52.197079    4892 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/44646b7f-870a-49c1-bd3d-8e3e7817d6a3-plugins-config-volume\") on node \"ip-10-0-132-82.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:00:52.197103 ip-10-0-132-82 kubenswrapper[4892]: I1030 20:00:52.197104    4892 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mpw2q\" (UniqueName: \"kubernetes.io/projected/44646b7f-870a-49c1-bd3d-8e3e7817d6a3-kube-api-access-mpw2q\") on node \"ip-10-0-132-82.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:00:52.308032 ip-10-0-132-82 kubenswrapper[4892]: I1030 20:00:52.306308    4892 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4bd85267-2235-4bc6-80cd-473a48b0098c-kube-api-access-qwskv\" (UniqueName: \"kubernetes.io/projected/4bd85267-2235-4bc6-80cd-473a48b0098c-kube-api-access-qwskv\") pod \"4bd85267-2235-4bc6-80cd-473a48b0098c\" (UID: \"4bd85267-2235-4bc6-80cd-473a48b0098c\") "
Oct 30 20:00:52.308032 ip-10-0-132-82 kubenswrapper[4892]: I1030 20:00:52.306410    4892 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/4bd85267-2235-4bc6-80cd-473a48b0098c-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4bd85267-2235-4bc6-80cd-473a48b0098c-plugins-config-volume\") pod \"4bd85267-2235-4bc6-80cd-473a48b0098c\" (UID: \"4bd85267-2235-4bc6-80cd-473a48b0098c\") "
Oct 30 20:00:52.308867 ip-10-0-132-82 kubenswrapper[4892]: I1030 20:00:52.308833    4892 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4bd85267-2235-4bc6-80cd-473a48b0098c-plugins-config-volume" pod "4bd85267-2235-4bc6-80cd-473a48b0098c" (UID: "4bd85267-2235-4bc6-80cd-473a48b0098c"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 20:00:52.311041 ip-10-0-132-82 kubenswrapper[4892]: I1030 20:00:52.311010    4892 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4bd85267-2235-4bc6-80cd-473a48b0098c-kube-api-access-qwskv" pod "4bd85267-2235-4bc6-80cd-473a48b0098c" (UID: "4bd85267-2235-4bc6-80cd-473a48b0098c"). InnerVolumeSpecName "kube-api-access-qwskv". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 20:00:52.408282 ip-10-0-132-82 kubenswrapper[4892]: I1030 20:00:52.408253    4892 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qwskv\" (UniqueName: \"kubernetes.io/projected/4bd85267-2235-4bc6-80cd-473a48b0098c-kube-api-access-qwskv\") on node \"ip-10-0-132-82.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:00:52.408282 ip-10-0-132-82 kubenswrapper[4892]: I1030 20:00:52.408279    4892 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4bd85267-2235-4bc6-80cd-473a48b0098c-plugins-config-volume\") on node \"ip-10-0-132-82.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:00:52.490082 ip-10-0-132-82 kubenswrapper[4892]: I1030 20:00:52.490051    4892 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w3-cf9b6f-7bb7c4ccf4-zc2s9"]
Oct 30 20:00:52.494925 ip-10-0-132-82 kubenswrapper[4892]: I1030 20:00:52.494904    4892 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w3-cf9b6f-7bb7c4ccf4-zc2s9"]
Oct 30 20:00:52.571580 ip-10-0-132-82 kubenswrapper[4892]: I1030 20:00:52.571523    4892 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="44646b7f-870a-49c1-bd3d-8e3e7817d6a3" path="/var/lib/kubelet/pods/44646b7f-870a-49c1-bd3d-8e3e7817d6a3/volumes"
Oct 30 20:00:53.079473 ip-10-0-132-82 kubenswrapper[4892]: I1030 20:00:53.079445    4892 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-cf9b6f-59d848ddcf-jxlzh"
Oct 30 20:00:53.079919 ip-10-0-132-82 kubenswrapper[4892]: I1030 20:00:53.079468    4892 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-cf9b6f-59d848ddcf-jxlzh" event={"ID":"4bd85267-2235-4bc6-80cd-473a48b0098c","Type":"ContainerDied","Data":"d9968232891cfdb768d249fec1a11fc94db1f847274e87aa519cfcbaf1f891b2"}
Oct 30 20:00:53.079919 ip-10-0-132-82 kubenswrapper[4892]: I1030 20:00:53.079570    4892 scope.go:122] "RemoveContainer" containerID="b14502fe4af0f3ad1bdf46b8cc4c5b7c04040ee3b14e2eb804d39f95a2741824"
Oct 30 20:00:53.244274 ip-10-0-132-82 kubenswrapper[4892]: I1030 20:00:53.244241    4892 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w3-cf9b6f-59d848ddcf-jxlzh"]
Oct 30 20:00:53.293529 ip-10-0-132-82 kubenswrapper[4892]: I1030 20:00:53.293497    4892 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w3-cf9b6f-59d848ddcf-jxlzh"]
Oct 30 20:00:53.342711 ip-10-0-132-82 kubenswrapper[4892]: I1030 20:00:53.342632    4892 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-tg5xj_11db9cec-653f-4aee-8937-0cf5b367efac/console-operator/3.log"
Oct 30 20:00:53.399049 ip-10-0-132-82 kubenswrapper[4892]: I1030 20:00:53.399026    4892 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-tg5xj_11db9cec-653f-4aee-8937-0cf5b367efac/console-operator/3.log"
Oct 30 20:00:54.569363 ip-10-0-132-82 kubenswrapper[4892]: I1030 20:00:54.569325    4892 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="4bd85267-2235-4bc6-80cd-473a48b0098c" path="/var/lib/kubelet/pods/4bd85267-2235-4bc6-80cd-473a48b0098c/volumes"
Oct 30 20:00:54.844545 ip-10-0-132-82 kubenswrapper[4892]: E1030 20:00:54.844463    4892 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-jrm87" containerName="extractor"
Oct 30 20:00:56.368725 ip-10-0-132-82 kubenswrapper[4892]: I1030 20:00:56.368693    4892 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w0-6ac304-openshift-default-8cdbdbc67-rgbzh"]
Oct 30 20:00:56.369267 ip-10-0-132-82 kubenswrapper[4892]: I1030 20:00:56.369015    4892 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-b-w0-6ac304-openshift-default-8cdbdbc67-rgbzh" podUID="0378dcfe-8d0a-43bb-808d-01a939801130" containerName="istio-proxy" containerID="cri-o://343ea86dd91bab14b3b2cdc43890a7dfe95c1eb3bc38abe0a14217acd0eb9d7d" gracePeriod=30
Oct 30 20:00:57.181031 ip-10-0-132-82 kubenswrapper[4892]: I1030 20:00:57.180959    4892 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-2d9861-b5kt4"]
Oct 30 20:00:57.181556 ip-10-0-132-82 kubenswrapper[4892]: I1030 20:00:57.181541    4892 cpu_manager.go:414] "containerMap: removing container" podUID="4bd85267-2235-4bc6-80cd-473a48b0098c" containerName="payload-pre-processing"
Oct 30 20:00:57.181599 ip-10-0-132-82 kubenswrapper[4892]: I1030 20:00:57.181558    4892 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4bd85267-2235-4bc6-80cd-473a48b0098c" containerName="payload-pre-processing"
Oct 30 20:00:57.181660 ip-10-0-132-82 kubenswrapper[4892]: I1030 20:00:57.181603    4892 cpu_manager.go:414] "containerMap: removing container" podUID="44646b7f-870a-49c1-bd3d-8e3e7817d6a3" containerName="payload-processing"
Oct 30 20:00:57.181660 ip-10-0-132-82 kubenswrapper[4892]: I1030 20:00:57.181610    4892 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="44646b7f-870a-49c1-bd3d-8e3e7817d6a3" containerName="payload-processing"
Oct 30 20:00:57.181660 ip-10-0-132-82 kubenswrapper[4892]: I1030 20:00:57.181620    4892 cpu_manager.go:414] "containerMap: removing container" podUID="05dbc393-f688-4634-ab42-8c00bf839520" containerName="revoke-keys"
Oct 30 20:00:57.181660 ip-10-0-132-82 kubenswrapper[4892]: I1030 20:00:57.181626    4892 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="05dbc393-f688-4634-ab42-8c00bf839520" containerName="revoke-keys"
Oct 30 20:00:57.181660 ip-10-0-132-82 kubenswrapper[4892]: I1030 20:00:57.181633    4892 cpu_manager.go:414] "containerMap: removing container" podUID="956121ad-97c4-4c74-8934-3367fbff6c24" containerName="revoke-keys"
Oct 30 20:00:57.181660 ip-10-0-132-82 kubenswrapper[4892]: I1030 20:00:57.181637    4892 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="956121ad-97c4-4c74-8934-3367fbff6c24" containerName="revoke-keys"
Oct 30 20:00:57.181660 ip-10-0-132-82 kubenswrapper[4892]: I1030 20:00:57.181642    4892 cpu_manager.go:414] "containerMap: removing container" podUID="03e83bd1-55e2-4aa5-956b-7964b3529ad4" containerName="revoke-keys"
Oct 30 20:00:57.181660 ip-10-0-132-82 kubenswrapper[4892]: I1030 20:00:57.181647    4892 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="03e83bd1-55e2-4aa5-956b-7964b3529ad4" containerName="revoke-keys"
Oct 30 20:00:57.181660 ip-10-0-132-82 kubenswrapper[4892]: I1030 20:00:57.181653    4892 cpu_manager.go:414] "containerMap: removing container" podUID="a4f83474-cd19-4c19-b6d5-2914706168e8" containerName="revoke-keys"
Oct 30 20:00:57.181660 ip-10-0-132-82 kubenswrapper[4892]: I1030 20:00:57.181657    4892 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a4f83474-cd19-4c19-b6d5-2914706168e8" containerName="revoke-keys"
Oct 30 20:00:57.182012 ip-10-0-132-82 kubenswrapper[4892]: I1030 20:00:57.181745    4892 memory_manager.go:365] "RemoveStaleState removing state" podUID="44646b7f-870a-49c1-bd3d-8e3e7817d6a3" containerName="payload-processing"
Oct 30 20:00:57.182012 ip-10-0-132-82 kubenswrapper[4892]: I1030 20:00:57.181755    4892 memory_manager.go:365] "RemoveStaleState removing state" podUID="05dbc393-f688-4634-ab42-8c00bf839520" containerName="revoke-keys"
Oct 30 20:00:57.182012 ip-10-0-132-82 kubenswrapper[4892]: I1030 20:00:57.181762    4892 memory_manager.go:365] "RemoveStaleState removing state" podUID="956121ad-97c4-4c74-8934-3367fbff6c24" containerName="revoke-keys"
Oct 30 20:00:57.182012 ip-10-0-132-82 kubenswrapper[4892]: I1030 20:00:57.181767    4892 memory_manager.go:365] "RemoveStaleState removing state" podUID="03e83bd1-55e2-4aa5-956b-7964b3529ad4" containerName="revoke-keys"
Oct 30 20:00:57.182012 ip-10-0-132-82 kubenswrapper[4892]: I1030 20:00:57.181772    4892 memory_manager.go:365] "RemoveStaleState removing state" podUID="a4f83474-cd19-4c19-b6d5-2914706168e8" containerName="revoke-keys"
Oct 30 20:00:57.182012 ip-10-0-132-82 kubenswrapper[4892]: I1030 20:00:57.181779    4892 memory_manager.go:365] "RemoveStaleState removing state" podUID="4bd85267-2235-4bc6-80cd-473a48b0098c" containerName="payload-pre-processing"
Oct 30 20:00:57.186491 ip-10-0-132-82 kubenswrapper[4892]: I1030 20:00:57.186475    4892 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-2d9861-b5kt4"
Oct 30 20:00:57.243280 ip-10-0-132-82 kubenswrapper[4892]: I1030 20:00:57.243248    4892 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 30 20:00:57.243426 ip-10-0-132-82 kubenswrapper[4892]: I1030 20:00:57.243373    4892 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 30 20:00:57.245395 ip-10-0-132-82 kubenswrapper[4892]: I1030 20:00:57.245378    4892 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 20:00:57.245601 ip-10-0-132-82 kubenswrapper[4892]: I1030 20:00:57.245587    4892 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-nlxp4\""
Oct 30 20:00:57.253596 ip-10-0-132-82 kubenswrapper[4892]: I1030 20:00:57.253567    4892 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/fc03b7fe-3210-4912-9437-f9a8fc838b28-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-w0-2d9861-b5kt4\" (UID: \"fc03b7fe-3210-4912-9437-f9a8fc838b28\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-2d9861-b5kt4"
Oct 30 20:00:57.253706 ip-10-0-132-82 kubenswrapper[4892]: I1030 20:00:57.253623    4892 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/fc03b7fe-3210-4912-9437-f9a8fc838b28-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-w0-2d9861-b5kt4\" (UID: \"fc03b7fe-3210-4912-9437-f9a8fc838b28\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-2d9861-b5kt4"
Oct 30 20:00:57.253834 ip-10-0-132-82 kubenswrapper[4892]: I1030 20:00:57.253818    4892 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bdtsf\" (UniqueName: \"kubernetes.io/projected/fc03b7fe-3210-4912-9437-f9a8fc838b28-kube-api-access-bdtsf\") pod \"maas-api-revoke-keys-e2e-worker-w0-2d9861-b5kt4\" (UID: \"fc03b7fe-3210-4912-9437-f9a8fc838b28\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-2d9861-b5kt4"
Oct 30 20:00:57.295931 ip-10-0-132-82 kubenswrapper[4892]: I1030 20:00:57.295905    4892 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-2d9861-b5kt4"]
Oct 30 20:00:57.354405 ip-10-0-132-82 kubenswrapper[4892]: I1030 20:00:57.354368    4892 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/fc03b7fe-3210-4912-9437-f9a8fc838b28-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-w0-2d9861-b5kt4\" (UID: \"fc03b7fe-3210-4912-9437-f9a8fc838b28\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-2d9861-b5kt4"
Oct 30 20:00:57.354405 ip-10-0-132-82 kubenswrapper[4892]: I1030 20:00:57.354407    4892 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/fc03b7fe-3210-4912-9437-f9a8fc838b28-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-w0-2d9861-b5kt4\" (UID: \"fc03b7fe-3210-4912-9437-f9a8fc838b28\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-2d9861-b5kt4"
Oct 30 20:00:57.354620 ip-10-0-132-82 kubenswrapper[4892]: I1030 20:00:57.354476    4892 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bdtsf\" (UniqueName: \"kubernetes.io/projected/fc03b7fe-3210-4912-9437-f9a8fc838b28-kube-api-access-bdtsf\") pod \"maas-api-revoke-keys-e2e-worker-w0-2d9861-b5kt4\" (UID: \"fc03b7fe-3210-4912-9437-f9a8fc838b28\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-2d9861-b5kt4"
Oct 30 20:00:57.355109 ip-10-0-132-82 kubenswrapper[4892]: I1030 20:00:57.355087    4892 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/fc03b7fe-3210-4912-9437-f9a8fc838b28-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-w0-2d9861-b5kt4\" (UID: \"fc03b7fe-3210-4912-9437-f9a8fc838b28\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-2d9861-b5kt4"
Oct 30 20:00:57.357122 ip-10-0-132-82 kubenswrapper[4892]: I1030 20:00:57.357098    4892 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/fc03b7fe-3210-4912-9437-f9a8fc838b28-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-w0-2d9861-b5kt4\" (UID: \"fc03b7fe-3210-4912-9437-f9a8fc838b28\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-2d9861-b5kt4"
Oct 30 20:00:57.411630 ip-10-0-132-82 kubenswrapper[4892]: I1030 20:00:57.411606    4892 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bdtsf\" (UniqueName: \"kubernetes.io/projected/fc03b7fe-3210-4912-9437-f9a8fc838b28-kube-api-access-bdtsf\") pod \"maas-api-revoke-keys-e2e-worker-w0-2d9861-b5kt4\" (UID: \"fc03b7fe-3210-4912-9437-f9a8fc838b28\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-2d9861-b5kt4"
Oct 30 20:00:57.499857 ip-10-0-132-82 kubenswrapper[4892]: I1030 20:00:57.499789    4892 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-2d9861-b5kt4"
Oct 30 20:00:57.884345 ip-10-0-132-82 kubenswrapper[4892]: I1030 20:00:57.884314    4892 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-2d9861-b5kt4"]
Oct 30 20:00:57.885329 ip-10-0-132-82 kubenswrapper[4892]: W1030 20:00:57.885302    4892 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfc03b7fe_3210_4912_9437_f9a8fc838b28.slice/crio-3971dc6b4c9211492bd4173f70791f7b3bd4c7f7326c36c4a4e6ef18e148dc5f WatchSource:0}: Error finding container 3971dc6b4c9211492bd4173f70791f7b3bd4c7f7326c36c4a4e6ef18e148dc5f: Status 404 returned error can't find the container with id 3971dc6b4c9211492bd4173f70791f7b3bd4c7f7326c36c4a4e6ef18e148dc5f
Oct 30 20:00:58.121805 ip-10-0-132-82 kubenswrapper[4892]: I1030 20:00:58.121774    4892 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-2d9861-b5kt4" event={"ID":"fc03b7fe-3210-4912-9437-f9a8fc838b28","Type":"ContainerStarted","Data":"3971dc6b4c9211492bd4173f70791f7b3bd4c7f7326c36c4a4e6ef18e148dc5f"}
Oct 30 20:00:59.174395 ip-10-0-132-82 kubenswrapper[4892]: I1030 20:00:59.173853    4892 generic.go:356] "Generic (PLEG): container finished" podID="fc03b7fe-3210-4912-9437-f9a8fc838b28" containerID="87784e77de2b9c57d01e135ffc4d04dd8cb3807a44e6c686fc17a1a3bcf72583" exitCode=0
Oct 30 20:00:59.174395 ip-10-0-132-82 kubenswrapper[4892]: I1030 20:00:59.173934    4892 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-2d9861-b5kt4" event={"ID":"fc03b7fe-3210-4912-9437-f9a8fc838b28","Type":"ContainerDied","Data":"87784e77de2b9c57d01e135ffc4d04dd8cb3807a44e6c686fc17a1a3bcf72583"}
Oct 30 20:01:00.324616 ip-10-0-132-82 kubenswrapper[4892]: I1030 20:01:00.324590    4892 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-2d9861-b5kt4"
Oct 30 20:01:00.391204 ip-10-0-132-82 kubenswrapper[4892]: I1030 20:01:00.391178    4892 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/fc03b7fe-3210-4912-9437-f9a8fc838b28-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/fc03b7fe-3210-4912-9437-f9a8fc838b28-maas-api-openshift-ca\") pod \"fc03b7fe-3210-4912-9437-f9a8fc838b28\" (UID: \"fc03b7fe-3210-4912-9437-f9a8fc838b28\") "
Oct 30 20:01:00.391336 ip-10-0-132-82 kubenswrapper[4892]: I1030 20:01:00.391226    4892 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/fc03b7fe-3210-4912-9437-f9a8fc838b28-kube-api-access-bdtsf\" (UniqueName: \"kubernetes.io/projected/fc03b7fe-3210-4912-9437-f9a8fc838b28-kube-api-access-bdtsf\") pod \"fc03b7fe-3210-4912-9437-f9a8fc838b28\" (UID: \"fc03b7fe-3210-4912-9437-f9a8fc838b28\") "
Oct 30 20:01:00.391336 ip-10-0-132-82 kubenswrapper[4892]: I1030 20:01:00.391267    4892 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/fc03b7fe-3210-4912-9437-f9a8fc838b28-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/fc03b7fe-3210-4912-9437-f9a8fc838b28-maas-api-xks-ca\") pod \"fc03b7fe-3210-4912-9437-f9a8fc838b28\" (UID: \"fc03b7fe-3210-4912-9437-f9a8fc838b28\") "
Oct 30 20:01:00.391569 ip-10-0-132-82 kubenswrapper[4892]: I1030 20:01:00.391546    4892 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/fc03b7fe-3210-4912-9437-f9a8fc838b28-maas-api-openshift-ca" pod "fc03b7fe-3210-4912-9437-f9a8fc838b28" (UID: "fc03b7fe-3210-4912-9437-f9a8fc838b28"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 20:01:00.393580 ip-10-0-132-82 kubenswrapper[4892]: I1030 20:01:00.393554    4892 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/fc03b7fe-3210-4912-9437-f9a8fc838b28-maas-api-xks-ca" pod "fc03b7fe-3210-4912-9437-f9a8fc838b28" (UID: "fc03b7fe-3210-4912-9437-f9a8fc838b28"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 20:01:00.393718 ip-10-0-132-82 kubenswrapper[4892]: I1030 20:01:00.393701    4892 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fc03b7fe-3210-4912-9437-f9a8fc838b28-kube-api-access-bdtsf" pod "fc03b7fe-3210-4912-9437-f9a8fc838b28" (UID: "fc03b7fe-3210-4912-9437-f9a8fc838b28"). InnerVolumeSpecName "kube-api-access-bdtsf". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 20:01:00.494327 ip-10-0-132-82 kubenswrapper[4892]: I1030 20:01:00.494249    4892 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/fc03b7fe-3210-4912-9437-f9a8fc838b28-maas-api-openshift-ca\") on node \"ip-10-0-132-82.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:01:00.494327 ip-10-0-132-82 kubenswrapper[4892]: I1030 20:01:00.494292    4892 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bdtsf\" (UniqueName: \"kubernetes.io/projected/fc03b7fe-3210-4912-9437-f9a8fc838b28-kube-api-access-bdtsf\") on node \"ip-10-0-132-82.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:01:00.494327 ip-10-0-132-82 kubenswrapper[4892]: I1030 20:01:00.494308    4892 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/fc03b7fe-3210-4912-9437-f9a8fc838b28-maas-api-xks-ca\") on node \"ip-10-0-132-82.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:01:01.190034 ip-10-0-132-82 kubenswrapper[4892]: I1030 20:01:01.189998    4892 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-2d9861-b5kt4"
Oct 30 20:01:01.190268 ip-10-0-132-82 kubenswrapper[4892]: I1030 20:01:01.190002    4892 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-2d9861-b5kt4" event={"ID":"fc03b7fe-3210-4912-9437-f9a8fc838b28","Type":"ContainerDied","Data":"3971dc6b4c9211492bd4173f70791f7b3bd4c7f7326c36c4a4e6ef18e148dc5f"}
Oct 30 20:01:01.190268 ip-10-0-132-82 kubenswrapper[4892]: I1030 20:01:01.190134    4892 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="3971dc6b4c9211492bd4173f70791f7b3bd4c7f7326c36c4a4e6ef18e148dc5f"
Oct 30 20:01:01.627333 ip-10-0-132-82 kubenswrapper[4892]: I1030 20:01:01.627311    4892 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w0-6ac304-openshift-default-8cdbdbc67-rgbzh"
Oct 30 20:01:01.706838 ip-10-0-132-82 kubenswrapper[4892]: I1030 20:01:01.706813    4892 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0378dcfe-8d0a-43bb-808d-01a939801130-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0378dcfe-8d0a-43bb-808d-01a939801130-workload-socket\") pod \"0378dcfe-8d0a-43bb-808d-01a939801130\" (UID: \"0378dcfe-8d0a-43bb-808d-01a939801130\") "
Oct 30 20:01:01.707013 ip-10-0-132-82 kubenswrapper[4892]: I1030 20:01:01.706866    4892 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0378dcfe-8d0a-43bb-808d-01a939801130-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0378dcfe-8d0a-43bb-808d-01a939801130-istio-envoy\") pod \"0378dcfe-8d0a-43bb-808d-01a939801130\" (UID: \"0378dcfe-8d0a-43bb-808d-01a939801130\") "
Oct 30 20:01:01.707013 ip-10-0-132-82 kubenswrapper[4892]: I1030 20:01:01.706911    4892 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0378dcfe-8d0a-43bb-808d-01a939801130-istio-token\" (UniqueName: \"kubernetes.io/projected/0378dcfe-8d0a-43bb-808d-01a939801130-istio-token\") pod \"0378dcfe-8d0a-43bb-808d-01a939801130\" (UID: \"0378dcfe-8d0a-43bb-808d-01a939801130\") "
Oct 30 20:01:01.707013 ip-10-0-132-82 kubenswrapper[4892]: I1030 20:01:01.706959    4892 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/0378dcfe-8d0a-43bb-808d-01a939801130-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0378dcfe-8d0a-43bb-808d-01a939801130-istiod-ca-cert\") pod \"0378dcfe-8d0a-43bb-808d-01a939801130\" (UID: \"0378dcfe-8d0a-43bb-808d-01a939801130\") "
Oct 30 20:01:01.707013 ip-10-0-132-82 kubenswrapper[4892]: I1030 20:01:01.707008    4892 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0378dcfe-8d0a-43bb-808d-01a939801130-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0378dcfe-8d0a-43bb-808d-01a939801130-credential-socket\") pod \"0378dcfe-8d0a-43bb-808d-01a939801130\" (UID: \"0378dcfe-8d0a-43bb-808d-01a939801130\") "
Oct 30 20:01:01.707251 ip-10-0-132-82 kubenswrapper[4892]: I1030 20:01:01.707057    4892 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0378dcfe-8d0a-43bb-808d-01a939801130-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0378dcfe-8d0a-43bb-808d-01a939801130-istio-data\") pod \"0378dcfe-8d0a-43bb-808d-01a939801130\" (UID: \"0378dcfe-8d0a-43bb-808d-01a939801130\") "
Oct 30 20:01:01.707251 ip-10-0-132-82 kubenswrapper[4892]: I1030 20:01:01.707124    4892 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/0378dcfe-8d0a-43bb-808d-01a939801130-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0378dcfe-8d0a-43bb-808d-01a939801130-istio-podinfo\") pod \"0378dcfe-8d0a-43bb-808d-01a939801130\" (UID: \"0378dcfe-8d0a-43bb-808d-01a939801130\") "
Oct 30 20:01:01.707251 ip-10-0-132-82 kubenswrapper[4892]: I1030 20:01:01.707131    4892 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0378dcfe-8d0a-43bb-808d-01a939801130-workload-socket" pod "0378dcfe-8d0a-43bb-808d-01a939801130" (UID: "0378dcfe-8d0a-43bb-808d-01a939801130"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 20:01:01.707251 ip-10-0-132-82 kubenswrapper[4892]: I1030 20:01:01.707209    4892 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0378dcfe-8d0a-43bb-808d-01a939801130-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0378dcfe-8d0a-43bb-808d-01a939801130-workload-certs\") pod \"0378dcfe-8d0a-43bb-808d-01a939801130\" (UID: \"0378dcfe-8d0a-43bb-808d-01a939801130\") "
Oct 30 20:01:01.707452 ip-10-0-132-82 kubenswrapper[4892]: I1030 20:01:01.707245    4892 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0378dcfe-8d0a-43bb-808d-01a939801130-kube-api-access-n2brw\" (UniqueName: \"kubernetes.io/projected/0378dcfe-8d0a-43bb-808d-01a939801130-kube-api-access-n2brw\") pod \"0378dcfe-8d0a-43bb-808d-01a939801130\" (UID: \"0378dcfe-8d0a-43bb-808d-01a939801130\") "
Oct 30 20:01:01.707452 ip-10-0-132-82 kubenswrapper[4892]: I1030 20:01:01.707362    4892 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0378dcfe-8d0a-43bb-808d-01a939801130-credential-socket" pod "0378dcfe-8d0a-43bb-808d-01a939801130" (UID: "0378dcfe-8d0a-43bb-808d-01a939801130"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 20:01:01.707452 ip-10-0-132-82 kubenswrapper[4892]: I1030 20:01:01.707375    4892 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0378dcfe-8d0a-43bb-808d-01a939801130-istio-data" pod "0378dcfe-8d0a-43bb-808d-01a939801130" (UID: "0378dcfe-8d0a-43bb-808d-01a939801130"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 20:01:01.707619 ip-10-0-132-82 kubenswrapper[4892]: I1030 20:01:01.707459    4892 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0378dcfe-8d0a-43bb-808d-01a939801130-istiod-ca-cert" pod "0378dcfe-8d0a-43bb-808d-01a939801130" (UID: "0378dcfe-8d0a-43bb-808d-01a939801130"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 20:01:01.707619 ip-10-0-132-82 kubenswrapper[4892]: I1030 20:01:01.707592    4892 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0378dcfe-8d0a-43bb-808d-01a939801130-workload-certs" pod "0378dcfe-8d0a-43bb-808d-01a939801130" (UID: "0378dcfe-8d0a-43bb-808d-01a939801130"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 20:01:01.708113 ip-10-0-132-82 kubenswrapper[4892]: I1030 20:01:01.708009    4892 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0378dcfe-8d0a-43bb-808d-01a939801130-credential-socket\") on node \"ip-10-0-132-82.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:01:01.708113 ip-10-0-132-82 kubenswrapper[4892]: I1030 20:01:01.708040    4892 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0378dcfe-8d0a-43bb-808d-01a939801130-istio-data\") on node \"ip-10-0-132-82.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:01:01.708113 ip-10-0-132-82 kubenswrapper[4892]: I1030 20:01:01.708055    4892 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0378dcfe-8d0a-43bb-808d-01a939801130-workload-certs\") on node \"ip-10-0-132-82.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:01:01.708113 ip-10-0-132-82 kubenswrapper[4892]: I1030 20:01:01.708071    4892 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0378dcfe-8d0a-43bb-808d-01a939801130-workload-socket\") on node \"ip-10-0-132-82.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:01:01.708113 ip-10-0-132-82 kubenswrapper[4892]: I1030 20:01:01.708085    4892 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0378dcfe-8d0a-43bb-808d-01a939801130-istiod-ca-cert\") on node \"ip-10-0-132-82.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:01:01.709308 ip-10-0-132-82 kubenswrapper[4892]: I1030 20:01:01.709288    4892 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0378dcfe-8d0a-43bb-808d-01a939801130-istio-token" pod "0378dcfe-8d0a-43bb-808d-01a939801130" (UID: "0378dcfe-8d0a-43bb-808d-01a939801130"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 20:01:01.709401 ip-10-0-132-82 kubenswrapper[4892]: I1030 20:01:01.709311    4892 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0378dcfe-8d0a-43bb-808d-01a939801130-istio-envoy" pod "0378dcfe-8d0a-43bb-808d-01a939801130" (UID: "0378dcfe-8d0a-43bb-808d-01a939801130"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 20:01:01.709586 ip-10-0-132-82 kubenswrapper[4892]: I1030 20:01:01.709571    4892 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/0378dcfe-8d0a-43bb-808d-01a939801130-istio-podinfo" pod "0378dcfe-8d0a-43bb-808d-01a939801130" (UID: "0378dcfe-8d0a-43bb-808d-01a939801130"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 20:01:01.709644 ip-10-0-132-82 kubenswrapper[4892]: I1030 20:01:01.709634    4892 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0378dcfe-8d0a-43bb-808d-01a939801130-kube-api-access-n2brw" pod "0378dcfe-8d0a-43bb-808d-01a939801130" (UID: "0378dcfe-8d0a-43bb-808d-01a939801130"). InnerVolumeSpecName "kube-api-access-n2brw". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 20:01:01.803462 ip-10-0-132-82 kubenswrapper[4892]: I1030 20:01:01.803393    4892 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w0-2d9861/distinct-2-w0-2d9861-kserve-679c6b56dd-nmhbm"]
Oct 30 20:01:01.809636 ip-10-0-132-82 kubenswrapper[4892]: I1030 20:01:01.809610    4892 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0378dcfe-8d0a-43bb-808d-01a939801130-istio-podinfo\") on node \"ip-10-0-132-82.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:01:01.809791 ip-10-0-132-82 kubenswrapper[4892]: I1030 20:01:01.809647    4892 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-n2brw\" (UniqueName: \"kubernetes.io/projected/0378dcfe-8d0a-43bb-808d-01a939801130-kube-api-access-n2brw\") on node \"ip-10-0-132-82.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:01:01.809791 ip-10-0-132-82 kubenswrapper[4892]: I1030 20:01:01.809664    4892 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0378dcfe-8d0a-43bb-808d-01a939801130-istio-envoy\") on node \"ip-10-0-132-82.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:01:01.809791 ip-10-0-132-82 kubenswrapper[4892]: I1030 20:01:01.809677    4892 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0378dcfe-8d0a-43bb-808d-01a939801130-istio-token\") on node \"ip-10-0-132-82.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:01:01.822729 ip-10-0-132-82 kubenswrapper[4892]: I1030 20:01:01.822701    4892 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w0-2d9861/premium-w0-2d9861-kserve-67945c6848-4n6kz"]
Oct 30 20:01:02.197918 ip-10-0-132-82 kubenswrapper[4892]: I1030 20:01:02.197878    4892 generic.go:356] "Generic (PLEG): container finished" podID="0378dcfe-8d0a-43bb-808d-01a939801130" containerID="343ea86dd91bab14b3b2cdc43890a7dfe95c1eb3bc38abe0a14217acd0eb9d7d" exitCode=0
Oct 30 20:01:02.198101 ip-10-0-132-82 kubenswrapper[4892]: I1030 20:01:02.197943    4892 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w0-6ac304-openshift-default-8cdbdbc67-rgbzh"
Oct 30 20:01:02.198101 ip-10-0-132-82 kubenswrapper[4892]: I1030 20:01:02.197948    4892 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w0-6ac304-openshift-default-8cdbdbc67-rgbzh" event={"ID":"0378dcfe-8d0a-43bb-808d-01a939801130","Type":"ContainerDied","Data":"343ea86dd91bab14b3b2cdc43890a7dfe95c1eb3bc38abe0a14217acd0eb9d7d"}
Oct 30 20:01:02.198101 ip-10-0-132-82 kubenswrapper[4892]: I1030 20:01:02.198032    4892 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w0-6ac304-openshift-default-8cdbdbc67-rgbzh" event={"ID":"0378dcfe-8d0a-43bb-808d-01a939801130","Type":"ContainerDied","Data":"0007b628d99c4db118be1064bae47abfc9d7dcd2fd0670ba91d16353b95520ee"}
Oct 30 20:01:02.198101 ip-10-0-132-82 kubenswrapper[4892]: I1030 20:01:02.198058    4892 scope.go:122] "RemoveContainer" containerID="343ea86dd91bab14b3b2cdc43890a7dfe95c1eb3bc38abe0a14217acd0eb9d7d"
Oct 30 20:01:02.209987 ip-10-0-132-82 kubenswrapper[4892]: I1030 20:01:02.209956    4892 scope.go:122] "RemoveContainer" containerID="b29ccaad7d805db2c688141f390e9322ab08c52aae531c10ef5dad204e09e713"
Oct 30 20:01:02.220056 ip-10-0-132-82 kubenswrapper[4892]: I1030 20:01:02.220038    4892 scope.go:122] "RemoveContainer" containerID="343ea86dd91bab14b3b2cdc43890a7dfe95c1eb3bc38abe0a14217acd0eb9d7d"
Oct 30 20:01:02.220347 ip-10-0-132-82 kubenswrapper[4892]: E1030 20:01:02.220325    4892 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"343ea86dd91bab14b3b2cdc43890a7dfe95c1eb3bc38abe0a14217acd0eb9d7d\": container with ID starting with 343ea86dd91bab14b3b2cdc43890a7dfe95c1eb3bc38abe0a14217acd0eb9d7d not found: ID does not exist" containerID="343ea86dd91bab14b3b2cdc43890a7dfe95c1eb3bc38abe0a14217acd0eb9d7d"
Oct 30 20:01:02.220391 ip-10-0-132-82 kubenswrapper[4892]: I1030 20:01:02.220356    4892 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"343ea86dd91bab14b3b2cdc43890a7dfe95c1eb3bc38abe0a14217acd0eb9d7d"} err="failed to get container status \"343ea86dd91bab14b3b2cdc43890a7dfe95c1eb3bc38abe0a14217acd0eb9d7d\": rpc error: code = NotFound desc = could not find container \"343ea86dd91bab14b3b2cdc43890a7dfe95c1eb3bc38abe0a14217acd0eb9d7d\": container with ID starting with 343ea86dd91bab14b3b2cdc43890a7dfe95c1eb3bc38abe0a14217acd0eb9d7d not found: ID does not exist"
Oct 30 20:01:02.220391 ip-10-0-132-82 kubenswrapper[4892]: I1030 20:01:02.220372    4892 scope.go:122] "RemoveContainer" containerID="b29ccaad7d805db2c688141f390e9322ab08c52aae531c10ef5dad204e09e713"
Oct 30 20:01:02.220657 ip-10-0-132-82 kubenswrapper[4892]: E1030 20:01:02.220640    4892 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b29ccaad7d805db2c688141f390e9322ab08c52aae531c10ef5dad204e09e713\": container with ID starting with b29ccaad7d805db2c688141f390e9322ab08c52aae531c10ef5dad204e09e713 not found: ID does not exist" containerID="b29ccaad7d805db2c688141f390e9322ab08c52aae531c10ef5dad204e09e713"
Oct 30 20:01:02.220701 ip-10-0-132-82 kubenswrapper[4892]: I1030 20:01:02.220664    4892 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b29ccaad7d805db2c688141f390e9322ab08c52aae531c10ef5dad204e09e713"} err="failed to get container status \"b29ccaad7d805db2c688141f390e9322ab08c52aae531c10ef5dad204e09e713\": rpc error: code = NotFound desc = could not find container \"b29ccaad7d805db2c688141f390e9322ab08c52aae531c10ef5dad204e09e713\": container with ID starting with b29ccaad7d805db2c688141f390e9322ab08c52aae531c10ef5dad204e09e713 not found: ID does not exist"
Oct 30 20:01:02.257713 ip-10-0-132-82 kubenswrapper[4892]: E1030 20:01:02.257680    4892 cadvisor_stats_provider.go:569] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0378dcfe_8d0a_43bb_808d_01a939801130.slice\": RecentStats: unable to find data in memory cache], [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0378dcfe_8d0a_43bb_808d_01a939801130.slice/crio-0007b628d99c4db118be1064bae47abfc9d7dcd2fd0670ba91d16353b95520ee\": RecentStats: unable to find data in memory cache]"
Oct 30 20:01:02.366175 ip-10-0-132-82 kubenswrapper[4892]: I1030 20:01:02.366115    4892 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w0-6ac304-openshift-default-8cdbdbc67-rgbzh"]
Oct 30 20:01:02.370111 ip-10-0-132-82 kubenswrapper[4892]: I1030 20:01:02.370087    4892 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-b-w0-6ac304-openshift-default-8cdbdbc67-rgbzh"]
Oct 30 20:01:02.570859 ip-10-0-132-82 kubenswrapper[4892]: I1030 20:01:02.570780    4892 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0378dcfe-8d0a-43bb-808d-01a939801130" path="/var/lib/kubelet/pods/0378dcfe-8d0a-43bb-808d-01a939801130/volumes"
Oct 30 20:01:05.636945 ip-10-0-132-82 kubenswrapper[4892]: I1030 20:01:05.636911    4892 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-cf9b6f-m26d6"]
Oct 30 20:01:05.685977 ip-10-0-132-82 kubenswrapper[4892]: I1030 20:01:05.685938    4892 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-cf9b6f-m26d6"]
Oct 30 20:01:06.570142 ip-10-0-132-82 kubenswrapper[4892]: I1030 20:01:06.570107    4892 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="aa6d7d96-304e-404f-b62e-68b3fa0a0266" path="/var/lib/kubelet/pods/aa6d7d96-304e-404f-b62e-68b3fa0a0266/volumes"
Oct 30 20:01:08.893402 ip-10-0-132-82 kubenswrapper[4892]: I1030 20:01:08.893366    4892 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w0-2d9861-d6c699c74-2cwqk"]
Oct 30 20:01:09.010785 ip-10-0-132-82 kubenswrapper[4892]: I1030 20:01:09.010749    4892 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w0-2d9861-78d7ffc559-lp7cm"]
Oct 30 20:01:12.753979 ip-10-0-132-82 kubenswrapper[4892]: I1030 20:01:12.753929    4892 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w3-cf9b6f-openshift-default-56db6769f7-xl4xv"]
Oct 30 20:01:12.754443 ip-10-0-132-82 kubenswrapper[4892]: I1030 20:01:12.754206    4892 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-a-w3-cf9b6f-openshift-default-56db6769f7-xl4xv" podUID="fd212b75-a818-45a3-946a-864a230b7707" containerName="istio-proxy" containerID="cri-o://a54da5a3cea214225b82ca4f392b05e4f74757f52c2e6374de65120122176f86" gracePeriod=30
Oct 30 20:01:13.382731 ip-10-0-132-82 kubenswrapper[4892]: I1030 20:01:13.382702    4892 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-c75ed4-jxjdl"]
Oct 30 20:01:13.383191 ip-10-0-132-82 kubenswrapper[4892]: I1030 20:01:13.383177    4892 cpu_manager.go:414] "containerMap: removing container" podUID="fc03b7fe-3210-4912-9437-f9a8fc838b28" containerName="revoke-keys"
Oct 30 20:01:13.383257 ip-10-0-132-82 kubenswrapper[4892]: I1030 20:01:13.383192    4892 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="fc03b7fe-3210-4912-9437-f9a8fc838b28" containerName="revoke-keys"
Oct 30 20:01:13.383257 ip-10-0-132-82 kubenswrapper[4892]: I1030 20:01:13.383213    4892 cpu_manager.go:414] "containerMap: removing container" podUID="0378dcfe-8d0a-43bb-808d-01a939801130" containerName="istio-proxy"
Oct 30 20:01:13.383257 ip-10-0-132-82 kubenswrapper[4892]: I1030 20:01:13.383219    4892 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0378dcfe-8d0a-43bb-808d-01a939801130" containerName="istio-proxy"
Oct 30 20:01:13.383368 ip-10-0-132-82 kubenswrapper[4892]: I1030 20:01:13.383307    4892 memory_manager.go:365] "RemoveStaleState removing state" podUID="0378dcfe-8d0a-43bb-808d-01a939801130" containerName="istio-proxy"
Oct 30 20:01:13.383368 ip-10-0-132-82 kubenswrapper[4892]: I1030 20:01:13.383318    4892 memory_manager.go:365] "RemoveStaleState removing state" podUID="fc03b7fe-3210-4912-9437-f9a8fc838b28" containerName="revoke-keys"
Oct 30 20:01:13.387995 ip-10-0-132-82 kubenswrapper[4892]: I1030 20:01:13.387946    4892 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-c75ed4-jxjdl"
Oct 30 20:01:13.456151 ip-10-0-132-82 kubenswrapper[4892]: I1030 20:01:13.456123    4892 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-c75ed4-jxjdl"]
Oct 30 20:01:13.457336 ip-10-0-132-82 kubenswrapper[4892]: I1030 20:01:13.457303    4892 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 20:01:13.457336 ip-10-0-132-82 kubenswrapper[4892]: I1030 20:01:13.457324    4892 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-nlxp4\""
Oct 30 20:01:13.457531 ip-10-0-132-82 kubenswrapper[4892]: I1030 20:01:13.457389    4892 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 30 20:01:13.457531 ip-10-0-132-82 kubenswrapper[4892]: I1030 20:01:13.457416    4892 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 30 20:01:13.458040 ip-10-0-132-82 kubenswrapper[4892]: I1030 20:01:13.458014    4892 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xwcf6\" (UniqueName: \"kubernetes.io/projected/75e5e791-3967-4357-a185-6aab26a05ce9-kube-api-access-xwcf6\") pod \"maas-api-revoke-keys-e2e-shared-b-w3-c75ed4-jxjdl\" (UID: \"75e5e791-3967-4357-a185-6aab26a05ce9\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-c75ed4-jxjdl"
Oct 30 20:01:13.458438 ip-10-0-132-82 kubenswrapper[4892]: I1030 20:01:13.458410    4892 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/75e5e791-3967-4357-a185-6aab26a05ce9-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w3-c75ed4-jxjdl\" (UID: \"75e5e791-3967-4357-a185-6aab26a05ce9\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-c75ed4-jxjdl"
Oct 30 20:01:13.458563 ip-10-0-132-82 kubenswrapper[4892]: I1030 20:01:13.458518    4892 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/75e5e791-3967-4357-a185-6aab26a05ce9-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w3-c75ed4-jxjdl\" (UID: \"75e5e791-3967-4357-a185-6aab26a05ce9\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-c75ed4-jxjdl"
Oct 30 20:01:13.559701 ip-10-0-132-82 kubenswrapper[4892]: I1030 20:01:13.559668    4892 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/75e5e791-3967-4357-a185-6aab26a05ce9-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w3-c75ed4-jxjdl\" (UID: \"75e5e791-3967-4357-a185-6aab26a05ce9\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-c75ed4-jxjdl"
Oct 30 20:01:13.559900 ip-10-0-132-82 kubenswrapper[4892]: I1030 20:01:13.559720    4892 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/75e5e791-3967-4357-a185-6aab26a05ce9-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w3-c75ed4-jxjdl\" (UID: \"75e5e791-3967-4357-a185-6aab26a05ce9\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-c75ed4-jxjdl"
Oct 30 20:01:13.559900 ip-10-0-132-82 kubenswrapper[4892]: I1030 20:01:13.559832    4892 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xwcf6\" (UniqueName: \"kubernetes.io/projected/75e5e791-3967-4357-a185-6aab26a05ce9-kube-api-access-xwcf6\") pod \"maas-api-revoke-keys-e2e-shared-b-w3-c75ed4-jxjdl\" (UID: \"75e5e791-3967-4357-a185-6aab26a05ce9\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-c75ed4-jxjdl"
Oct 30 20:01:13.560399 ip-10-0-132-82 kubenswrapper[4892]: I1030 20:01:13.560374    4892 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/75e5e791-3967-4357-a185-6aab26a05ce9-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w3-c75ed4-jxjdl\" (UID: \"75e5e791-3967-4357-a185-6aab26a05ce9\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-c75ed4-jxjdl"
Oct 30 20:01:13.562343 ip-10-0-132-82 kubenswrapper[4892]: I1030 20:01:13.562323    4892 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/75e5e791-3967-4357-a185-6aab26a05ce9-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w3-c75ed4-jxjdl\" (UID: \"75e5e791-3967-4357-a185-6aab26a05ce9\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-c75ed4-jxjdl"
Oct 30 20:01:13.620464 ip-10-0-132-82 kubenswrapper[4892]: I1030 20:01:13.620433    4892 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xwcf6\" (UniqueName: \"kubernetes.io/projected/75e5e791-3967-4357-a185-6aab26a05ce9-kube-api-access-xwcf6\") pod \"maas-api-revoke-keys-e2e-shared-b-w3-c75ed4-jxjdl\" (UID: \"75e5e791-3967-4357-a185-6aab26a05ce9\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-c75ed4-jxjdl"
Oct 30 20:01:13.701654 ip-10-0-132-82 kubenswrapper[4892]: I1030 20:01:13.701629    4892 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-c75ed4-jxjdl"
Oct 30 20:01:13.894533 ip-10-0-132-82 kubenswrapper[4892]: I1030 20:01:13.894488    4892 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-worker-w0-2d9861-d6c699c74-2cwqk" podUID="7ecedcf3-79c1-4637-8f4c-eefc4f056d7b" containerName="payload-processing" containerID="cri-o://98043c270dceec91d43f761ede75370fc06e3c2b3d23b18769f5ac469d1cbd0e" gracePeriod=25
Oct 30 20:01:14.011686 ip-10-0-132-82 kubenswrapper[4892]: I1030 20:01:14.011604    4892 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-2d9861-78d7ffc559-lp7cm" podUID="f1df6069-128b-4c8c-86ef-dae9bce19498" containerName="payload-pre-processing" containerID="cri-o://d827b779fa16786b1f673fc81b92d9e095af48da65264094e22be8a2c16a559c" gracePeriod=25
Oct 30 20:01:14.095134 ip-10-0-132-82 kubenswrapper[4892]: I1030 20:01:14.095104    4892 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-c75ed4-jxjdl"]
Oct 30 20:01:14.096053 ip-10-0-132-82 kubenswrapper[4892]: W1030 20:01:14.096020    4892 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod75e5e791_3967_4357_a185_6aab26a05ce9.slice/crio-6ea10562a86a5019ab288e6c1e8fbedf41475480afa8efa541ea289868122db6 WatchSource:0}: Error finding container 6ea10562a86a5019ab288e6c1e8fbedf41475480afa8efa541ea289868122db6: Status 404 returned error can't find the container with id 6ea10562a86a5019ab288e6c1e8fbedf41475480afa8efa541ea289868122db6
Oct 30 20:01:14.316717 ip-10-0-132-82 kubenswrapper[4892]: I1030 20:01:14.316621    4892 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-c75ed4-jxjdl" event={"ID":"75e5e791-3967-4357-a185-6aab26a05ce9","Type":"ContainerStarted","Data":"6ea10562a86a5019ab288e6c1e8fbedf41475480afa8efa541ea289868122db6"}
Oct 30 20:01:15.325214 ip-10-0-132-82 kubenswrapper[4892]: I1030 20:01:15.325181    4892 generic.go:356] "Generic (PLEG): container finished" podID="75e5e791-3967-4357-a185-6aab26a05ce9" containerID="c9f0039cbe33119de37b0bed2fbf5a0c8ee21d851638f12ead36ccd48eaf67f3" exitCode=0
Oct 30 20:01:15.325609 ip-10-0-132-82 kubenswrapper[4892]: I1030 20:01:15.325272    4892 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-c75ed4-jxjdl" event={"ID":"75e5e791-3967-4357-a185-6aab26a05ce9","Type":"ContainerDied","Data":"c9f0039cbe33119de37b0bed2fbf5a0c8ee21d851638f12ead36ccd48eaf67f3"}
Oct 30 20:01:15.394799 ip-10-0-132-82 kubenswrapper[4892]: I1030 20:01:15.394771    4892 patch_prober.go:29] interesting pod/e2e-shared-a-w3-cf9b6f-openshift-default-56db6769f7-xl4xv container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Oct 30 20:01:15.395015 ip-10-0-132-82 kubenswrapper[4892]: I1030 20:01:15.394822    4892 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-shared-a-w3-cf9b6f-openshift-default-56db6769f7-xl4xv" podUID="fd212b75-a818-45a3-946a-864a230b7707" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Oct 30 20:01:16.512397 ip-10-0-132-82 kubenswrapper[4892]: I1030 20:01:16.512373    4892 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-c75ed4-jxjdl"
Oct 30 20:01:16.593144 ip-10-0-132-82 kubenswrapper[4892]: I1030 20:01:16.593064    4892 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/75e5e791-3967-4357-a185-6aab26a05ce9-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/75e5e791-3967-4357-a185-6aab26a05ce9-maas-api-xks-ca\") pod \"75e5e791-3967-4357-a185-6aab26a05ce9\" (UID: \"75e5e791-3967-4357-a185-6aab26a05ce9\") "
Oct 30 20:01:16.593299 ip-10-0-132-82 kubenswrapper[4892]: I1030 20:01:16.593169    4892 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/75e5e791-3967-4357-a185-6aab26a05ce9-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/75e5e791-3967-4357-a185-6aab26a05ce9-maas-api-openshift-ca\") pod \"75e5e791-3967-4357-a185-6aab26a05ce9\" (UID: \"75e5e791-3967-4357-a185-6aab26a05ce9\") "
Oct 30 20:01:16.593299 ip-10-0-132-82 kubenswrapper[4892]: I1030 20:01:16.593211    4892 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/75e5e791-3967-4357-a185-6aab26a05ce9-kube-api-access-xwcf6\" (UniqueName: \"kubernetes.io/projected/75e5e791-3967-4357-a185-6aab26a05ce9-kube-api-access-xwcf6\") pod \"75e5e791-3967-4357-a185-6aab26a05ce9\" (UID: \"75e5e791-3967-4357-a185-6aab26a05ce9\") "
Oct 30 20:01:16.593560 ip-10-0-132-82 kubenswrapper[4892]: I1030 20:01:16.593535    4892 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/75e5e791-3967-4357-a185-6aab26a05ce9-maas-api-openshift-ca" pod "75e5e791-3967-4357-a185-6aab26a05ce9" (UID: "75e5e791-3967-4357-a185-6aab26a05ce9"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 20:01:16.593677 ip-10-0-132-82 kubenswrapper[4892]: I1030 20:01:16.593662    4892 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/75e5e791-3967-4357-a185-6aab26a05ce9-maas-api-openshift-ca\") on node \"ip-10-0-132-82.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:01:16.595387 ip-10-0-132-82 kubenswrapper[4892]: I1030 20:01:16.595359    4892 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/75e5e791-3967-4357-a185-6aab26a05ce9-maas-api-xks-ca" pod "75e5e791-3967-4357-a185-6aab26a05ce9" (UID: "75e5e791-3967-4357-a185-6aab26a05ce9"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 20:01:16.595515 ip-10-0-132-82 kubenswrapper[4892]: I1030 20:01:16.595409    4892 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/75e5e791-3967-4357-a185-6aab26a05ce9-kube-api-access-xwcf6" pod "75e5e791-3967-4357-a185-6aab26a05ce9" (UID: "75e5e791-3967-4357-a185-6aab26a05ce9"). InnerVolumeSpecName "kube-api-access-xwcf6". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 20:01:16.694990 ip-10-0-132-82 kubenswrapper[4892]: I1030 20:01:16.694932    4892 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/75e5e791-3967-4357-a185-6aab26a05ce9-maas-api-xks-ca\") on node \"ip-10-0-132-82.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:01:16.694990 ip-10-0-132-82 kubenswrapper[4892]: I1030 20:01:16.694985    4892 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xwcf6\" (UniqueName: \"kubernetes.io/projected/75e5e791-3967-4357-a185-6aab26a05ce9-kube-api-access-xwcf6\") on node \"ip-10-0-132-82.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:01:16.825214 ip-10-0-132-82 kubenswrapper[4892]: I1030 20:01:16.825167    4892 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w0-2d9861/distinct-2-w0-2d9861-kserve-679c6b56dd-nmhbm" podUID="65b0586a-6593-48cc-adf8-8ff48417ece4" containerName="main" containerID="cri-o://d666bdabcb260b813cdb2b7190a2f57774d6bbcd0232768e893ed49ef2d99b29" gracePeriod=45
Oct 30 20:01:16.843073 ip-10-0-132-82 kubenswrapper[4892]: I1030 20:01:16.843014    4892 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w0-2d9861/premium-w0-2d9861-kserve-67945c6848-4n6kz" podUID="798988b2-c769-4f73-a7c8-ca15de0beef1" containerName="main" containerID="cri-o://8d4825891af93b96a32d886ec52a856fd76916d0c7acaf73ef5311b66ae1bc6f" gracePeriod=45
Oct 30 20:01:17.115064 ip-10-0-132-82 kubenswrapper[4892]: I1030 20:01:17.115034    4892 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-2d9861/premium-w0-2d9861-kserve-67945c6848-4n6kz"
Oct 30 20:01:17.178103 ip-10-0-132-82 kubenswrapper[4892]: I1030 20:01:17.178067    4892 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-2d9861/distinct-2-w0-2d9861-kserve-679c6b56dd-nmhbm"
Oct 30 20:01:17.200295 ip-10-0-132-82 kubenswrapper[4892]: I1030 20:01:17.200263    4892 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/798988b2-c769-4f73-a7c8-ca15de0beef1-dshm\" (UniqueName: \"kubernetes.io/empty-dir/798988b2-c769-4f73-a7c8-ca15de0beef1-dshm\") pod \"798988b2-c769-4f73-a7c8-ca15de0beef1\" (UID: \"798988b2-c769-4f73-a7c8-ca15de0beef1\") "
Oct 30 20:01:17.200510 ip-10-0-132-82 kubenswrapper[4892]: I1030 20:01:17.200345    4892 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/798988b2-c769-4f73-a7c8-ca15de0beef1-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/798988b2-c769-4f73-a7c8-ca15de0beef1-tmp-dir\") pod \"798988b2-c769-4f73-a7c8-ca15de0beef1\" (UID: \"798988b2-c769-4f73-a7c8-ca15de0beef1\") "
Oct 30 20:01:17.200510 ip-10-0-132-82 kubenswrapper[4892]: I1030 20:01:17.200391    4892 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/798988b2-c769-4f73-a7c8-ca15de0beef1-home\" (UniqueName: \"kubernetes.io/empty-dir/798988b2-c769-4f73-a7c8-ca15de0beef1-home\") pod \"798988b2-c769-4f73-a7c8-ca15de0beef1\" (UID: \"798988b2-c769-4f73-a7c8-ca15de0beef1\") "
Oct 30 20:01:17.200510 ip-10-0-132-82 kubenswrapper[4892]: I1030 20:01:17.200444    4892 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/798988b2-c769-4f73-a7c8-ca15de0beef1-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/798988b2-c769-4f73-a7c8-ca15de0beef1-model-cache\") pod \"798988b2-c769-4f73-a7c8-ca15de0beef1\" (UID: \"798988b2-c769-4f73-a7c8-ca15de0beef1\") "
Oct 30 20:01:17.200510 ip-10-0-132-82 kubenswrapper[4892]: I1030 20:01:17.200491    4892 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/798988b2-c769-4f73-a7c8-ca15de0beef1-kube-api-access-bpvwz\" (UniqueName: \"kubernetes.io/projected/798988b2-c769-4f73-a7c8-ca15de0beef1-kube-api-access-bpvwz\") pod \"798988b2-c769-4f73-a7c8-ca15de0beef1\" (UID: \"798988b2-c769-4f73-a7c8-ca15de0beef1\") "
Oct 30 20:01:17.200747 ip-10-0-132-82 kubenswrapper[4892]: I1030 20:01:17.200526    4892 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/798988b2-c769-4f73-a7c8-ca15de0beef1-tls-certs\" (UniqueName: \"kubernetes.io/secret/798988b2-c769-4f73-a7c8-ca15de0beef1-tls-certs\") pod \"798988b2-c769-4f73-a7c8-ca15de0beef1\" (UID: \"798988b2-c769-4f73-a7c8-ca15de0beef1\") "
Oct 30 20:01:17.200747 ip-10-0-132-82 kubenswrapper[4892]: I1030 20:01:17.200707    4892 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/798988b2-c769-4f73-a7c8-ca15de0beef1-home" pod "798988b2-c769-4f73-a7c8-ca15de0beef1" (UID: "798988b2-c769-4f73-a7c8-ca15de0beef1"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 20:01:17.200833 ip-10-0-132-82 kubenswrapper[4892]: I1030 20:01:17.200738    4892 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/798988b2-c769-4f73-a7c8-ca15de0beef1-model-cache" pod "798988b2-c769-4f73-a7c8-ca15de0beef1" (UID: "798988b2-c769-4f73-a7c8-ca15de0beef1"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 20:01:17.201013 ip-10-0-132-82 kubenswrapper[4892]: I1030 20:01:17.200949    4892 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/798988b2-c769-4f73-a7c8-ca15de0beef1-model-cache\") on node \"ip-10-0-132-82.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:01:17.201113 ip-10-0-132-82 kubenswrapper[4892]: I1030 20:01:17.201016    4892 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/798988b2-c769-4f73-a7c8-ca15de0beef1-home\") on node \"ip-10-0-132-82.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:01:17.201113 ip-10-0-132-82 kubenswrapper[4892]: I1030 20:01:17.201040    4892 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/798988b2-c769-4f73-a7c8-ca15de0beef1-tmp-dir" pod "798988b2-c769-4f73-a7c8-ca15de0beef1" (UID: "798988b2-c769-4f73-a7c8-ca15de0beef1"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 20:01:17.202909 ip-10-0-132-82 kubenswrapper[4892]: I1030 20:01:17.202882    4892 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/798988b2-c769-4f73-a7c8-ca15de0beef1-kube-api-access-bpvwz" pod "798988b2-c769-4f73-a7c8-ca15de0beef1" (UID: "798988b2-c769-4f73-a7c8-ca15de0beef1"). InnerVolumeSpecName "kube-api-access-bpvwz". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 20:01:17.203272 ip-10-0-132-82 kubenswrapper[4892]: I1030 20:01:17.203247    4892 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/798988b2-c769-4f73-a7c8-ca15de0beef1-dshm" pod "798988b2-c769-4f73-a7c8-ca15de0beef1" (UID: "798988b2-c769-4f73-a7c8-ca15de0beef1"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 20:01:17.203395 ip-10-0-132-82 kubenswrapper[4892]: I1030 20:01:17.203247    4892 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/798988b2-c769-4f73-a7c8-ca15de0beef1-tls-certs" pod "798988b2-c769-4f73-a7c8-ca15de0beef1" (UID: "798988b2-c769-4f73-a7c8-ca15de0beef1"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 20:01:17.302229 ip-10-0-132-82 kubenswrapper[4892]: I1030 20:01:17.302200    4892 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/65b0586a-6593-48cc-adf8-8ff48417ece4-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/65b0586a-6593-48cc-adf8-8ff48417ece4-tmp-dir\") pod \"65b0586a-6593-48cc-adf8-8ff48417ece4\" (UID: \"65b0586a-6593-48cc-adf8-8ff48417ece4\") "
Oct 30 20:01:17.302229 ip-10-0-132-82 kubenswrapper[4892]: I1030 20:01:17.302244    4892 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/65b0586a-6593-48cc-adf8-8ff48417ece4-home\" (UniqueName: \"kubernetes.io/empty-dir/65b0586a-6593-48cc-adf8-8ff48417ece4-home\") pod \"65b0586a-6593-48cc-adf8-8ff48417ece4\" (UID: \"65b0586a-6593-48cc-adf8-8ff48417ece4\") "
Oct 30 20:01:17.302509 ip-10-0-132-82 kubenswrapper[4892]: I1030 20:01:17.302279    4892 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/65b0586a-6593-48cc-adf8-8ff48417ece4-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/65b0586a-6593-48cc-adf8-8ff48417ece4-model-cache\") pod \"65b0586a-6593-48cc-adf8-8ff48417ece4\" (UID: \"65b0586a-6593-48cc-adf8-8ff48417ece4\") "
Oct 30 20:01:17.302509 ip-10-0-132-82 kubenswrapper[4892]: I1030 20:01:17.302332    4892 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/65b0586a-6593-48cc-adf8-8ff48417ece4-kube-api-access-9grrd\" (UniqueName: \"kubernetes.io/projected/65b0586a-6593-48cc-adf8-8ff48417ece4-kube-api-access-9grrd\") pod \"65b0586a-6593-48cc-adf8-8ff48417ece4\" (UID: \"65b0586a-6593-48cc-adf8-8ff48417ece4\") "
Oct 30 20:01:17.302509 ip-10-0-132-82 kubenswrapper[4892]: I1030 20:01:17.302366    4892 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/65b0586a-6593-48cc-adf8-8ff48417ece4-tls-certs\" (UniqueName: \"kubernetes.io/secret/65b0586a-6593-48cc-adf8-8ff48417ece4-tls-certs\") pod \"65b0586a-6593-48cc-adf8-8ff48417ece4\" (UID: \"65b0586a-6593-48cc-adf8-8ff48417ece4\") "
Oct 30 20:01:17.302509 ip-10-0-132-82 kubenswrapper[4892]: I1030 20:01:17.302406    4892 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/65b0586a-6593-48cc-adf8-8ff48417ece4-dshm\" (UniqueName: \"kubernetes.io/empty-dir/65b0586a-6593-48cc-adf8-8ff48417ece4-dshm\") pod \"65b0586a-6593-48cc-adf8-8ff48417ece4\" (UID: \"65b0586a-6593-48cc-adf8-8ff48417ece4\") "
Oct 30 20:01:17.303223 ip-10-0-132-82 kubenswrapper[4892]: I1030 20:01:17.302519    4892 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/65b0586a-6593-48cc-adf8-8ff48417ece4-tmp-dir" pod "65b0586a-6593-48cc-adf8-8ff48417ece4" (UID: "65b0586a-6593-48cc-adf8-8ff48417ece4"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 20:01:17.303223 ip-10-0-132-82 kubenswrapper[4892]: I1030 20:01:17.302542    4892 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/65b0586a-6593-48cc-adf8-8ff48417ece4-home" pod "65b0586a-6593-48cc-adf8-8ff48417ece4" (UID: "65b0586a-6593-48cc-adf8-8ff48417ece4"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 20:01:17.303346 ip-10-0-132-82 kubenswrapper[4892]: I1030 20:01:17.303254    4892 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/65b0586a-6593-48cc-adf8-8ff48417ece4-tmp-dir\") on node \"ip-10-0-132-82.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:01:17.303346 ip-10-0-132-82 kubenswrapper[4892]: I1030 20:01:17.303277    4892 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/798988b2-c769-4f73-a7c8-ca15de0beef1-dshm\") on node \"ip-10-0-132-82.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:01:17.303346 ip-10-0-132-82 kubenswrapper[4892]: I1030 20:01:17.303291    4892 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/65b0586a-6593-48cc-adf8-8ff48417ece4-home\") on node \"ip-10-0-132-82.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:01:17.303346 ip-10-0-132-82 kubenswrapper[4892]: I1030 20:01:17.303302    4892 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/798988b2-c769-4f73-a7c8-ca15de0beef1-tmp-dir\") on node \"ip-10-0-132-82.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:01:17.303346 ip-10-0-132-82 kubenswrapper[4892]: I1030 20:01:17.303318    4892 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bpvwz\" (UniqueName: \"kubernetes.io/projected/798988b2-c769-4f73-a7c8-ca15de0beef1-kube-api-access-bpvwz\") on node \"ip-10-0-132-82.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:01:17.303346 ip-10-0-132-82 kubenswrapper[4892]: I1030 20:01:17.303335    4892 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/798988b2-c769-4f73-a7c8-ca15de0beef1-tls-certs\") on node \"ip-10-0-132-82.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:01:17.303649 ip-10-0-132-82 kubenswrapper[4892]: I1030 20:01:17.303626    4892 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/65b0586a-6593-48cc-adf8-8ff48417ece4-model-cache" pod "65b0586a-6593-48cc-adf8-8ff48417ece4" (UID: "65b0586a-6593-48cc-adf8-8ff48417ece4"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 20:01:17.304912 ip-10-0-132-82 kubenswrapper[4892]: I1030 20:01:17.304888    4892 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/65b0586a-6593-48cc-adf8-8ff48417ece4-kube-api-access-9grrd" pod "65b0586a-6593-48cc-adf8-8ff48417ece4" (UID: "65b0586a-6593-48cc-adf8-8ff48417ece4"). InnerVolumeSpecName "kube-api-access-9grrd". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 20:01:17.305752 ip-10-0-132-82 kubenswrapper[4892]: I1030 20:01:17.305732    4892 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/65b0586a-6593-48cc-adf8-8ff48417ece4-dshm" pod "65b0586a-6593-48cc-adf8-8ff48417ece4" (UID: "65b0586a-6593-48cc-adf8-8ff48417ece4"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 20:01:17.306290 ip-10-0-132-82 kubenswrapper[4892]: I1030 20:01:17.306269    4892 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/65b0586a-6593-48cc-adf8-8ff48417ece4-tls-certs" pod "65b0586a-6593-48cc-adf8-8ff48417ece4" (UID: "65b0586a-6593-48cc-adf8-8ff48417ece4"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 20:01:17.342696 ip-10-0-132-82 kubenswrapper[4892]: I1030 20:01:17.342664    4892 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-c75ed4-jxjdl"
Oct 30 20:01:17.342879 ip-10-0-132-82 kubenswrapper[4892]: I1030 20:01:17.342660    4892 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-c75ed4-jxjdl" event={"ID":"75e5e791-3967-4357-a185-6aab26a05ce9","Type":"ContainerDied","Data":"6ea10562a86a5019ab288e6c1e8fbedf41475480afa8efa541ea289868122db6"}
Oct 30 20:01:17.342879 ip-10-0-132-82 kubenswrapper[4892]: I1030 20:01:17.342832    4892 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="6ea10562a86a5019ab288e6c1e8fbedf41475480afa8efa541ea289868122db6"
Oct 30 20:01:17.344172 ip-10-0-132-82 kubenswrapper[4892]: I1030 20:01:17.344147    4892 generic.go:356] "Generic (PLEG): container finished" podID="798988b2-c769-4f73-a7c8-ca15de0beef1" containerID="8d4825891af93b96a32d886ec52a856fd76916d0c7acaf73ef5311b66ae1bc6f" exitCode=0
Oct 30 20:01:17.344316 ip-10-0-132-82 kubenswrapper[4892]: I1030 20:01:17.344184    4892 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-2d9861/premium-w0-2d9861-kserve-67945c6848-4n6kz"
Oct 30 20:01:17.344316 ip-10-0-132-82 kubenswrapper[4892]: I1030 20:01:17.344227    4892 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-2d9861/premium-w0-2d9861-kserve-67945c6848-4n6kz" event={"ID":"798988b2-c769-4f73-a7c8-ca15de0beef1","Type":"ContainerDied","Data":"8d4825891af93b96a32d886ec52a856fd76916d0c7acaf73ef5311b66ae1bc6f"}
Oct 30 20:01:17.344316 ip-10-0-132-82 kubenswrapper[4892]: I1030 20:01:17.344270    4892 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-2d9861/premium-w0-2d9861-kserve-67945c6848-4n6kz" event={"ID":"798988b2-c769-4f73-a7c8-ca15de0beef1","Type":"ContainerDied","Data":"81935d55ef963ebc8372addb46ae497b06bd3006acc6bf54203797c976d5c709"}
Oct 30 20:01:17.344316 ip-10-0-132-82 kubenswrapper[4892]: I1030 20:01:17.344291    4892 scope.go:122] "RemoveContainer" containerID="8d4825891af93b96a32d886ec52a856fd76916d0c7acaf73ef5311b66ae1bc6f"
Oct 30 20:01:17.347189 ip-10-0-132-82 kubenswrapper[4892]: I1030 20:01:17.347165    4892 generic.go:356] "Generic (PLEG): container finished" podID="65b0586a-6593-48cc-adf8-8ff48417ece4" containerID="d666bdabcb260b813cdb2b7190a2f57774d6bbcd0232768e893ed49ef2d99b29" exitCode=0
Oct 30 20:01:17.347319 ip-10-0-132-82 kubenswrapper[4892]: I1030 20:01:17.347304    4892 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-2d9861/distinct-2-w0-2d9861-kserve-679c6b56dd-nmhbm" event={"ID":"65b0586a-6593-48cc-adf8-8ff48417ece4","Type":"ContainerDied","Data":"d666bdabcb260b813cdb2b7190a2f57774d6bbcd0232768e893ed49ef2d99b29"}
Oct 30 20:01:17.347388 ip-10-0-132-82 kubenswrapper[4892]: I1030 20:01:17.347336    4892 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-2d9861/distinct-2-w0-2d9861-kserve-679c6b56dd-nmhbm" event={"ID":"65b0586a-6593-48cc-adf8-8ff48417ece4","Type":"ContainerDied","Data":"c5b217d2de2cd2b38101716994243231ac57195716d24dc1bf6e839eb2efe482"}
Oct 30 20:01:17.347460 ip-10-0-132-82 kubenswrapper[4892]: I1030 20:01:17.347394    4892 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-2d9861/distinct-2-w0-2d9861-kserve-679c6b56dd-nmhbm"
Oct 30 20:01:17.369828 ip-10-0-132-82 kubenswrapper[4892]: I1030 20:01:17.369809    4892 scope.go:122] "RemoveContainer" containerID="8d4825891af93b96a32d886ec52a856fd76916d0c7acaf73ef5311b66ae1bc6f"
Oct 30 20:01:17.370279 ip-10-0-132-82 kubenswrapper[4892]: E1030 20:01:17.370248    4892 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8d4825891af93b96a32d886ec52a856fd76916d0c7acaf73ef5311b66ae1bc6f\": container with ID starting with 8d4825891af93b96a32d886ec52a856fd76916d0c7acaf73ef5311b66ae1bc6f not found: ID does not exist" containerID="8d4825891af93b96a32d886ec52a856fd76916d0c7acaf73ef5311b66ae1bc6f"
Oct 30 20:01:17.370387 ip-10-0-132-82 kubenswrapper[4892]: I1030 20:01:17.370281    4892 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8d4825891af93b96a32d886ec52a856fd76916d0c7acaf73ef5311b66ae1bc6f"} err="failed to get container status \"8d4825891af93b96a32d886ec52a856fd76916d0c7acaf73ef5311b66ae1bc6f\": rpc error: code = NotFound desc = could not find container \"8d4825891af93b96a32d886ec52a856fd76916d0c7acaf73ef5311b66ae1bc6f\": container with ID starting with 8d4825891af93b96a32d886ec52a856fd76916d0c7acaf73ef5311b66ae1bc6f not found: ID does not exist"
Oct 30 20:01:17.370387 ip-10-0-132-82 kubenswrapper[4892]: I1030 20:01:17.370301    4892 scope.go:122] "RemoveContainer" containerID="d666bdabcb260b813cdb2b7190a2f57774d6bbcd0232768e893ed49ef2d99b29"
Oct 30 20:01:17.383087 ip-10-0-132-82 kubenswrapper[4892]: I1030 20:01:17.383066    4892 scope.go:122] "RemoveContainer" containerID="d666bdabcb260b813cdb2b7190a2f57774d6bbcd0232768e893ed49ef2d99b29"
Oct 30 20:01:17.383419 ip-10-0-132-82 kubenswrapper[4892]: E1030 20:01:17.383394    4892 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d666bdabcb260b813cdb2b7190a2f57774d6bbcd0232768e893ed49ef2d99b29\": container with ID starting with d666bdabcb260b813cdb2b7190a2f57774d6bbcd0232768e893ed49ef2d99b29 not found: ID does not exist" containerID="d666bdabcb260b813cdb2b7190a2f57774d6bbcd0232768e893ed49ef2d99b29"
Oct 30 20:01:17.383500 ip-10-0-132-82 kubenswrapper[4892]: I1030 20:01:17.383433    4892 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d666bdabcb260b813cdb2b7190a2f57774d6bbcd0232768e893ed49ef2d99b29"} err="failed to get container status \"d666bdabcb260b813cdb2b7190a2f57774d6bbcd0232768e893ed49ef2d99b29\": rpc error: code = NotFound desc = could not find container \"d666bdabcb260b813cdb2b7190a2f57774d6bbcd0232768e893ed49ef2d99b29\": container with ID starting with d666bdabcb260b813cdb2b7190a2f57774d6bbcd0232768e893ed49ef2d99b29 not found: ID does not exist"
Oct 30 20:01:17.404119 ip-10-0-132-82 kubenswrapper[4892]: I1030 20:01:17.404090    4892 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/65b0586a-6593-48cc-adf8-8ff48417ece4-model-cache\") on node \"ip-10-0-132-82.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:01:17.404119 ip-10-0-132-82 kubenswrapper[4892]: I1030 20:01:17.404116    4892 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9grrd\" (UniqueName: \"kubernetes.io/projected/65b0586a-6593-48cc-adf8-8ff48417ece4-kube-api-access-9grrd\") on node \"ip-10-0-132-82.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:01:17.404265 ip-10-0-132-82 kubenswrapper[4892]: I1030 20:01:17.404126    4892 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/65b0586a-6593-48cc-adf8-8ff48417ece4-tls-certs\") on node \"ip-10-0-132-82.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:01:17.404265 ip-10-0-132-82 kubenswrapper[4892]: I1030 20:01:17.404135    4892 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/65b0586a-6593-48cc-adf8-8ff48417ece4-dshm\") on node \"ip-10-0-132-82.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:01:17.668642 ip-10-0-132-82 kubenswrapper[4892]: I1030 20:01:17.668604    4892 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w0-2d9861/distinct-2-w0-2d9861-kserve-679c6b56dd-nmhbm"]
Oct 30 20:01:17.672509 ip-10-0-132-82 kubenswrapper[4892]: I1030 20:01:17.672483    4892 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w0-2d9861/distinct-2-w0-2d9861-kserve-679c6b56dd-nmhbm"]
Oct 30 20:01:17.784127 ip-10-0-132-82 kubenswrapper[4892]: I1030 20:01:17.784081    4892 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w0-2d9861/premium-w0-2d9861-kserve-67945c6848-4n6kz"]
Oct 30 20:01:17.787690 ip-10-0-132-82 kubenswrapper[4892]: I1030 20:01:17.787668    4892 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w0-2d9861/premium-w0-2d9861-kserve-67945c6848-4n6kz"]
Oct 30 20:01:18.018949 ip-10-0-132-82 kubenswrapper[4892]: I1030 20:01:18.018922    4892 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w3-cf9b6f-openshift-default-56db6769f7-xl4xv"
Oct 30 20:01:18.112726 ip-10-0-132-82 kubenswrapper[4892]: I1030 20:01:18.112697    4892 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/fd212b75-a818-45a3-946a-864a230b7707-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/fd212b75-a818-45a3-946a-864a230b7707-istio-envoy\") pod \"fd212b75-a818-45a3-946a-864a230b7707\" (UID: \"fd212b75-a818-45a3-946a-864a230b7707\") "
Oct 30 20:01:18.112882 ip-10-0-132-82 kubenswrapper[4892]: I1030 20:01:18.112736    4892 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/fd212b75-a818-45a3-946a-864a230b7707-istio-token\" (UniqueName: \"kubernetes.io/projected/fd212b75-a818-45a3-946a-864a230b7707-istio-token\") pod \"fd212b75-a818-45a3-946a-864a230b7707\" (UID: \"fd212b75-a818-45a3-946a-864a230b7707\") "
Oct 30 20:01:18.112882 ip-10-0-132-82 kubenswrapper[4892]: I1030 20:01:18.112799    4892 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/fd212b75-a818-45a3-946a-864a230b7707-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/fd212b75-a818-45a3-946a-864a230b7707-credential-socket\") pod \"fd212b75-a818-45a3-946a-864a230b7707\" (UID: \"fd212b75-a818-45a3-946a-864a230b7707\") "
Oct 30 20:01:18.112882 ip-10-0-132-82 kubenswrapper[4892]: I1030 20:01:18.112832    4892 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/fd212b75-a818-45a3-946a-864a230b7707-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/fd212b75-a818-45a3-946a-864a230b7707-istio-podinfo\") pod \"fd212b75-a818-45a3-946a-864a230b7707\" (UID: \"fd212b75-a818-45a3-946a-864a230b7707\") "
Oct 30 20:01:18.112882 ip-10-0-132-82 kubenswrapper[4892]: I1030 20:01:18.112852    4892 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/fd212b75-a818-45a3-946a-864a230b7707-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/fd212b75-a818-45a3-946a-864a230b7707-istio-data\") pod \"fd212b75-a818-45a3-946a-864a230b7707\" (UID: \"fd212b75-a818-45a3-946a-864a230b7707\") "
Oct 30 20:01:18.112882 ip-10-0-132-82 kubenswrapper[4892]: I1030 20:01:18.112879    4892 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/fd212b75-a818-45a3-946a-864a230b7707-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/fd212b75-a818-45a3-946a-864a230b7707-workload-socket\") pod \"fd212b75-a818-45a3-946a-864a230b7707\" (UID: \"fd212b75-a818-45a3-946a-864a230b7707\") "
Oct 30 20:01:18.113276 ip-10-0-132-82 kubenswrapper[4892]: I1030 20:01:18.112935    4892 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/fd212b75-a818-45a3-946a-864a230b7707-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/fd212b75-a818-45a3-946a-864a230b7707-workload-certs\") pod \"fd212b75-a818-45a3-946a-864a230b7707\" (UID: \"fd212b75-a818-45a3-946a-864a230b7707\") "
Oct 30 20:01:18.113276 ip-10-0-132-82 kubenswrapper[4892]: I1030 20:01:18.112997    4892 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/fd212b75-a818-45a3-946a-864a230b7707-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/fd212b75-a818-45a3-946a-864a230b7707-istiod-ca-cert\") pod \"fd212b75-a818-45a3-946a-864a230b7707\" (UID: \"fd212b75-a818-45a3-946a-864a230b7707\") "
Oct 30 20:01:18.113276 ip-10-0-132-82 kubenswrapper[4892]: I1030 20:01:18.113040    4892 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/fd212b75-a818-45a3-946a-864a230b7707-kube-api-access-ntlts\" (UniqueName: \"kubernetes.io/projected/fd212b75-a818-45a3-946a-864a230b7707-kube-api-access-ntlts\") pod \"fd212b75-a818-45a3-946a-864a230b7707\" (UID: \"fd212b75-a818-45a3-946a-864a230b7707\") "
Oct 30 20:01:18.113276 ip-10-0-132-82 kubenswrapper[4892]: I1030 20:01:18.113221    4892 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fd212b75-a818-45a3-946a-864a230b7707-workload-socket" pod "fd212b75-a818-45a3-946a-864a230b7707" (UID: "fd212b75-a818-45a3-946a-864a230b7707"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 20:01:18.113276 ip-10-0-132-82 kubenswrapper[4892]: I1030 20:01:18.113240    4892 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fd212b75-a818-45a3-946a-864a230b7707-istio-data" pod "fd212b75-a818-45a3-946a-864a230b7707" (UID: "fd212b75-a818-45a3-946a-864a230b7707"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 20:01:18.113725 ip-10-0-132-82 kubenswrapper[4892]: I1030 20:01:18.113316    4892 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/fd212b75-a818-45a3-946a-864a230b7707-istiod-ca-cert" pod "fd212b75-a818-45a3-946a-864a230b7707" (UID: "fd212b75-a818-45a3-946a-864a230b7707"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 20:01:18.113725 ip-10-0-132-82 kubenswrapper[4892]: I1030 20:01:18.113432    4892 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fd212b75-a818-45a3-946a-864a230b7707-credential-socket" pod "fd212b75-a818-45a3-946a-864a230b7707" (UID: "fd212b75-a818-45a3-946a-864a230b7707"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 20:01:18.113725 ip-10-0-132-82 kubenswrapper[4892]: I1030 20:01:18.113581    4892 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fd212b75-a818-45a3-946a-864a230b7707-workload-certs" pod "fd212b75-a818-45a3-946a-864a230b7707" (UID: "fd212b75-a818-45a3-946a-864a230b7707"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 20:01:18.113725 ip-10-0-132-82 kubenswrapper[4892]: I1030 20:01:18.113721    4892 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/fd212b75-a818-45a3-946a-864a230b7707-credential-socket\") on node \"ip-10-0-132-82.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:01:18.114016 ip-10-0-132-82 kubenswrapper[4892]: I1030 20:01:18.113735    4892 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/fd212b75-a818-45a3-946a-864a230b7707-istio-data\") on node \"ip-10-0-132-82.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:01:18.114016 ip-10-0-132-82 kubenswrapper[4892]: I1030 20:01:18.113748    4892 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/fd212b75-a818-45a3-946a-864a230b7707-workload-socket\") on node \"ip-10-0-132-82.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:01:18.114016 ip-10-0-132-82 kubenswrapper[4892]: I1030 20:01:18.113762    4892 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/fd212b75-a818-45a3-946a-864a230b7707-workload-certs\") on node \"ip-10-0-132-82.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:01:18.114016 ip-10-0-132-82 kubenswrapper[4892]: I1030 20:01:18.113776    4892 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/fd212b75-a818-45a3-946a-864a230b7707-istiod-ca-cert\") on node \"ip-10-0-132-82.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:01:18.115872 ip-10-0-132-82 kubenswrapper[4892]: I1030 20:01:18.115772    4892 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fd212b75-a818-45a3-946a-864a230b7707-istio-envoy" pod "fd212b75-a818-45a3-946a-864a230b7707" (UID: "fd212b75-a818-45a3-946a-864a230b7707"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 20:01:18.115872 ip-10-0-132-82 kubenswrapper[4892]: I1030 20:01:18.115793    4892 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/fd212b75-a818-45a3-946a-864a230b7707-istio-podinfo" pod "fd212b75-a818-45a3-946a-864a230b7707" (UID: "fd212b75-a818-45a3-946a-864a230b7707"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 20:01:18.115872 ip-10-0-132-82 kubenswrapper[4892]: I1030 20:01:18.115804    4892 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fd212b75-a818-45a3-946a-864a230b7707-istio-token" pod "fd212b75-a818-45a3-946a-864a230b7707" (UID: "fd212b75-a818-45a3-946a-864a230b7707"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 20:01:18.116170 ip-10-0-132-82 kubenswrapper[4892]: I1030 20:01:18.115919    4892 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fd212b75-a818-45a3-946a-864a230b7707-kube-api-access-ntlts" pod "fd212b75-a818-45a3-946a-864a230b7707" (UID: "fd212b75-a818-45a3-946a-864a230b7707"). InnerVolumeSpecName "kube-api-access-ntlts". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 20:01:18.214389 ip-10-0-132-82 kubenswrapper[4892]: I1030 20:01:18.214325    4892 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/fd212b75-a818-45a3-946a-864a230b7707-istio-podinfo\") on node \"ip-10-0-132-82.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:01:18.214389 ip-10-0-132-82 kubenswrapper[4892]: I1030 20:01:18.214348    4892 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ntlts\" (UniqueName: \"kubernetes.io/projected/fd212b75-a818-45a3-946a-864a230b7707-kube-api-access-ntlts\") on node \"ip-10-0-132-82.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:01:18.214389 ip-10-0-132-82 kubenswrapper[4892]: I1030 20:01:18.214359    4892 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/fd212b75-a818-45a3-946a-864a230b7707-istio-envoy\") on node \"ip-10-0-132-82.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:01:18.214389 ip-10-0-132-82 kubenswrapper[4892]: I1030 20:01:18.214369    4892 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/fd212b75-a818-45a3-946a-864a230b7707-istio-token\") on node \"ip-10-0-132-82.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:01:18.357498 ip-10-0-132-82 kubenswrapper[4892]: I1030 20:01:18.357457    4892 generic.go:356] "Generic (PLEG): container finished" podID="fd212b75-a818-45a3-946a-864a230b7707" containerID="a54da5a3cea214225b82ca4f392b05e4f74757f52c2e6374de65120122176f86" exitCode=0
Oct 30 20:01:18.357697 ip-10-0-132-82 kubenswrapper[4892]: I1030 20:01:18.357512    4892 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w3-cf9b6f-openshift-default-56db6769f7-xl4xv"
Oct 30 20:01:18.357697 ip-10-0-132-82 kubenswrapper[4892]: I1030 20:01:18.357546    4892 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w3-cf9b6f-openshift-default-56db6769f7-xl4xv" event={"ID":"fd212b75-a818-45a3-946a-864a230b7707","Type":"ContainerDied","Data":"a54da5a3cea214225b82ca4f392b05e4f74757f52c2e6374de65120122176f86"}
Oct 30 20:01:18.357697 ip-10-0-132-82 kubenswrapper[4892]: I1030 20:01:18.357643    4892 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w3-cf9b6f-openshift-default-56db6769f7-xl4xv" event={"ID":"fd212b75-a818-45a3-946a-864a230b7707","Type":"ContainerDied","Data":"f0dbedfeca2a18f12365cac61a397772359acb31916d626a2010eb1eca587504"}
Oct 30 20:01:18.357697 ip-10-0-132-82 kubenswrapper[4892]: I1030 20:01:18.357675    4892 scope.go:122] "RemoveContainer" containerID="a54da5a3cea214225b82ca4f392b05e4f74757f52c2e6374de65120122176f86"
Oct 30 20:01:18.369067 ip-10-0-132-82 kubenswrapper[4892]: I1030 20:01:18.369048    4892 scope.go:122] "RemoveContainer" containerID="658c14d7f26cca7441ed312ad29eb8f102c041b9138faff109a3707e3f723c75"
Oct 30 20:01:18.378869 ip-10-0-132-82 kubenswrapper[4892]: I1030 20:01:18.378847    4892 scope.go:122] "RemoveContainer" containerID="a54da5a3cea214225b82ca4f392b05e4f74757f52c2e6374de65120122176f86"
Oct 30 20:01:18.379198 ip-10-0-132-82 kubenswrapper[4892]: E1030 20:01:18.379178    4892 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a54da5a3cea214225b82ca4f392b05e4f74757f52c2e6374de65120122176f86\": container with ID starting with a54da5a3cea214225b82ca4f392b05e4f74757f52c2e6374de65120122176f86 not found: ID does not exist" containerID="a54da5a3cea214225b82ca4f392b05e4f74757f52c2e6374de65120122176f86"
Oct 30 20:01:18.379276 ip-10-0-132-82 kubenswrapper[4892]: I1030 20:01:18.379206    4892 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a54da5a3cea214225b82ca4f392b05e4f74757f52c2e6374de65120122176f86"} err="failed to get container status \"a54da5a3cea214225b82ca4f392b05e4f74757f52c2e6374de65120122176f86\": rpc error: code = NotFound desc = could not find container \"a54da5a3cea214225b82ca4f392b05e4f74757f52c2e6374de65120122176f86\": container with ID starting with a54da5a3cea214225b82ca4f392b05e4f74757f52c2e6374de65120122176f86 not found: ID does not exist"
Oct 30 20:01:18.379276 ip-10-0-132-82 kubenswrapper[4892]: I1030 20:01:18.379230    4892 scope.go:122] "RemoveContainer" containerID="658c14d7f26cca7441ed312ad29eb8f102c041b9138faff109a3707e3f723c75"
Oct 30 20:01:18.379495 ip-10-0-132-82 kubenswrapper[4892]: E1030 20:01:18.379473    4892 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"658c14d7f26cca7441ed312ad29eb8f102c041b9138faff109a3707e3f723c75\": container with ID starting with 658c14d7f26cca7441ed312ad29eb8f102c041b9138faff109a3707e3f723c75 not found: ID does not exist" containerID="658c14d7f26cca7441ed312ad29eb8f102c041b9138faff109a3707e3f723c75"
Oct 30 20:01:18.379557 ip-10-0-132-82 kubenswrapper[4892]: I1030 20:01:18.379502    4892 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"658c14d7f26cca7441ed312ad29eb8f102c041b9138faff109a3707e3f723c75"} err="failed to get container status \"658c14d7f26cca7441ed312ad29eb8f102c041b9138faff109a3707e3f723c75\": rpc error: code = NotFound desc = could not find container \"658c14d7f26cca7441ed312ad29eb8f102c041b9138faff109a3707e3f723c75\": container with ID starting with 658c14d7f26cca7441ed312ad29eb8f102c041b9138faff109a3707e3f723c75 not found: ID does not exist"
Oct 30 20:01:18.524460 ip-10-0-132-82 kubenswrapper[4892]: I1030 20:01:18.524392    4892 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w3-cf9b6f-openshift-default-56db6769f7-xl4xv"]
Oct 30 20:01:18.569432 ip-10-0-132-82 kubenswrapper[4892]: I1030 20:01:18.569404    4892 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="65b0586a-6593-48cc-adf8-8ff48417ece4" path="/var/lib/kubelet/pods/65b0586a-6593-48cc-adf8-8ff48417ece4/volumes"
Oct 30 20:01:18.569937 ip-10-0-132-82 kubenswrapper[4892]: I1030 20:01:18.569919    4892 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="798988b2-c769-4f73-a7c8-ca15de0beef1" path="/var/lib/kubelet/pods/798988b2-c769-4f73-a7c8-ca15de0beef1/volumes"
Oct 30 20:01:18.573364 ip-10-0-132-82 kubenswrapper[4892]: I1030 20:01:18.573345    4892 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-a-w3-cf9b6f-openshift-default-56db6769f7-xl4xv"]
Oct 30 20:01:19.339098 ip-10-0-132-82 kubenswrapper[4892]: I1030 20:01:19.339063    4892 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-2d9861-78d7ffc559-lp7cm" podUID="f1df6069-128b-4c8c-86ef-dae9bce19498" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 30 20:01:19.339513 ip-10-0-132-82 kubenswrapper[4892]: I1030 20:01:19.339158    4892 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-worker-w0-2d9861-d6c699c74-2cwqk" podUID="7ecedcf3-79c1-4637-8f4c-eefc4f056d7b" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 30 20:01:20.398787 ip-10-0-132-82 kubenswrapper[4892]: I1030 20:01:20.398758    4892 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-96f166af-wv982"]
Oct 30 20:01:20.399239 ip-10-0-132-82 kubenswrapper[4892]: I1030 20:01:20.399224    4892 cpu_manager.go:414] "containerMap: removing container" podUID="75e5e791-3967-4357-a185-6aab26a05ce9" containerName="revoke-keys"
Oct 30 20:01:20.399285 ip-10-0-132-82 kubenswrapper[4892]: I1030 20:01:20.399240    4892 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="75e5e791-3967-4357-a185-6aab26a05ce9" containerName="revoke-keys"
Oct 30 20:01:20.399285 ip-10-0-132-82 kubenswrapper[4892]: I1030 20:01:20.399258    4892 cpu_manager.go:414] "containerMap: removing container" podUID="fd212b75-a818-45a3-946a-864a230b7707" containerName="istio-proxy"
Oct 30 20:01:20.399285 ip-10-0-132-82 kubenswrapper[4892]: I1030 20:01:20.399264    4892 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="fd212b75-a818-45a3-946a-864a230b7707" containerName="istio-proxy"
Oct 30 20:01:20.399285 ip-10-0-132-82 kubenswrapper[4892]: I1030 20:01:20.399284    4892 cpu_manager.go:414] "containerMap: removing container" podUID="fd212b75-a818-45a3-946a-864a230b7707" containerName="istio-proxy"
Oct 30 20:01:20.399419 ip-10-0-132-82 kubenswrapper[4892]: I1030 20:01:20.399290    4892 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="fd212b75-a818-45a3-946a-864a230b7707" containerName="istio-proxy"
Oct 30 20:01:20.399419 ip-10-0-132-82 kubenswrapper[4892]: I1030 20:01:20.399298    4892 cpu_manager.go:414] "containerMap: removing container" podUID="0378dcfe-8d0a-43bb-808d-01a939801130" containerName="istio-proxy"
Oct 30 20:01:20.399419 ip-10-0-132-82 kubenswrapper[4892]: I1030 20:01:20.399303    4892 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0378dcfe-8d0a-43bb-808d-01a939801130" containerName="istio-proxy"
Oct 30 20:01:20.399419 ip-10-0-132-82 kubenswrapper[4892]: I1030 20:01:20.399309    4892 cpu_manager.go:414] "containerMap: removing container" podUID="65b0586a-6593-48cc-adf8-8ff48417ece4" containerName="main"
Oct 30 20:01:20.399419 ip-10-0-132-82 kubenswrapper[4892]: I1030 20:01:20.399313    4892 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="65b0586a-6593-48cc-adf8-8ff48417ece4" containerName="main"
Oct 30 20:01:20.399419 ip-10-0-132-82 kubenswrapper[4892]: I1030 20:01:20.399326    4892 cpu_manager.go:414] "containerMap: removing container" podUID="798988b2-c769-4f73-a7c8-ca15de0beef1" containerName="main"
Oct 30 20:01:20.399419 ip-10-0-132-82 kubenswrapper[4892]: I1030 20:01:20.399330    4892 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="798988b2-c769-4f73-a7c8-ca15de0beef1" containerName="main"
Oct 30 20:01:20.399419 ip-10-0-132-82 kubenswrapper[4892]: I1030 20:01:20.399400    4892 memory_manager.go:365] "RemoveStaleState removing state" podUID="65b0586a-6593-48cc-adf8-8ff48417ece4" containerName="main"
Oct 30 20:01:20.399419 ip-10-0-132-82 kubenswrapper[4892]: I1030 20:01:20.399409    4892 memory_manager.go:365] "RemoveStaleState removing state" podUID="798988b2-c769-4f73-a7c8-ca15de0beef1" containerName="main"
Oct 30 20:01:20.399419 ip-10-0-132-82 kubenswrapper[4892]: I1030 20:01:20.399418    4892 memory_manager.go:365] "RemoveStaleState removing state" podUID="75e5e791-3967-4357-a185-6aab26a05ce9" containerName="revoke-keys"
Oct 30 20:01:20.399714 ip-10-0-132-82 kubenswrapper[4892]: I1030 20:01:20.399423    4892 memory_manager.go:365] "RemoveStaleState removing state" podUID="0378dcfe-8d0a-43bb-808d-01a939801130" containerName="istio-proxy"
Oct 30 20:01:20.399714 ip-10-0-132-82 kubenswrapper[4892]: I1030 20:01:20.399430    4892 memory_manager.go:365] "RemoveStaleState removing state" podUID="fd212b75-a818-45a3-946a-864a230b7707" containerName="istio-proxy"
Oct 30 20:01:20.403700 ip-10-0-132-82 kubenswrapper[4892]: I1030 20:01:20.403681    4892 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-96f166af-wv982"
Oct 30 20:01:20.457077 ip-10-0-132-82 kubenswrapper[4892]: I1030 20:01:20.457051    4892 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-cc81bdfd-pprtv"]
Oct 30 20:01:20.457721 ip-10-0-132-82 kubenswrapper[4892]: I1030 20:01:20.457700    4892 memory_manager.go:365] "RemoveStaleState removing state" podUID="fd212b75-a818-45a3-946a-864a230b7707" containerName="istio-proxy"
Oct 30 20:01:20.461199 ip-10-0-132-82 kubenswrapper[4892]: I1030 20:01:20.461180    4892 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-cc81bdfd-pprtv"
Oct 30 20:01:20.503336 ip-10-0-132-82 kubenswrapper[4892]: I1030 20:01:20.503314    4892 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 30 20:01:20.503336 ip-10-0-132-82 kubenswrapper[4892]: I1030 20:01:20.503315    4892 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 20:01:20.503497 ip-10-0-132-82 kubenswrapper[4892]: I1030 20:01:20.503342    4892 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 30 20:01:20.505749 ip-10-0-132-82 kubenswrapper[4892]: I1030 20:01:20.505731    4892 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-nlxp4\""
Oct 30 20:01:20.507479 ip-10-0-132-82 kubenswrapper[4892]: I1030 20:01:20.507459    4892 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-96f166af-wv982"]
Oct 30 20:01:20.516411 ip-10-0-132-82 kubenswrapper[4892]: I1030 20:01:20.516390    4892 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-cc81bdfd-pprtv"]
Oct 30 20:01:20.536980 ip-10-0-132-82 kubenswrapper[4892]: I1030 20:01:20.536937    4892 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/46408b41-3826-472e-ab55-22cf9c43ebb5-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-96f166af-wv982\" (UID: \"46408b41-3826-472e-ab55-22cf9c43ebb5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-96f166af-wv982"
Oct 30 20:01:20.537099 ip-10-0-132-82 kubenswrapper[4892]: I1030 20:01:20.537069    4892 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kbjfh\" (UniqueName: \"kubernetes.io/projected/46408b41-3826-472e-ab55-22cf9c43ebb5-kube-api-access-kbjfh\") pod \"maas-api-revoke-sub-96f166af-wv982\" (UID: \"46408b41-3826-472e-ab55-22cf9c43ebb5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-96f166af-wv982"
Oct 30 20:01:20.537175 ip-10-0-132-82 kubenswrapper[4892]: I1030 20:01:20.537112    4892 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/46408b41-3826-472e-ab55-22cf9c43ebb5-maas-api-xks-ca\") pod \"maas-api-revoke-sub-96f166af-wv982\" (UID: \"46408b41-3826-472e-ab55-22cf9c43ebb5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-96f166af-wv982"
Oct 30 20:01:20.569554 ip-10-0-132-82 kubenswrapper[4892]: I1030 20:01:20.569518    4892 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="fd212b75-a818-45a3-946a-864a230b7707" path="/var/lib/kubelet/pods/fd212b75-a818-45a3-946a-864a230b7707/volumes"
Oct 30 20:01:20.637865 ip-10-0-132-82 kubenswrapper[4892]: I1030 20:01:20.637839    4892 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/46408b41-3826-472e-ab55-22cf9c43ebb5-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-96f166af-wv982\" (UID: \"46408b41-3826-472e-ab55-22cf9c43ebb5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-96f166af-wv982"
Oct 30 20:01:20.638009 ip-10-0-132-82 kubenswrapper[4892]: I1030 20:01:20.637887    4892 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-54qdc\" (UniqueName: \"kubernetes.io/projected/714775e4-a883-4b7c-bea6-cdb90ae3bff4-kube-api-access-54qdc\") pod \"maas-api-revoke-sub-cc81bdfd-pprtv\" (UID: \"714775e4-a883-4b7c-bea6-cdb90ae3bff4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-cc81bdfd-pprtv"
Oct 30 20:01:20.638009 ip-10-0-132-82 kubenswrapper[4892]: I1030 20:01:20.637922    4892 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kbjfh\" (UniqueName: \"kubernetes.io/projected/46408b41-3826-472e-ab55-22cf9c43ebb5-kube-api-access-kbjfh\") pod \"maas-api-revoke-sub-96f166af-wv982\" (UID: \"46408b41-3826-472e-ab55-22cf9c43ebb5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-96f166af-wv982"
Oct 30 20:01:20.638009 ip-10-0-132-82 kubenswrapper[4892]: I1030 20:01:20.637942    4892 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/46408b41-3826-472e-ab55-22cf9c43ebb5-maas-api-xks-ca\") pod \"maas-api-revoke-sub-96f166af-wv982\" (UID: \"46408b41-3826-472e-ab55-22cf9c43ebb5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-96f166af-wv982"
Oct 30 20:01:20.638154 ip-10-0-132-82 kubenswrapper[4892]: I1030 20:01:20.638043    4892 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/714775e4-a883-4b7c-bea6-cdb90ae3bff4-maas-api-xks-ca\") pod \"maas-api-revoke-sub-cc81bdfd-pprtv\" (UID: \"714775e4-a883-4b7c-bea6-cdb90ae3bff4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-cc81bdfd-pprtv"
Oct 30 20:01:20.638154 ip-10-0-132-82 kubenswrapper[4892]: I1030 20:01:20.638094    4892 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/714775e4-a883-4b7c-bea6-cdb90ae3bff4-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-cc81bdfd-pprtv\" (UID: \"714775e4-a883-4b7c-bea6-cdb90ae3bff4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-cc81bdfd-pprtv"
Oct 30 20:01:20.638665 ip-10-0-132-82 kubenswrapper[4892]: I1030 20:01:20.638643    4892 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/46408b41-3826-472e-ab55-22cf9c43ebb5-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-96f166af-wv982\" (UID: \"46408b41-3826-472e-ab55-22cf9c43ebb5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-96f166af-wv982"
Oct 30 20:01:20.640596 ip-10-0-132-82 kubenswrapper[4892]: I1030 20:01:20.640575    4892 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/46408b41-3826-472e-ab55-22cf9c43ebb5-maas-api-xks-ca\") pod \"maas-api-revoke-sub-96f166af-wv982\" (UID: \"46408b41-3826-472e-ab55-22cf9c43ebb5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-96f166af-wv982"
Oct 30 20:01:20.694807 ip-10-0-132-82 kubenswrapper[4892]: I1030 20:01:20.694752    4892 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-kbjfh\" (UniqueName: \"kubernetes.io/projected/46408b41-3826-472e-ab55-22cf9c43ebb5-kube-api-access-kbjfh\") pod \"maas-api-revoke-sub-96f166af-wv982\" (UID: \"46408b41-3826-472e-ab55-22cf9c43ebb5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-96f166af-wv982"
Oct 30 20:01:20.717458 ip-10-0-132-82 kubenswrapper[4892]: I1030 20:01:20.717440    4892 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-96f166af-wv982"
Oct 30 20:01:20.739684 ip-10-0-132-82 kubenswrapper[4892]: I1030 20:01:20.739662    4892 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/714775e4-a883-4b7c-bea6-cdb90ae3bff4-maas-api-xks-ca\") pod \"maas-api-revoke-sub-cc81bdfd-pprtv\" (UID: \"714775e4-a883-4b7c-bea6-cdb90ae3bff4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-cc81bdfd-pprtv"
Oct 30 20:01:20.739813 ip-10-0-132-82 kubenswrapper[4892]: I1030 20:01:20.739724    4892 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/714775e4-a883-4b7c-bea6-cdb90ae3bff4-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-cc81bdfd-pprtv\" (UID: \"714775e4-a883-4b7c-bea6-cdb90ae3bff4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-cc81bdfd-pprtv"
Oct 30 20:01:20.739905 ip-10-0-132-82 kubenswrapper[4892]: I1030 20:01:20.739888    4892 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-54qdc\" (UniqueName: \"kubernetes.io/projected/714775e4-a883-4b7c-bea6-cdb90ae3bff4-kube-api-access-54qdc\") pod \"maas-api-revoke-sub-cc81bdfd-pprtv\" (UID: \"714775e4-a883-4b7c-bea6-cdb90ae3bff4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-cc81bdfd-pprtv"
Oct 30 20:01:20.740845 ip-10-0-132-82 kubenswrapper[4892]: I1030 20:01:20.740818    4892 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/714775e4-a883-4b7c-bea6-cdb90ae3bff4-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-cc81bdfd-pprtv\" (UID: \"714775e4-a883-4b7c-bea6-cdb90ae3bff4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-cc81bdfd-pprtv"
Oct 30 20:01:20.742179 ip-10-0-132-82 kubenswrapper[4892]: I1030 20:01:20.742156    4892 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/714775e4-a883-4b7c-bea6-cdb90ae3bff4-maas-api-xks-ca\") pod \"maas-api-revoke-sub-cc81bdfd-pprtv\" (UID: \"714775e4-a883-4b7c-bea6-cdb90ae3bff4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-cc81bdfd-pprtv"
Oct 30 20:01:20.796038 ip-10-0-132-82 kubenswrapper[4892]: I1030 20:01:20.796011    4892 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-54qdc\" (UniqueName: \"kubernetes.io/projected/714775e4-a883-4b7c-bea6-cdb90ae3bff4-kube-api-access-54qdc\") pod \"maas-api-revoke-sub-cc81bdfd-pprtv\" (UID: \"714775e4-a883-4b7c-bea6-cdb90ae3bff4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-cc81bdfd-pprtv"
Oct 30 20:01:21.077698 ip-10-0-132-82 kubenswrapper[4892]: I1030 20:01:21.077630    4892 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-cc81bdfd-pprtv"
Oct 30 20:01:21.095039 ip-10-0-132-82 kubenswrapper[4892]: I1030 20:01:21.095010    4892 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-96f166af-wv982"]
Oct 30 20:01:21.095459 ip-10-0-132-82 kubenswrapper[4892]: W1030 20:01:21.095440    4892 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod46408b41_3826_472e_ab55_22cf9c43ebb5.slice/crio-a3d7f8494cca94a6872f81b0fe2141073ecb6442dc91a15ad1624b40b37983a5 WatchSource:0}: Error finding container a3d7f8494cca94a6872f81b0fe2141073ecb6442dc91a15ad1624b40b37983a5: Status 404 returned error can't find the container with id a3d7f8494cca94a6872f81b0fe2141073ecb6442dc91a15ad1624b40b37983a5
Oct 30 20:01:21.382068 ip-10-0-132-82 kubenswrapper[4892]: I1030 20:01:21.382030    4892 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-96f166af-wv982" event={"ID":"46408b41-3826-472e-ab55-22cf9c43ebb5","Type":"ContainerStarted","Data":"a3d7f8494cca94a6872f81b0fe2141073ecb6442dc91a15ad1624b40b37983a5"}
Oct 30 20:01:21.453846 ip-10-0-132-82 kubenswrapper[4892]: I1030 20:01:21.453819    4892 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-cc81bdfd-pprtv"]
Oct 30 20:01:21.455654 ip-10-0-132-82 kubenswrapper[4892]: W1030 20:01:21.455627    4892 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod714775e4_a883_4b7c_bea6_cdb90ae3bff4.slice/crio-6748e2b0d4ae7cee3e36f4817c796caca7032ae2935809895a05ad9d1745b5f7 WatchSource:0}: Error finding container 6748e2b0d4ae7cee3e36f4817c796caca7032ae2935809895a05ad9d1745b5f7: Status 404 returned error can't find the container with id 6748e2b0d4ae7cee3e36f4817c796caca7032ae2935809895a05ad9d1745b5f7
Oct 30 20:01:22.397542 ip-10-0-132-82 kubenswrapper[4892]: I1030 20:01:22.397505    4892 generic.go:356] "Generic (PLEG): container finished" podID="714775e4-a883-4b7c-bea6-cdb90ae3bff4" containerID="c2955dbce66a6d69474bfacfc27986e2c512185941ace175bdd1d87971953e46" exitCode=0
Oct 30 20:01:22.397711 ip-10-0-132-82 kubenswrapper[4892]: I1030 20:01:22.397597    4892 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-cc81bdfd-pprtv" event={"ID":"714775e4-a883-4b7c-bea6-cdb90ae3bff4","Type":"ContainerDied","Data":"c2955dbce66a6d69474bfacfc27986e2c512185941ace175bdd1d87971953e46"}
Oct 30 20:01:22.397711 ip-10-0-132-82 kubenswrapper[4892]: I1030 20:01:22.397688    4892 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-cc81bdfd-pprtv" event={"ID":"714775e4-a883-4b7c-bea6-cdb90ae3bff4","Type":"ContainerStarted","Data":"6748e2b0d4ae7cee3e36f4817c796caca7032ae2935809895a05ad9d1745b5f7"}
Oct 30 20:01:22.399348 ip-10-0-132-82 kubenswrapper[4892]: I1030 20:01:22.399323    4892 generic.go:356] "Generic (PLEG): container finished" podID="46408b41-3826-472e-ab55-22cf9c43ebb5" containerID="231fbf87cbd4f8237c06f7eedfb0b2994b2ba049d0332d1bc213ef4d8d7894c9" exitCode=0
Oct 30 20:01:22.399460 ip-10-0-132-82 kubenswrapper[4892]: I1030 20:01:22.399366    4892 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-96f166af-wv982" event={"ID":"46408b41-3826-472e-ab55-22cf9c43ebb5","Type":"ContainerDied","Data":"231fbf87cbd4f8237c06f7eedfb0b2994b2ba049d0332d1bc213ef4d8d7894c9"}
Oct 30 20:01:23.583918 ip-10-0-132-82 kubenswrapper[4892]: I1030 20:01:23.583893    4892 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-cc81bdfd-pprtv"
Oct 30 20:01:23.587085 ip-10-0-132-82 kubenswrapper[4892]: I1030 20:01:23.587068    4892 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-96f166af-wv982"
Oct 30 20:01:23.770996 ip-10-0-132-82 kubenswrapper[4892]: I1030 20:01:23.770887    4892 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/46408b41-3826-472e-ab55-22cf9c43ebb5-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/46408b41-3826-472e-ab55-22cf9c43ebb5-maas-api-openshift-ca\") pod \"46408b41-3826-472e-ab55-22cf9c43ebb5\" (UID: \"46408b41-3826-472e-ab55-22cf9c43ebb5\") "
Oct 30 20:01:23.770996 ip-10-0-132-82 kubenswrapper[4892]: I1030 20:01:23.770947    4892 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/714775e4-a883-4b7c-bea6-cdb90ae3bff4-kube-api-access-54qdc\" (UniqueName: \"kubernetes.io/projected/714775e4-a883-4b7c-bea6-cdb90ae3bff4-kube-api-access-54qdc\") pod \"714775e4-a883-4b7c-bea6-cdb90ae3bff4\" (UID: \"714775e4-a883-4b7c-bea6-cdb90ae3bff4\") "
Oct 30 20:01:23.770996 ip-10-0-132-82 kubenswrapper[4892]: I1030 20:01:23.770992    4892 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/714775e4-a883-4b7c-bea6-cdb90ae3bff4-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/714775e4-a883-4b7c-bea6-cdb90ae3bff4-maas-api-xks-ca\") pod \"714775e4-a883-4b7c-bea6-cdb90ae3bff4\" (UID: \"714775e4-a883-4b7c-bea6-cdb90ae3bff4\") "
Oct 30 20:01:23.771301 ip-10-0-132-82 kubenswrapper[4892]: I1030 20:01:23.771065    4892 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/46408b41-3826-472e-ab55-22cf9c43ebb5-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/46408b41-3826-472e-ab55-22cf9c43ebb5-maas-api-xks-ca\") pod \"46408b41-3826-472e-ab55-22cf9c43ebb5\" (UID: \"46408b41-3826-472e-ab55-22cf9c43ebb5\") "
Oct 30 20:01:23.771301 ip-10-0-132-82 kubenswrapper[4892]: I1030 20:01:23.771123    4892 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/46408b41-3826-472e-ab55-22cf9c43ebb5-kube-api-access-kbjfh\" (UniqueName: \"kubernetes.io/projected/46408b41-3826-472e-ab55-22cf9c43ebb5-kube-api-access-kbjfh\") pod \"46408b41-3826-472e-ab55-22cf9c43ebb5\" (UID: \"46408b41-3826-472e-ab55-22cf9c43ebb5\") "
Oct 30 20:01:23.771301 ip-10-0-132-82 kubenswrapper[4892]: I1030 20:01:23.771174    4892 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/714775e4-a883-4b7c-bea6-cdb90ae3bff4-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/714775e4-a883-4b7c-bea6-cdb90ae3bff4-maas-api-openshift-ca\") pod \"714775e4-a883-4b7c-bea6-cdb90ae3bff4\" (UID: \"714775e4-a883-4b7c-bea6-cdb90ae3bff4\") "
Oct 30 20:01:23.771776 ip-10-0-132-82 kubenswrapper[4892]: I1030 20:01:23.771738    4892 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/46408b41-3826-472e-ab55-22cf9c43ebb5-maas-api-openshift-ca" pod "46408b41-3826-472e-ab55-22cf9c43ebb5" (UID: "46408b41-3826-472e-ab55-22cf9c43ebb5"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 20:01:23.771992 ip-10-0-132-82 kubenswrapper[4892]: I1030 20:01:23.771864    4892 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/714775e4-a883-4b7c-bea6-cdb90ae3bff4-maas-api-openshift-ca" pod "714775e4-a883-4b7c-bea6-cdb90ae3bff4" (UID: "714775e4-a883-4b7c-bea6-cdb90ae3bff4"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 20:01:23.773700 ip-10-0-132-82 kubenswrapper[4892]: I1030 20:01:23.773660    4892 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/714775e4-a883-4b7c-bea6-cdb90ae3bff4-maas-api-xks-ca" pod "714775e4-a883-4b7c-bea6-cdb90ae3bff4" (UID: "714775e4-a883-4b7c-bea6-cdb90ae3bff4"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 20:01:23.773946 ip-10-0-132-82 kubenswrapper[4892]: I1030 20:01:23.773925    4892 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/46408b41-3826-472e-ab55-22cf9c43ebb5-maas-api-xks-ca" pod "46408b41-3826-472e-ab55-22cf9c43ebb5" (UID: "46408b41-3826-472e-ab55-22cf9c43ebb5"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 20:01:23.774084 ip-10-0-132-82 kubenswrapper[4892]: I1030 20:01:23.774061    4892 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/46408b41-3826-472e-ab55-22cf9c43ebb5-kube-api-access-kbjfh" pod "46408b41-3826-472e-ab55-22cf9c43ebb5" (UID: "46408b41-3826-472e-ab55-22cf9c43ebb5"). InnerVolumeSpecName "kube-api-access-kbjfh". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 20:01:23.774146 ip-10-0-132-82 kubenswrapper[4892]: I1030 20:01:23.774126    4892 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/714775e4-a883-4b7c-bea6-cdb90ae3bff4-kube-api-access-54qdc" pod "714775e4-a883-4b7c-bea6-cdb90ae3bff4" (UID: "714775e4-a883-4b7c-bea6-cdb90ae3bff4"). InnerVolumeSpecName "kube-api-access-54qdc". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 20:01:23.872684 ip-10-0-132-82 kubenswrapper[4892]: I1030 20:01:23.872647    4892 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/46408b41-3826-472e-ab55-22cf9c43ebb5-maas-api-openshift-ca\") on node \"ip-10-0-132-82.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:01:23.872684 ip-10-0-132-82 kubenswrapper[4892]: I1030 20:01:23.872674    4892 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-54qdc\" (UniqueName: \"kubernetes.io/projected/714775e4-a883-4b7c-bea6-cdb90ae3bff4-kube-api-access-54qdc\") on node \"ip-10-0-132-82.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:01:23.872684 ip-10-0-132-82 kubenswrapper[4892]: I1030 20:01:23.872685    4892 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/714775e4-a883-4b7c-bea6-cdb90ae3bff4-maas-api-xks-ca\") on node \"ip-10-0-132-82.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:01:23.872684 ip-10-0-132-82 kubenswrapper[4892]: I1030 20:01:23.872695    4892 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/46408b41-3826-472e-ab55-22cf9c43ebb5-maas-api-xks-ca\") on node \"ip-10-0-132-82.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:01:23.873096 ip-10-0-132-82 kubenswrapper[4892]: I1030 20:01:23.872705    4892 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kbjfh\" (UniqueName: \"kubernetes.io/projected/46408b41-3826-472e-ab55-22cf9c43ebb5-kube-api-access-kbjfh\") on node \"ip-10-0-132-82.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:01:23.873096 ip-10-0-132-82 kubenswrapper[4892]: I1030 20:01:23.872714    4892 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/714775e4-a883-4b7c-bea6-cdb90ae3bff4-maas-api-openshift-ca\") on node \"ip-10-0-132-82.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:01:24.415824 ip-10-0-132-82 kubenswrapper[4892]: I1030 20:01:24.415799    4892 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-cc81bdfd-pprtv"
Oct 30 20:01:24.415824 ip-10-0-132-82 kubenswrapper[4892]: I1030 20:01:24.415818    4892 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-cc81bdfd-pprtv" event={"ID":"714775e4-a883-4b7c-bea6-cdb90ae3bff4","Type":"ContainerDied","Data":"6748e2b0d4ae7cee3e36f4817c796caca7032ae2935809895a05ad9d1745b5f7"}
Oct 30 20:01:24.416078 ip-10-0-132-82 kubenswrapper[4892]: I1030 20:01:24.415869    4892 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="6748e2b0d4ae7cee3e36f4817c796caca7032ae2935809895a05ad9d1745b5f7"
Oct 30 20:01:24.417390 ip-10-0-132-82 kubenswrapper[4892]: I1030 20:01:24.417369    4892 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-96f166af-wv982" event={"ID":"46408b41-3826-472e-ab55-22cf9c43ebb5","Type":"ContainerDied","Data":"a3d7f8494cca94a6872f81b0fe2141073ecb6442dc91a15ad1624b40b37983a5"}
Oct 30 20:01:24.417513 ip-10-0-132-82 kubenswrapper[4892]: I1030 20:01:24.417392    4892 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-96f166af-wv982"
Oct 30 20:01:24.417513 ip-10-0-132-82 kubenswrapper[4892]: I1030 20:01:24.417402    4892 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="a3d7f8494cca94a6872f81b0fe2141073ecb6442dc91a15ad1624b40b37983a5"
Oct 30 20:01:24.845316 ip-10-0-132-82 kubenswrapper[4892]: E1030 20:01:24.845166    4892 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-jrm87" containerName="extractor"
Oct 30 20:01:25.034158 ip-10-0-132-82 kubenswrapper[4892]: I1030 20:01:25.034119    4892 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w3-c75ed4-545bd7675b-lj8lp"]
Oct 30 20:01:25.141530 ip-10-0-132-82 kubenswrapper[4892]: I1030 20:01:25.141491    4892 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w3-c75ed4-b45dd89bc-pd54p"]
Oct 30 20:01:29.039580 ip-10-0-132-82 kubenswrapper[4892]: I1030 20:01:29.039546    4892 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w6-5f55cd-2nmh7"]
Oct 30 20:01:29.089142 ip-10-0-132-82 kubenswrapper[4892]: I1030 20:01:29.089111    4892 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w6-5f55cd-2nmh7"]
Oct 30 20:01:29.254732 ip-10-0-132-82 kubenswrapper[4892]: I1030 20:01:29.254698    4892 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-mt-dd9b8d79-openshift-default-857b46944f-4zd2h"]
Oct 30 20:01:29.254993 ip-10-0-132-82 kubenswrapper[4892]: I1030 20:01:29.254944    4892 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-mt-dd9b8d79-openshift-default-857b46944f-4zd2h" podUID="332cc10b-8e86-49c3-8381-d48fa8bd4782" containerName="istio-proxy" containerID="cri-o://4696f8f805d751f4d0130504b60b6a5b82a6f0d4157e02818fdaf7fdb041d9e6" gracePeriod=30
Oct 30 20:01:29.268634 ip-10-0-132-82 kubenswrapper[4892]: I1030 20:01:29.268606    4892 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-mt-dd9b8d79-openshift-default-857b46944f-5wjxt"]
Oct 30 20:01:29.268865 ip-10-0-132-82 kubenswrapper[4892]: I1030 20:01:29.268840    4892 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-mt-dd9b8d79-openshift-default-857b46944f-5wjxt" podUID="0d3f37ec-9b98-4a8c-a38c-d02ffd5417b1" containerName="istio-proxy" containerID="cri-o://8362b168aab1751b927549e61df4fbf77947cd9d1642f02367196ef1356ac97b" gracePeriod=30
Oct 30 20:01:29.343573 ip-10-0-132-82 kubenswrapper[4892]: I1030 20:01:29.343474    4892 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-2d9861-78d7ffc559-lp7cm" podUID="f1df6069-128b-4c8c-86ef-dae9bce19498" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 30 20:01:29.343768 ip-10-0-132-82 kubenswrapper[4892]: I1030 20:01:29.343591    4892 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-worker-w0-2d9861-d6c699c74-2cwqk" podUID="7ecedcf3-79c1-4637-8f4c-eefc4f056d7b" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 30 20:01:30.035347 ip-10-0-132-82 kubenswrapper[4892]: I1030 20:01:30.035302    4892 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-shared-b-w3-c75ed4-545bd7675b-lj8lp" podUID="83a6bdb5-d704-47cf-bdc4-202978ec386b" containerName="payload-processing" containerID="cri-o://d66c5248b5b79ab3eb73d7a4637c56df358b9891acc0b9ef45da901ad23c581f" gracePeriod=25
Oct 30 20:01:30.142447 ip-10-0-132-82 kubenswrapper[4892]: I1030 20:01:30.142347    4892 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-c75ed4-b45dd89bc-pd54p" podUID="6003eaae-92d0-41c3-ab2d-118574a64955" containerName="payload-pre-processing" containerID="cri-o://325e4ec21ace3346672609aa23bcb8f70e4dcf65f46ccf3bfaabc66dc581230f" gracePeriod=25
Oct 30 20:01:30.305892 ip-10-0-132-82 kubenswrapper[4892]: I1030 20:01:30.305825    4892 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w3-c75ed4-545bd7675b-lj8lp"
Oct 30 20:01:30.358103 ip-10-0-132-82 kubenswrapper[4892]: I1030 20:01:30.358071    4892 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/83a6bdb5-d704-47cf-bdc4-202978ec386b-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/83a6bdb5-d704-47cf-bdc4-202978ec386b-plugins-config-volume\") pod \"83a6bdb5-d704-47cf-bdc4-202978ec386b\" (UID: \"83a6bdb5-d704-47cf-bdc4-202978ec386b\") "
Oct 30 20:01:30.358292 ip-10-0-132-82 kubenswrapper[4892]: I1030 20:01:30.358160    4892 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/83a6bdb5-d704-47cf-bdc4-202978ec386b-kube-api-access-k7k2w\" (UniqueName: \"kubernetes.io/projected/83a6bdb5-d704-47cf-bdc4-202978ec386b-kube-api-access-k7k2w\") pod \"83a6bdb5-d704-47cf-bdc4-202978ec386b\" (UID: \"83a6bdb5-d704-47cf-bdc4-202978ec386b\") "
Oct 30 20:01:30.358508 ip-10-0-132-82 kubenswrapper[4892]: I1030 20:01:30.358483    4892 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/83a6bdb5-d704-47cf-bdc4-202978ec386b-plugins-config-volume" pod "83a6bdb5-d704-47cf-bdc4-202978ec386b" (UID: "83a6bdb5-d704-47cf-bdc4-202978ec386b"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 20:01:30.358637 ip-10-0-132-82 kubenswrapper[4892]: I1030 20:01:30.358616    4892 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/83a6bdb5-d704-47cf-bdc4-202978ec386b-plugins-config-volume\") on node \"ip-10-0-132-82.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:01:30.360794 ip-10-0-132-82 kubenswrapper[4892]: I1030 20:01:30.360756    4892 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/83a6bdb5-d704-47cf-bdc4-202978ec386b-kube-api-access-k7k2w" pod "83a6bdb5-d704-47cf-bdc4-202978ec386b" (UID: "83a6bdb5-d704-47cf-bdc4-202978ec386b"). InnerVolumeSpecName "kube-api-access-k7k2w". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 20:01:30.399660 ip-10-0-132-82 kubenswrapper[4892]: I1030 20:01:30.399639    4892 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-c75ed4-b45dd89bc-pd54p"
Oct 30 20:01:30.459233 ip-10-0-132-82 kubenswrapper[4892]: I1030 20:01:30.459207    4892 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6003eaae-92d0-41c3-ab2d-118574a64955-kube-api-access-55ccq\" (UniqueName: \"kubernetes.io/projected/6003eaae-92d0-41c3-ab2d-118574a64955-kube-api-access-55ccq\") pod \"6003eaae-92d0-41c3-ab2d-118574a64955\" (UID: \"6003eaae-92d0-41c3-ab2d-118574a64955\") "
Oct 30 20:01:30.459418 ip-10-0-132-82 kubenswrapper[4892]: I1030 20:01:30.459348    4892 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/6003eaae-92d0-41c3-ab2d-118574a64955-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6003eaae-92d0-41c3-ab2d-118574a64955-plugins-config-volume\") pod \"6003eaae-92d0-41c3-ab2d-118574a64955\" (UID: \"6003eaae-92d0-41c3-ab2d-118574a64955\") "
Oct 30 20:01:30.459697 ip-10-0-132-82 kubenswrapper[4892]: I1030 20:01:30.459677    4892 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-k7k2w\" (UniqueName: \"kubernetes.io/projected/83a6bdb5-d704-47cf-bdc4-202978ec386b-kube-api-access-k7k2w\") on node \"ip-10-0-132-82.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:01:30.459817 ip-10-0-132-82 kubenswrapper[4892]: I1030 20:01:30.459790    4892 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6003eaae-92d0-41c3-ab2d-118574a64955-plugins-config-volume" pod "6003eaae-92d0-41c3-ab2d-118574a64955" (UID: "6003eaae-92d0-41c3-ab2d-118574a64955"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 20:01:30.461433 ip-10-0-132-82 kubenswrapper[4892]: I1030 20:01:30.461415    4892 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6003eaae-92d0-41c3-ab2d-118574a64955-kube-api-access-55ccq" pod "6003eaae-92d0-41c3-ab2d-118574a64955" (UID: "6003eaae-92d0-41c3-ab2d-118574a64955"). InnerVolumeSpecName "kube-api-access-55ccq". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 20:01:30.466404 ip-10-0-132-82 kubenswrapper[4892]: I1030 20:01:30.466381    4892 generic.go:356] "Generic (PLEG): container finished" podID="6003eaae-92d0-41c3-ab2d-118574a64955" containerID="325e4ec21ace3346672609aa23bcb8f70e4dcf65f46ccf3bfaabc66dc581230f" exitCode=0
Oct 30 20:01:30.466513 ip-10-0-132-82 kubenswrapper[4892]: I1030 20:01:30.466428    4892 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-c75ed4-b45dd89bc-pd54p"
Oct 30 20:01:30.466513 ip-10-0-132-82 kubenswrapper[4892]: I1030 20:01:30.466465    4892 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-c75ed4-b45dd89bc-pd54p" event={"ID":"6003eaae-92d0-41c3-ab2d-118574a64955","Type":"ContainerDied","Data":"325e4ec21ace3346672609aa23bcb8f70e4dcf65f46ccf3bfaabc66dc581230f"}
Oct 30 20:01:30.466513 ip-10-0-132-82 kubenswrapper[4892]: I1030 20:01:30.466510    4892 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-c75ed4-b45dd89bc-pd54p" event={"ID":"6003eaae-92d0-41c3-ab2d-118574a64955","Type":"ContainerDied","Data":"d20addaec98073821a1b39389acaca002338be0bd9c5a99ae2ea9e484e8903e0"}
Oct 30 20:01:30.466648 ip-10-0-132-82 kubenswrapper[4892]: I1030 20:01:30.466534    4892 scope.go:122] "RemoveContainer" containerID="325e4ec21ace3346672609aa23bcb8f70e4dcf65f46ccf3bfaabc66dc581230f"
Oct 30 20:01:30.468117 ip-10-0-132-82 kubenswrapper[4892]: I1030 20:01:30.468091    4892 generic.go:356] "Generic (PLEG): container finished" podID="83a6bdb5-d704-47cf-bdc4-202978ec386b" containerID="d66c5248b5b79ab3eb73d7a4637c56df358b9891acc0b9ef45da901ad23c581f" exitCode=0
Oct 30 20:01:30.468228 ip-10-0-132-82 kubenswrapper[4892]: I1030 20:01:30.468133    4892 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w3-c75ed4-545bd7675b-lj8lp"
Oct 30 20:01:30.468228 ip-10-0-132-82 kubenswrapper[4892]: I1030 20:01:30.468175    4892 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w3-c75ed4-545bd7675b-lj8lp" event={"ID":"83a6bdb5-d704-47cf-bdc4-202978ec386b","Type":"ContainerDied","Data":"d66c5248b5b79ab3eb73d7a4637c56df358b9891acc0b9ef45da901ad23c581f"}
Oct 30 20:01:30.468228 ip-10-0-132-82 kubenswrapper[4892]: I1030 20:01:30.468220    4892 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w3-c75ed4-545bd7675b-lj8lp" event={"ID":"83a6bdb5-d704-47cf-bdc4-202978ec386b","Type":"ContainerDied","Data":"023e4d1ab631941e492cbaf4fb8f1e8f564bba62ca3788a02f7cced9eef25a12"}
Oct 30 20:01:30.477110 ip-10-0-132-82 kubenswrapper[4892]: I1030 20:01:30.477090    4892 scope.go:122] "RemoveContainer" containerID="325e4ec21ace3346672609aa23bcb8f70e4dcf65f46ccf3bfaabc66dc581230f"
Oct 30 20:01:30.477363 ip-10-0-132-82 kubenswrapper[4892]: E1030 20:01:30.477346    4892 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"325e4ec21ace3346672609aa23bcb8f70e4dcf65f46ccf3bfaabc66dc581230f\": container with ID starting with 325e4ec21ace3346672609aa23bcb8f70e4dcf65f46ccf3bfaabc66dc581230f not found: ID does not exist" containerID="325e4ec21ace3346672609aa23bcb8f70e4dcf65f46ccf3bfaabc66dc581230f"
Oct 30 20:01:30.477436 ip-10-0-132-82 kubenswrapper[4892]: I1030 20:01:30.477369    4892 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"325e4ec21ace3346672609aa23bcb8f70e4dcf65f46ccf3bfaabc66dc581230f"} err="failed to get container status \"325e4ec21ace3346672609aa23bcb8f70e4dcf65f46ccf3bfaabc66dc581230f\": rpc error: code = NotFound desc = could not find container \"325e4ec21ace3346672609aa23bcb8f70e4dcf65f46ccf3bfaabc66dc581230f\": container with ID starting with 325e4ec21ace3346672609aa23bcb8f70e4dcf65f46ccf3bfaabc66dc581230f not found: ID does not exist"
Oct 30 20:01:30.477436 ip-10-0-132-82 kubenswrapper[4892]: I1030 20:01:30.477385    4892 scope.go:122] "RemoveContainer" containerID="d66c5248b5b79ab3eb73d7a4637c56df358b9891acc0b9ef45da901ad23c581f"
Oct 30 20:01:30.486575 ip-10-0-132-82 kubenswrapper[4892]: I1030 20:01:30.486558    4892 scope.go:122] "RemoveContainer" containerID="d66c5248b5b79ab3eb73d7a4637c56df358b9891acc0b9ef45da901ad23c581f"
Oct 30 20:01:30.486842 ip-10-0-132-82 kubenswrapper[4892]: E1030 20:01:30.486823    4892 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d66c5248b5b79ab3eb73d7a4637c56df358b9891acc0b9ef45da901ad23c581f\": container with ID starting with d66c5248b5b79ab3eb73d7a4637c56df358b9891acc0b9ef45da901ad23c581f not found: ID does not exist" containerID="d66c5248b5b79ab3eb73d7a4637c56df358b9891acc0b9ef45da901ad23c581f"
Oct 30 20:01:30.486901 ip-10-0-132-82 kubenswrapper[4892]: I1030 20:01:30.486847    4892 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d66c5248b5b79ab3eb73d7a4637c56df358b9891acc0b9ef45da901ad23c581f"} err="failed to get container status \"d66c5248b5b79ab3eb73d7a4637c56df358b9891acc0b9ef45da901ad23c581f\": rpc error: code = NotFound desc = could not find container \"d66c5248b5b79ab3eb73d7a4637c56df358b9891acc0b9ef45da901ad23c581f\": container with ID starting with d66c5248b5b79ab3eb73d7a4637c56df358b9891acc0b9ef45da901ad23c581f not found: ID does not exist"
Oct 30 20:01:30.561651 ip-10-0-132-82 kubenswrapper[4892]: I1030 20:01:30.561585    4892 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6003eaae-92d0-41c3-ab2d-118574a64955-plugins-config-volume\") on node \"ip-10-0-132-82.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:01:30.561651 ip-10-0-132-82 kubenswrapper[4892]: I1030 20:01:30.561620    4892 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-55ccq\" (UniqueName: \"kubernetes.io/projected/6003eaae-92d0-41c3-ab2d-118574a64955-kube-api-access-55ccq\") on node \"ip-10-0-132-82.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:01:30.569312 ip-10-0-132-82 kubenswrapper[4892]: I1030 20:01:30.569286    4892 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a48896fd-ad37-43b3-af20-4bc3028c5460" path="/var/lib/kubelet/pods/a48896fd-ad37-43b3-af20-4bc3028c5460/volumes"
Oct 30 20:01:30.662939 ip-10-0-132-82 kubenswrapper[4892]: I1030 20:01:30.662896    4892 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w3-c75ed4-b45dd89bc-pd54p"]
Oct 30 20:01:30.666228 ip-10-0-132-82 kubenswrapper[4892]: I1030 20:01:30.666201    4892 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w3-c75ed4-b45dd89bc-pd54p"]
Oct 30 20:01:30.774622 ip-10-0-132-82 kubenswrapper[4892]: I1030 20:01:30.774586    4892 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w3-c75ed4-545bd7675b-lj8lp"]
Oct 30 20:01:30.779352 ip-10-0-132-82 kubenswrapper[4892]: I1030 20:01:30.779323    4892 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w3-c75ed4-545bd7675b-lj8lp"]
Oct 30 20:01:32.341031 ip-10-0-132-82 kubenswrapper[4892]: I1030 20:01:32.340993    4892 patch_prober.go:29] interesting pod/e2e-mt-dd9b8d79-openshift-default-857b46944f-4zd2h container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Oct 30 20:01:32.341424 ip-10-0-132-82 kubenswrapper[4892]: I1030 20:01:32.341061    4892 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-mt-dd9b8d79-openshift-default-857b46944f-4zd2h" podUID="332cc10b-8e86-49c3-8381-d48fa8bd4782" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Oct 30 20:01:32.358751 ip-10-0-132-82 kubenswrapper[4892]: I1030 20:01:32.358712    4892 patch_prober.go:29] interesting pod/e2e-mt-dd9b8d79-openshift-default-857b46944f-5wjxt container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Oct 30 20:01:32.358891 ip-10-0-132-82 kubenswrapper[4892]: I1030 20:01:32.358779    4892 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-mt-dd9b8d79-openshift-default-857b46944f-5wjxt" podUID="0d3f37ec-9b98-4a8c-a38c-d02ffd5417b1" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Oct 30 20:01:32.571183 ip-10-0-132-82 kubenswrapper[4892]: I1030 20:01:32.571151    4892 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="6003eaae-92d0-41c3-ab2d-118574a64955" path="/var/lib/kubelet/pods/6003eaae-92d0-41c3-ab2d-118574a64955/volumes"
Oct 30 20:01:32.571544 ip-10-0-132-82 kubenswrapper[4892]: I1030 20:01:32.571530    4892 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="83a6bdb5-d704-47cf-bdc4-202978ec386b" path="/var/lib/kubelet/pods/83a6bdb5-d704-47cf-bdc4-202978ec386b/volumes"
Oct 30 20:01:34.155625 ip-10-0-132-82 kubenswrapper[4892]: I1030 20:01:34.155590    4892 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w0-2d9861-d6c699c74-2cwqk"
Oct 30 20:01:34.193407 ip-10-0-132-82 kubenswrapper[4892]: I1030 20:01:34.192308    4892 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/7ecedcf3-79c1-4637-8f4c-eefc4f056d7b-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7ecedcf3-79c1-4637-8f4c-eefc4f056d7b-plugins-config-volume\") pod \"7ecedcf3-79c1-4637-8f4c-eefc4f056d7b\" (UID: \"7ecedcf3-79c1-4637-8f4c-eefc4f056d7b\") "
Oct 30 20:01:34.193407 ip-10-0-132-82 kubenswrapper[4892]: I1030 20:01:34.192390    4892 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7ecedcf3-79c1-4637-8f4c-eefc4f056d7b-kube-api-access-g6mnq\" (UniqueName: \"kubernetes.io/projected/7ecedcf3-79c1-4637-8f4c-eefc4f056d7b-kube-api-access-g6mnq\") pod \"7ecedcf3-79c1-4637-8f4c-eefc4f056d7b\" (UID: \"7ecedcf3-79c1-4637-8f4c-eefc4f056d7b\") "
Oct 30 20:01:34.193407 ip-10-0-132-82 kubenswrapper[4892]: I1030 20:01:34.193366    4892 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7ecedcf3-79c1-4637-8f4c-eefc4f056d7b-plugins-config-volume" pod "7ecedcf3-79c1-4637-8f4c-eefc4f056d7b" (UID: "7ecedcf3-79c1-4637-8f4c-eefc4f056d7b"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 20:01:34.198364 ip-10-0-132-82 kubenswrapper[4892]: I1030 20:01:34.197058    4892 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7ecedcf3-79c1-4637-8f4c-eefc4f056d7b-kube-api-access-g6mnq" pod "7ecedcf3-79c1-4637-8f4c-eefc4f056d7b" (UID: "7ecedcf3-79c1-4637-8f4c-eefc4f056d7b"). InnerVolumeSpecName "kube-api-access-g6mnq". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 20:01:34.266792 ip-10-0-132-82 kubenswrapper[4892]: I1030 20:01:34.261117    4892 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-2d9861-78d7ffc559-lp7cm"
Oct 30 20:01:34.293363 ip-10-0-132-82 kubenswrapper[4892]: I1030 20:01:34.293335    4892 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f1df6069-128b-4c8c-86ef-dae9bce19498-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f1df6069-128b-4c8c-86ef-dae9bce19498-plugins-config-volume\") pod \"f1df6069-128b-4c8c-86ef-dae9bce19498\" (UID: \"f1df6069-128b-4c8c-86ef-dae9bce19498\") "
Oct 30 20:01:34.293520 ip-10-0-132-82 kubenswrapper[4892]: I1030 20:01:34.293466    4892 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f1df6069-128b-4c8c-86ef-dae9bce19498-kube-api-access-gll9v\" (UniqueName: \"kubernetes.io/projected/f1df6069-128b-4c8c-86ef-dae9bce19498-kube-api-access-gll9v\") pod \"f1df6069-128b-4c8c-86ef-dae9bce19498\" (UID: \"f1df6069-128b-4c8c-86ef-dae9bce19498\") "
Oct 30 20:01:34.293706 ip-10-0-132-82 kubenswrapper[4892]: I1030 20:01:34.293682    4892 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f1df6069-128b-4c8c-86ef-dae9bce19498-plugins-config-volume" pod "f1df6069-128b-4c8c-86ef-dae9bce19498" (UID: "f1df6069-128b-4c8c-86ef-dae9bce19498"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 20:01:34.293798 ip-10-0-132-82 kubenswrapper[4892]: I1030 20:01:34.293784    4892 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-g6mnq\" (UniqueName: \"kubernetes.io/projected/7ecedcf3-79c1-4637-8f4c-eefc4f056d7b-kube-api-access-g6mnq\") on node \"ip-10-0-132-82.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:01:34.293836 ip-10-0-132-82 kubenswrapper[4892]: I1030 20:01:34.293800    4892 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7ecedcf3-79c1-4637-8f4c-eefc4f056d7b-plugins-config-volume\") on node \"ip-10-0-132-82.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:01:34.293836 ip-10-0-132-82 kubenswrapper[4892]: I1030 20:01:34.293809    4892 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f1df6069-128b-4c8c-86ef-dae9bce19498-plugins-config-volume\") on node \"ip-10-0-132-82.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:01:34.295929 ip-10-0-132-82 kubenswrapper[4892]: I1030 20:01:34.295901    4892 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f1df6069-128b-4c8c-86ef-dae9bce19498-kube-api-access-gll9v" pod "f1df6069-128b-4c8c-86ef-dae9bce19498" (UID: "f1df6069-128b-4c8c-86ef-dae9bce19498"). InnerVolumeSpecName "kube-api-access-gll9v". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 20:01:34.394477 ip-10-0-132-82 kubenswrapper[4892]: I1030 20:01:34.394447    4892 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gll9v\" (UniqueName: \"kubernetes.io/projected/f1df6069-128b-4c8c-86ef-dae9bce19498-kube-api-access-gll9v\") on node \"ip-10-0-132-82.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:01:34.504664 ip-10-0-132-82 kubenswrapper[4892]: I1030 20:01:34.504624    4892 generic.go:356] "Generic (PLEG): container finished" podID="0d3f37ec-9b98-4a8c-a38c-d02ffd5417b1" containerID="8362b168aab1751b927549e61df4fbf77947cd9d1642f02367196ef1356ac97b" exitCode=0
Oct 30 20:01:34.504831 ip-10-0-132-82 kubenswrapper[4892]: I1030 20:01:34.504731    4892 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-dd9b8d79-openshift-default-857b46944f-5wjxt" event={"ID":"0d3f37ec-9b98-4a8c-a38c-d02ffd5417b1","Type":"ContainerDied","Data":"8362b168aab1751b927549e61df4fbf77947cd9d1642f02367196ef1356ac97b"}
Oct 30 20:01:34.506756 ip-10-0-132-82 kubenswrapper[4892]: I1030 20:01:34.506728    4892 generic.go:356] "Generic (PLEG): container finished" podID="f1df6069-128b-4c8c-86ef-dae9bce19498" containerID="d827b779fa16786b1f673fc81b92d9e095af48da65264094e22be8a2c16a559c" exitCode=0
Oct 30 20:01:34.507040 ip-10-0-132-82 kubenswrapper[4892]: I1030 20:01:34.507020    4892 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-2d9861-78d7ffc559-lp7cm" event={"ID":"f1df6069-128b-4c8c-86ef-dae9bce19498","Type":"ContainerDied","Data":"d827b779fa16786b1f673fc81b92d9e095af48da65264094e22be8a2c16a559c"}
Oct 30 20:01:34.507181 ip-10-0-132-82 kubenswrapper[4892]: I1030 20:01:34.507167    4892 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-2d9861-78d7ffc559-lp7cm" event={"ID":"f1df6069-128b-4c8c-86ef-dae9bce19498","Type":"ContainerDied","Data":"4415f3b45d736f0b4e0d512ae98b7015dd9987a3d053a573abe4da0659a985c1"}
Oct 30 20:01:34.507485 ip-10-0-132-82 kubenswrapper[4892]: I1030 20:01:34.507466    4892 scope.go:122] "RemoveContainer" containerID="d827b779fa16786b1f673fc81b92d9e095af48da65264094e22be8a2c16a559c"
Oct 30 20:01:34.507775 ip-10-0-132-82 kubenswrapper[4892]: I1030 20:01:34.507760    4892 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-2d9861-78d7ffc559-lp7cm"
Oct 30 20:01:34.552024 ip-10-0-132-82 kubenswrapper[4892]: I1030 20:01:34.549109    4892 generic.go:356] "Generic (PLEG): container finished" podID="332cc10b-8e86-49c3-8381-d48fa8bd4782" containerID="4696f8f805d751f4d0130504b60b6a5b82a6f0d4157e02818fdaf7fdb041d9e6" exitCode=0
Oct 30 20:01:34.552024 ip-10-0-132-82 kubenswrapper[4892]: I1030 20:01:34.549184    4892 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-dd9b8d79-openshift-default-857b46944f-4zd2h" event={"ID":"332cc10b-8e86-49c3-8381-d48fa8bd4782","Type":"ContainerDied","Data":"4696f8f805d751f4d0130504b60b6a5b82a6f0d4157e02818fdaf7fdb041d9e6"}
Oct 30 20:01:34.552024 ip-10-0-132-82 kubenswrapper[4892]: I1030 20:01:34.551323    4892 generic.go:356] "Generic (PLEG): container finished" podID="7ecedcf3-79c1-4637-8f4c-eefc4f056d7b" containerID="98043c270dceec91d43f761ede75370fc06e3c2b3d23b18769f5ac469d1cbd0e" exitCode=0
Oct 30 20:01:34.552024 ip-10-0-132-82 kubenswrapper[4892]: I1030 20:01:34.551354    4892 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w0-2d9861-d6c699c74-2cwqk" event={"ID":"7ecedcf3-79c1-4637-8f4c-eefc4f056d7b","Type":"ContainerDied","Data":"98043c270dceec91d43f761ede75370fc06e3c2b3d23b18769f5ac469d1cbd0e"}
Oct 30 20:01:34.552024 ip-10-0-132-82 kubenswrapper[4892]: I1030 20:01:34.551390    4892 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w0-2d9861-d6c699c74-2cwqk" event={"ID":"7ecedcf3-79c1-4637-8f4c-eefc4f056d7b","Type":"ContainerDied","Data":"8e6d18773dcc7db5fb710c735525d9c763912378d88de12fa5303e3a7033c800"}
Oct 30 20:01:34.552024 ip-10-0-132-82 kubenswrapper[4892]: I1030 20:01:34.551466    4892 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w0-2d9861-d6c699c74-2cwqk"
Oct 30 20:01:34.580303 ip-10-0-132-82 kubenswrapper[4892]: I1030 20:01:34.571436    4892 scope.go:122] "RemoveContainer" containerID="d827b779fa16786b1f673fc81b92d9e095af48da65264094e22be8a2c16a559c"
Oct 30 20:01:34.580303 ip-10-0-132-82 kubenswrapper[4892]: E1030 20:01:34.571793    4892 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d827b779fa16786b1f673fc81b92d9e095af48da65264094e22be8a2c16a559c\": container with ID starting with d827b779fa16786b1f673fc81b92d9e095af48da65264094e22be8a2c16a559c not found: ID does not exist" containerID="d827b779fa16786b1f673fc81b92d9e095af48da65264094e22be8a2c16a559c"
Oct 30 20:01:34.580303 ip-10-0-132-82 kubenswrapper[4892]: I1030 20:01:34.571830    4892 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d827b779fa16786b1f673fc81b92d9e095af48da65264094e22be8a2c16a559c"} err="failed to get container status \"d827b779fa16786b1f673fc81b92d9e095af48da65264094e22be8a2c16a559c\": rpc error: code = NotFound desc = could not find container \"d827b779fa16786b1f673fc81b92d9e095af48da65264094e22be8a2c16a559c\": container with ID starting with d827b779fa16786b1f673fc81b92d9e095af48da65264094e22be8a2c16a559c not found: ID does not exist"
Oct 30 20:01:34.580303 ip-10-0-132-82 kubenswrapper[4892]: I1030 20:01:34.571853    4892 scope.go:122] "RemoveContainer" containerID="98043c270dceec91d43f761ede75370fc06e3c2b3d23b18769f5ac469d1cbd0e"
Oct 30 20:01:34.591451 ip-10-0-132-82 kubenswrapper[4892]: I1030 20:01:34.591427    4892 scope.go:122] "RemoveContainer" containerID="98043c270dceec91d43f761ede75370fc06e3c2b3d23b18769f5ac469d1cbd0e"
Oct 30 20:01:34.591862 ip-10-0-132-82 kubenswrapper[4892]: E1030 20:01:34.591835    4892 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"98043c270dceec91d43f761ede75370fc06e3c2b3d23b18769f5ac469d1cbd0e\": container with ID starting with 98043c270dceec91d43f761ede75370fc06e3c2b3d23b18769f5ac469d1cbd0e not found: ID does not exist" containerID="98043c270dceec91d43f761ede75370fc06e3c2b3d23b18769f5ac469d1cbd0e"
Oct 30 20:01:34.591958 ip-10-0-132-82 kubenswrapper[4892]: I1030 20:01:34.591892    4892 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"98043c270dceec91d43f761ede75370fc06e3c2b3d23b18769f5ac469d1cbd0e"} err="failed to get container status \"98043c270dceec91d43f761ede75370fc06e3c2b3d23b18769f5ac469d1cbd0e\": rpc error: code = NotFound desc = could not find container \"98043c270dceec91d43f761ede75370fc06e3c2b3d23b18769f5ac469d1cbd0e\": container with ID starting with 98043c270dceec91d43f761ede75370fc06e3c2b3d23b18769f5ac469d1cbd0e not found: ID does not exist"
Oct 30 20:01:34.602798 ip-10-0-132-82 kubenswrapper[4892]: I1030 20:01:34.602774    4892 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-dd9b8d79-openshift-default-857b46944f-4zd2h"
Oct 30 20:01:34.664799 ip-10-0-132-82 kubenswrapper[4892]: I1030 20:01:34.664778    4892 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-dd9b8d79-openshift-default-857b46944f-5wjxt"
Oct 30 20:01:34.696870 ip-10-0-132-82 kubenswrapper[4892]: I1030 20:01:34.696806    4892 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w0-2d9861-78d7ffc559-lp7cm"]
Oct 30 20:01:34.699066 ip-10-0-132-82 kubenswrapper[4892]: I1030 20:01:34.699041    4892 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0d3f37ec-9b98-4a8c-a38c-d02ffd5417b1-kube-api-access-pzvgb\" (UniqueName: \"kubernetes.io/projected/0d3f37ec-9b98-4a8c-a38c-d02ffd5417b1-kube-api-access-pzvgb\") pod \"0d3f37ec-9b98-4a8c-a38c-d02ffd5417b1\" (UID: \"0d3f37ec-9b98-4a8c-a38c-d02ffd5417b1\") "
Oct 30 20:01:34.699192 ip-10-0-132-82 kubenswrapper[4892]: I1030 20:01:34.699082    4892 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0d3f37ec-9b98-4a8c-a38c-d02ffd5417b1-istio-token\" (UniqueName: \"kubernetes.io/projected/0d3f37ec-9b98-4a8c-a38c-d02ffd5417b1-istio-token\") pod \"0d3f37ec-9b98-4a8c-a38c-d02ffd5417b1\" (UID: \"0d3f37ec-9b98-4a8c-a38c-d02ffd5417b1\") "
Oct 30 20:01:34.699256 ip-10-0-132-82 kubenswrapper[4892]: I1030 20:01:34.699193    4892 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0d3f37ec-9b98-4a8c-a38c-d02ffd5417b1-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0d3f37ec-9b98-4a8c-a38c-d02ffd5417b1-istio-envoy\") pod \"0d3f37ec-9b98-4a8c-a38c-d02ffd5417b1\" (UID: \"0d3f37ec-9b98-4a8c-a38c-d02ffd5417b1\") "
Oct 30 20:01:34.699328 ip-10-0-132-82 kubenswrapper[4892]: I1030 20:01:34.699270    4892 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/332cc10b-8e86-49c3-8381-d48fa8bd4782-kube-api-access-8hxk9\" (UniqueName: \"kubernetes.io/projected/332cc10b-8e86-49c3-8381-d48fa8bd4782-kube-api-access-8hxk9\") pod \"332cc10b-8e86-49c3-8381-d48fa8bd4782\" (UID: \"332cc10b-8e86-49c3-8381-d48fa8bd4782\") "
Oct 30 20:01:34.699386 ip-10-0-132-82 kubenswrapper[4892]: I1030 20:01:34.699339    4892 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/332cc10b-8e86-49c3-8381-d48fa8bd4782-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/332cc10b-8e86-49c3-8381-d48fa8bd4782-workload-certs\") pod \"332cc10b-8e86-49c3-8381-d48fa8bd4782\" (UID: \"332cc10b-8e86-49c3-8381-d48fa8bd4782\") "
Oct 30 20:01:34.699445 ip-10-0-132-82 kubenswrapper[4892]: I1030 20:01:34.699396    4892 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/332cc10b-8e86-49c3-8381-d48fa8bd4782-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/332cc10b-8e86-49c3-8381-d48fa8bd4782-istio-data\") pod \"332cc10b-8e86-49c3-8381-d48fa8bd4782\" (UID: \"332cc10b-8e86-49c3-8381-d48fa8bd4782\") "
Oct 30 20:01:34.699501 ip-10-0-132-82 kubenswrapper[4892]: I1030 20:01:34.699486    4892 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0d3f37ec-9b98-4a8c-a38c-d02ffd5417b1-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0d3f37ec-9b98-4a8c-a38c-d02ffd5417b1-credential-socket\") pod \"0d3f37ec-9b98-4a8c-a38c-d02ffd5417b1\" (UID: \"0d3f37ec-9b98-4a8c-a38c-d02ffd5417b1\") "
Oct 30 20:01:34.699563 ip-10-0-132-82 kubenswrapper[4892]: I1030 20:01:34.699519    4892 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/0d3f37ec-9b98-4a8c-a38c-d02ffd5417b1-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0d3f37ec-9b98-4a8c-a38c-d02ffd5417b1-istio-podinfo\") pod \"0d3f37ec-9b98-4a8c-a38c-d02ffd5417b1\" (UID: \"0d3f37ec-9b98-4a8c-a38c-d02ffd5417b1\") "
Oct 30 20:01:34.699627 ip-10-0-132-82 kubenswrapper[4892]: I1030 20:01:34.699572    4892 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0d3f37ec-9b98-4a8c-a38c-d02ffd5417b1-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0d3f37ec-9b98-4a8c-a38c-d02ffd5417b1-workload-socket\") pod \"0d3f37ec-9b98-4a8c-a38c-d02ffd5417b1\" (UID: \"0d3f37ec-9b98-4a8c-a38c-d02ffd5417b1\") "
Oct 30 20:01:34.699686 ip-10-0-132-82 kubenswrapper[4892]: I1030 20:01:34.699649    4892 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0d3f37ec-9b98-4a8c-a38c-d02ffd5417b1-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0d3f37ec-9b98-4a8c-a38c-d02ffd5417b1-workload-certs\") pod \"0d3f37ec-9b98-4a8c-a38c-d02ffd5417b1\" (UID: \"0d3f37ec-9b98-4a8c-a38c-d02ffd5417b1\") "
Oct 30 20:01:34.699741 ip-10-0-132-82 kubenswrapper[4892]: I1030 20:01:34.699720    4892 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/332cc10b-8e86-49c3-8381-d48fa8bd4782-istio-data" pod "332cc10b-8e86-49c3-8381-d48fa8bd4782" (UID: "332cc10b-8e86-49c3-8381-d48fa8bd4782"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 20:01:34.699741 ip-10-0-132-82 kubenswrapper[4892]: I1030 20:01:34.699731    4892 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/332cc10b-8e86-49c3-8381-d48fa8bd4782-workload-certs" pod "332cc10b-8e86-49c3-8381-d48fa8bd4782" (UID: "332cc10b-8e86-49c3-8381-d48fa8bd4782"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 20:01:34.699862 ip-10-0-132-82 kubenswrapper[4892]: I1030 20:01:34.699740    4892 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/332cc10b-8e86-49c3-8381-d48fa8bd4782-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/332cc10b-8e86-49c3-8381-d48fa8bd4782-credential-socket\") pod \"332cc10b-8e86-49c3-8381-d48fa8bd4782\" (UID: \"332cc10b-8e86-49c3-8381-d48fa8bd4782\") "
Oct 30 20:01:34.699862 ip-10-0-132-82 kubenswrapper[4892]: I1030 20:01:34.699799    4892 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/0d3f37ec-9b98-4a8c-a38c-d02ffd5417b1-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0d3f37ec-9b98-4a8c-a38c-d02ffd5417b1-istiod-ca-cert\") pod \"0d3f37ec-9b98-4a8c-a38c-d02ffd5417b1\" (UID: \"0d3f37ec-9b98-4a8c-a38c-d02ffd5417b1\") "
Oct 30 20:01:34.699862 ip-10-0-132-82 kubenswrapper[4892]: I1030 20:01:34.699826    4892 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/332cc10b-8e86-49c3-8381-d48fa8bd4782-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/332cc10b-8e86-49c3-8381-d48fa8bd4782-istiod-ca-cert\") pod \"332cc10b-8e86-49c3-8381-d48fa8bd4782\" (UID: \"332cc10b-8e86-49c3-8381-d48fa8bd4782\") "
Oct 30 20:01:34.700053 ip-10-0-132-82 kubenswrapper[4892]: I1030 20:01:34.699869    4892 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/332cc10b-8e86-49c3-8381-d48fa8bd4782-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/332cc10b-8e86-49c3-8381-d48fa8bd4782-istio-podinfo\") pod \"332cc10b-8e86-49c3-8381-d48fa8bd4782\" (UID: \"332cc10b-8e86-49c3-8381-d48fa8bd4782\") "
Oct 30 20:01:34.700053 ip-10-0-132-82 kubenswrapper[4892]: I1030 20:01:34.699904    4892 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0d3f37ec-9b98-4a8c-a38c-d02ffd5417b1-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0d3f37ec-9b98-4a8c-a38c-d02ffd5417b1-istio-data\") pod \"0d3f37ec-9b98-4a8c-a38c-d02ffd5417b1\" (UID: \"0d3f37ec-9b98-4a8c-a38c-d02ffd5417b1\") "
Oct 30 20:01:34.700053 ip-10-0-132-82 kubenswrapper[4892]: I1030 20:01:34.699938    4892 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/332cc10b-8e86-49c3-8381-d48fa8bd4782-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/332cc10b-8e86-49c3-8381-d48fa8bd4782-istio-envoy\") pod \"332cc10b-8e86-49c3-8381-d48fa8bd4782\" (UID: \"332cc10b-8e86-49c3-8381-d48fa8bd4782\") "
Oct 30 20:01:34.700053 ip-10-0-132-82 kubenswrapper[4892]: I1030 20:01:34.699985    4892 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/332cc10b-8e86-49c3-8381-d48fa8bd4782-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/332cc10b-8e86-49c3-8381-d48fa8bd4782-workload-socket\") pod \"332cc10b-8e86-49c3-8381-d48fa8bd4782\" (UID: \"332cc10b-8e86-49c3-8381-d48fa8bd4782\") "
Oct 30 20:01:34.700053 ip-10-0-132-82 kubenswrapper[4892]: I1030 20:01:34.699986    4892 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0d3f37ec-9b98-4a8c-a38c-d02ffd5417b1-workload-socket" pod "0d3f37ec-9b98-4a8c-a38c-d02ffd5417b1" (UID: "0d3f37ec-9b98-4a8c-a38c-d02ffd5417b1"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 20:01:34.700053 ip-10-0-132-82 kubenswrapper[4892]: I1030 20:01:34.700027    4892 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/332cc10b-8e86-49c3-8381-d48fa8bd4782-istio-token\" (UniqueName: \"kubernetes.io/projected/332cc10b-8e86-49c3-8381-d48fa8bd4782-istio-token\") pod \"332cc10b-8e86-49c3-8381-d48fa8bd4782\" (UID: \"332cc10b-8e86-49c3-8381-d48fa8bd4782\") "
Oct 30 20:01:34.701110 ip-10-0-132-82 kubenswrapper[4892]: I1030 20:01:34.700316    4892 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/332cc10b-8e86-49c3-8381-d48fa8bd4782-istiod-ca-cert" pod "332cc10b-8e86-49c3-8381-d48fa8bd4782" (UID: "332cc10b-8e86-49c3-8381-d48fa8bd4782"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 20:01:34.701110 ip-10-0-132-82 kubenswrapper[4892]: I1030 20:01:34.700412    4892 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/332cc10b-8e86-49c3-8381-d48fa8bd4782-workload-certs\") on node \"ip-10-0-132-82.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:01:34.701110 ip-10-0-132-82 kubenswrapper[4892]: I1030 20:01:34.700431    4892 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/332cc10b-8e86-49c3-8381-d48fa8bd4782-istio-data\") on node \"ip-10-0-132-82.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:01:34.701110 ip-10-0-132-82 kubenswrapper[4892]: I1030 20:01:34.700446    4892 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0d3f37ec-9b98-4a8c-a38c-d02ffd5417b1-workload-socket\") on node \"ip-10-0-132-82.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:01:34.701110 ip-10-0-132-82 kubenswrapper[4892]: I1030 20:01:34.700642    4892 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0d3f37ec-9b98-4a8c-a38c-d02ffd5417b1-credential-socket" pod "0d3f37ec-9b98-4a8c-a38c-d02ffd5417b1" (UID: "0d3f37ec-9b98-4a8c-a38c-d02ffd5417b1"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 20:01:34.701110 ip-10-0-132-82 kubenswrapper[4892]: I1030 20:01:34.700768    4892 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/332cc10b-8e86-49c3-8381-d48fa8bd4782-credential-socket" pod "332cc10b-8e86-49c3-8381-d48fa8bd4782" (UID: "332cc10b-8e86-49c3-8381-d48fa8bd4782"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 20:01:34.701110 ip-10-0-132-82 kubenswrapper[4892]: I1030 20:01:34.701011    4892 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0d3f37ec-9b98-4a8c-a38c-d02ffd5417b1-istio-data" pod "0d3f37ec-9b98-4a8c-a38c-d02ffd5417b1" (UID: "0d3f37ec-9b98-4a8c-a38c-d02ffd5417b1"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 20:01:34.701570 ip-10-0-132-82 kubenswrapper[4892]: I1030 20:01:34.701251    4892 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/332cc10b-8e86-49c3-8381-d48fa8bd4782-workload-socket" pod "332cc10b-8e86-49c3-8381-d48fa8bd4782" (UID: "332cc10b-8e86-49c3-8381-d48fa8bd4782"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 20:01:34.701570 ip-10-0-132-82 kubenswrapper[4892]: I1030 20:01:34.701303    4892 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0d3f37ec-9b98-4a8c-a38c-d02ffd5417b1-istiod-ca-cert" pod "0d3f37ec-9b98-4a8c-a38c-d02ffd5417b1" (UID: "0d3f37ec-9b98-4a8c-a38c-d02ffd5417b1"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 20:01:34.701782 ip-10-0-132-82 kubenswrapper[4892]: I1030 20:01:34.701753    4892 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0d3f37ec-9b98-4a8c-a38c-d02ffd5417b1-workload-certs" pod "0d3f37ec-9b98-4a8c-a38c-d02ffd5417b1" (UID: "0d3f37ec-9b98-4a8c-a38c-d02ffd5417b1"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 20:01:34.702357 ip-10-0-132-82 kubenswrapper[4892]: I1030 20:01:34.702332    4892 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/0d3f37ec-9b98-4a8c-a38c-d02ffd5417b1-istio-podinfo" pod "0d3f37ec-9b98-4a8c-a38c-d02ffd5417b1" (UID: "0d3f37ec-9b98-4a8c-a38c-d02ffd5417b1"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 20:01:34.702644 ip-10-0-132-82 kubenswrapper[4892]: I1030 20:01:34.702617    4892 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0d3f37ec-9b98-4a8c-a38c-d02ffd5417b1-istio-envoy" pod "0d3f37ec-9b98-4a8c-a38c-d02ffd5417b1" (UID: "0d3f37ec-9b98-4a8c-a38c-d02ffd5417b1"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 20:01:34.702644 ip-10-0-132-82 kubenswrapper[4892]: I1030 20:01:34.702626    4892 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w0-2d9861-78d7ffc559-lp7cm"]
Oct 30 20:01:34.702780 ip-10-0-132-82 kubenswrapper[4892]: I1030 20:01:34.702708    4892 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/332cc10b-8e86-49c3-8381-d48fa8bd4782-istio-token" pod "332cc10b-8e86-49c3-8381-d48fa8bd4782" (UID: "332cc10b-8e86-49c3-8381-d48fa8bd4782"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 20:01:34.702843 ip-10-0-132-82 kubenswrapper[4892]: I1030 20:01:34.702828    4892 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0d3f37ec-9b98-4a8c-a38c-d02ffd5417b1-kube-api-access-pzvgb" pod "0d3f37ec-9b98-4a8c-a38c-d02ffd5417b1" (UID: "0d3f37ec-9b98-4a8c-a38c-d02ffd5417b1"). InnerVolumeSpecName "kube-api-access-pzvgb". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 20:01:34.704273 ip-10-0-132-82 kubenswrapper[4892]: I1030 20:01:34.704248    4892 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/332cc10b-8e86-49c3-8381-d48fa8bd4782-istio-podinfo" pod "332cc10b-8e86-49c3-8381-d48fa8bd4782" (UID: "332cc10b-8e86-49c3-8381-d48fa8bd4782"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 20:01:34.704365 ip-10-0-132-82 kubenswrapper[4892]: I1030 20:01:34.704294    4892 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/332cc10b-8e86-49c3-8381-d48fa8bd4782-kube-api-access-8hxk9" pod "332cc10b-8e86-49c3-8381-d48fa8bd4782" (UID: "332cc10b-8e86-49c3-8381-d48fa8bd4782"). InnerVolumeSpecName "kube-api-access-8hxk9". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 20:01:34.704457 ip-10-0-132-82 kubenswrapper[4892]: I1030 20:01:34.704438    4892 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/332cc10b-8e86-49c3-8381-d48fa8bd4782-istio-envoy" pod "332cc10b-8e86-49c3-8381-d48fa8bd4782" (UID: "332cc10b-8e86-49c3-8381-d48fa8bd4782"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 20:01:34.704595 ip-10-0-132-82 kubenswrapper[4892]: I1030 20:01:34.704582    4892 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0d3f37ec-9b98-4a8c-a38c-d02ffd5417b1-istio-token" pod "0d3f37ec-9b98-4a8c-a38c-d02ffd5417b1" (UID: "0d3f37ec-9b98-4a8c-a38c-d02ffd5417b1"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 20:01:34.801589 ip-10-0-132-82 kubenswrapper[4892]: I1030 20:01:34.801560    4892 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0d3f37ec-9b98-4a8c-a38c-d02ffd5417b1-credential-socket\") on node \"ip-10-0-132-82.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:01:34.801589 ip-10-0-132-82 kubenswrapper[4892]: I1030 20:01:34.801584    4892 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0d3f37ec-9b98-4a8c-a38c-d02ffd5417b1-istio-podinfo\") on node \"ip-10-0-132-82.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:01:34.801589 ip-10-0-132-82 kubenswrapper[4892]: I1030 20:01:34.801594    4892 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0d3f37ec-9b98-4a8c-a38c-d02ffd5417b1-workload-certs\") on node \"ip-10-0-132-82.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:01:34.801797 ip-10-0-132-82 kubenswrapper[4892]: I1030 20:01:34.801602    4892 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/332cc10b-8e86-49c3-8381-d48fa8bd4782-credential-socket\") on node \"ip-10-0-132-82.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:01:34.801797 ip-10-0-132-82 kubenswrapper[4892]: I1030 20:01:34.801612    4892 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0d3f37ec-9b98-4a8c-a38c-d02ffd5417b1-istiod-ca-cert\") on node \"ip-10-0-132-82.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:01:34.801797 ip-10-0-132-82 kubenswrapper[4892]: I1030 20:01:34.801620    4892 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/332cc10b-8e86-49c3-8381-d48fa8bd4782-istiod-ca-cert\") on node \"ip-10-0-132-82.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:01:34.801797 ip-10-0-132-82 kubenswrapper[4892]: I1030 20:01:34.801629    4892 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/332cc10b-8e86-49c3-8381-d48fa8bd4782-istio-podinfo\") on node \"ip-10-0-132-82.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:01:34.801797 ip-10-0-132-82 kubenswrapper[4892]: I1030 20:01:34.801637    4892 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0d3f37ec-9b98-4a8c-a38c-d02ffd5417b1-istio-data\") on node \"ip-10-0-132-82.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:01:34.801797 ip-10-0-132-82 kubenswrapper[4892]: I1030 20:01:34.801645    4892 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/332cc10b-8e86-49c3-8381-d48fa8bd4782-istio-envoy\") on node \"ip-10-0-132-82.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:01:34.801797 ip-10-0-132-82 kubenswrapper[4892]: I1030 20:01:34.801653    4892 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/332cc10b-8e86-49c3-8381-d48fa8bd4782-workload-socket\") on node \"ip-10-0-132-82.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:01:34.801797 ip-10-0-132-82 kubenswrapper[4892]: I1030 20:01:34.801661    4892 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/332cc10b-8e86-49c3-8381-d48fa8bd4782-istio-token\") on node \"ip-10-0-132-82.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:01:34.801797 ip-10-0-132-82 kubenswrapper[4892]: I1030 20:01:34.801668    4892 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pzvgb\" (UniqueName: \"kubernetes.io/projected/0d3f37ec-9b98-4a8c-a38c-d02ffd5417b1-kube-api-access-pzvgb\") on node \"ip-10-0-132-82.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:01:34.801797 ip-10-0-132-82 kubenswrapper[4892]: I1030 20:01:34.801677    4892 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0d3f37ec-9b98-4a8c-a38c-d02ffd5417b1-istio-token\") on node \"ip-10-0-132-82.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:01:34.801797 ip-10-0-132-82 kubenswrapper[4892]: I1030 20:01:34.801685    4892 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0d3f37ec-9b98-4a8c-a38c-d02ffd5417b1-istio-envoy\") on node \"ip-10-0-132-82.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:01:34.801797 ip-10-0-132-82 kubenswrapper[4892]: I1030 20:01:34.801693    4892 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8hxk9\" (UniqueName: \"kubernetes.io/projected/332cc10b-8e86-49c3-8381-d48fa8bd4782-kube-api-access-8hxk9\") on node \"ip-10-0-132-82.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:01:34.814535 ip-10-0-132-82 kubenswrapper[4892]: I1030 20:01:34.814508    4892 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w0-2d9861-d6c699c74-2cwqk"]
Oct 30 20:01:34.820299 ip-10-0-132-82 kubenswrapper[4892]: I1030 20:01:34.820277    4892 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w0-2d9861-d6c699c74-2cwqk"]
Oct 30 20:01:35.559775 ip-10-0-132-82 kubenswrapper[4892]: I1030 20:01:35.559726    4892 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-dd9b8d79-openshift-default-857b46944f-5wjxt"
Oct 30 20:01:35.560232 ip-10-0-132-82 kubenswrapper[4892]: I1030 20:01:35.559728    4892 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-dd9b8d79-openshift-default-857b46944f-5wjxt" event={"ID":"0d3f37ec-9b98-4a8c-a38c-d02ffd5417b1","Type":"ContainerDied","Data":"2959d833a96babb3ad079c1ec4102ffeaaad3744c4e0cb4254b1e5157b215f02"}
Oct 30 20:01:35.560232 ip-10-0-132-82 kubenswrapper[4892]: I1030 20:01:35.559916    4892 scope.go:122] "RemoveContainer" containerID="8362b168aab1751b927549e61df4fbf77947cd9d1642f02367196ef1356ac97b"
Oct 30 20:01:35.562428 ip-10-0-132-82 kubenswrapper[4892]: I1030 20:01:35.562409    4892 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-dd9b8d79-openshift-default-857b46944f-4zd2h"
Oct 30 20:01:35.562550 ip-10-0-132-82 kubenswrapper[4892]: I1030 20:01:35.562402    4892 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-dd9b8d79-openshift-default-857b46944f-4zd2h" event={"ID":"332cc10b-8e86-49c3-8381-d48fa8bd4782","Type":"ContainerDied","Data":"df828233221d53ec7aedb240efaa7c9d2085b6789d78a26e771f531dfb8d943d"}
Oct 30 20:01:35.572636 ip-10-0-132-82 kubenswrapper[4892]: I1030 20:01:35.572616    4892 scope.go:122] "RemoveContainer" containerID="4696f8f805d751f4d0130504b60b6a5b82a6f0d4157e02818fdaf7fdb041d9e6"
Oct 30 20:01:35.726389 ip-10-0-132-82 kubenswrapper[4892]: I1030 20:01:35.726359    4892 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-mt-dd9b8d79-openshift-default-857b46944f-4zd2h"]
Oct 30 20:01:35.775534 ip-10-0-132-82 kubenswrapper[4892]: I1030 20:01:35.775510    4892 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-mt-dd9b8d79-openshift-default-857b46944f-4zd2h"]
Oct 30 20:01:35.838709 ip-10-0-132-82 kubenswrapper[4892]: I1030 20:01:35.838642    4892 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-mt-dd9b8d79-openshift-default-857b46944f-5wjxt"]
Oct 30 20:01:35.842756 ip-10-0-132-82 kubenswrapper[4892]: I1030 20:01:35.842735    4892 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-mt-dd9b8d79-openshift-default-857b46944f-5wjxt"]
Oct 30 20:01:36.569844 ip-10-0-132-82 kubenswrapper[4892]: I1030 20:01:36.569816    4892 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0d3f37ec-9b98-4a8c-a38c-d02ffd5417b1" path="/var/lib/kubelet/pods/0d3f37ec-9b98-4a8c-a38c-d02ffd5417b1/volumes"
Oct 30 20:01:36.570386 ip-10-0-132-82 kubenswrapper[4892]: I1030 20:01:36.570368    4892 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="332cc10b-8e86-49c3-8381-d48fa8bd4782" path="/var/lib/kubelet/pods/332cc10b-8e86-49c3-8381-d48fa8bd4782/volumes"
Oct 30 20:01:36.570773 ip-10-0-132-82 kubenswrapper[4892]: I1030 20:01:36.570760    4892 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="7ecedcf3-79c1-4637-8f4c-eefc4f056d7b" path="/var/lib/kubelet/pods/7ecedcf3-79c1-4637-8f4c-eefc4f056d7b/volumes"
Oct 30 20:01:36.571641 ip-10-0-132-82 kubenswrapper[4892]: I1030 20:01:36.571623    4892 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f1df6069-128b-4c8c-86ef-dae9bce19498" path="/var/lib/kubelet/pods/f1df6069-128b-4c8c-86ef-dae9bce19498/volumes"
Oct 30 20:01:43.480595 ip-10-0-132-82 kubenswrapper[4892]: I1030 20:01:43.480559    4892 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-fsfvk"]
Oct 30 20:01:46.568488 ip-10-0-132-82 kubenswrapper[4892]: I1030 20:01:46.568454    4892 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-8bdbd0b9-qpqgz"]
Oct 30 20:01:46.568876 ip-10-0-132-82 kubenswrapper[4892]: I1030 20:01:46.568819    4892 cpu_manager.go:414] "containerMap: removing container" podUID="7ecedcf3-79c1-4637-8f4c-eefc4f056d7b" containerName="payload-processing"
Oct 30 20:01:46.568876 ip-10-0-132-82 kubenswrapper[4892]: I1030 20:01:46.568829    4892 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7ecedcf3-79c1-4637-8f4c-eefc4f056d7b" containerName="payload-processing"
Oct 30 20:01:46.568876 ip-10-0-132-82 kubenswrapper[4892]: I1030 20:01:46.568842    4892 cpu_manager.go:414] "containerMap: removing container" podUID="83a6bdb5-d704-47cf-bdc4-202978ec386b" containerName="payload-processing"
Oct 30 20:01:46.568876 ip-10-0-132-82 kubenswrapper[4892]: I1030 20:01:46.568847    4892 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="83a6bdb5-d704-47cf-bdc4-202978ec386b" containerName="payload-processing"
Oct 30 20:01:46.568876 ip-10-0-132-82 kubenswrapper[4892]: I1030 20:01:46.568866    4892 cpu_manager.go:414] "containerMap: removing container" podUID="0d3f37ec-9b98-4a8c-a38c-d02ffd5417b1" containerName="istio-proxy"
Oct 30 20:01:46.568876 ip-10-0-132-82 kubenswrapper[4892]: I1030 20:01:46.568871    4892 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0d3f37ec-9b98-4a8c-a38c-d02ffd5417b1" containerName="istio-proxy"
Oct 30 20:01:46.568876 ip-10-0-132-82 kubenswrapper[4892]: I1030 20:01:46.568880    4892 cpu_manager.go:414] "containerMap: removing container" podUID="f1df6069-128b-4c8c-86ef-dae9bce19498" containerName="payload-pre-processing"
Oct 30 20:01:46.569146 ip-10-0-132-82 kubenswrapper[4892]: I1030 20:01:46.568884    4892 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f1df6069-128b-4c8c-86ef-dae9bce19498" containerName="payload-pre-processing"
Oct 30 20:01:46.569146 ip-10-0-132-82 kubenswrapper[4892]: I1030 20:01:46.568892    4892 cpu_manager.go:414] "containerMap: removing container" podUID="6003eaae-92d0-41c3-ab2d-118574a64955" containerName="payload-pre-processing"
Oct 30 20:01:46.569146 ip-10-0-132-82 kubenswrapper[4892]: I1030 20:01:46.568897    4892 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6003eaae-92d0-41c3-ab2d-118574a64955" containerName="payload-pre-processing"
Oct 30 20:01:46.569146 ip-10-0-132-82 kubenswrapper[4892]: I1030 20:01:46.568908    4892 cpu_manager.go:414] "containerMap: removing container" podUID="46408b41-3826-472e-ab55-22cf9c43ebb5" containerName="revoke-keys"
Oct 30 20:01:46.569146 ip-10-0-132-82 kubenswrapper[4892]: I1030 20:01:46.568912    4892 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="46408b41-3826-472e-ab55-22cf9c43ebb5" containerName="revoke-keys"
Oct 30 20:01:46.569146 ip-10-0-132-82 kubenswrapper[4892]: I1030 20:01:46.568920    4892 cpu_manager.go:414] "containerMap: removing container" podUID="332cc10b-8e86-49c3-8381-d48fa8bd4782" containerName="istio-proxy"
Oct 30 20:01:46.569146 ip-10-0-132-82 kubenswrapper[4892]: I1030 20:01:46.568924    4892 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="332cc10b-8e86-49c3-8381-d48fa8bd4782" containerName="istio-proxy"
Oct 30 20:01:46.569146 ip-10-0-132-82 kubenswrapper[4892]: I1030 20:01:46.568940    4892 cpu_manager.go:414] "containerMap: removing container" podUID="714775e4-a883-4b7c-bea6-cdb90ae3bff4" containerName="revoke-keys"
Oct 30 20:01:46.569146 ip-10-0-132-82 kubenswrapper[4892]: I1030 20:01:46.568944    4892 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="714775e4-a883-4b7c-bea6-cdb90ae3bff4" containerName="revoke-keys"
Oct 30 20:01:46.569146 ip-10-0-132-82 kubenswrapper[4892]: I1030 20:01:46.569033    4892 memory_manager.go:365] "RemoveStaleState removing state" podUID="7ecedcf3-79c1-4637-8f4c-eefc4f056d7b" containerName="payload-processing"
Oct 30 20:01:46.569146 ip-10-0-132-82 kubenswrapper[4892]: I1030 20:01:46.569041    4892 memory_manager.go:365] "RemoveStaleState removing state" podUID="83a6bdb5-d704-47cf-bdc4-202978ec386b" containerName="payload-processing"
Oct 30 20:01:46.569146 ip-10-0-132-82 kubenswrapper[4892]: I1030 20:01:46.569050    4892 memory_manager.go:365] "RemoveStaleState removing state" podUID="0d3f37ec-9b98-4a8c-a38c-d02ffd5417b1" containerName="istio-proxy"
Oct 30 20:01:46.569146 ip-10-0-132-82 kubenswrapper[4892]: I1030 20:01:46.569056    4892 memory_manager.go:365] "RemoveStaleState removing state" podUID="f1df6069-128b-4c8c-86ef-dae9bce19498" containerName="payload-pre-processing"
Oct 30 20:01:46.569146 ip-10-0-132-82 kubenswrapper[4892]: I1030 20:01:46.569063    4892 memory_manager.go:365] "RemoveStaleState removing state" podUID="6003eaae-92d0-41c3-ab2d-118574a64955" containerName="payload-pre-processing"
Oct 30 20:01:46.569146 ip-10-0-132-82 kubenswrapper[4892]: I1030 20:01:46.569069    4892 memory_manager.go:365] "RemoveStaleState removing state" podUID="46408b41-3826-472e-ab55-22cf9c43ebb5" containerName="revoke-keys"
Oct 30 20:01:46.569146 ip-10-0-132-82 kubenswrapper[4892]: I1030 20:01:46.569074    4892 memory_manager.go:365] "RemoveStaleState removing state" podUID="332cc10b-8e86-49c3-8381-d48fa8bd4782" containerName="istio-proxy"
Oct 30 20:01:46.569146 ip-10-0-132-82 kubenswrapper[4892]: I1030 20:01:46.569081    4892 memory_manager.go:365] "RemoveStaleState removing state" podUID="714775e4-a883-4b7c-bea6-cdb90ae3bff4" containerName="revoke-keys"
Oct 30 20:01:46.575389 ip-10-0-132-82 kubenswrapper[4892]: I1030 20:01:46.575357    4892 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8bdbd0b9-qpqgz"
Oct 30 20:01:46.612027 ip-10-0-132-82 kubenswrapper[4892]: I1030 20:01:46.611994    4892 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/56bf0c05-25c3-4fc3-9299-2eaa33c76d07-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-8bdbd0b9-qpqgz\" (UID: \"56bf0c05-25c3-4fc3-9299-2eaa33c76d07\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8bdbd0b9-qpqgz"
Oct 30 20:01:46.612179 ip-10-0-132-82 kubenswrapper[4892]: I1030 20:01:46.612062    4892 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/56bf0c05-25c3-4fc3-9299-2eaa33c76d07-maas-api-xks-ca\") pod \"maas-api-revoke-sub-8bdbd0b9-qpqgz\" (UID: \"56bf0c05-25c3-4fc3-9299-2eaa33c76d07\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8bdbd0b9-qpqgz"
Oct 30 20:01:46.612242 ip-10-0-132-82 kubenswrapper[4892]: I1030 20:01:46.612229    4892 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dkd6r\" (UniqueName: \"kubernetes.io/projected/56bf0c05-25c3-4fc3-9299-2eaa33c76d07-kube-api-access-dkd6r\") pod \"maas-api-revoke-sub-8bdbd0b9-qpqgz\" (UID: \"56bf0c05-25c3-4fc3-9299-2eaa33c76d07\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8bdbd0b9-qpqgz"
Oct 30 20:01:46.640262 ip-10-0-132-82 kubenswrapper[4892]: I1030 20:01:46.639679    4892 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-8bdbd0b9-qpqgz"]
Oct 30 20:01:46.684311 ip-10-0-132-82 kubenswrapper[4892]: I1030 20:01:46.684275    4892 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 20:01:46.684311 ip-10-0-132-82 kubenswrapper[4892]: I1030 20:01:46.684287    4892 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 30 20:01:46.684311 ip-10-0-132-82 kubenswrapper[4892]: I1030 20:01:46.684290    4892 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 30 20:01:46.684573 ip-10-0-132-82 kubenswrapper[4892]: I1030 20:01:46.684533    4892 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-nlxp4\""
Oct 30 20:01:46.713437 ip-10-0-132-82 kubenswrapper[4892]: I1030 20:01:46.713413    4892 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/56bf0c05-25c3-4fc3-9299-2eaa33c76d07-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-8bdbd0b9-qpqgz\" (UID: \"56bf0c05-25c3-4fc3-9299-2eaa33c76d07\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8bdbd0b9-qpqgz"
Oct 30 20:01:46.713567 ip-10-0-132-82 kubenswrapper[4892]: I1030 20:01:46.713466    4892 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/56bf0c05-25c3-4fc3-9299-2eaa33c76d07-maas-api-xks-ca\") pod \"maas-api-revoke-sub-8bdbd0b9-qpqgz\" (UID: \"56bf0c05-25c3-4fc3-9299-2eaa33c76d07\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8bdbd0b9-qpqgz"
Oct 30 20:01:46.713567 ip-10-0-132-82 kubenswrapper[4892]: I1030 20:01:46.713511    4892 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dkd6r\" (UniqueName: \"kubernetes.io/projected/56bf0c05-25c3-4fc3-9299-2eaa33c76d07-kube-api-access-dkd6r\") pod \"maas-api-revoke-sub-8bdbd0b9-qpqgz\" (UID: \"56bf0c05-25c3-4fc3-9299-2eaa33c76d07\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8bdbd0b9-qpqgz"
Oct 30 20:01:46.714076 ip-10-0-132-82 kubenswrapper[4892]: I1030 20:01:46.714055    4892 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/56bf0c05-25c3-4fc3-9299-2eaa33c76d07-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-8bdbd0b9-qpqgz\" (UID: \"56bf0c05-25c3-4fc3-9299-2eaa33c76d07\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8bdbd0b9-qpqgz"
Oct 30 20:01:46.715920 ip-10-0-132-82 kubenswrapper[4892]: I1030 20:01:46.715898    4892 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/56bf0c05-25c3-4fc3-9299-2eaa33c76d07-maas-api-xks-ca\") pod \"maas-api-revoke-sub-8bdbd0b9-qpqgz\" (UID: \"56bf0c05-25c3-4fc3-9299-2eaa33c76d07\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8bdbd0b9-qpqgz"
Oct 30 20:01:46.778139 ip-10-0-132-82 kubenswrapper[4892]: I1030 20:01:46.778107    4892 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dkd6r\" (UniqueName: \"kubernetes.io/projected/56bf0c05-25c3-4fc3-9299-2eaa33c76d07-kube-api-access-dkd6r\") pod \"maas-api-revoke-sub-8bdbd0b9-qpqgz\" (UID: \"56bf0c05-25c3-4fc3-9299-2eaa33c76d07\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8bdbd0b9-qpqgz"
Oct 30 20:01:46.893495 ip-10-0-132-82 kubenswrapper[4892]: I1030 20:01:46.893471    4892 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8bdbd0b9-qpqgz"
Oct 30 20:01:47.290376 ip-10-0-132-82 kubenswrapper[4892]: I1030 20:01:47.290338    4892 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-8bdbd0b9-qpqgz"]
Oct 30 20:01:47.291633 ip-10-0-132-82 kubenswrapper[4892]: W1030 20:01:47.291593    4892 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod56bf0c05_25c3_4fc3_9299_2eaa33c76d07.slice/crio-84b68d4166bf33b789e00017f5adeb0a82ca5726509f1950869e4353c9141544 WatchSource:0}: Error finding container 84b68d4166bf33b789e00017f5adeb0a82ca5726509f1950869e4353c9141544: Status 404 returned error can't find the container with id 84b68d4166bf33b789e00017f5adeb0a82ca5726509f1950869e4353c9141544
Oct 30 20:01:47.417067 ip-10-0-132-82 kubenswrapper[4892]: I1030 20:01:47.417027    4892 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w6-5f55cd-openshift-default-9dc497cb5-q2fbg"]
Oct 30 20:01:47.417243 ip-10-0-132-82 kubenswrapper[4892]: I1030 20:01:47.417224    4892 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w6-5f55cd-openshift-default-9dc497cb5-q2fbg" podUID="fac4fe86-a38e-434e-bfc1-4c3dbceffd27" containerName="istio-proxy" containerID="cri-o://e3df77051ab71dd38f8567e41d2b95dcd600ea2f8b9dbe7bc16540bfdc010985" gracePeriod=30
Oct 30 20:01:47.654604 ip-10-0-132-82 kubenswrapper[4892]: I1030 20:01:47.654568    4892 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8bdbd0b9-qpqgz" event={"ID":"56bf0c05-25c3-4fc3-9299-2eaa33c76d07","Type":"ContainerStarted","Data":"3eafeae8988c1c0b0eefcb83d62e59b90f941a740bdfa2169906b78c48cdb07f"}
Oct 30 20:01:47.654994 ip-10-0-132-82 kubenswrapper[4892]: I1030 20:01:47.654637    4892 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8bdbd0b9-qpqgz" event={"ID":"56bf0c05-25c3-4fc3-9299-2eaa33c76d07","Type":"ContainerStarted","Data":"84b68d4166bf33b789e00017f5adeb0a82ca5726509f1950869e4353c9141544"}
Oct 30 20:01:47.817386 ip-10-0-132-82 kubenswrapper[4892]: I1030 20:01:47.817289    4892 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8bdbd0b9-qpqgz" podStartSLOduration=1.8172749929999998 podStartE2EDuration="1.817274993s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 20:01:46 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 20:01:47.815491708 +0000 UTC m=+2755.385957307" watchObservedRunningTime="2026-10-30 20:01:47.817274993 +0000 UTC m=+2755.387740591"
Oct 30 20:01:48.663208 ip-10-0-132-82 kubenswrapper[4892]: I1030 20:01:48.663177    4892 generic.go:356] "Generic (PLEG): container finished" podID="56bf0c05-25c3-4fc3-9299-2eaa33c76d07" containerID="3eafeae8988c1c0b0eefcb83d62e59b90f941a740bdfa2169906b78c48cdb07f" exitCode=0
Oct 30 20:01:48.663691 ip-10-0-132-82 kubenswrapper[4892]: I1030 20:01:48.663317    4892 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8bdbd0b9-qpqgz" event={"ID":"56bf0c05-25c3-4fc3-9299-2eaa33c76d07","Type":"ContainerDied","Data":"3eafeae8988c1c0b0eefcb83d62e59b90f941a740bdfa2169906b78c48cdb07f"}
Oct 30 20:01:49.841282 ip-10-0-132-82 kubenswrapper[4892]: I1030 20:01:49.841250    4892 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8bdbd0b9-qpqgz"
Oct 30 20:01:49.942928 ip-10-0-132-82 kubenswrapper[4892]: I1030 20:01:49.942841    4892 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/56bf0c05-25c3-4fc3-9299-2eaa33c76d07-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/56bf0c05-25c3-4fc3-9299-2eaa33c76d07-maas-api-xks-ca\") pod \"56bf0c05-25c3-4fc3-9299-2eaa33c76d07\" (UID: \"56bf0c05-25c3-4fc3-9299-2eaa33c76d07\") "
Oct 30 20:01:49.942928 ip-10-0-132-82 kubenswrapper[4892]: I1030 20:01:49.942887    4892 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/56bf0c05-25c3-4fc3-9299-2eaa33c76d07-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/56bf0c05-25c3-4fc3-9299-2eaa33c76d07-maas-api-openshift-ca\") pod \"56bf0c05-25c3-4fc3-9299-2eaa33c76d07\" (UID: \"56bf0c05-25c3-4fc3-9299-2eaa33c76d07\") "
Oct 30 20:01:49.943194 ip-10-0-132-82 kubenswrapper[4892]: I1030 20:01:49.943056    4892 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/56bf0c05-25c3-4fc3-9299-2eaa33c76d07-kube-api-access-dkd6r\" (UniqueName: \"kubernetes.io/projected/56bf0c05-25c3-4fc3-9299-2eaa33c76d07-kube-api-access-dkd6r\") pod \"56bf0c05-25c3-4fc3-9299-2eaa33c76d07\" (UID: \"56bf0c05-25c3-4fc3-9299-2eaa33c76d07\") "
Oct 30 20:01:49.943330 ip-10-0-132-82 kubenswrapper[4892]: I1030 20:01:49.943298    4892 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/56bf0c05-25c3-4fc3-9299-2eaa33c76d07-maas-api-openshift-ca" pod "56bf0c05-25c3-4fc3-9299-2eaa33c76d07" (UID: "56bf0c05-25c3-4fc3-9299-2eaa33c76d07"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 20:01:49.943466 ip-10-0-132-82 kubenswrapper[4892]: I1030 20:01:49.943376    4892 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/56bf0c05-25c3-4fc3-9299-2eaa33c76d07-maas-api-openshift-ca\") on node \"ip-10-0-132-82.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:01:49.945168 ip-10-0-132-82 kubenswrapper[4892]: I1030 20:01:49.945145    4892 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/56bf0c05-25c3-4fc3-9299-2eaa33c76d07-maas-api-xks-ca" pod "56bf0c05-25c3-4fc3-9299-2eaa33c76d07" (UID: "56bf0c05-25c3-4fc3-9299-2eaa33c76d07"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 20:01:49.945551 ip-10-0-132-82 kubenswrapper[4892]: I1030 20:01:49.945531    4892 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/56bf0c05-25c3-4fc3-9299-2eaa33c76d07-kube-api-access-dkd6r" pod "56bf0c05-25c3-4fc3-9299-2eaa33c76d07" (UID: "56bf0c05-25c3-4fc3-9299-2eaa33c76d07"). InnerVolumeSpecName "kube-api-access-dkd6r". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 20:01:50.044124 ip-10-0-132-82 kubenswrapper[4892]: I1030 20:01:50.044088    4892 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/56bf0c05-25c3-4fc3-9299-2eaa33c76d07-maas-api-xks-ca\") on node \"ip-10-0-132-82.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:01:50.044124 ip-10-0-132-82 kubenswrapper[4892]: I1030 20:01:50.044125    4892 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dkd6r\" (UniqueName: \"kubernetes.io/projected/56bf0c05-25c3-4fc3-9299-2eaa33c76d07-kube-api-access-dkd6r\") on node \"ip-10-0-132-82.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:01:50.681897 ip-10-0-132-82 kubenswrapper[4892]: I1030 20:01:50.681861    4892 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8bdbd0b9-qpqgz"
Oct 30 20:01:50.682096 ip-10-0-132-82 kubenswrapper[4892]: I1030 20:01:50.681886    4892 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8bdbd0b9-qpqgz" event={"ID":"56bf0c05-25c3-4fc3-9299-2eaa33c76d07","Type":"ContainerDied","Data":"84b68d4166bf33b789e00017f5adeb0a82ca5726509f1950869e4353c9141544"}
Oct 30 20:01:50.682096 ip-10-0-132-82 kubenswrapper[4892]: I1030 20:01:50.681940    4892 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="84b68d4166bf33b789e00017f5adeb0a82ca5726509f1950869e4353c9141544"
Oct 30 20:01:52.671589 ip-10-0-132-82 kubenswrapper[4892]: I1030 20:01:52.671566    4892 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-5f55cd-openshift-default-9dc497cb5-q2fbg"
Oct 30 20:01:52.695935 ip-10-0-132-82 kubenswrapper[4892]: I1030 20:01:52.695869    4892 generic.go:356] "Generic (PLEG): container finished" podID="fac4fe86-a38e-434e-bfc1-4c3dbceffd27" containerID="e3df77051ab71dd38f8567e41d2b95dcd600ea2f8b9dbe7bc16540bfdc010985" exitCode=0
Oct 30 20:01:52.695935 ip-10-0-132-82 kubenswrapper[4892]: I1030 20:01:52.695914    4892 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-5f55cd-openshift-default-9dc497cb5-q2fbg"
Oct 30 20:01:52.696147 ip-10-0-132-82 kubenswrapper[4892]: I1030 20:01:52.695950    4892 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-5f55cd-openshift-default-9dc497cb5-q2fbg" event={"ID":"fac4fe86-a38e-434e-bfc1-4c3dbceffd27","Type":"ContainerDied","Data":"e3df77051ab71dd38f8567e41d2b95dcd600ea2f8b9dbe7bc16540bfdc010985"}
Oct 30 20:01:52.696147 ip-10-0-132-82 kubenswrapper[4892]: I1030 20:01:52.696031    4892 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-5f55cd-openshift-default-9dc497cb5-q2fbg" event={"ID":"fac4fe86-a38e-434e-bfc1-4c3dbceffd27","Type":"ContainerDied","Data":"7e1583030fce73aed574c4372208fad061d5c3d3bdc799ff9927177febbf4382"}
Oct 30 20:01:52.696147 ip-10-0-132-82 kubenswrapper[4892]: I1030 20:01:52.696063    4892 scope.go:122] "RemoveContainer" containerID="e3df77051ab71dd38f8567e41d2b95dcd600ea2f8b9dbe7bc16540bfdc010985"
Oct 30 20:01:52.705867 ip-10-0-132-82 kubenswrapper[4892]: I1030 20:01:52.705850    4892 scope.go:122] "RemoveContainer" containerID="e3df77051ab71dd38f8567e41d2b95dcd600ea2f8b9dbe7bc16540bfdc010985"
Oct 30 20:01:52.706177 ip-10-0-132-82 kubenswrapper[4892]: E1030 20:01:52.706158    4892 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e3df77051ab71dd38f8567e41d2b95dcd600ea2f8b9dbe7bc16540bfdc010985\": container with ID starting with e3df77051ab71dd38f8567e41d2b95dcd600ea2f8b9dbe7bc16540bfdc010985 not found: ID does not exist" containerID="e3df77051ab71dd38f8567e41d2b95dcd600ea2f8b9dbe7bc16540bfdc010985"
Oct 30 20:01:52.706251 ip-10-0-132-82 kubenswrapper[4892]: I1030 20:01:52.706184    4892 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e3df77051ab71dd38f8567e41d2b95dcd600ea2f8b9dbe7bc16540bfdc010985"} err="failed to get container status \"e3df77051ab71dd38f8567e41d2b95dcd600ea2f8b9dbe7bc16540bfdc010985\": rpc error: code = NotFound desc = could not find container \"e3df77051ab71dd38f8567e41d2b95dcd600ea2f8b9dbe7bc16540bfdc010985\": container with ID starting with e3df77051ab71dd38f8567e41d2b95dcd600ea2f8b9dbe7bc16540bfdc010985 not found: ID does not exist"
Oct 30 20:01:52.769027 ip-10-0-132-82 kubenswrapper[4892]: I1030 20:01:52.769008    4892 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/fac4fe86-a38e-434e-bfc1-4c3dbceffd27-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/fac4fe86-a38e-434e-bfc1-4c3dbceffd27-istio-envoy\") pod \"fac4fe86-a38e-434e-bfc1-4c3dbceffd27\" (UID: \"fac4fe86-a38e-434e-bfc1-4c3dbceffd27\") "
Oct 30 20:01:52.769142 ip-10-0-132-82 kubenswrapper[4892]: I1030 20:01:52.769043    4892 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/fac4fe86-a38e-434e-bfc1-4c3dbceffd27-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/fac4fe86-a38e-434e-bfc1-4c3dbceffd27-workload-certs\") pod \"fac4fe86-a38e-434e-bfc1-4c3dbceffd27\" (UID: \"fac4fe86-a38e-434e-bfc1-4c3dbceffd27\") "
Oct 30 20:01:52.769142 ip-10-0-132-82 kubenswrapper[4892]: I1030 20:01:52.769082    4892 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/fac4fe86-a38e-434e-bfc1-4c3dbceffd27-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/fac4fe86-a38e-434e-bfc1-4c3dbceffd27-istiod-ca-cert\") pod \"fac4fe86-a38e-434e-bfc1-4c3dbceffd27\" (UID: \"fac4fe86-a38e-434e-bfc1-4c3dbceffd27\") "
Oct 30 20:01:52.769142 ip-10-0-132-82 kubenswrapper[4892]: I1030 20:01:52.769107    4892 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/fac4fe86-a38e-434e-bfc1-4c3dbceffd27-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/fac4fe86-a38e-434e-bfc1-4c3dbceffd27-workload-socket\") pod \"fac4fe86-a38e-434e-bfc1-4c3dbceffd27\" (UID: \"fac4fe86-a38e-434e-bfc1-4c3dbceffd27\") "
Oct 30 20:01:52.769142 ip-10-0-132-82 kubenswrapper[4892]: I1030 20:01:52.769135    4892 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/fac4fe86-a38e-434e-bfc1-4c3dbceffd27-istio-token\" (UniqueName: \"kubernetes.io/projected/fac4fe86-a38e-434e-bfc1-4c3dbceffd27-istio-token\") pod \"fac4fe86-a38e-434e-bfc1-4c3dbceffd27\" (UID: \"fac4fe86-a38e-434e-bfc1-4c3dbceffd27\") "
Oct 30 20:01:52.769374 ip-10-0-132-82 kubenswrapper[4892]: I1030 20:01:52.769158    4892 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/fac4fe86-a38e-434e-bfc1-4c3dbceffd27-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/fac4fe86-a38e-434e-bfc1-4c3dbceffd27-credential-socket\") pod \"fac4fe86-a38e-434e-bfc1-4c3dbceffd27\" (UID: \"fac4fe86-a38e-434e-bfc1-4c3dbceffd27\") "
Oct 30 20:01:52.769374 ip-10-0-132-82 kubenswrapper[4892]: I1030 20:01:52.769189    4892 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/fac4fe86-a38e-434e-bfc1-4c3dbceffd27-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/fac4fe86-a38e-434e-bfc1-4c3dbceffd27-istio-data\") pod \"fac4fe86-a38e-434e-bfc1-4c3dbceffd27\" (UID: \"fac4fe86-a38e-434e-bfc1-4c3dbceffd27\") "
Oct 30 20:01:52.769374 ip-10-0-132-82 kubenswrapper[4892]: I1030 20:01:52.769256    4892 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/fac4fe86-a38e-434e-bfc1-4c3dbceffd27-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/fac4fe86-a38e-434e-bfc1-4c3dbceffd27-istio-podinfo\") pod \"fac4fe86-a38e-434e-bfc1-4c3dbceffd27\" (UID: \"fac4fe86-a38e-434e-bfc1-4c3dbceffd27\") "
Oct 30 20:01:52.769374 ip-10-0-132-82 kubenswrapper[4892]: I1030 20:01:52.769284    4892 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/fac4fe86-a38e-434e-bfc1-4c3dbceffd27-kube-api-access-vv9kc\" (UniqueName: \"kubernetes.io/projected/fac4fe86-a38e-434e-bfc1-4c3dbceffd27-kube-api-access-vv9kc\") pod \"fac4fe86-a38e-434e-bfc1-4c3dbceffd27\" (UID: \"fac4fe86-a38e-434e-bfc1-4c3dbceffd27\") "
Oct 30 20:01:52.769374 ip-10-0-132-82 kubenswrapper[4892]: I1030 20:01:52.769333    4892 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fac4fe86-a38e-434e-bfc1-4c3dbceffd27-workload-certs" pod "fac4fe86-a38e-434e-bfc1-4c3dbceffd27" (UID: "fac4fe86-a38e-434e-bfc1-4c3dbceffd27"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 20:01:52.769656 ip-10-0-132-82 kubenswrapper[4892]: I1030 20:01:52.769405    4892 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fac4fe86-a38e-434e-bfc1-4c3dbceffd27-workload-socket" pod "fac4fe86-a38e-434e-bfc1-4c3dbceffd27" (UID: "fac4fe86-a38e-434e-bfc1-4c3dbceffd27"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 20:01:52.769656 ip-10-0-132-82 kubenswrapper[4892]: I1030 20:01:52.769531    4892 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/fac4fe86-a38e-434e-bfc1-4c3dbceffd27-istiod-ca-cert" pod "fac4fe86-a38e-434e-bfc1-4c3dbceffd27" (UID: "fac4fe86-a38e-434e-bfc1-4c3dbceffd27"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 20:01:52.769656 ip-10-0-132-82 kubenswrapper[4892]: I1030 20:01:52.769549    4892 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fac4fe86-a38e-434e-bfc1-4c3dbceffd27-istio-data" pod "fac4fe86-a38e-434e-bfc1-4c3dbceffd27" (UID: "fac4fe86-a38e-434e-bfc1-4c3dbceffd27"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 20:01:52.769807 ip-10-0-132-82 kubenswrapper[4892]: I1030 20:01:52.769679    4892 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/fac4fe86-a38e-434e-bfc1-4c3dbceffd27-workload-certs\") on node \"ip-10-0-132-82.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:01:52.769807 ip-10-0-132-82 kubenswrapper[4892]: I1030 20:01:52.769700    4892 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/fac4fe86-a38e-434e-bfc1-4c3dbceffd27-istiod-ca-cert\") on node \"ip-10-0-132-82.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:01:52.769807 ip-10-0-132-82 kubenswrapper[4892]: I1030 20:01:52.769715    4892 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/fac4fe86-a38e-434e-bfc1-4c3dbceffd27-workload-socket\") on node \"ip-10-0-132-82.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:01:52.769807 ip-10-0-132-82 kubenswrapper[4892]: I1030 20:01:52.769730    4892 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/fac4fe86-a38e-434e-bfc1-4c3dbceffd27-istio-data\") on node \"ip-10-0-132-82.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:01:52.769807 ip-10-0-132-82 kubenswrapper[4892]: I1030 20:01:52.769707    4892 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fac4fe86-a38e-434e-bfc1-4c3dbceffd27-credential-socket" pod "fac4fe86-a38e-434e-bfc1-4c3dbceffd27" (UID: "fac4fe86-a38e-434e-bfc1-4c3dbceffd27"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 20:01:52.771257 ip-10-0-132-82 kubenswrapper[4892]: I1030 20:01:52.771232    4892 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fac4fe86-a38e-434e-bfc1-4c3dbceffd27-istio-envoy" pod "fac4fe86-a38e-434e-bfc1-4c3dbceffd27" (UID: "fac4fe86-a38e-434e-bfc1-4c3dbceffd27"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 20:01:52.771473 ip-10-0-132-82 kubenswrapper[4892]: I1030 20:01:52.771456    4892 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fac4fe86-a38e-434e-bfc1-4c3dbceffd27-kube-api-access-vv9kc" pod "fac4fe86-a38e-434e-bfc1-4c3dbceffd27" (UID: "fac4fe86-a38e-434e-bfc1-4c3dbceffd27"). InnerVolumeSpecName "kube-api-access-vv9kc". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 20:01:52.771594 ip-10-0-132-82 kubenswrapper[4892]: I1030 20:01:52.771579    4892 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fac4fe86-a38e-434e-bfc1-4c3dbceffd27-istio-token" pod "fac4fe86-a38e-434e-bfc1-4c3dbceffd27" (UID: "fac4fe86-a38e-434e-bfc1-4c3dbceffd27"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 20:01:52.771731 ip-10-0-132-82 kubenswrapper[4892]: I1030 20:01:52.771712    4892 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/fac4fe86-a38e-434e-bfc1-4c3dbceffd27-istio-podinfo" pod "fac4fe86-a38e-434e-bfc1-4c3dbceffd27" (UID: "fac4fe86-a38e-434e-bfc1-4c3dbceffd27"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 20:01:52.870543 ip-10-0-132-82 kubenswrapper[4892]: I1030 20:01:52.870498    4892 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/fac4fe86-a38e-434e-bfc1-4c3dbceffd27-istio-token\") on node \"ip-10-0-132-82.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:01:52.870711 ip-10-0-132-82 kubenswrapper[4892]: I1030 20:01:52.870548    4892 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/fac4fe86-a38e-434e-bfc1-4c3dbceffd27-credential-socket\") on node \"ip-10-0-132-82.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:01:52.870711 ip-10-0-132-82 kubenswrapper[4892]: I1030 20:01:52.870564    4892 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/fac4fe86-a38e-434e-bfc1-4c3dbceffd27-istio-podinfo\") on node \"ip-10-0-132-82.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:01:52.870711 ip-10-0-132-82 kubenswrapper[4892]: I1030 20:01:52.870579    4892 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vv9kc\" (UniqueName: \"kubernetes.io/projected/fac4fe86-a38e-434e-bfc1-4c3dbceffd27-kube-api-access-vv9kc\") on node \"ip-10-0-132-82.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:01:52.870711 ip-10-0-132-82 kubenswrapper[4892]: I1030 20:01:52.870594    4892 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/fac4fe86-a38e-434e-bfc1-4c3dbceffd27-istio-envoy\") on node \"ip-10-0-132-82.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:01:53.117773 ip-10-0-132-82 kubenswrapper[4892]: I1030 20:01:53.117744    4892 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w6-5f55cd-openshift-default-9dc497cb5-q2fbg"]
Oct 30 20:01:53.121875 ip-10-0-132-82 kubenswrapper[4892]: I1030 20:01:53.121851    4892 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w6-5f55cd-openshift-default-9dc497cb5-q2fbg"]
Oct 30 20:01:54.568662 ip-10-0-132-82 kubenswrapper[4892]: I1030 20:01:54.568628    4892 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="fac4fe86-a38e-434e-bfc1-4c3dbceffd27" path="/var/lib/kubelet/pods/fac4fe86-a38e-434e-bfc1-4c3dbceffd27/volumes"
Oct 30 20:01:54.842919 ip-10-0-132-82 kubenswrapper[4892]: E1030 20:01:54.842829    4892 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-jrm87" containerName="extractor"
Oct 30 20:01:59.443779 ip-10-0-132-82 kubenswrapper[4892]: I1030 20:01:59.443715    4892 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-2d9861-b5kt4"]
Oct 30 20:01:59.492479 ip-10-0-132-82 kubenswrapper[4892]: I1030 20:01:59.492450    4892 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-2d9861-b5kt4"]
Oct 30 20:02:00.568774 ip-10-0-132-82 kubenswrapper[4892]: I1030 20:02:00.568738    4892 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="fc03b7fe-3210-4912-9437-f9a8fc838b28" path="/var/lib/kubelet/pods/fc03b7fe-3210-4912-9437-f9a8fc838b28/volumes"
Oct 30 20:02:00.685578 ip-10-0-132-82 kubenswrapper[4892]: I1030 20:02:00.685548    4892 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-c75ed4-jxjdl"]
Oct 30 20:02:00.734885 ip-10-0-132-82 kubenswrapper[4892]: I1030 20:02:00.734847    4892 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-c75ed4-jxjdl"]
Oct 30 20:02:02.102319 ip-10-0-132-82 kubenswrapper[4892]: I1030 20:02:02.102287    4892 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w2-a3f9b7/simulator-w2-a3f9b7-kserve-79bbc4cb5f-8m488"]
Oct 30 20:02:02.568542 ip-10-0-132-82 kubenswrapper[4892]: I1030 20:02:02.568504    4892 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="75e5e791-3967-4357-a185-6aab26a05ce9" path="/var/lib/kubelet/pods/75e5e791-3967-4357-a185-6aab26a05ce9/volumes"
Oct 30 20:02:07.396822 ip-10-0-132-82 kubenswrapper[4892]: I1030 20:02:07.396789    4892 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-a3f9b7-gfgvh"]
Oct 30 20:02:07.397246 ip-10-0-132-82 kubenswrapper[4892]: I1030 20:02:07.397228    4892 cpu_manager.go:414] "containerMap: removing container" podUID="fac4fe86-a38e-434e-bfc1-4c3dbceffd27" containerName="istio-proxy"
Oct 30 20:02:07.397246 ip-10-0-132-82 kubenswrapper[4892]: I1030 20:02:07.397240    4892 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="fac4fe86-a38e-434e-bfc1-4c3dbceffd27" containerName="istio-proxy"
Oct 30 20:02:07.397329 ip-10-0-132-82 kubenswrapper[4892]: I1030 20:02:07.397256    4892 cpu_manager.go:414] "containerMap: removing container" podUID="56bf0c05-25c3-4fc3-9299-2eaa33c76d07" containerName="revoke-keys"
Oct 30 20:02:07.397329 ip-10-0-132-82 kubenswrapper[4892]: I1030 20:02:07.397261    4892 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="56bf0c05-25c3-4fc3-9299-2eaa33c76d07" containerName="revoke-keys"
Oct 30 20:02:07.397394 ip-10-0-132-82 kubenswrapper[4892]: I1030 20:02:07.397336    4892 memory_manager.go:365] "RemoveStaleState removing state" podUID="fac4fe86-a38e-434e-bfc1-4c3dbceffd27" containerName="istio-proxy"
Oct 30 20:02:07.397394 ip-10-0-132-82 kubenswrapper[4892]: I1030 20:02:07.397346    4892 memory_manager.go:365] "RemoveStaleState removing state" podUID="56bf0c05-25c3-4fc3-9299-2eaa33c76d07" containerName="revoke-keys"
Oct 30 20:02:07.402449 ip-10-0-132-82 kubenswrapper[4892]: I1030 20:02:07.402433    4892 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-a3f9b7-gfgvh"
Oct 30 20:02:07.504911 ip-10-0-132-82 kubenswrapper[4892]: I1030 20:02:07.504883    4892 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 20:02:07.504911 ip-10-0-132-82 kubenswrapper[4892]: I1030 20:02:07.504908    4892 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 30 20:02:07.505142 ip-10-0-132-82 kubenswrapper[4892]: I1030 20:02:07.504947    4892 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 30 20:02:07.505142 ip-10-0-132-82 kubenswrapper[4892]: I1030 20:02:07.504998    4892 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-nlxp4\""
Oct 30 20:02:07.507628 ip-10-0-132-82 kubenswrapper[4892]: I1030 20:02:07.507610    4892 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-a3f9b7-gfgvh"]
Oct 30 20:02:07.515561 ip-10-0-132-82 kubenswrapper[4892]: I1030 20:02:07.515538    4892 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xs55d\" (UniqueName: \"kubernetes.io/projected/497e96db-0aa8-4f9f-be89-4797e1fc6d0a-kube-api-access-xs55d\") pod \"maas-api-revoke-keys-e2e-worker-w2-a3f9b7-gfgvh\" (UID: \"497e96db-0aa8-4f9f-be89-4797e1fc6d0a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-a3f9b7-gfgvh"
Oct 30 20:02:07.515679 ip-10-0-132-82 kubenswrapper[4892]: I1030 20:02:07.515589    4892 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/497e96db-0aa8-4f9f-be89-4797e1fc6d0a-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-w2-a3f9b7-gfgvh\" (UID: \"497e96db-0aa8-4f9f-be89-4797e1fc6d0a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-a3f9b7-gfgvh"
Oct 30 20:02:07.515781 ip-10-0-132-82 kubenswrapper[4892]: I1030 20:02:07.515754    4892 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/497e96db-0aa8-4f9f-be89-4797e1fc6d0a-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-w2-a3f9b7-gfgvh\" (UID: \"497e96db-0aa8-4f9f-be89-4797e1fc6d0a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-a3f9b7-gfgvh"
Oct 30 20:02:07.616675 ip-10-0-132-82 kubenswrapper[4892]: I1030 20:02:07.616643    4892 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/497e96db-0aa8-4f9f-be89-4797e1fc6d0a-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-w2-a3f9b7-gfgvh\" (UID: \"497e96db-0aa8-4f9f-be89-4797e1fc6d0a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-a3f9b7-gfgvh"
Oct 30 20:02:07.616808 ip-10-0-132-82 kubenswrapper[4892]: I1030 20:02:07.616747    4892 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xs55d\" (UniqueName: \"kubernetes.io/projected/497e96db-0aa8-4f9f-be89-4797e1fc6d0a-kube-api-access-xs55d\") pod \"maas-api-revoke-keys-e2e-worker-w2-a3f9b7-gfgvh\" (UID: \"497e96db-0aa8-4f9f-be89-4797e1fc6d0a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-a3f9b7-gfgvh"
Oct 30 20:02:07.616808 ip-10-0-132-82 kubenswrapper[4892]: I1030 20:02:07.616795    4892 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/497e96db-0aa8-4f9f-be89-4797e1fc6d0a-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-w2-a3f9b7-gfgvh\" (UID: \"497e96db-0aa8-4f9f-be89-4797e1fc6d0a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-a3f9b7-gfgvh"
Oct 30 20:02:07.617502 ip-10-0-132-82 kubenswrapper[4892]: I1030 20:02:07.617475    4892 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/497e96db-0aa8-4f9f-be89-4797e1fc6d0a-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-w2-a3f9b7-gfgvh\" (UID: \"497e96db-0aa8-4f9f-be89-4797e1fc6d0a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-a3f9b7-gfgvh"
Oct 30 20:02:07.619092 ip-10-0-132-82 kubenswrapper[4892]: I1030 20:02:07.619066    4892 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/497e96db-0aa8-4f9f-be89-4797e1fc6d0a-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-w2-a3f9b7-gfgvh\" (UID: \"497e96db-0aa8-4f9f-be89-4797e1fc6d0a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-a3f9b7-gfgvh"
Oct 30 20:02:07.674882 ip-10-0-132-82 kubenswrapper[4892]: I1030 20:02:07.674828    4892 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xs55d\" (UniqueName: \"kubernetes.io/projected/497e96db-0aa8-4f9f-be89-4797e1fc6d0a-kube-api-access-xs55d\") pod \"maas-api-revoke-keys-e2e-worker-w2-a3f9b7-gfgvh\" (UID: \"497e96db-0aa8-4f9f-be89-4797e1fc6d0a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-a3f9b7-gfgvh"
Oct 30 20:02:07.715277 ip-10-0-132-82 kubenswrapper[4892]: I1030 20:02:07.715253    4892 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-a3f9b7-gfgvh"
Oct 30 20:02:08.099957 ip-10-0-132-82 kubenswrapper[4892]: I1030 20:02:08.099926    4892 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-a3f9b7-gfgvh"]
Oct 30 20:02:08.101123 ip-10-0-132-82 kubenswrapper[4892]: W1030 20:02:08.101096    4892 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod497e96db_0aa8_4f9f_be89_4797e1fc6d0a.slice/crio-e55374b1a2f2e2942c0dd0ca2cc0cf268edf77eae5146067a5e8930b17d923b2 WatchSource:0}: Error finding container e55374b1a2f2e2942c0dd0ca2cc0cf268edf77eae5146067a5e8930b17d923b2: Status 404 returned error can't find the container with id e55374b1a2f2e2942c0dd0ca2cc0cf268edf77eae5146067a5e8930b17d923b2
Oct 30 20:02:08.815566 ip-10-0-132-82 kubenswrapper[4892]: I1030 20:02:08.815529    4892 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-a3f9b7-gfgvh" event={"ID":"497e96db-0aa8-4f9f-be89-4797e1fc6d0a","Type":"ContainerStarted","Data":"530375a37374506ec1d13e2ffefc8cbb2c4c939ad609ca8a15d08fa6b116e2c0"}
Oct 30 20:02:08.816008 ip-10-0-132-82 kubenswrapper[4892]: I1030 20:02:08.815585    4892 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-a3f9b7-gfgvh" event={"ID":"497e96db-0aa8-4f9f-be89-4797e1fc6d0a","Type":"ContainerStarted","Data":"e55374b1a2f2e2942c0dd0ca2cc0cf268edf77eae5146067a5e8930b17d923b2"}
Oct 30 20:02:08.936338 ip-10-0-132-82 kubenswrapper[4892]: I1030 20:02:08.936273    4892 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-a3f9b7-gfgvh" podStartSLOduration=1.93625603 podStartE2EDuration="1.93625603s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 20:02:07 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 20:02:08.932442542 +0000 UTC m=+2776.502908141" watchObservedRunningTime="2026-10-30 20:02:08.93625603 +0000 UTC m=+2776.506721633"
Oct 30 20:02:09.186481 ip-10-0-132-82 kubenswrapper[4892]: I1030 20:02:09.186452    4892 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-ada8d7-ss6z5"]
Oct 30 20:02:09.190068 ip-10-0-132-82 kubenswrapper[4892]: I1030 20:02:09.190046    4892 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-ada8d7-ss6z5"
Oct 30 20:02:09.265717 ip-10-0-132-82 kubenswrapper[4892]: I1030 20:02:09.265305    4892 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-ada8d7-ss6z5"]
Oct 30 20:02:09.334222 ip-10-0-132-82 kubenswrapper[4892]: I1030 20:02:09.334193    4892 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/73e40af0-f49a-41bf-9dbd-e52de5027c21-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-w3-ada8d7-ss6z5\" (UID: \"73e40af0-f49a-41bf-9dbd-e52de5027c21\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-ada8d7-ss6z5"
Oct 30 20:02:09.334399 ip-10-0-132-82 kubenswrapper[4892]: I1030 20:02:09.334252    4892 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9gttt\" (UniqueName: \"kubernetes.io/projected/73e40af0-f49a-41bf-9dbd-e52de5027c21-kube-api-access-9gttt\") pod \"maas-api-revoke-keys-e2e-worker-w3-ada8d7-ss6z5\" (UID: \"73e40af0-f49a-41bf-9dbd-e52de5027c21\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-ada8d7-ss6z5"
Oct 30 20:02:09.334399 ip-10-0-132-82 kubenswrapper[4892]: I1030 20:02:09.334318    4892 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/73e40af0-f49a-41bf-9dbd-e52de5027c21-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-w3-ada8d7-ss6z5\" (UID: \"73e40af0-f49a-41bf-9dbd-e52de5027c21\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-ada8d7-ss6z5"
Oct 30 20:02:09.435297 ip-10-0-132-82 kubenswrapper[4892]: I1030 20:02:09.435265    4892 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/73e40af0-f49a-41bf-9dbd-e52de5027c21-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-w3-ada8d7-ss6z5\" (UID: \"73e40af0-f49a-41bf-9dbd-e52de5027c21\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-ada8d7-ss6z5"
Oct 30 20:02:09.435493 ip-10-0-132-82 kubenswrapper[4892]: I1030 20:02:09.435325    4892 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9gttt\" (UniqueName: \"kubernetes.io/projected/73e40af0-f49a-41bf-9dbd-e52de5027c21-kube-api-access-9gttt\") pod \"maas-api-revoke-keys-e2e-worker-w3-ada8d7-ss6z5\" (UID: \"73e40af0-f49a-41bf-9dbd-e52de5027c21\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-ada8d7-ss6z5"
Oct 30 20:02:09.435493 ip-10-0-132-82 kubenswrapper[4892]: I1030 20:02:09.435350    4892 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/73e40af0-f49a-41bf-9dbd-e52de5027c21-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-w3-ada8d7-ss6z5\" (UID: \"73e40af0-f49a-41bf-9dbd-e52de5027c21\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-ada8d7-ss6z5"
Oct 30 20:02:09.436038 ip-10-0-132-82 kubenswrapper[4892]: I1030 20:02:09.436011    4892 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/73e40af0-f49a-41bf-9dbd-e52de5027c21-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-w3-ada8d7-ss6z5\" (UID: \"73e40af0-f49a-41bf-9dbd-e52de5027c21\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-ada8d7-ss6z5"
Oct 30 20:02:09.438173 ip-10-0-132-82 kubenswrapper[4892]: I1030 20:02:09.438108    4892 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/73e40af0-f49a-41bf-9dbd-e52de5027c21-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-w3-ada8d7-ss6z5\" (UID: \"73e40af0-f49a-41bf-9dbd-e52de5027c21\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-ada8d7-ss6z5"
Oct 30 20:02:09.492946 ip-10-0-132-82 kubenswrapper[4892]: I1030 20:02:09.492920    4892 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9gttt\" (UniqueName: \"kubernetes.io/projected/73e40af0-f49a-41bf-9dbd-e52de5027c21-kube-api-access-9gttt\") pod \"maas-api-revoke-keys-e2e-worker-w3-ada8d7-ss6z5\" (UID: \"73e40af0-f49a-41bf-9dbd-e52de5027c21\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-ada8d7-ss6z5"
Oct 30 20:02:09.502751 ip-10-0-132-82 kubenswrapper[4892]: I1030 20:02:09.502731    4892 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-ada8d7-ss6z5"
Oct 30 20:02:09.824712 ip-10-0-132-82 kubenswrapper[4892]: I1030 20:02:09.824627    4892 generic.go:356] "Generic (PLEG): container finished" podID="497e96db-0aa8-4f9f-be89-4797e1fc6d0a" containerID="530375a37374506ec1d13e2ffefc8cbb2c4c939ad609ca8a15d08fa6b116e2c0" exitCode=0
Oct 30 20:02:09.824712 ip-10-0-132-82 kubenswrapper[4892]: I1030 20:02:09.824698    4892 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-a3f9b7-gfgvh" event={"ID":"497e96db-0aa8-4f9f-be89-4797e1fc6d0a","Type":"ContainerDied","Data":"530375a37374506ec1d13e2ffefc8cbb2c4c939ad609ca8a15d08fa6b116e2c0"}
Oct 30 20:02:09.888170 ip-10-0-132-82 kubenswrapper[4892]: W1030 20:02:09.888136    4892 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod73e40af0_f49a_41bf_9dbd_e52de5027c21.slice/crio-0e8e5a2da1dbe2772e279205a4554ce26a83667d4151fb0a6784c5a6a997abcf WatchSource:0}: Error finding container 0e8e5a2da1dbe2772e279205a4554ce26a83667d4151fb0a6784c5a6a997abcf: Status 404 returned error can't find the container with id 0e8e5a2da1dbe2772e279205a4554ce26a83667d4151fb0a6784c5a6a997abcf
Oct 30 20:02:09.888429 ip-10-0-132-82 kubenswrapper[4892]: I1030 20:02:09.888407    4892 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-ada8d7-ss6z5"]
Oct 30 20:02:10.835676 ip-10-0-132-82 kubenswrapper[4892]: I1030 20:02:10.835634    4892 generic.go:356] "Generic (PLEG): container finished" podID="73e40af0-f49a-41bf-9dbd-e52de5027c21" containerID="6e2f1bb171ba7408ba5268f7d248482cd9666a47606ddee2b30525ef31256a7b" exitCode=0
Oct 30 20:02:10.836111 ip-10-0-132-82 kubenswrapper[4892]: I1030 20:02:10.835703    4892 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-ada8d7-ss6z5" event={"ID":"73e40af0-f49a-41bf-9dbd-e52de5027c21","Type":"ContainerDied","Data":"6e2f1bb171ba7408ba5268f7d248482cd9666a47606ddee2b30525ef31256a7b"}
Oct 30 20:02:10.836111 ip-10-0-132-82 kubenswrapper[4892]: I1030 20:02:10.835768    4892 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-ada8d7-ss6z5" event={"ID":"73e40af0-f49a-41bf-9dbd-e52de5027c21","Type":"ContainerStarted","Data":"0e8e5a2da1dbe2772e279205a4554ce26a83667d4151fb0a6784c5a6a997abcf"}
Oct 30 20:02:10.989134 ip-10-0-132-82 kubenswrapper[4892]: I1030 20:02:10.989107    4892 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-a3f9b7-gfgvh"
Oct 30 20:02:11.001756 ip-10-0-132-82 kubenswrapper[4892]: I1030 20:02:11.001729    4892 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-p7m7x"]
Oct 30 20:02:11.002396 ip-10-0-132-82 kubenswrapper[4892]: I1030 20:02:11.002373    4892 cpu_manager.go:414] "containerMap: removing container" podUID="497e96db-0aa8-4f9f-be89-4797e1fc6d0a" containerName="revoke-keys"
Oct 30 20:02:11.002396 ip-10-0-132-82 kubenswrapper[4892]: I1030 20:02:11.002396    4892 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="497e96db-0aa8-4f9f-be89-4797e1fc6d0a" containerName="revoke-keys"
Oct 30 20:02:11.002532 ip-10-0-132-82 kubenswrapper[4892]: I1030 20:02:11.002500    4892 memory_manager.go:365] "RemoveStaleState removing state" podUID="497e96db-0aa8-4f9f-be89-4797e1fc6d0a" containerName="revoke-keys"
Oct 30 20:02:11.005561 ip-10-0-132-82 kubenswrapper[4892]: I1030 20:02:11.005544    4892 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-p7m7x"
Oct 30 20:02:11.007198 ip-10-0-132-82 kubenswrapper[4892]: I1030 20:02:11.007177    4892 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-mb59v"]
Oct 30 20:02:11.010942 ip-10-0-132-82 kubenswrapper[4892]: I1030 20:02:11.010922    4892 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-mb59v"
Oct 30 20:02:11.058248 ip-10-0-132-82 kubenswrapper[4892]: I1030 20:02:11.058226    4892 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"maas-default-gateway-openshift-default-dockercfg-f2npg\""
Oct 30 20:02:11.072740 ip-10-0-132-82 kubenswrapper[4892]: I1030 20:02:11.072705    4892 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-p7m7x"]
Oct 30 20:02:11.118832 ip-10-0-132-82 kubenswrapper[4892]: I1030 20:02:11.118799    4892 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-mb59v"]
Oct 30 20:02:11.150741 ip-10-0-132-82 kubenswrapper[4892]: I1030 20:02:11.150705    4892 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/497e96db-0aa8-4f9f-be89-4797e1fc6d0a-kube-api-access-xs55d\" (UniqueName: \"kubernetes.io/projected/497e96db-0aa8-4f9f-be89-4797e1fc6d0a-kube-api-access-xs55d\") pod \"497e96db-0aa8-4f9f-be89-4797e1fc6d0a\" (UID: \"497e96db-0aa8-4f9f-be89-4797e1fc6d0a\") "
Oct 30 20:02:11.150927 ip-10-0-132-82 kubenswrapper[4892]: I1030 20:02:11.150791    4892 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/497e96db-0aa8-4f9f-be89-4797e1fc6d0a-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/497e96db-0aa8-4f9f-be89-4797e1fc6d0a-maas-api-xks-ca\") pod \"497e96db-0aa8-4f9f-be89-4797e1fc6d0a\" (UID: \"497e96db-0aa8-4f9f-be89-4797e1fc6d0a\") "
Oct 30 20:02:11.150927 ip-10-0-132-82 kubenswrapper[4892]: I1030 20:02:11.150875    4892 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/497e96db-0aa8-4f9f-be89-4797e1fc6d0a-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/497e96db-0aa8-4f9f-be89-4797e1fc6d0a-maas-api-openshift-ca\") pod \"497e96db-0aa8-4f9f-be89-4797e1fc6d0a\" (UID: \"497e96db-0aa8-4f9f-be89-4797e1fc6d0a\") "
Oct 30 20:02:11.151103 ip-10-0-132-82 kubenswrapper[4892]: I1030 20:02:11.151086    4892 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/72926e14-705b-493a-879f-f595c35a7490-istio-envoy\") pod \"maas-default-gateway-openshift-default-6db586dd6c-mb59v\" (UID: \"72926e14-705b-493a-879f-f595c35a7490\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-mb59v"
Oct 30 20:02:11.151179 ip-10-0-132-82 kubenswrapper[4892]: I1030 20:02:11.151126    4892 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/72926e14-705b-493a-879f-f595c35a7490-istio-data\") pod \"maas-default-gateway-openshift-default-6db586dd6c-mb59v\" (UID: \"72926e14-705b-493a-879f-f595c35a7490\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-mb59v"
Oct 30 20:02:11.151179 ip-10-0-132-82 kubenswrapper[4892]: I1030 20:02:11.151152    4892 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/72926e14-705b-493a-879f-f595c35a7490-workload-socket\") pod \"maas-default-gateway-openshift-default-6db586dd6c-mb59v\" (UID: \"72926e14-705b-493a-879f-f595c35a7490\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-mb59v"
Oct 30 20:02:11.151301 ip-10-0-132-82 kubenswrapper[4892]: I1030 20:02:11.151175    4892 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/741c37d3-758a-4ced-ab0e-9038baea2d88-workload-certs\") pod \"maas-default-gateway-openshift-default-6db586dd6c-p7m7x\" (UID: \"741c37d3-758a-4ced-ab0e-9038baea2d88\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-p7m7x"
Oct 30 20:02:11.151301 ip-10-0-132-82 kubenswrapper[4892]: I1030 20:02:11.151251    4892 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/741c37d3-758a-4ced-ab0e-9038baea2d88-istio-data\") pod \"maas-default-gateway-openshift-default-6db586dd6c-p7m7x\" (UID: \"741c37d3-758a-4ced-ab0e-9038baea2d88\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-p7m7x"
Oct 30 20:02:11.151301 ip-10-0-132-82 kubenswrapper[4892]: I1030 20:02:11.151295    4892 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hgml7\" (UniqueName: \"kubernetes.io/projected/72926e14-705b-493a-879f-f595c35a7490-kube-api-access-hgml7\") pod \"maas-default-gateway-openshift-default-6db586dd6c-mb59v\" (UID: \"72926e14-705b-493a-879f-f595c35a7490\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-mb59v"
Oct 30 20:02:11.151473 ip-10-0-132-82 kubenswrapper[4892]: I1030 20:02:11.151319    4892 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/72926e14-705b-493a-879f-f595c35a7490-istio-token\") pod \"maas-default-gateway-openshift-default-6db586dd6c-mb59v\" (UID: \"72926e14-705b-493a-879f-f595c35a7490\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-mb59v"
Oct 30 20:02:11.151473 ip-10-0-132-82 kubenswrapper[4892]: I1030 20:02:11.151341    4892 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/741c37d3-758a-4ced-ab0e-9038baea2d88-credential-socket\") pod \"maas-default-gateway-openshift-default-6db586dd6c-p7m7x\" (UID: \"741c37d3-758a-4ced-ab0e-9038baea2d88\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-p7m7x"
Oct 30 20:02:11.151473 ip-10-0-132-82 kubenswrapper[4892]: I1030 20:02:11.151378    4892 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/741c37d3-758a-4ced-ab0e-9038baea2d88-istio-envoy\") pod \"maas-default-gateway-openshift-default-6db586dd6c-p7m7x\" (UID: \"741c37d3-758a-4ced-ab0e-9038baea2d88\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-p7m7x"
Oct 30 20:02:11.151473 ip-10-0-132-82 kubenswrapper[4892]: I1030 20:02:11.151408    4892 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/741c37d3-758a-4ced-ab0e-9038baea2d88-istio-podinfo\") pod \"maas-default-gateway-openshift-default-6db586dd6c-p7m7x\" (UID: \"741c37d3-758a-4ced-ab0e-9038baea2d88\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-p7m7x"
Oct 30 20:02:11.151473 ip-10-0-132-82 kubenswrapper[4892]: I1030 20:02:11.151439    4892 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cvv5r\" (UniqueName: \"kubernetes.io/projected/741c37d3-758a-4ced-ab0e-9038baea2d88-kube-api-access-cvv5r\") pod \"maas-default-gateway-openshift-default-6db586dd6c-p7m7x\" (UID: \"741c37d3-758a-4ced-ab0e-9038baea2d88\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-p7m7x"
Oct 30 20:02:11.151758 ip-10-0-132-82 kubenswrapper[4892]: I1030 20:02:11.151473    4892 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/72926e14-705b-493a-879f-f595c35a7490-workload-certs\") pod \"maas-default-gateway-openshift-default-6db586dd6c-mb59v\" (UID: \"72926e14-705b-493a-879f-f595c35a7490\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-mb59v"
Oct 30 20:02:11.151758 ip-10-0-132-82 kubenswrapper[4892]: I1030 20:02:11.151498    4892 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/72926e14-705b-493a-879f-f595c35a7490-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-6db586dd6c-mb59v\" (UID: \"72926e14-705b-493a-879f-f595c35a7490\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-mb59v"
Oct 30 20:02:11.151758 ip-10-0-132-82 kubenswrapper[4892]: I1030 20:02:11.151549    4892 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/741c37d3-758a-4ced-ab0e-9038baea2d88-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-6db586dd6c-p7m7x\" (UID: \"741c37d3-758a-4ced-ab0e-9038baea2d88\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-p7m7x"
Oct 30 20:02:11.151758 ip-10-0-132-82 kubenswrapper[4892]: I1030 20:02:11.151576    4892 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/72926e14-705b-493a-879f-f595c35a7490-credential-socket\") pod \"maas-default-gateway-openshift-default-6db586dd6c-mb59v\" (UID: \"72926e14-705b-493a-879f-f595c35a7490\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-mb59v"
Oct 30 20:02:11.151758 ip-10-0-132-82 kubenswrapper[4892]: I1030 20:02:11.151604    4892 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/741c37d3-758a-4ced-ab0e-9038baea2d88-istio-token\") pod \"maas-default-gateway-openshift-default-6db586dd6c-p7m7x\" (UID: \"741c37d3-758a-4ced-ab0e-9038baea2d88\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-p7m7x"
Oct 30 20:02:11.151758 ip-10-0-132-82 kubenswrapper[4892]: I1030 20:02:11.151628    4892 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/72926e14-705b-493a-879f-f595c35a7490-istio-podinfo\") pod \"maas-default-gateway-openshift-default-6db586dd6c-mb59v\" (UID: \"72926e14-705b-493a-879f-f595c35a7490\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-mb59v"
Oct 30 20:02:11.151758 ip-10-0-132-82 kubenswrapper[4892]: I1030 20:02:11.151654    4892 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/741c37d3-758a-4ced-ab0e-9038baea2d88-workload-socket\") pod \"maas-default-gateway-openshift-default-6db586dd6c-p7m7x\" (UID: \"741c37d3-758a-4ced-ab0e-9038baea2d88\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-p7m7x"
Oct 30 20:02:11.152768 ip-10-0-132-82 kubenswrapper[4892]: I1030 20:02:11.152740    4892 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/497e96db-0aa8-4f9f-be89-4797e1fc6d0a-maas-api-openshift-ca" pod "497e96db-0aa8-4f9f-be89-4797e1fc6d0a" (UID: "497e96db-0aa8-4f9f-be89-4797e1fc6d0a"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 20:02:11.154976 ip-10-0-132-82 kubenswrapper[4892]: I1030 20:02:11.154923    4892 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/497e96db-0aa8-4f9f-be89-4797e1fc6d0a-maas-api-xks-ca" pod "497e96db-0aa8-4f9f-be89-4797e1fc6d0a" (UID: "497e96db-0aa8-4f9f-be89-4797e1fc6d0a"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 20:02:11.156004 ip-10-0-132-82 kubenswrapper[4892]: I1030 20:02:11.155945    4892 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/497e96db-0aa8-4f9f-be89-4797e1fc6d0a-kube-api-access-xs55d" pod "497e96db-0aa8-4f9f-be89-4797e1fc6d0a" (UID: "497e96db-0aa8-4f9f-be89-4797e1fc6d0a"). InnerVolumeSpecName "kube-api-access-xs55d". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 20:02:11.252957 ip-10-0-132-82 kubenswrapper[4892]: I1030 20:02:11.252874    4892 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/741c37d3-758a-4ced-ab0e-9038baea2d88-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-6db586dd6c-p7m7x\" (UID: \"741c37d3-758a-4ced-ab0e-9038baea2d88\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-p7m7x"
Oct 30 20:02:11.252957 ip-10-0-132-82 kubenswrapper[4892]: I1030 20:02:11.252910    4892 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/72926e14-705b-493a-879f-f595c35a7490-credential-socket\") pod \"maas-default-gateway-openshift-default-6db586dd6c-mb59v\" (UID: \"72926e14-705b-493a-879f-f595c35a7490\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-mb59v"
Oct 30 20:02:11.252957 ip-10-0-132-82 kubenswrapper[4892]: I1030 20:02:11.252930    4892 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/741c37d3-758a-4ced-ab0e-9038baea2d88-istio-token\") pod \"maas-default-gateway-openshift-default-6db586dd6c-p7m7x\" (UID: \"741c37d3-758a-4ced-ab0e-9038baea2d88\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-p7m7x"
Oct 30 20:02:11.252957 ip-10-0-132-82 kubenswrapper[4892]: I1030 20:02:11.252946    4892 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/72926e14-705b-493a-879f-f595c35a7490-istio-podinfo\") pod \"maas-default-gateway-openshift-default-6db586dd6c-mb59v\" (UID: \"72926e14-705b-493a-879f-f595c35a7490\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-mb59v"
Oct 30 20:02:11.253316 ip-10-0-132-82 kubenswrapper[4892]: I1030 20:02:11.253001    4892 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/741c37d3-758a-4ced-ab0e-9038baea2d88-workload-socket\") pod \"maas-default-gateway-openshift-default-6db586dd6c-p7m7x\" (UID: \"741c37d3-758a-4ced-ab0e-9038baea2d88\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-p7m7x"
Oct 30 20:02:11.253316 ip-10-0-132-82 kubenswrapper[4892]: I1030 20:02:11.253295    4892 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/72926e14-705b-493a-879f-f595c35a7490-istio-envoy\") pod \"maas-default-gateway-openshift-default-6db586dd6c-mb59v\" (UID: \"72926e14-705b-493a-879f-f595c35a7490\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-mb59v"
Oct 30 20:02:11.253450 ip-10-0-132-82 kubenswrapper[4892]: I1030 20:02:11.253349    4892 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/72926e14-705b-493a-879f-f595c35a7490-credential-socket\") pod \"maas-default-gateway-openshift-default-6db586dd6c-mb59v\" (UID: \"72926e14-705b-493a-879f-f595c35a7490\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-mb59v"
Oct 30 20:02:11.253742 ip-10-0-132-82 kubenswrapper[4892]: I1030 20:02:11.253704    4892 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/741c37d3-758a-4ced-ab0e-9038baea2d88-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-6db586dd6c-p7m7x\" (UID: \"741c37d3-758a-4ced-ab0e-9038baea2d88\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-p7m7x"
Oct 30 20:02:11.253890 ip-10-0-132-82 kubenswrapper[4892]: I1030 20:02:11.253706    4892 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/72926e14-705b-493a-879f-f595c35a7490-istio-data\") pod \"maas-default-gateway-openshift-default-6db586dd6c-mb59v\" (UID: \"72926e14-705b-493a-879f-f595c35a7490\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-mb59v"
Oct 30 20:02:11.253890 ip-10-0-132-82 kubenswrapper[4892]: I1030 20:02:11.253778    4892 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/72926e14-705b-493a-879f-f595c35a7490-workload-socket\") pod \"maas-default-gateway-openshift-default-6db586dd6c-mb59v\" (UID: \"72926e14-705b-493a-879f-f595c35a7490\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-mb59v"
Oct 30 20:02:11.253890 ip-10-0-132-82 kubenswrapper[4892]: I1030 20:02:11.253802    4892 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/741c37d3-758a-4ced-ab0e-9038baea2d88-workload-certs\") pod \"maas-default-gateway-openshift-default-6db586dd6c-p7m7x\" (UID: \"741c37d3-758a-4ced-ab0e-9038baea2d88\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-p7m7x"
Oct 30 20:02:11.253890 ip-10-0-132-82 kubenswrapper[4892]: I1030 20:02:11.253878    4892 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/741c37d3-758a-4ced-ab0e-9038baea2d88-istio-data\") pod \"maas-default-gateway-openshift-default-6db586dd6c-p7m7x\" (UID: \"741c37d3-758a-4ced-ab0e-9038baea2d88\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-p7m7x"
Oct 30 20:02:11.254171 ip-10-0-132-82 kubenswrapper[4892]: I1030 20:02:11.253954    4892 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/72926e14-705b-493a-879f-f595c35a7490-istio-data\") pod \"maas-default-gateway-openshift-default-6db586dd6c-mb59v\" (UID: \"72926e14-705b-493a-879f-f595c35a7490\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-mb59v"
Oct 30 20:02:11.254171 ip-10-0-132-82 kubenswrapper[4892]: I1030 20:02:11.254056    4892 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hgml7\" (UniqueName: \"kubernetes.io/projected/72926e14-705b-493a-879f-f595c35a7490-kube-api-access-hgml7\") pod \"maas-default-gateway-openshift-default-6db586dd6c-mb59v\" (UID: \"72926e14-705b-493a-879f-f595c35a7490\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-mb59v"
Oct 30 20:02:11.254171 ip-10-0-132-82 kubenswrapper[4892]: I1030 20:02:11.254090    4892 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/72926e14-705b-493a-879f-f595c35a7490-istio-token\") pod \"maas-default-gateway-openshift-default-6db586dd6c-mb59v\" (UID: \"72926e14-705b-493a-879f-f595c35a7490\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-mb59v"
Oct 30 20:02:11.254171 ip-10-0-132-82 kubenswrapper[4892]: I1030 20:02:11.254132    4892 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/741c37d3-758a-4ced-ab0e-9038baea2d88-credential-socket\") pod \"maas-default-gateway-openshift-default-6db586dd6c-p7m7x\" (UID: \"741c37d3-758a-4ced-ab0e-9038baea2d88\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-p7m7x"
Oct 30 20:02:11.254389 ip-10-0-132-82 kubenswrapper[4892]: I1030 20:02:11.254172    4892 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/741c37d3-758a-4ced-ab0e-9038baea2d88-istio-envoy\") pod \"maas-default-gateway-openshift-default-6db586dd6c-p7m7x\" (UID: \"741c37d3-758a-4ced-ab0e-9038baea2d88\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-p7m7x"
Oct 30 20:02:11.254389 ip-10-0-132-82 kubenswrapper[4892]: I1030 20:02:11.254217    4892 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/741c37d3-758a-4ced-ab0e-9038baea2d88-istio-podinfo\") pod \"maas-default-gateway-openshift-default-6db586dd6c-p7m7x\" (UID: \"741c37d3-758a-4ced-ab0e-9038baea2d88\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-p7m7x"
Oct 30 20:02:11.254389 ip-10-0-132-82 kubenswrapper[4892]: I1030 20:02:11.254240    4892 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/72926e14-705b-493a-879f-f595c35a7490-workload-socket\") pod \"maas-default-gateway-openshift-default-6db586dd6c-mb59v\" (UID: \"72926e14-705b-493a-879f-f595c35a7490\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-mb59v"
Oct 30 20:02:11.254389 ip-10-0-132-82 kubenswrapper[4892]: I1030 20:02:11.254251    4892 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cvv5r\" (UniqueName: \"kubernetes.io/projected/741c37d3-758a-4ced-ab0e-9038baea2d88-kube-api-access-cvv5r\") pod \"maas-default-gateway-openshift-default-6db586dd6c-p7m7x\" (UID: \"741c37d3-758a-4ced-ab0e-9038baea2d88\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-p7m7x"
Oct 30 20:02:11.254389 ip-10-0-132-82 kubenswrapper[4892]: I1030 20:02:11.254327    4892 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/72926e14-705b-493a-879f-f595c35a7490-workload-certs\") pod \"maas-default-gateway-openshift-default-6db586dd6c-mb59v\" (UID: \"72926e14-705b-493a-879f-f595c35a7490\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-mb59v"
Oct 30 20:02:11.254389 ip-10-0-132-82 kubenswrapper[4892]: I1030 20:02:11.254357    4892 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/72926e14-705b-493a-879f-f595c35a7490-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-6db586dd6c-mb59v\" (UID: \"72926e14-705b-493a-879f-f595c35a7490\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-mb59v"
Oct 30 20:02:11.254730 ip-10-0-132-82 kubenswrapper[4892]: I1030 20:02:11.254437    4892 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xs55d\" (UniqueName: \"kubernetes.io/projected/497e96db-0aa8-4f9f-be89-4797e1fc6d0a-kube-api-access-xs55d\") on node \"ip-10-0-132-82.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:02:11.254730 ip-10-0-132-82 kubenswrapper[4892]: I1030 20:02:11.254455    4892 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/497e96db-0aa8-4f9f-be89-4797e1fc6d0a-maas-api-xks-ca\") on node \"ip-10-0-132-82.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:02:11.254730 ip-10-0-132-82 kubenswrapper[4892]: I1030 20:02:11.254471    4892 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/497e96db-0aa8-4f9f-be89-4797e1fc6d0a-maas-api-openshift-ca\") on node \"ip-10-0-132-82.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:02:11.255006 ip-10-0-132-82 kubenswrapper[4892]: I1030 20:02:11.254945    4892 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/72926e14-705b-493a-879f-f595c35a7490-workload-certs\") pod \"maas-default-gateway-openshift-default-6db586dd6c-mb59v\" (UID: \"72926e14-705b-493a-879f-f595c35a7490\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-mb59v"
Oct 30 20:02:11.255137 ip-10-0-132-82 kubenswrapper[4892]: I1030 20:02:11.255118    4892 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/741c37d3-758a-4ced-ab0e-9038baea2d88-credential-socket\") pod \"maas-default-gateway-openshift-default-6db586dd6c-p7m7x\" (UID: \"741c37d3-758a-4ced-ab0e-9038baea2d88\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-p7m7x"
Oct 30 20:02:11.255137 ip-10-0-132-82 kubenswrapper[4892]: I1030 20:02:11.255127    4892 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/72926e14-705b-493a-879f-f595c35a7490-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-6db586dd6c-mb59v\" (UID: \"72926e14-705b-493a-879f-f595c35a7490\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-mb59v"
Oct 30 20:02:11.255274 ip-10-0-132-82 kubenswrapper[4892]: I1030 20:02:11.255220    4892 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/741c37d3-758a-4ced-ab0e-9038baea2d88-istio-data\") pod \"maas-default-gateway-openshift-default-6db586dd6c-p7m7x\" (UID: \"741c37d3-758a-4ced-ab0e-9038baea2d88\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-p7m7x"
Oct 30 20:02:11.255622 ip-10-0-132-82 kubenswrapper[4892]: I1030 20:02:11.255597    4892 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/741c37d3-758a-4ced-ab0e-9038baea2d88-workload-certs\") pod \"maas-default-gateway-openshift-default-6db586dd6c-p7m7x\" (UID: \"741c37d3-758a-4ced-ab0e-9038baea2d88\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-p7m7x"
Oct 30 20:02:11.255724 ip-10-0-132-82 kubenswrapper[4892]: I1030 20:02:11.255695    4892 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/741c37d3-758a-4ced-ab0e-9038baea2d88-workload-socket\") pod \"maas-default-gateway-openshift-default-6db586dd6c-p7m7x\" (UID: \"741c37d3-758a-4ced-ab0e-9038baea2d88\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-p7m7x"
Oct 30 20:02:11.256314 ip-10-0-132-82 kubenswrapper[4892]: I1030 20:02:11.256284    4892 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/72926e14-705b-493a-879f-f595c35a7490-istio-podinfo\") pod \"maas-default-gateway-openshift-default-6db586dd6c-mb59v\" (UID: \"72926e14-705b-493a-879f-f595c35a7490\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-mb59v"
Oct 30 20:02:11.256512 ip-10-0-132-82 kubenswrapper[4892]: I1030 20:02:11.256494    4892 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/72926e14-705b-493a-879f-f595c35a7490-istio-envoy\") pod \"maas-default-gateway-openshift-default-6db586dd6c-mb59v\" (UID: \"72926e14-705b-493a-879f-f595c35a7490\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-mb59v"
Oct 30 20:02:11.257463 ip-10-0-132-82 kubenswrapper[4892]: I1030 20:02:11.257438    4892 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/741c37d3-758a-4ced-ab0e-9038baea2d88-istio-podinfo\") pod \"maas-default-gateway-openshift-default-6db586dd6c-p7m7x\" (UID: \"741c37d3-758a-4ced-ab0e-9038baea2d88\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-p7m7x"
Oct 30 20:02:11.257558 ip-10-0-132-82 kubenswrapper[4892]: I1030 20:02:11.257532    4892 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/741c37d3-758a-4ced-ab0e-9038baea2d88-istio-envoy\") pod \"maas-default-gateway-openshift-default-6db586dd6c-p7m7x\" (UID: \"741c37d3-758a-4ced-ab0e-9038baea2d88\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-p7m7x"
Oct 30 20:02:11.311449 ip-10-0-132-82 kubenswrapper[4892]: I1030 20:02:11.311416    4892 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/72926e14-705b-493a-879f-f595c35a7490-istio-token\") pod \"maas-default-gateway-openshift-default-6db586dd6c-mb59v\" (UID: \"72926e14-705b-493a-879f-f595c35a7490\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-mb59v"
Oct 30 20:02:11.311617 ip-10-0-132-82 kubenswrapper[4892]: I1030 20:02:11.311502    4892 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/741c37d3-758a-4ced-ab0e-9038baea2d88-istio-token\") pod \"maas-default-gateway-openshift-default-6db586dd6c-p7m7x\" (UID: \"741c37d3-758a-4ced-ab0e-9038baea2d88\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-p7m7x"
Oct 30 20:02:11.312006 ip-10-0-132-82 kubenswrapper[4892]: I1030 20:02:11.311952    4892 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-cvv5r\" (UniqueName: \"kubernetes.io/projected/741c37d3-758a-4ced-ab0e-9038baea2d88-kube-api-access-cvv5r\") pod \"maas-default-gateway-openshift-default-6db586dd6c-p7m7x\" (UID: \"741c37d3-758a-4ced-ab0e-9038baea2d88\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-p7m7x"
Oct 30 20:02:11.312129 ip-10-0-132-82 kubenswrapper[4892]: I1030 20:02:11.312107    4892 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hgml7\" (UniqueName: \"kubernetes.io/projected/72926e14-705b-493a-879f-f595c35a7490-kube-api-access-hgml7\") pod \"maas-default-gateway-openshift-default-6db586dd6c-mb59v\" (UID: \"72926e14-705b-493a-879f-f595c35a7490\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-mb59v"
Oct 30 20:02:11.320875 ip-10-0-132-82 kubenswrapper[4892]: I1030 20:02:11.320855    4892 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-p7m7x"
Oct 30 20:02:11.327117 ip-10-0-132-82 kubenswrapper[4892]: I1030 20:02:11.327098    4892 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-mb59v"
Oct 30 20:02:11.767501 ip-10-0-132-82 kubenswrapper[4892]: I1030 20:02:11.767450    4892 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-p7m7x"]
Oct 30 20:02:11.769222 ip-10-0-132-82 kubenswrapper[4892]: W1030 20:02:11.769194    4892 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod741c37d3_758a_4ced_ab0e_9038baea2d88.slice/crio-ad74710bb76347152ab4105a1ef41992b8d1ce7f66b43f49739ce2f674cec83e WatchSource:0}: Error finding container ad74710bb76347152ab4105a1ef41992b8d1ce7f66b43f49739ce2f674cec83e: Status 404 returned error can't find the container with id ad74710bb76347152ab4105a1ef41992b8d1ce7f66b43f49739ce2f674cec83e
Oct 30 20:02:11.771459 ip-10-0-132-82 kubenswrapper[4892]: I1030 20:02:11.771434    4892 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-mb59v"]
Oct 30 20:02:11.772345 ip-10-0-132-82 kubenswrapper[4892]: I1030 20:02:11.772305    4892 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892000Ki","pods":"250"}
Oct 30 20:02:11.772456 ip-10-0-132-82 kubenswrapper[4892]: I1030 20:02:11.772417    4892 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892000Ki","pods":"250"}
Oct 30 20:02:11.772859 ip-10-0-132-82 kubenswrapper[4892]: I1030 20:02:11.772477    4892 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892000Ki","pods":"250"}
Oct 30 20:02:11.776609 ip-10-0-132-82 kubenswrapper[4892]: W1030 20:02:11.776581    4892 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod72926e14_705b_493a_879f_f595c35a7490.slice/crio-b197489a8cf534162a60956e538a6087a8c508c0ebde88c49cb86f69daab9e2a WatchSource:0}: Error finding container b197489a8cf534162a60956e538a6087a8c508c0ebde88c49cb86f69daab9e2a: Status 404 returned error can't find the container with id b197489a8cf534162a60956e538a6087a8c508c0ebde88c49cb86f69daab9e2a
Oct 30 20:02:11.782274 ip-10-0-132-82 kubenswrapper[4892]: I1030 20:02:11.782237    4892 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892000Ki","pods":"250"}
Oct 30 20:02:11.782381 ip-10-0-132-82 kubenswrapper[4892]: I1030 20:02:11.782316    4892 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892000Ki","pods":"250"}
Oct 30 20:02:11.782381 ip-10-0-132-82 kubenswrapper[4892]: I1030 20:02:11.782356    4892 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892000Ki","pods":"250"}
Oct 30 20:02:11.846936 ip-10-0-132-82 kubenswrapper[4892]: I1030 20:02:11.846860    4892 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-mb59v" event={"ID":"72926e14-705b-493a-879f-f595c35a7490","Type":"ContainerStarted","Data":"b197489a8cf534162a60956e538a6087a8c508c0ebde88c49cb86f69daab9e2a"}
Oct 30 20:02:11.848397 ip-10-0-132-82 kubenswrapper[4892]: I1030 20:02:11.848376    4892 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-a3f9b7-gfgvh"
Oct 30 20:02:11.848517 ip-10-0-132-82 kubenswrapper[4892]: I1030 20:02:11.848405    4892 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-a3f9b7-gfgvh" event={"ID":"497e96db-0aa8-4f9f-be89-4797e1fc6d0a","Type":"ContainerDied","Data":"e55374b1a2f2e2942c0dd0ca2cc0cf268edf77eae5146067a5e8930b17d923b2"}
Oct 30 20:02:11.848517 ip-10-0-132-82 kubenswrapper[4892]: I1030 20:02:11.848449    4892 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="e55374b1a2f2e2942c0dd0ca2cc0cf268edf77eae5146067a5e8930b17d923b2"
Oct 30 20:02:11.850186 ip-10-0-132-82 kubenswrapper[4892]: I1030 20:02:11.850158    4892 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-p7m7x" event={"ID":"741c37d3-758a-4ced-ab0e-9038baea2d88","Type":"ContainerStarted","Data":"ad74710bb76347152ab4105a1ef41992b8d1ce7f66b43f49739ce2f674cec83e"}
Oct 30 20:02:12.038227 ip-10-0-132-82 kubenswrapper[4892]: I1030 20:02:12.038195    4892 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-ada8d7-ss6z5"
Oct 30 20:02:12.172506 ip-10-0-132-82 kubenswrapper[4892]: I1030 20:02:12.172460    4892 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/73e40af0-f49a-41bf-9dbd-e52de5027c21-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/73e40af0-f49a-41bf-9dbd-e52de5027c21-maas-api-xks-ca\") pod \"73e40af0-f49a-41bf-9dbd-e52de5027c21\" (UID: \"73e40af0-f49a-41bf-9dbd-e52de5027c21\") "
Oct 30 20:02:12.172683 ip-10-0-132-82 kubenswrapper[4892]: I1030 20:02:12.172549    4892 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/73e40af0-f49a-41bf-9dbd-e52de5027c21-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/73e40af0-f49a-41bf-9dbd-e52de5027c21-maas-api-openshift-ca\") pod \"73e40af0-f49a-41bf-9dbd-e52de5027c21\" (UID: \"73e40af0-f49a-41bf-9dbd-e52de5027c21\") "
Oct 30 20:02:12.172683 ip-10-0-132-82 kubenswrapper[4892]: I1030 20:02:12.172662    4892 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/73e40af0-f49a-41bf-9dbd-e52de5027c21-kube-api-access-9gttt\" (UniqueName: \"kubernetes.io/projected/73e40af0-f49a-41bf-9dbd-e52de5027c21-kube-api-access-9gttt\") pod \"73e40af0-f49a-41bf-9dbd-e52de5027c21\" (UID: \"73e40af0-f49a-41bf-9dbd-e52de5027c21\") "
Oct 30 20:02:12.173804 ip-10-0-132-82 kubenswrapper[4892]: I1030 20:02:12.173778    4892 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/73e40af0-f49a-41bf-9dbd-e52de5027c21-maas-api-openshift-ca" pod "73e40af0-f49a-41bf-9dbd-e52de5027c21" (UID: "73e40af0-f49a-41bf-9dbd-e52de5027c21"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 20:02:12.177461 ip-10-0-132-82 kubenswrapper[4892]: I1030 20:02:12.177390    4892 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/73e40af0-f49a-41bf-9dbd-e52de5027c21-kube-api-access-9gttt" pod "73e40af0-f49a-41bf-9dbd-e52de5027c21" (UID: "73e40af0-f49a-41bf-9dbd-e52de5027c21"). InnerVolumeSpecName "kube-api-access-9gttt". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 20:02:12.181502 ip-10-0-132-82 kubenswrapper[4892]: I1030 20:02:12.181460    4892 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/73e40af0-f49a-41bf-9dbd-e52de5027c21-maas-api-xks-ca" pod "73e40af0-f49a-41bf-9dbd-e52de5027c21" (UID: "73e40af0-f49a-41bf-9dbd-e52de5027c21"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 20:02:12.274123 ip-10-0-132-82 kubenswrapper[4892]: I1030 20:02:12.274043    4892 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/73e40af0-f49a-41bf-9dbd-e52de5027c21-maas-api-xks-ca\") on node \"ip-10-0-132-82.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:02:12.274123 ip-10-0-132-82 kubenswrapper[4892]: I1030 20:02:12.274080    4892 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/73e40af0-f49a-41bf-9dbd-e52de5027c21-maas-api-openshift-ca\") on node \"ip-10-0-132-82.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:02:12.274123 ip-10-0-132-82 kubenswrapper[4892]: I1030 20:02:12.274096    4892 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9gttt\" (UniqueName: \"kubernetes.io/projected/73e40af0-f49a-41bf-9dbd-e52de5027c21-kube-api-access-9gttt\") on node \"ip-10-0-132-82.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:02:12.872044 ip-10-0-132-82 kubenswrapper[4892]: I1030 20:02:12.870612    4892 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-mb59v" event={"ID":"72926e14-705b-493a-879f-f595c35a7490","Type":"ContainerStarted","Data":"daa545f6b5ed58fff46391eee8e6f47d0db43d87a6fee8b2c07925792dfcda7c"}
Oct 30 20:02:12.875996 ip-10-0-132-82 kubenswrapper[4892]: I1030 20:02:12.873376    4892 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-ada8d7-ss6z5" event={"ID":"73e40af0-f49a-41bf-9dbd-e52de5027c21","Type":"ContainerDied","Data":"0e8e5a2da1dbe2772e279205a4554ce26a83667d4151fb0a6784c5a6a997abcf"}
Oct 30 20:02:12.875996 ip-10-0-132-82 kubenswrapper[4892]: I1030 20:02:12.873427    4892 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="0e8e5a2da1dbe2772e279205a4554ce26a83667d4151fb0a6784c5a6a997abcf"
Oct 30 20:02:12.875996 ip-10-0-132-82 kubenswrapper[4892]: I1030 20:02:12.873492    4892 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-ada8d7-ss6z5"
Oct 30 20:02:12.892722 ip-10-0-132-82 kubenswrapper[4892]: I1030 20:02:12.892622    4892 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-p7m7x" event={"ID":"741c37d3-758a-4ced-ab0e-9038baea2d88","Type":"ContainerStarted","Data":"9989808b24a5895221146c0e35de04e3a5077e014a7900af7241e8077e77f2cf"}
Oct 30 20:02:13.043589 ip-10-0-132-82 kubenswrapper[4892]: I1030 20:02:13.043527    4892 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-mb59v" podStartSLOduration=3.043507621 podStartE2EDuration="3.043507621s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 20:02:10 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 20:02:13.041449811 +0000 UTC m=+2780.611915414" watchObservedRunningTime="2026-10-30 20:02:13.043507621 +0000 UTC m=+2780.613973220"
Oct 30 20:02:13.214776 ip-10-0-132-82 kubenswrapper[4892]: I1030 20:02:13.214719    4892 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-p7m7x" podStartSLOduration=3.214703993 podStartE2EDuration="3.214703993s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 20:02:10 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 20:02:13.211714167 +0000 UTC m=+2780.782179767" watchObservedRunningTime="2026-10-30 20:02:13.214703993 +0000 UTC m=+2780.785169644"
Oct 30 20:02:13.322006 ip-10-0-132-82 kubenswrapper[4892]: I1030 20:02:13.321447    4892 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-p7m7x"
Oct 30 20:02:13.322903 ip-10-0-132-82 kubenswrapper[4892]: I1030 20:02:13.322879    4892 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6db586dd6c-p7m7x container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.252:15021/healthz/ready\": dial tcp 10.132.0.252:15021: connect: connection refused" start-of-body=
Oct 30 20:02:13.323026 ip-10-0-132-82 kubenswrapper[4892]: I1030 20:02:13.322929    4892 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-p7m7x" podUID="741c37d3-758a-4ced-ab0e-9038baea2d88" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.252:15021/healthz/ready\": dial tcp 10.132.0.252:15021: connect: connection refused"
Oct 30 20:02:13.332616 ip-10-0-132-82 kubenswrapper[4892]: I1030 20:02:13.332587    4892 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-mb59v"
Oct 30 20:02:13.332893 ip-10-0-132-82 kubenswrapper[4892]: I1030 20:02:13.332871    4892 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6db586dd6c-mb59v container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.253:15021/healthz/ready\": dial tcp 10.132.0.253:15021: connect: connection refused" start-of-body=
Oct 30 20:02:13.333088 ip-10-0-132-82 kubenswrapper[4892]: I1030 20:02:13.333058    4892 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-mb59v" podUID="72926e14-705b-493a-879f-f595c35a7490" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.253:15021/healthz/ready\": dial tcp 10.132.0.253:15021: connect: connection refused"
Oct 30 20:02:13.884533 ip-10-0-132-82 kubenswrapper[4892]: I1030 20:02:13.884051    4892 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w3-ada8d7/simulator-w3-ada8d7-kserve-7d575cfcb5-5txgn"]
Oct 30 20:02:14.321838 ip-10-0-132-82 kubenswrapper[4892]: I1030 20:02:14.321752    4892 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6db586dd6c-p7m7x container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.252:15021/healthz/ready\": dial tcp 10.132.0.252:15021: connect: connection refused" start-of-body=
Oct 30 20:02:14.321838 ip-10-0-132-82 kubenswrapper[4892]: I1030 20:02:14.321810    4892 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-p7m7x" podUID="741c37d3-758a-4ced-ab0e-9038baea2d88" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.252:15021/healthz/ready\": dial tcp 10.132.0.252:15021: connect: connection refused"
Oct 30 20:02:14.335988 ip-10-0-132-82 kubenswrapper[4892]: I1030 20:02:14.331278    4892 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6db586dd6c-mb59v container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.253:15021/healthz/ready\": dial tcp 10.132.0.253:15021: connect: connection refused" start-of-body=
Oct 30 20:02:14.335988 ip-10-0-132-82 kubenswrapper[4892]: I1030 20:02:14.331331    4892 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-mb59v" podUID="72926e14-705b-493a-879f-f595c35a7490" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.253:15021/healthz/ready\": dial tcp 10.132.0.253:15021: connect: connection refused"
Oct 30 20:02:15.321363 ip-10-0-132-82 kubenswrapper[4892]: I1030 20:02:15.321326    4892 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6db586dd6c-p7m7x container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.252:15021/healthz/ready\": dial tcp 10.132.0.252:15021: connect: connection refused" start-of-body=
Oct 30 20:02:15.321760 ip-10-0-132-82 kubenswrapper[4892]: I1030 20:02:15.321394    4892 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-p7m7x" podUID="741c37d3-758a-4ced-ab0e-9038baea2d88" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.252:15021/healthz/ready\": dial tcp 10.132.0.252:15021: connect: connection refused"
Oct 30 20:02:15.331420 ip-10-0-132-82 kubenswrapper[4892]: I1030 20:02:15.331373    4892 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6db586dd6c-mb59v container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.253:15021/healthz/ready\": dial tcp 10.132.0.253:15021: connect: connection refused" start-of-body=
Oct 30 20:02:15.331670 ip-10-0-132-82 kubenswrapper[4892]: I1030 20:02:15.331628    4892 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-mb59v" podUID="72926e14-705b-493a-879f-f595c35a7490" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.253:15021/healthz/ready\": dial tcp 10.132.0.253:15021: connect: connection refused"
Oct 30 20:02:16.321560 ip-10-0-132-82 kubenswrapper[4892]: I1030 20:02:16.321524    4892 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6db586dd6c-p7m7x container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.252:15021/healthz/ready\": dial tcp 10.132.0.252:15021: connect: connection refused" start-of-body=
Oct 30 20:02:16.321951 ip-10-0-132-82 kubenswrapper[4892]: I1030 20:02:16.321584    4892 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-p7m7x" podUID="741c37d3-758a-4ced-ab0e-9038baea2d88" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.252:15021/healthz/ready\": dial tcp 10.132.0.252:15021: connect: connection refused"
Oct 30 20:02:16.332991 ip-10-0-132-82 kubenswrapper[4892]: I1030 20:02:16.331263    4892 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6db586dd6c-mb59v container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.253:15021/healthz/ready\": dial tcp 10.132.0.253:15021: connect: connection refused" start-of-body=
Oct 30 20:02:16.332991 ip-10-0-132-82 kubenswrapper[4892]: I1030 20:02:16.331309    4892 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-mb59v" podUID="72926e14-705b-493a-879f-f595c35a7490" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.253:15021/healthz/ready\": dial tcp 10.132.0.253:15021: connect: connection refused"
Oct 30 20:02:17.123198 ip-10-0-132-82 kubenswrapper[4892]: I1030 20:02:17.123102    4892 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w2-a3f9b7/simulator-w2-a3f9b7-kserve-79bbc4cb5f-8m488" podUID="50fb2283-bf3f-41da-835b-f407ee1a04f3" containerName="main" containerID="cri-o://222b1703c0f545b6c40b1052718766033672f7cc4aacaa0061ab5fafce867dc9" gracePeriod=45
Oct 30 20:02:17.212775 ip-10-0-132-82 kubenswrapper[4892]: E1030 20:02:17.212741    4892 cadvisor_stats_provider.go:569] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod50fb2283_bf3f_41da_835b_f407ee1a04f3.slice/crio-222b1703c0f545b6c40b1052718766033672f7cc4aacaa0061ab5fafce867dc9.scope\": RecentStats: unable to find data in memory cache]"
Oct 30 20:02:17.321483 ip-10-0-132-82 kubenswrapper[4892]: I1030 20:02:17.321449    4892 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6db586dd6c-p7m7x container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.252:15021/healthz/ready\": dial tcp 10.132.0.252:15021: connect: connection refused" start-of-body=
Oct 30 20:02:17.321665 ip-10-0-132-82 kubenswrapper[4892]: I1030 20:02:17.321501    4892 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-p7m7x" podUID="741c37d3-758a-4ced-ab0e-9038baea2d88" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.252:15021/healthz/ready\": dial tcp 10.132.0.252:15021: connect: connection refused"
Oct 30 20:02:17.351989 ip-10-0-132-82 kubenswrapper[4892]: I1030 20:02:17.350079    4892 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6db586dd6c-mb59v container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.253:15021/healthz/ready\": dial tcp 10.132.0.253:15021: connect: connection refused" start-of-body=
Oct 30 20:02:17.351989 ip-10-0-132-82 kubenswrapper[4892]: I1030 20:02:17.350138    4892 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-mb59v" podUID="72926e14-705b-493a-879f-f595c35a7490" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.253:15021/healthz/ready\": dial tcp 10.132.0.253:15021: connect: connection refused"
Oct 30 20:02:17.596770 ip-10-0-132-82 kubenswrapper[4892]: I1030 20:02:17.596459    4892 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w2-a3f9b7/simulator-w2-a3f9b7-kserve-79bbc4cb5f-8m488"
Oct 30 20:02:17.689386 ip-10-0-132-82 kubenswrapper[4892]: I1030 20:02:17.689310    4892 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/50fb2283-bf3f-41da-835b-f407ee1a04f3-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/50fb2283-bf3f-41da-835b-f407ee1a04f3-model-cache\") pod \"50fb2283-bf3f-41da-835b-f407ee1a04f3\" (UID: \"50fb2283-bf3f-41da-835b-f407ee1a04f3\") "
Oct 30 20:02:17.689386 ip-10-0-132-82 kubenswrapper[4892]: I1030 20:02:17.689354    4892 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/50fb2283-bf3f-41da-835b-f407ee1a04f3-tls-certs\" (UniqueName: \"kubernetes.io/secret/50fb2283-bf3f-41da-835b-f407ee1a04f3-tls-certs\") pod \"50fb2283-bf3f-41da-835b-f407ee1a04f3\" (UID: \"50fb2283-bf3f-41da-835b-f407ee1a04f3\") "
Oct 30 20:02:17.689577 ip-10-0-132-82 kubenswrapper[4892]: I1030 20:02:17.689393    4892 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/50fb2283-bf3f-41da-835b-f407ee1a04f3-home\" (UniqueName: \"kubernetes.io/empty-dir/50fb2283-bf3f-41da-835b-f407ee1a04f3-home\") pod \"50fb2283-bf3f-41da-835b-f407ee1a04f3\" (UID: \"50fb2283-bf3f-41da-835b-f407ee1a04f3\") "
Oct 30 20:02:17.689577 ip-10-0-132-82 kubenswrapper[4892]: I1030 20:02:17.689454    4892 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/50fb2283-bf3f-41da-835b-f407ee1a04f3-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/50fb2283-bf3f-41da-835b-f407ee1a04f3-tmp-dir\") pod \"50fb2283-bf3f-41da-835b-f407ee1a04f3\" (UID: \"50fb2283-bf3f-41da-835b-f407ee1a04f3\") "
Oct 30 20:02:17.689577 ip-10-0-132-82 kubenswrapper[4892]: I1030 20:02:17.689476    4892 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/50fb2283-bf3f-41da-835b-f407ee1a04f3-dshm\" (UniqueName: \"kubernetes.io/empty-dir/50fb2283-bf3f-41da-835b-f407ee1a04f3-dshm\") pod \"50fb2283-bf3f-41da-835b-f407ee1a04f3\" (UID: \"50fb2283-bf3f-41da-835b-f407ee1a04f3\") "
Oct 30 20:02:17.689577 ip-10-0-132-82 kubenswrapper[4892]: I1030 20:02:17.689526    4892 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/50fb2283-bf3f-41da-835b-f407ee1a04f3-kube-api-access-qx6vd\" (UniqueName: \"kubernetes.io/projected/50fb2283-bf3f-41da-835b-f407ee1a04f3-kube-api-access-qx6vd\") pod \"50fb2283-bf3f-41da-835b-f407ee1a04f3\" (UID: \"50fb2283-bf3f-41da-835b-f407ee1a04f3\") "
Oct 30 20:02:17.689577 ip-10-0-132-82 kubenswrapper[4892]: I1030 20:02:17.689548    4892 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/50fb2283-bf3f-41da-835b-f407ee1a04f3-model-cache" pod "50fb2283-bf3f-41da-835b-f407ee1a04f3" (UID: "50fb2283-bf3f-41da-835b-f407ee1a04f3"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 20:02:17.689743 ip-10-0-132-82 kubenswrapper[4892]: I1030 20:02:17.689678    4892 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/50fb2283-bf3f-41da-835b-f407ee1a04f3-home" pod "50fb2283-bf3f-41da-835b-f407ee1a04f3" (UID: "50fb2283-bf3f-41da-835b-f407ee1a04f3"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 20:02:17.689804 ip-10-0-132-82 kubenswrapper[4892]: I1030 20:02:17.689791    4892 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/50fb2283-bf3f-41da-835b-f407ee1a04f3-model-cache\") on node \"ip-10-0-132-82.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:02:17.689841 ip-10-0-132-82 kubenswrapper[4892]: I1030 20:02:17.689808    4892 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/50fb2283-bf3f-41da-835b-f407ee1a04f3-home\") on node \"ip-10-0-132-82.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:02:17.690125 ip-10-0-132-82 kubenswrapper[4892]: I1030 20:02:17.690107    4892 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/50fb2283-bf3f-41da-835b-f407ee1a04f3-tmp-dir" pod "50fb2283-bf3f-41da-835b-f407ee1a04f3" (UID: "50fb2283-bf3f-41da-835b-f407ee1a04f3"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 20:02:17.692336 ip-10-0-132-82 kubenswrapper[4892]: I1030 20:02:17.692311    4892 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/50fb2283-bf3f-41da-835b-f407ee1a04f3-dshm" pod "50fb2283-bf3f-41da-835b-f407ee1a04f3" (UID: "50fb2283-bf3f-41da-835b-f407ee1a04f3"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 20:02:17.692460 ip-10-0-132-82 kubenswrapper[4892]: I1030 20:02:17.692369    4892 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/50fb2283-bf3f-41da-835b-f407ee1a04f3-kube-api-access-qx6vd" pod "50fb2283-bf3f-41da-835b-f407ee1a04f3" (UID: "50fb2283-bf3f-41da-835b-f407ee1a04f3"). InnerVolumeSpecName "kube-api-access-qx6vd". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 20:02:17.692936 ip-10-0-132-82 kubenswrapper[4892]: I1030 20:02:17.692911    4892 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/50fb2283-bf3f-41da-835b-f407ee1a04f3-tls-certs" pod "50fb2283-bf3f-41da-835b-f407ee1a04f3" (UID: "50fb2283-bf3f-41da-835b-f407ee1a04f3"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 20:02:17.791137 ip-10-0-132-82 kubenswrapper[4892]: I1030 20:02:17.791034    4892 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/50fb2283-bf3f-41da-835b-f407ee1a04f3-tmp-dir\") on node \"ip-10-0-132-82.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:02:17.791137 ip-10-0-132-82 kubenswrapper[4892]: I1030 20:02:17.791073    4892 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/50fb2283-bf3f-41da-835b-f407ee1a04f3-dshm\") on node \"ip-10-0-132-82.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:02:17.791137 ip-10-0-132-82 kubenswrapper[4892]: I1030 20:02:17.791089    4892 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qx6vd\" (UniqueName: \"kubernetes.io/projected/50fb2283-bf3f-41da-835b-f407ee1a04f3-kube-api-access-qx6vd\") on node \"ip-10-0-132-82.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:02:17.791137 ip-10-0-132-82 kubenswrapper[4892]: I1030 20:02:17.791107    4892 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/50fb2283-bf3f-41da-835b-f407ee1a04f3-tls-certs\") on node \"ip-10-0-132-82.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:02:17.960345 ip-10-0-132-82 kubenswrapper[4892]: I1030 20:02:17.960243    4892 generic.go:356] "Generic (PLEG): container finished" podID="50fb2283-bf3f-41da-835b-f407ee1a04f3" containerID="222b1703c0f545b6c40b1052718766033672f7cc4aacaa0061ab5fafce867dc9" exitCode=0
Oct 30 20:02:17.960345 ip-10-0-132-82 kubenswrapper[4892]: I1030 20:02:17.960290    4892 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w2-a3f9b7/simulator-w2-a3f9b7-kserve-79bbc4cb5f-8m488"
Oct 30 20:02:17.960587 ip-10-0-132-82 kubenswrapper[4892]: I1030 20:02:17.960364    4892 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w2-a3f9b7/simulator-w2-a3f9b7-kserve-79bbc4cb5f-8m488" event={"ID":"50fb2283-bf3f-41da-835b-f407ee1a04f3","Type":"ContainerDied","Data":"222b1703c0f545b6c40b1052718766033672f7cc4aacaa0061ab5fafce867dc9"}
Oct 30 20:02:17.961089 ip-10-0-132-82 kubenswrapper[4892]: I1030 20:02:17.961018    4892 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w2-a3f9b7/simulator-w2-a3f9b7-kserve-79bbc4cb5f-8m488" event={"ID":"50fb2283-bf3f-41da-835b-f407ee1a04f3","Type":"ContainerDied","Data":"feb4c41f668c3ab5520bd5aad6c2b701f236b226b31ee5b48d3c5f77b93c3eb5"}
Oct 30 20:02:17.961344 ip-10-0-132-82 kubenswrapper[4892]: I1030 20:02:17.961048    4892 scope.go:122] "RemoveContainer" containerID="222b1703c0f545b6c40b1052718766033672f7cc4aacaa0061ab5fafce867dc9"
Oct 30 20:02:17.978187 ip-10-0-132-82 kubenswrapper[4892]: I1030 20:02:17.978161    4892 scope.go:122] "RemoveContainer" containerID="222b1703c0f545b6c40b1052718766033672f7cc4aacaa0061ab5fafce867dc9"
Oct 30 20:02:17.978586 ip-10-0-132-82 kubenswrapper[4892]: E1030 20:02:17.978556    4892 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"222b1703c0f545b6c40b1052718766033672f7cc4aacaa0061ab5fafce867dc9\": container with ID starting with 222b1703c0f545b6c40b1052718766033672f7cc4aacaa0061ab5fafce867dc9 not found: ID does not exist" containerID="222b1703c0f545b6c40b1052718766033672f7cc4aacaa0061ab5fafce867dc9"
Oct 30 20:02:17.978707 ip-10-0-132-82 kubenswrapper[4892]: I1030 20:02:17.978596    4892 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"222b1703c0f545b6c40b1052718766033672f7cc4aacaa0061ab5fafce867dc9"} err="failed to get container status \"222b1703c0f545b6c40b1052718766033672f7cc4aacaa0061ab5fafce867dc9\": rpc error: code = NotFound desc = could not find container \"222b1703c0f545b6c40b1052718766033672f7cc4aacaa0061ab5fafce867dc9\": container with ID starting with 222b1703c0f545b6c40b1052718766033672f7cc4aacaa0061ab5fafce867dc9 not found: ID does not exist"
Oct 30 20:02:18.125958 ip-10-0-132-82 kubenswrapper[4892]: I1030 20:02:18.125921    4892 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w2-a3f9b7/simulator-w2-a3f9b7-kserve-79bbc4cb5f-8m488"]
Oct 30 20:02:18.174981 ip-10-0-132-82 kubenswrapper[4892]: I1030 20:02:18.174937    4892 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w2-a3f9b7/simulator-w2-a3f9b7-kserve-79bbc4cb5f-8m488"]
Oct 30 20:02:18.322220 ip-10-0-132-82 kubenswrapper[4892]: I1030 20:02:18.322141    4892 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6db586dd6c-p7m7x container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.252:15021/healthz/ready\": dial tcp 10.132.0.252:15021: connect: connection refused" start-of-body=
Oct 30 20:02:18.322220 ip-10-0-132-82 kubenswrapper[4892]: I1030 20:02:18.322202    4892 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-p7m7x" podUID="741c37d3-758a-4ced-ab0e-9038baea2d88" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.252:15021/healthz/ready\": dial tcp 10.132.0.252:15021: connect: connection refused"
Oct 30 20:02:18.327399 ip-10-0-132-82 kubenswrapper[4892]: I1030 20:02:18.327372    4892 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6db586dd6c-mb59v container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.253:15021/healthz/ready\": dial tcp 10.132.0.253:15021: connect: connection refused" start-of-body=
Oct 30 20:02:18.327504 ip-10-0-132-82 kubenswrapper[4892]: I1030 20:02:18.327423    4892 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-mb59v" podUID="72926e14-705b-493a-879f-f595c35a7490" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.253:15021/healthz/ready\": dial tcp 10.132.0.253:15021: connect: connection refused"
Oct 30 20:02:18.569050 ip-10-0-132-82 kubenswrapper[4892]: I1030 20:02:18.569010    4892 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="50fb2283-bf3f-41da-835b-f407ee1a04f3" path="/var/lib/kubelet/pods/50fb2283-bf3f-41da-835b-f407ee1a04f3/volumes"
Oct 30 20:02:19.321832 ip-10-0-132-82 kubenswrapper[4892]: I1030 20:02:19.321789    4892 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6db586dd6c-p7m7x container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.252:15021/healthz/ready\": dial tcp 10.132.0.252:15021: connect: connection refused" start-of-body=
Oct 30 20:02:19.322121 ip-10-0-132-82 kubenswrapper[4892]: I1030 20:02:19.321886    4892 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-p7m7x" podUID="741c37d3-758a-4ced-ab0e-9038baea2d88" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.252:15021/healthz/ready\": dial tcp 10.132.0.252:15021: connect: connection refused"
Oct 30 20:02:19.328216 ip-10-0-132-82 kubenswrapper[4892]: I1030 20:02:19.328188    4892 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6db586dd6c-mb59v container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.253:15021/healthz/ready\": dial tcp 10.132.0.253:15021: connect: connection refused" start-of-body=
Oct 30 20:02:19.328533 ip-10-0-132-82 kubenswrapper[4892]: I1030 20:02:19.328245    4892 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-mb59v" podUID="72926e14-705b-493a-879f-f595c35a7490" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.253:15021/healthz/ready\": dial tcp 10.132.0.253:15021: connect: connection refused"
Oct 30 20:02:20.496095 ip-10-0-132-82 kubenswrapper[4892]: I1030 20:02:20.496056    4892 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-fcb89cf2-pfwt8"]
Oct 30 20:02:20.497115 ip-10-0-132-82 kubenswrapper[4892]: I1030 20:02:20.496619    4892 cpu_manager.go:414] "containerMap: removing container" podUID="50fb2283-bf3f-41da-835b-f407ee1a04f3" containerName="main"
Oct 30 20:02:20.497115 ip-10-0-132-82 kubenswrapper[4892]: I1030 20:02:20.496639    4892 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="50fb2283-bf3f-41da-835b-f407ee1a04f3" containerName="main"
Oct 30 20:02:20.497115 ip-10-0-132-82 kubenswrapper[4892]: I1030 20:02:20.496670    4892 cpu_manager.go:414] "containerMap: removing container" podUID="73e40af0-f49a-41bf-9dbd-e52de5027c21" containerName="revoke-keys"
Oct 30 20:02:20.497115 ip-10-0-132-82 kubenswrapper[4892]: I1030 20:02:20.496676    4892 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="73e40af0-f49a-41bf-9dbd-e52de5027c21" containerName="revoke-keys"
Oct 30 20:02:20.497115 ip-10-0-132-82 kubenswrapper[4892]: I1030 20:02:20.496755    4892 memory_manager.go:365] "RemoveStaleState removing state" podUID="50fb2283-bf3f-41da-835b-f407ee1a04f3" containerName="main"
Oct 30 20:02:20.497115 ip-10-0-132-82 kubenswrapper[4892]: I1030 20:02:20.496767    4892 memory_manager.go:365] "RemoveStaleState removing state" podUID="73e40af0-f49a-41bf-9dbd-e52de5027c21" containerName="revoke-keys"
Oct 30 20:02:20.500493 ip-10-0-132-82 kubenswrapper[4892]: I1030 20:02:20.500469    4892 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-fcb89cf2-pfwt8"
Oct 30 20:02:20.511041 ip-10-0-132-82 kubenswrapper[4892]: I1030 20:02:20.511011    4892 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-p7m7x"
Oct 30 20:02:20.511454 ip-10-0-132-82 kubenswrapper[4892]: I1030 20:02:20.511394    4892 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-p7m7x"
Oct 30 20:02:20.512902 ip-10-0-132-82 kubenswrapper[4892]: I1030 20:02:20.512880    4892 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-p7m7x"
Oct 30 20:02:20.544705 ip-10-0-132-82 kubenswrapper[4892]: I1030 20:02:20.544672    4892 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-7a1cd64a-9lx2c"]
Oct 30 20:02:20.548588 ip-10-0-132-82 kubenswrapper[4892]: I1030 20:02:20.548569    4892 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7a1cd64a-9lx2c"
Oct 30 20:02:20.589484 ip-10-0-132-82 kubenswrapper[4892]: I1030 20:02:20.589455    4892 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 20:02:20.589484 ip-10-0-132-82 kubenswrapper[4892]: I1030 20:02:20.589484    4892 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 30 20:02:20.589717 ip-10-0-132-82 kubenswrapper[4892]: I1030 20:02:20.589525    4892 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-nlxp4\""
Oct 30 20:02:20.589717 ip-10-0-132-82 kubenswrapper[4892]: I1030 20:02:20.589533    4892 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 30 20:02:20.606661 ip-10-0-132-82 kubenswrapper[4892]: I1030 20:02:20.606624    4892 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-fcb89cf2-pfwt8"]
Oct 30 20:02:20.610540 ip-10-0-132-82 kubenswrapper[4892]: I1030 20:02:20.610517    4892 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-7a1cd64a-9lx2c"]
Oct 30 20:02:20.623623 ip-10-0-132-82 kubenswrapper[4892]: I1030 20:02:20.623591    4892 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-mb59v"
Oct 30 20:02:20.624166 ip-10-0-132-82 kubenswrapper[4892]: I1030 20:02:20.624116    4892 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-mb59v"
Oct 30 20:02:20.624954 ip-10-0-132-82 kubenswrapper[4892]: I1030 20:02:20.624937    4892 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-mb59v"
Oct 30 20:02:20.648389 ip-10-0-132-82 kubenswrapper[4892]: I1030 20:02:20.648363    4892 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5dec99a8-bafc-496d-baa2-8974fdcd53eb-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-fcb89cf2-pfwt8\" (UID: \"5dec99a8-bafc-496d-baa2-8974fdcd53eb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fcb89cf2-pfwt8"
Oct 30 20:02:20.648577 ip-10-0-132-82 kubenswrapper[4892]: I1030 20:02:20.648546    4892 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0fb0d83f-df47-49bc-aa9b-16b4eb5708dc-maas-api-xks-ca\") pod \"maas-api-revoke-sub-7a1cd64a-9lx2c\" (UID: \"0fb0d83f-df47-49bc-aa9b-16b4eb5708dc\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7a1cd64a-9lx2c"
Oct 30 20:02:20.648728 ip-10-0-132-82 kubenswrapper[4892]: I1030 20:02:20.648625    4892 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5dec99a8-bafc-496d-baa2-8974fdcd53eb-maas-api-xks-ca\") pod \"maas-api-revoke-sub-fcb89cf2-pfwt8\" (UID: \"5dec99a8-bafc-496d-baa2-8974fdcd53eb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fcb89cf2-pfwt8"
Oct 30 20:02:20.648728 ip-10-0-132-82 kubenswrapper[4892]: I1030 20:02:20.648653    4892 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tvkn8\" (UniqueName: \"kubernetes.io/projected/5dec99a8-bafc-496d-baa2-8974fdcd53eb-kube-api-access-tvkn8\") pod \"maas-api-revoke-sub-fcb89cf2-pfwt8\" (UID: \"5dec99a8-bafc-496d-baa2-8974fdcd53eb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fcb89cf2-pfwt8"
Oct 30 20:02:20.648866 ip-10-0-132-82 kubenswrapper[4892]: I1030 20:02:20.648727    4892 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-d84hg\" (UniqueName: \"kubernetes.io/projected/0fb0d83f-df47-49bc-aa9b-16b4eb5708dc-kube-api-access-d84hg\") pod \"maas-api-revoke-sub-7a1cd64a-9lx2c\" (UID: \"0fb0d83f-df47-49bc-aa9b-16b4eb5708dc\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7a1cd64a-9lx2c"
Oct 30 20:02:20.648995 ip-10-0-132-82 kubenswrapper[4892]: I1030 20:02:20.648948    4892 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0fb0d83f-df47-49bc-aa9b-16b4eb5708dc-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-7a1cd64a-9lx2c\" (UID: \"0fb0d83f-df47-49bc-aa9b-16b4eb5708dc\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7a1cd64a-9lx2c"
Oct 30 20:02:20.727385 ip-10-0-132-82 kubenswrapper[4892]: I1030 20:02:20.727361    4892 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3556bc06-5mltw"]
Oct 30 20:02:20.731098 ip-10-0-132-82 kubenswrapper[4892]: I1030 20:02:20.731083    4892 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3556bc06-5mltw"
Oct 30 20:02:20.750360 ip-10-0-132-82 kubenswrapper[4892]: I1030 20:02:20.750312    4892 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0fb0d83f-df47-49bc-aa9b-16b4eb5708dc-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-7a1cd64a-9lx2c\" (UID: \"0fb0d83f-df47-49bc-aa9b-16b4eb5708dc\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7a1cd64a-9lx2c"
Oct 30 20:02:20.750360 ip-10-0-132-82 kubenswrapper[4892]: I1030 20:02:20.750342    4892 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5dec99a8-bafc-496d-baa2-8974fdcd53eb-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-fcb89cf2-pfwt8\" (UID: \"5dec99a8-bafc-496d-baa2-8974fdcd53eb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fcb89cf2-pfwt8"
Oct 30 20:02:20.750548 ip-10-0-132-82 kubenswrapper[4892]: I1030 20:02:20.750522    4892 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0fb0d83f-df47-49bc-aa9b-16b4eb5708dc-maas-api-xks-ca\") pod \"maas-api-revoke-sub-7a1cd64a-9lx2c\" (UID: \"0fb0d83f-df47-49bc-aa9b-16b4eb5708dc\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7a1cd64a-9lx2c"
Oct 30 20:02:20.750624 ip-10-0-132-82 kubenswrapper[4892]: I1030 20:02:20.750606    4892 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5dec99a8-bafc-496d-baa2-8974fdcd53eb-maas-api-xks-ca\") pod \"maas-api-revoke-sub-fcb89cf2-pfwt8\" (UID: \"5dec99a8-bafc-496d-baa2-8974fdcd53eb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fcb89cf2-pfwt8"
Oct 30 20:02:20.750684 ip-10-0-132-82 kubenswrapper[4892]: I1030 20:02:20.750634    4892 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tvkn8\" (UniqueName: \"kubernetes.io/projected/5dec99a8-bafc-496d-baa2-8974fdcd53eb-kube-api-access-tvkn8\") pod \"maas-api-revoke-sub-fcb89cf2-pfwt8\" (UID: \"5dec99a8-bafc-496d-baa2-8974fdcd53eb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fcb89cf2-pfwt8"
Oct 30 20:02:20.750741 ip-10-0-132-82 kubenswrapper[4892]: I1030 20:02:20.750705    4892 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-d84hg\" (UniqueName: \"kubernetes.io/projected/0fb0d83f-df47-49bc-aa9b-16b4eb5708dc-kube-api-access-d84hg\") pod \"maas-api-revoke-sub-7a1cd64a-9lx2c\" (UID: \"0fb0d83f-df47-49bc-aa9b-16b4eb5708dc\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7a1cd64a-9lx2c"
Oct 30 20:02:20.751032 ip-10-0-132-82 kubenswrapper[4892]: I1030 20:02:20.750994    4892 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0fb0d83f-df47-49bc-aa9b-16b4eb5708dc-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-7a1cd64a-9lx2c\" (UID: \"0fb0d83f-df47-49bc-aa9b-16b4eb5708dc\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7a1cd64a-9lx2c"
Oct 30 20:02:20.751181 ip-10-0-132-82 kubenswrapper[4892]: I1030 20:02:20.751159    4892 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5dec99a8-bafc-496d-baa2-8974fdcd53eb-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-fcb89cf2-pfwt8\" (UID: \"5dec99a8-bafc-496d-baa2-8974fdcd53eb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fcb89cf2-pfwt8"
Oct 30 20:02:20.752937 ip-10-0-132-82 kubenswrapper[4892]: I1030 20:02:20.752912    4892 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0fb0d83f-df47-49bc-aa9b-16b4eb5708dc-maas-api-xks-ca\") pod \"maas-api-revoke-sub-7a1cd64a-9lx2c\" (UID: \"0fb0d83f-df47-49bc-aa9b-16b4eb5708dc\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7a1cd64a-9lx2c"
Oct 30 20:02:20.753048 ip-10-0-132-82 kubenswrapper[4892]: I1030 20:02:20.752953    4892 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5dec99a8-bafc-496d-baa2-8974fdcd53eb-maas-api-xks-ca\") pod \"maas-api-revoke-sub-fcb89cf2-pfwt8\" (UID: \"5dec99a8-bafc-496d-baa2-8974fdcd53eb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fcb89cf2-pfwt8"
Oct 30 20:02:20.797350 ip-10-0-132-82 kubenswrapper[4892]: I1030 20:02:20.797312    4892 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3556bc06-5mltw"]
Oct 30 20:02:20.811479 ip-10-0-132-82 kubenswrapper[4892]: I1030 20:02:20.811444    4892 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-d84hg\" (UniqueName: \"kubernetes.io/projected/0fb0d83f-df47-49bc-aa9b-16b4eb5708dc-kube-api-access-d84hg\") pod \"maas-api-revoke-sub-7a1cd64a-9lx2c\" (UID: \"0fb0d83f-df47-49bc-aa9b-16b4eb5708dc\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7a1cd64a-9lx2c"
Oct 30 20:02:20.818989 ip-10-0-132-82 kubenswrapper[4892]: I1030 20:02:20.818941    4892 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tvkn8\" (UniqueName: \"kubernetes.io/projected/5dec99a8-bafc-496d-baa2-8974fdcd53eb-kube-api-access-tvkn8\") pod \"maas-api-revoke-sub-fcb89cf2-pfwt8\" (UID: \"5dec99a8-bafc-496d-baa2-8974fdcd53eb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fcb89cf2-pfwt8"
Oct 30 20:02:20.852085 ip-10-0-132-82 kubenswrapper[4892]: I1030 20:02:20.852048    4892 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/83cb2031-9ebe-487f-8053-361008386451-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3556bc06-5mltw\" (UID: \"83cb2031-9ebe-487f-8053-361008386451\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3556bc06-5mltw"
Oct 30 20:02:20.852270 ip-10-0-132-82 kubenswrapper[4892]: I1030 20:02:20.852126    4892 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/83cb2031-9ebe-487f-8053-361008386451-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3556bc06-5mltw\" (UID: \"83cb2031-9ebe-487f-8053-361008386451\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3556bc06-5mltw"
Oct 30 20:02:20.852270 ip-10-0-132-82 kubenswrapper[4892]: I1030 20:02:20.852252    4892 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jzdwm\" (UniqueName: \"kubernetes.io/projected/83cb2031-9ebe-487f-8053-361008386451-kube-api-access-jzdwm\") pod \"maas-api-revoke-sub-3556bc06-5mltw\" (UID: \"83cb2031-9ebe-487f-8053-361008386451\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3556bc06-5mltw"
Oct 30 20:02:20.862887 ip-10-0-132-82 kubenswrapper[4892]: I1030 20:02:20.862856    4892 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7a1cd64a-9lx2c"
Oct 30 20:02:20.953075 ip-10-0-132-82 kubenswrapper[4892]: I1030 20:02:20.953032    4892 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jzdwm\" (UniqueName: \"kubernetes.io/projected/83cb2031-9ebe-487f-8053-361008386451-kube-api-access-jzdwm\") pod \"maas-api-revoke-sub-3556bc06-5mltw\" (UID: \"83cb2031-9ebe-487f-8053-361008386451\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3556bc06-5mltw"
Oct 30 20:02:20.956062 ip-10-0-132-82 kubenswrapper[4892]: I1030 20:02:20.953100    4892 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/83cb2031-9ebe-487f-8053-361008386451-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3556bc06-5mltw\" (UID: \"83cb2031-9ebe-487f-8053-361008386451\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3556bc06-5mltw"
Oct 30 20:02:20.956062 ip-10-0-132-82 kubenswrapper[4892]: I1030 20:02:20.953156    4892 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/83cb2031-9ebe-487f-8053-361008386451-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3556bc06-5mltw\" (UID: \"83cb2031-9ebe-487f-8053-361008386451\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3556bc06-5mltw"
Oct 30 20:02:20.956062 ip-10-0-132-82 kubenswrapper[4892]: I1030 20:02:20.953922    4892 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/83cb2031-9ebe-487f-8053-361008386451-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3556bc06-5mltw\" (UID: \"83cb2031-9ebe-487f-8053-361008386451\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3556bc06-5mltw"
Oct 30 20:02:20.957513 ip-10-0-132-82 kubenswrapper[4892]: I1030 20:02:20.957318    4892 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/83cb2031-9ebe-487f-8053-361008386451-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3556bc06-5mltw\" (UID: \"83cb2031-9ebe-487f-8053-361008386451\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3556bc06-5mltw"
Oct 30 20:02:21.012195 ip-10-0-132-82 kubenswrapper[4892]: I1030 20:02:21.012107    4892 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jzdwm\" (UniqueName: \"kubernetes.io/projected/83cb2031-9ebe-487f-8053-361008386451-kube-api-access-jzdwm\") pod \"maas-api-revoke-sub-3556bc06-5mltw\" (UID: \"83cb2031-9ebe-487f-8053-361008386451\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3556bc06-5mltw"
Oct 30 20:02:21.043883 ip-10-0-132-82 kubenswrapper[4892]: I1030 20:02:21.043855    4892 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3556bc06-5mltw"
Oct 30 20:02:21.116563 ip-10-0-132-82 kubenswrapper[4892]: I1030 20:02:21.116536    4892 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-fcb89cf2-pfwt8"
Oct 30 20:02:21.262602 ip-10-0-132-82 kubenswrapper[4892]: I1030 20:02:21.262557    4892 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-7a1cd64a-9lx2c"]
Oct 30 20:02:21.263998 ip-10-0-132-82 kubenswrapper[4892]: W1030 20:02:21.263953    4892 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0fb0d83f_df47_49bc_aa9b_16b4eb5708dc.slice/crio-96f9c33acc36350878df6ad29d46584b20a3bcd3941faef1140106b4ae8f7423 WatchSource:0}: Error finding container 96f9c33acc36350878df6ad29d46584b20a3bcd3941faef1140106b4ae8f7423: Status 404 returned error can't find the container with id 96f9c33acc36350878df6ad29d46584b20a3bcd3941faef1140106b4ae8f7423
Oct 30 20:02:21.419202 ip-10-0-132-82 kubenswrapper[4892]: I1030 20:02:21.419171    4892 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3556bc06-5mltw"]
Oct 30 20:02:21.420840 ip-10-0-132-82 kubenswrapper[4892]: W1030 20:02:21.420816    4892 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod83cb2031_9ebe_487f_8053_361008386451.slice/crio-667c2c36c48f372bba70239e74433be7dc77e8ab7a010c13f88f739f64983e42 WatchSource:0}: Error finding container 667c2c36c48f372bba70239e74433be7dc77e8ab7a010c13f88f739f64983e42: Status 404 returned error can't find the container with id 667c2c36c48f372bba70239e74433be7dc77e8ab7a010c13f88f739f64983e42
Oct 30 20:02:21.497795 ip-10-0-132-82 kubenswrapper[4892]: I1030 20:02:21.497769    4892 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-fcb89cf2-pfwt8"]
Oct 30 20:02:21.498540 ip-10-0-132-82 kubenswrapper[4892]: W1030 20:02:21.498510    4892 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5dec99a8_bafc_496d_baa2_8974fdcd53eb.slice/crio-79d42771e58acf5f967154cc7eee2f360fd88ba62e2f847a3b4fa89bf631489e WatchSource:0}: Error finding container 79d42771e58acf5f967154cc7eee2f360fd88ba62e2f847a3b4fa89bf631489e: Status 404 returned error can't find the container with id 79d42771e58acf5f967154cc7eee2f360fd88ba62e2f847a3b4fa89bf631489e
Oct 30 20:02:22.002787 ip-10-0-132-82 kubenswrapper[4892]: I1030 20:02:22.002750    4892 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7a1cd64a-9lx2c" event={"ID":"0fb0d83f-df47-49bc-aa9b-16b4eb5708dc","Type":"ContainerStarted","Data":"46ffd73a30a924f956acc4f0c93f6bfe18ad7428170697876eb41c7ae2197cca"}
Oct 30 20:02:22.003034 ip-10-0-132-82 kubenswrapper[4892]: I1030 20:02:22.002813    4892 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7a1cd64a-9lx2c" event={"ID":"0fb0d83f-df47-49bc-aa9b-16b4eb5708dc","Type":"ContainerStarted","Data":"96f9c33acc36350878df6ad29d46584b20a3bcd3941faef1140106b4ae8f7423"}
Oct 30 20:02:22.004627 ip-10-0-132-82 kubenswrapper[4892]: I1030 20:02:22.004588    4892 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-fcb89cf2-pfwt8" event={"ID":"5dec99a8-bafc-496d-baa2-8974fdcd53eb","Type":"ContainerStarted","Data":"c31c586e2429efe19c8075f4d13e541005ae6f8e68432d3ebddc471b1c56e4ba"}
Oct 30 20:02:22.004777 ip-10-0-132-82 kubenswrapper[4892]: I1030 20:02:22.004640    4892 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-fcb89cf2-pfwt8" event={"ID":"5dec99a8-bafc-496d-baa2-8974fdcd53eb","Type":"ContainerStarted","Data":"79d42771e58acf5f967154cc7eee2f360fd88ba62e2f847a3b4fa89bf631489e"}
Oct 30 20:02:22.006277 ip-10-0-132-82 kubenswrapper[4892]: I1030 20:02:22.006252    4892 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3556bc06-5mltw" event={"ID":"83cb2031-9ebe-487f-8053-361008386451","Type":"ContainerStarted","Data":"616a2776cc0caf27afe59f3085659095da970931699b815c9ba30f32362ed90f"}
Oct 30 20:02:22.006365 ip-10-0-132-82 kubenswrapper[4892]: I1030 20:02:22.006300    4892 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3556bc06-5mltw" event={"ID":"83cb2031-9ebe-487f-8053-361008386451","Type":"ContainerStarted","Data":"667c2c36c48f372bba70239e74433be7dc77e8ab7a010c13f88f739f64983e42"}
Oct 30 20:02:22.118838 ip-10-0-132-82 kubenswrapper[4892]: I1030 20:02:22.118736    4892 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7a1cd64a-9lx2c" podStartSLOduration=2.118718496 podStartE2EDuration="2.118718496s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 20:02:20 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 20:02:22.115642518 +0000 UTC m=+2789.686108117" watchObservedRunningTime="2026-10-30 20:02:22.118718496 +0000 UTC m=+2789.689184096"
Oct 30 20:02:22.231410 ip-10-0-132-82 kubenswrapper[4892]: I1030 20:02:22.231356    4892 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-fcb89cf2-pfwt8" podStartSLOduration=2.231341351 podStartE2EDuration="2.231341351s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 20:02:20 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 20:02:22.228218394 +0000 UTC m=+2789.798683993" watchObservedRunningTime="2026-10-30 20:02:22.231341351 +0000 UTC m=+2789.801806950"
Oct 30 20:02:22.341851 ip-10-0-132-82 kubenswrapper[4892]: I1030 20:02:22.341798    4892 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3556bc06-5mltw" podStartSLOduration=2.34178307 podStartE2EDuration="2.34178307s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 20:02:20 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 20:02:22.340558531 +0000 UTC m=+2789.911024129" watchObservedRunningTime="2026-10-30 20:02:22.34178307 +0000 UTC m=+2789.912248680"
Oct 30 20:02:23.014494 ip-10-0-132-82 kubenswrapper[4892]: I1030 20:02:23.014456    4892 generic.go:356] "Generic (PLEG): container finished" podID="5dec99a8-bafc-496d-baa2-8974fdcd53eb" containerID="c31c586e2429efe19c8075f4d13e541005ae6f8e68432d3ebddc471b1c56e4ba" exitCode=0
Oct 30 20:02:23.015016 ip-10-0-132-82 kubenswrapper[4892]: I1030 20:02:23.014543    4892 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-fcb89cf2-pfwt8" event={"ID":"5dec99a8-bafc-496d-baa2-8974fdcd53eb","Type":"ContainerDied","Data":"c31c586e2429efe19c8075f4d13e541005ae6f8e68432d3ebddc471b1c56e4ba"}
Oct 30 20:02:23.016499 ip-10-0-132-82 kubenswrapper[4892]: I1030 20:02:23.016471    4892 generic.go:356] "Generic (PLEG): container finished" podID="83cb2031-9ebe-487f-8053-361008386451" containerID="616a2776cc0caf27afe59f3085659095da970931699b815c9ba30f32362ed90f" exitCode=0
Oct 30 20:02:23.016630 ip-10-0-132-82 kubenswrapper[4892]: I1030 20:02:23.016555    4892 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3556bc06-5mltw" event={"ID":"83cb2031-9ebe-487f-8053-361008386451","Type":"ContainerDied","Data":"616a2776cc0caf27afe59f3085659095da970931699b815c9ba30f32362ed90f"}
Oct 30 20:02:23.018390 ip-10-0-132-82 kubenswrapper[4892]: I1030 20:02:23.018367    4892 generic.go:356] "Generic (PLEG): container finished" podID="0fb0d83f-df47-49bc-aa9b-16b4eb5708dc" containerID="46ffd73a30a924f956acc4f0c93f6bfe18ad7428170697876eb41c7ae2197cca" exitCode=0
Oct 30 20:02:23.018488 ip-10-0-132-82 kubenswrapper[4892]: I1030 20:02:23.018436    4892 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7a1cd64a-9lx2c" event={"ID":"0fb0d83f-df47-49bc-aa9b-16b4eb5708dc","Type":"ContainerDied","Data":"46ffd73a30a924f956acc4f0c93f6bfe18ad7428170697876eb41c7ae2197cca"}
Oct 30 20:02:24.219989 ip-10-0-132-82 kubenswrapper[4892]: I1030 20:02:24.219950    4892 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3556bc06-5mltw"
Oct 30 20:02:24.237010 ip-10-0-132-82 kubenswrapper[4892]: I1030 20:02:24.234476    4892 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7a1cd64a-9lx2c"
Oct 30 20:02:24.238130 ip-10-0-132-82 kubenswrapper[4892]: I1030 20:02:24.238110    4892 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-fcb89cf2-pfwt8"
Oct 30 20:02:24.284495 ip-10-0-132-82 kubenswrapper[4892]: I1030 20:02:24.284434    4892 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/83cb2031-9ebe-487f-8053-361008386451-kube-api-access-jzdwm\" (UniqueName: \"kubernetes.io/projected/83cb2031-9ebe-487f-8053-361008386451-kube-api-access-jzdwm\") pod \"83cb2031-9ebe-487f-8053-361008386451\" (UID: \"83cb2031-9ebe-487f-8053-361008386451\") "
Oct 30 20:02:24.284609 ip-10-0-132-82 kubenswrapper[4892]: I1030 20:02:24.284534    4892 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/83cb2031-9ebe-487f-8053-361008386451-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/83cb2031-9ebe-487f-8053-361008386451-maas-api-xks-ca\") pod \"83cb2031-9ebe-487f-8053-361008386451\" (UID: \"83cb2031-9ebe-487f-8053-361008386451\") "
Oct 30 20:02:24.284609 ip-10-0-132-82 kubenswrapper[4892]: I1030 20:02:24.284567    4892 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/83cb2031-9ebe-487f-8053-361008386451-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/83cb2031-9ebe-487f-8053-361008386451-maas-api-openshift-ca\") pod \"83cb2031-9ebe-487f-8053-361008386451\" (UID: \"83cb2031-9ebe-487f-8053-361008386451\") "
Oct 30 20:02:24.285019 ip-10-0-132-82 kubenswrapper[4892]: I1030 20:02:24.284990    4892 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/83cb2031-9ebe-487f-8053-361008386451-maas-api-openshift-ca" pod "83cb2031-9ebe-487f-8053-361008386451" (UID: "83cb2031-9ebe-487f-8053-361008386451"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 20:02:24.286715 ip-10-0-132-82 kubenswrapper[4892]: I1030 20:02:24.286693    4892 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/83cb2031-9ebe-487f-8053-361008386451-kube-api-access-jzdwm" pod "83cb2031-9ebe-487f-8053-361008386451" (UID: "83cb2031-9ebe-487f-8053-361008386451"). InnerVolumeSpecName "kube-api-access-jzdwm". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 20:02:24.286800 ip-10-0-132-82 kubenswrapper[4892]: I1030 20:02:24.286743    4892 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/83cb2031-9ebe-487f-8053-361008386451-maas-api-xks-ca" pod "83cb2031-9ebe-487f-8053-361008386451" (UID: "83cb2031-9ebe-487f-8053-361008386451"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 20:02:24.385521 ip-10-0-132-82 kubenswrapper[4892]: I1030 20:02:24.385493    4892 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/5dec99a8-bafc-496d-baa2-8974fdcd53eb-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5dec99a8-bafc-496d-baa2-8974fdcd53eb-maas-api-xks-ca\") pod \"5dec99a8-bafc-496d-baa2-8974fdcd53eb\" (UID: \"5dec99a8-bafc-496d-baa2-8974fdcd53eb\") "
Oct 30 20:02:24.385694 ip-10-0-132-82 kubenswrapper[4892]: I1030 20:02:24.385527    4892 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/0fb0d83f-df47-49bc-aa9b-16b4eb5708dc-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0fb0d83f-df47-49bc-aa9b-16b4eb5708dc-maas-api-openshift-ca\") pod \"0fb0d83f-df47-49bc-aa9b-16b4eb5708dc\" (UID: \"0fb0d83f-df47-49bc-aa9b-16b4eb5708dc\") "
Oct 30 20:02:24.385694 ip-10-0-132-82 kubenswrapper[4892]: I1030 20:02:24.385576    4892 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0fb0d83f-df47-49bc-aa9b-16b4eb5708dc-kube-api-access-d84hg\" (UniqueName: \"kubernetes.io/projected/0fb0d83f-df47-49bc-aa9b-16b4eb5708dc-kube-api-access-d84hg\") pod \"0fb0d83f-df47-49bc-aa9b-16b4eb5708dc\" (UID: \"0fb0d83f-df47-49bc-aa9b-16b4eb5708dc\") "
Oct 30 20:02:24.385694 ip-10-0-132-82 kubenswrapper[4892]: I1030 20:02:24.385617    4892 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5dec99a8-bafc-496d-baa2-8974fdcd53eb-kube-api-access-tvkn8\" (UniqueName: \"kubernetes.io/projected/5dec99a8-bafc-496d-baa2-8974fdcd53eb-kube-api-access-tvkn8\") pod \"5dec99a8-bafc-496d-baa2-8974fdcd53eb\" (UID: \"5dec99a8-bafc-496d-baa2-8974fdcd53eb\") "
Oct 30 20:02:24.385843 ip-10-0-132-82 kubenswrapper[4892]: I1030 20:02:24.385711    4892 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/0fb0d83f-df47-49bc-aa9b-16b4eb5708dc-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0fb0d83f-df47-49bc-aa9b-16b4eb5708dc-maas-api-xks-ca\") pod \"0fb0d83f-df47-49bc-aa9b-16b4eb5708dc\" (UID: \"0fb0d83f-df47-49bc-aa9b-16b4eb5708dc\") "
Oct 30 20:02:24.385843 ip-10-0-132-82 kubenswrapper[4892]: I1030 20:02:24.385733    4892 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/5dec99a8-bafc-496d-baa2-8974fdcd53eb-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5dec99a8-bafc-496d-baa2-8974fdcd53eb-maas-api-openshift-ca\") pod \"5dec99a8-bafc-496d-baa2-8974fdcd53eb\" (UID: \"5dec99a8-bafc-496d-baa2-8974fdcd53eb\") "
Oct 30 20:02:24.386075 ip-10-0-132-82 kubenswrapper[4892]: I1030 20:02:24.386046    4892 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0fb0d83f-df47-49bc-aa9b-16b4eb5708dc-maas-api-openshift-ca" pod "0fb0d83f-df47-49bc-aa9b-16b4eb5708dc" (UID: "0fb0d83f-df47-49bc-aa9b-16b4eb5708dc"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 20:02:24.386161 ip-10-0-132-82 kubenswrapper[4892]: I1030 20:02:24.386107    4892 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jzdwm\" (UniqueName: \"kubernetes.io/projected/83cb2031-9ebe-487f-8053-361008386451-kube-api-access-jzdwm\") on node \"ip-10-0-132-82.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:02:24.386232 ip-10-0-132-82 kubenswrapper[4892]: I1030 20:02:24.386159    4892 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/83cb2031-9ebe-487f-8053-361008386451-maas-api-xks-ca\") on node \"ip-10-0-132-82.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:02:24.386232 ip-10-0-132-82 kubenswrapper[4892]: I1030 20:02:24.386186    4892 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/83cb2031-9ebe-487f-8053-361008386451-maas-api-openshift-ca\") on node \"ip-10-0-132-82.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:02:24.386325 ip-10-0-132-82 kubenswrapper[4892]: I1030 20:02:24.386240    4892 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5dec99a8-bafc-496d-baa2-8974fdcd53eb-maas-api-openshift-ca" pod "5dec99a8-bafc-496d-baa2-8974fdcd53eb" (UID: "5dec99a8-bafc-496d-baa2-8974fdcd53eb"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 20:02:24.387872 ip-10-0-132-82 kubenswrapper[4892]: I1030 20:02:24.387846    4892 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0fb0d83f-df47-49bc-aa9b-16b4eb5708dc-maas-api-xks-ca" pod "0fb0d83f-df47-49bc-aa9b-16b4eb5708dc" (UID: "0fb0d83f-df47-49bc-aa9b-16b4eb5708dc"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 20:02:24.388132 ip-10-0-132-82 kubenswrapper[4892]: I1030 20:02:24.388112    4892 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0fb0d83f-df47-49bc-aa9b-16b4eb5708dc-kube-api-access-d84hg" pod "0fb0d83f-df47-49bc-aa9b-16b4eb5708dc" (UID: "0fb0d83f-df47-49bc-aa9b-16b4eb5708dc"). InnerVolumeSpecName "kube-api-access-d84hg". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 20:02:24.388307 ip-10-0-132-82 kubenswrapper[4892]: I1030 20:02:24.388290    4892 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5dec99a8-bafc-496d-baa2-8974fdcd53eb-kube-api-access-tvkn8" pod "5dec99a8-bafc-496d-baa2-8974fdcd53eb" (UID: "5dec99a8-bafc-496d-baa2-8974fdcd53eb"). InnerVolumeSpecName "kube-api-access-tvkn8". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 20:02:24.388358 ip-10-0-132-82 kubenswrapper[4892]: I1030 20:02:24.388291    4892 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5dec99a8-bafc-496d-baa2-8974fdcd53eb-maas-api-xks-ca" pod "5dec99a8-bafc-496d-baa2-8974fdcd53eb" (UID: "5dec99a8-bafc-496d-baa2-8974fdcd53eb"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 20:02:24.487177 ip-10-0-132-82 kubenswrapper[4892]: I1030 20:02:24.487138    4892 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-d84hg\" (UniqueName: \"kubernetes.io/projected/0fb0d83f-df47-49bc-aa9b-16b4eb5708dc-kube-api-access-d84hg\") on node \"ip-10-0-132-82.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:02:24.487177 ip-10-0-132-82 kubenswrapper[4892]: I1030 20:02:24.487165    4892 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tvkn8\" (UniqueName: \"kubernetes.io/projected/5dec99a8-bafc-496d-baa2-8974fdcd53eb-kube-api-access-tvkn8\") on node \"ip-10-0-132-82.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:02:24.487177 ip-10-0-132-82 kubenswrapper[4892]: I1030 20:02:24.487175    4892 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0fb0d83f-df47-49bc-aa9b-16b4eb5708dc-maas-api-xks-ca\") on node \"ip-10-0-132-82.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:02:24.487177 ip-10-0-132-82 kubenswrapper[4892]: I1030 20:02:24.487185    4892 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5dec99a8-bafc-496d-baa2-8974fdcd53eb-maas-api-openshift-ca\") on node \"ip-10-0-132-82.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:02:24.487536 ip-10-0-132-82 kubenswrapper[4892]: I1030 20:02:24.487193    4892 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5dec99a8-bafc-496d-baa2-8974fdcd53eb-maas-api-xks-ca\") on node \"ip-10-0-132-82.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:02:24.487536 ip-10-0-132-82 kubenswrapper[4892]: I1030 20:02:24.487202    4892 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0fb0d83f-df47-49bc-aa9b-16b4eb5708dc-maas-api-openshift-ca\") on node \"ip-10-0-132-82.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:02:24.843553 ip-10-0-132-82 kubenswrapper[4892]: E1030 20:02:24.843468    4892 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-jrm87" containerName="extractor"
Oct 30 20:02:25.020492 ip-10-0-132-82 kubenswrapper[4892]: I1030 20:02:25.020451    4892 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-fsfvk"]
Oct 30 20:02:25.034452 ip-10-0-132-82 kubenswrapper[4892]: I1030 20:02:25.034426    4892 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3556bc06-5mltw"
Oct 30 20:02:25.034644 ip-10-0-132-82 kubenswrapper[4892]: I1030 20:02:25.034468    4892 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3556bc06-5mltw" event={"ID":"83cb2031-9ebe-487f-8053-361008386451","Type":"ContainerDied","Data":"667c2c36c48f372bba70239e74433be7dc77e8ab7a010c13f88f739f64983e42"}
Oct 30 20:02:25.034644 ip-10-0-132-82 kubenswrapper[4892]: I1030 20:02:25.034525    4892 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="667c2c36c48f372bba70239e74433be7dc77e8ab7a010c13f88f739f64983e42"
Oct 30 20:02:25.036333 ip-10-0-132-82 kubenswrapper[4892]: I1030 20:02:25.036312    4892 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7a1cd64a-9lx2c"
Oct 30 20:02:25.036493 ip-10-0-132-82 kubenswrapper[4892]: I1030 20:02:25.036352    4892 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7a1cd64a-9lx2c" event={"ID":"0fb0d83f-df47-49bc-aa9b-16b4eb5708dc","Type":"ContainerDied","Data":"96f9c33acc36350878df6ad29d46584b20a3bcd3941faef1140106b4ae8f7423"}
Oct 30 20:02:25.036493 ip-10-0-132-82 kubenswrapper[4892]: I1030 20:02:25.036403    4892 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="96f9c33acc36350878df6ad29d46584b20a3bcd3941faef1140106b4ae8f7423"
Oct 30 20:02:25.038697 ip-10-0-132-82 kubenswrapper[4892]: I1030 20:02:25.038671    4892 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-fcb89cf2-pfwt8" event={"ID":"5dec99a8-bafc-496d-baa2-8974fdcd53eb","Type":"ContainerDied","Data":"79d42771e58acf5f967154cc7eee2f360fd88ba62e2f847a3b4fa89bf631489e"}
Oct 30 20:02:25.038804 ip-10-0-132-82 kubenswrapper[4892]: I1030 20:02:25.038719    4892 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="79d42771e58acf5f967154cc7eee2f360fd88ba62e2f847a3b4fa89bf631489e"
Oct 30 20:02:25.039438 ip-10-0-132-82 kubenswrapper[4892]: I1030 20:02:25.038946    4892 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-fcb89cf2-pfwt8"
Oct 30 20:02:25.989282 ip-10-0-132-82 kubenswrapper[4892]: I1030 20:02:25.989250    4892 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-k85mx"]
Oct 30 20:02:25.989687 ip-10-0-132-82 kubenswrapper[4892]: I1030 20:02:25.989658    4892 cpu_manager.go:414] "containerMap: removing container" podUID="83cb2031-9ebe-487f-8053-361008386451" containerName="revoke-keys"
Oct 30 20:02:25.989687 ip-10-0-132-82 kubenswrapper[4892]: I1030 20:02:25.989669    4892 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="83cb2031-9ebe-487f-8053-361008386451" containerName="revoke-keys"
Oct 30 20:02:25.989687 ip-10-0-132-82 kubenswrapper[4892]: I1030 20:02:25.989685    4892 cpu_manager.go:414] "containerMap: removing container" podUID="0fb0d83f-df47-49bc-aa9b-16b4eb5708dc" containerName="revoke-keys"
Oct 30 20:02:25.989797 ip-10-0-132-82 kubenswrapper[4892]: I1030 20:02:25.989690    4892 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0fb0d83f-df47-49bc-aa9b-16b4eb5708dc" containerName="revoke-keys"
Oct 30 20:02:25.989797 ip-10-0-132-82 kubenswrapper[4892]: I1030 20:02:25.989713    4892 cpu_manager.go:414] "containerMap: removing container" podUID="5dec99a8-bafc-496d-baa2-8974fdcd53eb" containerName="revoke-keys"
Oct 30 20:02:25.989797 ip-10-0-132-82 kubenswrapper[4892]: I1030 20:02:25.989719    4892 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5dec99a8-bafc-496d-baa2-8974fdcd53eb" containerName="revoke-keys"
Oct 30 20:02:25.989797 ip-10-0-132-82 kubenswrapper[4892]: I1030 20:02:25.989790    4892 memory_manager.go:365] "RemoveStaleState removing state" podUID="83cb2031-9ebe-487f-8053-361008386451" containerName="revoke-keys"
Oct 30 20:02:25.989930 ip-10-0-132-82 kubenswrapper[4892]: I1030 20:02:25.989799    4892 memory_manager.go:365] "RemoveStaleState removing state" podUID="0fb0d83f-df47-49bc-aa9b-16b4eb5708dc" containerName="revoke-keys"
Oct 30 20:02:25.989930 ip-10-0-132-82 kubenswrapper[4892]: I1030 20:02:25.989809    4892 memory_manager.go:365] "RemoveStaleState removing state" podUID="5dec99a8-bafc-496d-baa2-8974fdcd53eb" containerName="revoke-keys"
Oct 30 20:02:25.994876 ip-10-0-132-82 kubenswrapper[4892]: I1030 20:02:25.994854    4892 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-k85mx"
Oct 30 20:02:26.099372 ip-10-0-132-82 kubenswrapper[4892]: I1030 20:02:26.099331    4892 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-k85mx"]
Oct 30 20:02:26.107346 ip-10-0-132-82 kubenswrapper[4892]: I1030 20:02:26.107319    4892 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6f05c89a-d631-4c95-aad6-d08d6c87cffd-istio-podinfo\") pod \"maas-default-gateway-openshift-default-6db586dd6c-k85mx\" (UID: \"6f05c89a-d631-4c95-aad6-d08d6c87cffd\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-k85mx"
Oct 30 20:02:26.107493 ip-10-0-132-82 kubenswrapper[4892]: I1030 20:02:26.107354    4892 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6f05c89a-d631-4c95-aad6-d08d6c87cffd-credential-socket\") pod \"maas-default-gateway-openshift-default-6db586dd6c-k85mx\" (UID: \"6f05c89a-d631-4c95-aad6-d08d6c87cffd\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-k85mx"
Oct 30 20:02:26.107493 ip-10-0-132-82 kubenswrapper[4892]: I1030 20:02:26.107386    4892 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6f05c89a-d631-4c95-aad6-d08d6c87cffd-istio-envoy\") pod \"maas-default-gateway-openshift-default-6db586dd6c-k85mx\" (UID: \"6f05c89a-d631-4c95-aad6-d08d6c87cffd\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-k85mx"
Oct 30 20:02:26.107592 ip-10-0-132-82 kubenswrapper[4892]: I1030 20:02:26.107538    4892 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sxn6r\" (UniqueName: \"kubernetes.io/projected/6f05c89a-d631-4c95-aad6-d08d6c87cffd-kube-api-access-sxn6r\") pod \"maas-default-gateway-openshift-default-6db586dd6c-k85mx\" (UID: \"6f05c89a-d631-4c95-aad6-d08d6c87cffd\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-k85mx"
Oct 30 20:02:26.107680 ip-10-0-132-82 kubenswrapper[4892]: I1030 20:02:26.107660    4892 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6f05c89a-d631-4c95-aad6-d08d6c87cffd-workload-socket\") pod \"maas-default-gateway-openshift-default-6db586dd6c-k85mx\" (UID: \"6f05c89a-d631-4c95-aad6-d08d6c87cffd\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-k85mx"
Oct 30 20:02:26.107721 ip-10-0-132-82 kubenswrapper[4892]: I1030 20:02:26.107694    4892 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6f05c89a-d631-4c95-aad6-d08d6c87cffd-workload-certs\") pod \"maas-default-gateway-openshift-default-6db586dd6c-k85mx\" (UID: \"6f05c89a-d631-4c95-aad6-d08d6c87cffd\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-k85mx"
Oct 30 20:02:26.107769 ip-10-0-132-82 kubenswrapper[4892]: I1030 20:02:26.107725    4892 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6f05c89a-d631-4c95-aad6-d08d6c87cffd-istio-token\") pod \"maas-default-gateway-openshift-default-6db586dd6c-k85mx\" (UID: \"6f05c89a-d631-4c95-aad6-d08d6c87cffd\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-k85mx"
Oct 30 20:02:26.107769 ip-10-0-132-82 kubenswrapper[4892]: I1030 20:02:26.107759    4892 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6f05c89a-d631-4c95-aad6-d08d6c87cffd-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-6db586dd6c-k85mx\" (UID: \"6f05c89a-d631-4c95-aad6-d08d6c87cffd\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-k85mx"
Oct 30 20:02:26.107850 ip-10-0-132-82 kubenswrapper[4892]: I1030 20:02:26.107834    4892 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6f05c89a-d631-4c95-aad6-d08d6c87cffd-istio-data\") pod \"maas-default-gateway-openshift-default-6db586dd6c-k85mx\" (UID: \"6f05c89a-d631-4c95-aad6-d08d6c87cffd\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-k85mx"
Oct 30 20:02:26.208528 ip-10-0-132-82 kubenswrapper[4892]: I1030 20:02:26.208493    4892 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6f05c89a-d631-4c95-aad6-d08d6c87cffd-workload-socket\") pod \"maas-default-gateway-openshift-default-6db586dd6c-k85mx\" (UID: \"6f05c89a-d631-4c95-aad6-d08d6c87cffd\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-k85mx"
Oct 30 20:02:26.208528 ip-10-0-132-82 kubenswrapper[4892]: I1030 20:02:26.208528    4892 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6f05c89a-d631-4c95-aad6-d08d6c87cffd-workload-certs\") pod \"maas-default-gateway-openshift-default-6db586dd6c-k85mx\" (UID: \"6f05c89a-d631-4c95-aad6-d08d6c87cffd\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-k85mx"
Oct 30 20:02:26.208734 ip-10-0-132-82 kubenswrapper[4892]: I1030 20:02:26.208545    4892 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6f05c89a-d631-4c95-aad6-d08d6c87cffd-istio-token\") pod \"maas-default-gateway-openshift-default-6db586dd6c-k85mx\" (UID: \"6f05c89a-d631-4c95-aad6-d08d6c87cffd\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-k85mx"
Oct 30 20:02:26.208734 ip-10-0-132-82 kubenswrapper[4892]: I1030 20:02:26.208562    4892 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6f05c89a-d631-4c95-aad6-d08d6c87cffd-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-6db586dd6c-k85mx\" (UID: \"6f05c89a-d631-4c95-aad6-d08d6c87cffd\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-k85mx"
Oct 30 20:02:26.208734 ip-10-0-132-82 kubenswrapper[4892]: I1030 20:02:26.208590    4892 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6f05c89a-d631-4c95-aad6-d08d6c87cffd-istio-data\") pod \"maas-default-gateway-openshift-default-6db586dd6c-k85mx\" (UID: \"6f05c89a-d631-4c95-aad6-d08d6c87cffd\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-k85mx"
Oct 30 20:02:26.208734 ip-10-0-132-82 kubenswrapper[4892]: I1030 20:02:26.208670    4892 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6f05c89a-d631-4c95-aad6-d08d6c87cffd-istio-podinfo\") pod \"maas-default-gateway-openshift-default-6db586dd6c-k85mx\" (UID: \"6f05c89a-d631-4c95-aad6-d08d6c87cffd\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-k85mx"
Oct 30 20:02:26.208734 ip-10-0-132-82 kubenswrapper[4892]: I1030 20:02:26.208692    4892 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6f05c89a-d631-4c95-aad6-d08d6c87cffd-credential-socket\") pod \"maas-default-gateway-openshift-default-6db586dd6c-k85mx\" (UID: \"6f05c89a-d631-4c95-aad6-d08d6c87cffd\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-k85mx"
Oct 30 20:02:26.209045 ip-10-0-132-82 kubenswrapper[4892]: I1030 20:02:26.208736    4892 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6f05c89a-d631-4c95-aad6-d08d6c87cffd-istio-envoy\") pod \"maas-default-gateway-openshift-default-6db586dd6c-k85mx\" (UID: \"6f05c89a-d631-4c95-aad6-d08d6c87cffd\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-k85mx"
Oct 30 20:02:26.209045 ip-10-0-132-82 kubenswrapper[4892]: I1030 20:02:26.208772    4892 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sxn6r\" (UniqueName: \"kubernetes.io/projected/6f05c89a-d631-4c95-aad6-d08d6c87cffd-kube-api-access-sxn6r\") pod \"maas-default-gateway-openshift-default-6db586dd6c-k85mx\" (UID: \"6f05c89a-d631-4c95-aad6-d08d6c87cffd\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-k85mx"
Oct 30 20:02:26.209045 ip-10-0-132-82 kubenswrapper[4892]: I1030 20:02:26.209004    4892 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6f05c89a-d631-4c95-aad6-d08d6c87cffd-workload-socket\") pod \"maas-default-gateway-openshift-default-6db586dd6c-k85mx\" (UID: \"6f05c89a-d631-4c95-aad6-d08d6c87cffd\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-k85mx"
Oct 30 20:02:26.209218 ip-10-0-132-82 kubenswrapper[4892]: I1030 20:02:26.209070    4892 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6f05c89a-d631-4c95-aad6-d08d6c87cffd-istio-data\") pod \"maas-default-gateway-openshift-default-6db586dd6c-k85mx\" (UID: \"6f05c89a-d631-4c95-aad6-d08d6c87cffd\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-k85mx"
Oct 30 20:02:26.209218 ip-10-0-132-82 kubenswrapper[4892]: I1030 20:02:26.209125    4892 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6f05c89a-d631-4c95-aad6-d08d6c87cffd-credential-socket\") pod \"maas-default-gateway-openshift-default-6db586dd6c-k85mx\" (UID: \"6f05c89a-d631-4c95-aad6-d08d6c87cffd\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-k85mx"
Oct 30 20:02:26.209336 ip-10-0-132-82 kubenswrapper[4892]: I1030 20:02:26.209312    4892 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6f05c89a-d631-4c95-aad6-d08d6c87cffd-workload-certs\") pod \"maas-default-gateway-openshift-default-6db586dd6c-k85mx\" (UID: \"6f05c89a-d631-4c95-aad6-d08d6c87cffd\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-k85mx"
Oct 30 20:02:26.209414 ip-10-0-132-82 kubenswrapper[4892]: I1030 20:02:26.209396    4892 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6f05c89a-d631-4c95-aad6-d08d6c87cffd-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-6db586dd6c-k85mx\" (UID: \"6f05c89a-d631-4c95-aad6-d08d6c87cffd\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-k85mx"
Oct 30 20:02:26.211258 ip-10-0-132-82 kubenswrapper[4892]: I1030 20:02:26.211238    4892 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6f05c89a-d631-4c95-aad6-d08d6c87cffd-istio-envoy\") pod \"maas-default-gateway-openshift-default-6db586dd6c-k85mx\" (UID: \"6f05c89a-d631-4c95-aad6-d08d6c87cffd\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-k85mx"
Oct 30 20:02:26.211403 ip-10-0-132-82 kubenswrapper[4892]: I1030 20:02:26.211386    4892 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6f05c89a-d631-4c95-aad6-d08d6c87cffd-istio-podinfo\") pod \"maas-default-gateway-openshift-default-6db586dd6c-k85mx\" (UID: \"6f05c89a-d631-4c95-aad6-d08d6c87cffd\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-k85mx"
Oct 30 20:02:26.265182 ip-10-0-132-82 kubenswrapper[4892]: I1030 20:02:26.265117    4892 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6f05c89a-d631-4c95-aad6-d08d6c87cffd-istio-token\") pod \"maas-default-gateway-openshift-default-6db586dd6c-k85mx\" (UID: \"6f05c89a-d631-4c95-aad6-d08d6c87cffd\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-k85mx"
Oct 30 20:02:26.265389 ip-10-0-132-82 kubenswrapper[4892]: I1030 20:02:26.265371    4892 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-sxn6r\" (UniqueName: \"kubernetes.io/projected/6f05c89a-d631-4c95-aad6-d08d6c87cffd-kube-api-access-sxn6r\") pod \"maas-default-gateway-openshift-default-6db586dd6c-k85mx\" (UID: \"6f05c89a-d631-4c95-aad6-d08d6c87cffd\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-k85mx"
Oct 30 20:02:26.308369 ip-10-0-132-82 kubenswrapper[4892]: I1030 20:02:26.308335    4892 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-k85mx"
Oct 30 20:02:26.738552 ip-10-0-132-82 kubenswrapper[4892]: I1030 20:02:26.738527    4892 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-k85mx"]
Oct 30 20:02:26.739304 ip-10-0-132-82 kubenswrapper[4892]: W1030 20:02:26.739280    4892 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6f05c89a_d631_4c95_aad6_d08d6c87cffd.slice/crio-984a32728c751b3ca9653a8908c58a423c7a71015b2573fd91b0cedf79eba725 WatchSource:0}: Error finding container 984a32728c751b3ca9653a8908c58a423c7a71015b2573fd91b0cedf79eba725: Status 404 returned error can't find the container with id 984a32728c751b3ca9653a8908c58a423c7a71015b2573fd91b0cedf79eba725
Oct 30 20:02:26.742137 ip-10-0-132-82 kubenswrapper[4892]: I1030 20:02:26.742104    4892 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892000Ki","pods":"250"}
Oct 30 20:02:26.742221 ip-10-0-132-82 kubenswrapper[4892]: I1030 20:02:26.742170    4892 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892000Ki","pods":"250"}
Oct 30 20:02:26.742221 ip-10-0-132-82 kubenswrapper[4892]: I1030 20:02:26.742201    4892 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892000Ki","pods":"250"}
Oct 30 20:02:27.055875 ip-10-0-132-82 kubenswrapper[4892]: I1030 20:02:27.055782    4892 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-k85mx" event={"ID":"6f05c89a-d631-4c95-aad6-d08d6c87cffd","Type":"ContainerStarted","Data":"d87139aa86bf1b1aacecc2c7327610a42fda8698ccf202823208d646da9c409d"}
Oct 30 20:02:27.055875 ip-10-0-132-82 kubenswrapper[4892]: I1030 20:02:27.055848    4892 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-k85mx" event={"ID":"6f05c89a-d631-4c95-aad6-d08d6c87cffd","Type":"ContainerStarted","Data":"984a32728c751b3ca9653a8908c58a423c7a71015b2573fd91b0cedf79eba725"}
Oct 30 20:02:27.171847 ip-10-0-132-82 kubenswrapper[4892]: I1030 20:02:27.171789    4892 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-k85mx" podStartSLOduration=2.171771002 podStartE2EDuration="2.171771002s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 20:02:25 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 20:02:27.171097697 +0000 UTC m=+2794.741563295" watchObservedRunningTime="2026-10-30 20:02:27.171771002 +0000 UTC m=+2794.742236598"
Oct 30 20:02:27.308582 ip-10-0-132-82 kubenswrapper[4892]: I1030 20:02:27.308486    4892 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-k85mx"
Oct 30 20:02:27.309927 ip-10-0-132-82 kubenswrapper[4892]: I1030 20:02:27.309894    4892 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6db586dd6c-k85mx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.1:15021/healthz/ready\": dial tcp 10.132.1.1:15021: connect: connection refused" start-of-body=
Oct 30 20:02:27.310175 ip-10-0-132-82 kubenswrapper[4892]: I1030 20:02:27.309950    4892 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-k85mx" podUID="6f05c89a-d631-4c95-aad6-d08d6c87cffd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.1:15021/healthz/ready\": dial tcp 10.132.1.1:15021: connect: connection refused"
Oct 30 20:02:28.308716 ip-10-0-132-82 kubenswrapper[4892]: I1030 20:02:28.308679    4892 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6db586dd6c-k85mx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.1:15021/healthz/ready\": dial tcp 10.132.1.1:15021: connect: connection refused" start-of-body=
Oct 30 20:02:28.309247 ip-10-0-132-82 kubenswrapper[4892]: I1030 20:02:28.308739    4892 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-k85mx" podUID="6f05c89a-d631-4c95-aad6-d08d6c87cffd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.1:15021/healthz/ready\": dial tcp 10.132.1.1:15021: connect: connection refused"
Oct 30 20:02:28.908914 ip-10-0-132-82 kubenswrapper[4892]: I1030 20:02:28.908865    4892 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w3-ada8d7/simulator-w3-ada8d7-kserve-7d575cfcb5-5txgn" podUID="10a5c7c8-6146-47b0-9643-bb137cf10bf5" containerName="main" containerID="cri-o://b16fc553522b87ac281529364f1d4a580ead182ca2d88e0462cce03fa304a0b9" gracePeriod=45
Oct 30 20:02:29.071727 ip-10-0-132-82 kubenswrapper[4892]: I1030 20:02:29.071695    4892 generic.go:356] "Generic (PLEG): container finished" podID="10a5c7c8-6146-47b0-9643-bb137cf10bf5" containerID="b16fc553522b87ac281529364f1d4a580ead182ca2d88e0462cce03fa304a0b9" exitCode=0
Oct 30 20:02:29.071898 ip-10-0-132-82 kubenswrapper[4892]: I1030 20:02:29.071776    4892 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w3-ada8d7/simulator-w3-ada8d7-kserve-7d575cfcb5-5txgn" event={"ID":"10a5c7c8-6146-47b0-9643-bb137cf10bf5","Type":"ContainerDied","Data":"b16fc553522b87ac281529364f1d4a580ead182ca2d88e0462cce03fa304a0b9"}
Oct 30 20:02:29.217319 ip-10-0-132-82 kubenswrapper[4892]: I1030 20:02:29.217296    4892 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w3-ada8d7/simulator-w3-ada8d7-kserve-7d575cfcb5-5txgn"
Oct 30 20:02:29.309280 ip-10-0-132-82 kubenswrapper[4892]: I1030 20:02:29.309250    4892 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6db586dd6c-k85mx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.1:15021/healthz/ready\": dial tcp 10.132.1.1:15021: connect: connection refused" start-of-body=
Oct 30 20:02:29.309691 ip-10-0-132-82 kubenswrapper[4892]: I1030 20:02:29.309300    4892 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-k85mx" podUID="6f05c89a-d631-4c95-aad6-d08d6c87cffd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.1:15021/healthz/ready\": dial tcp 10.132.1.1:15021: connect: connection refused"
Oct 30 20:02:29.368578 ip-10-0-132-82 kubenswrapper[4892]: I1030 20:02:29.368543    4892 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/10a5c7c8-6146-47b0-9643-bb137cf10bf5-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/10a5c7c8-6146-47b0-9643-bb137cf10bf5-tmp-dir\") pod \"10a5c7c8-6146-47b0-9643-bb137cf10bf5\" (UID: \"10a5c7c8-6146-47b0-9643-bb137cf10bf5\") "
Oct 30 20:02:29.368750 ip-10-0-132-82 kubenswrapper[4892]: I1030 20:02:29.368655    4892 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/10a5c7c8-6146-47b0-9643-bb137cf10bf5-home\" (UniqueName: \"kubernetes.io/empty-dir/10a5c7c8-6146-47b0-9643-bb137cf10bf5-home\") pod \"10a5c7c8-6146-47b0-9643-bb137cf10bf5\" (UID: \"10a5c7c8-6146-47b0-9643-bb137cf10bf5\") "
Oct 30 20:02:29.368825 ip-10-0-132-82 kubenswrapper[4892]: I1030 20:02:29.368752    4892 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/10a5c7c8-6146-47b0-9643-bb137cf10bf5-dshm\" (UniqueName: \"kubernetes.io/empty-dir/10a5c7c8-6146-47b0-9643-bb137cf10bf5-dshm\") pod \"10a5c7c8-6146-47b0-9643-bb137cf10bf5\" (UID: \"10a5c7c8-6146-47b0-9643-bb137cf10bf5\") "
Oct 30 20:02:29.368825 ip-10-0-132-82 kubenswrapper[4892]: I1030 20:02:29.368747    4892 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/10a5c7c8-6146-47b0-9643-bb137cf10bf5-tmp-dir" pod "10a5c7c8-6146-47b0-9643-bb137cf10bf5" (UID: "10a5c7c8-6146-47b0-9643-bb137cf10bf5"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 20:02:29.368825 ip-10-0-132-82 kubenswrapper[4892]: I1030 20:02:29.368778    4892 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/10a5c7c8-6146-47b0-9643-bb137cf10bf5-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/10a5c7c8-6146-47b0-9643-bb137cf10bf5-model-cache\") pod \"10a5c7c8-6146-47b0-9643-bb137cf10bf5\" (UID: \"10a5c7c8-6146-47b0-9643-bb137cf10bf5\") "
Oct 30 20:02:29.368825 ip-10-0-132-82 kubenswrapper[4892]: I1030 20:02:29.368820    4892 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/10a5c7c8-6146-47b0-9643-bb137cf10bf5-tls-certs\" (UniqueName: \"kubernetes.io/secret/10a5c7c8-6146-47b0-9643-bb137cf10bf5-tls-certs\") pod \"10a5c7c8-6146-47b0-9643-bb137cf10bf5\" (UID: \"10a5c7c8-6146-47b0-9643-bb137cf10bf5\") "
Oct 30 20:02:29.369086 ip-10-0-132-82 kubenswrapper[4892]: I1030 20:02:29.368848    4892 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/10a5c7c8-6146-47b0-9643-bb137cf10bf5-kube-api-access-qxp7s\" (UniqueName: \"kubernetes.io/projected/10a5c7c8-6146-47b0-9643-bb137cf10bf5-kube-api-access-qxp7s\") pod \"10a5c7c8-6146-47b0-9643-bb137cf10bf5\" (UID: \"10a5c7c8-6146-47b0-9643-bb137cf10bf5\") "
Oct 30 20:02:29.369086 ip-10-0-132-82 kubenswrapper[4892]: I1030 20:02:29.368875    4892 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/10a5c7c8-6146-47b0-9643-bb137cf10bf5-home" pod "10a5c7c8-6146-47b0-9643-bb137cf10bf5" (UID: "10a5c7c8-6146-47b0-9643-bb137cf10bf5"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 20:02:29.369086 ip-10-0-132-82 kubenswrapper[4892]: I1030 20:02:29.368983    4892 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/10a5c7c8-6146-47b0-9643-bb137cf10bf5-model-cache" pod "10a5c7c8-6146-47b0-9643-bb137cf10bf5" (UID: "10a5c7c8-6146-47b0-9643-bb137cf10bf5"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 20:02:29.369268 ip-10-0-132-82 kubenswrapper[4892]: I1030 20:02:29.369216    4892 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/10a5c7c8-6146-47b0-9643-bb137cf10bf5-model-cache\") on node \"ip-10-0-132-82.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:02:29.369268 ip-10-0-132-82 kubenswrapper[4892]: I1030 20:02:29.369236    4892 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/10a5c7c8-6146-47b0-9643-bb137cf10bf5-tmp-dir\") on node \"ip-10-0-132-82.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:02:29.369268 ip-10-0-132-82 kubenswrapper[4892]: I1030 20:02:29.369250    4892 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/10a5c7c8-6146-47b0-9643-bb137cf10bf5-home\") on node \"ip-10-0-132-82.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:02:29.371575 ip-10-0-132-82 kubenswrapper[4892]: I1030 20:02:29.371545    4892 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/10a5c7c8-6146-47b0-9643-bb137cf10bf5-tls-certs" pod "10a5c7c8-6146-47b0-9643-bb137cf10bf5" (UID: "10a5c7c8-6146-47b0-9643-bb137cf10bf5"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 20:02:29.377383 ip-10-0-132-82 kubenswrapper[4892]: I1030 20:02:29.377356    4892 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/10a5c7c8-6146-47b0-9643-bb137cf10bf5-dshm" pod "10a5c7c8-6146-47b0-9643-bb137cf10bf5" (UID: "10a5c7c8-6146-47b0-9643-bb137cf10bf5"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 20:02:29.377549 ip-10-0-132-82 kubenswrapper[4892]: I1030 20:02:29.377376    4892 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/10a5c7c8-6146-47b0-9643-bb137cf10bf5-kube-api-access-qxp7s" pod "10a5c7c8-6146-47b0-9643-bb137cf10bf5" (UID: "10a5c7c8-6146-47b0-9643-bb137cf10bf5"). InnerVolumeSpecName "kube-api-access-qxp7s". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 20:02:29.470784 ip-10-0-132-82 kubenswrapper[4892]: I1030 20:02:29.470706    4892 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/10a5c7c8-6146-47b0-9643-bb137cf10bf5-dshm\") on node \"ip-10-0-132-82.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:02:29.470784 ip-10-0-132-82 kubenswrapper[4892]: I1030 20:02:29.470744    4892 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/10a5c7c8-6146-47b0-9643-bb137cf10bf5-tls-certs\") on node \"ip-10-0-132-82.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:02:29.470784 ip-10-0-132-82 kubenswrapper[4892]: I1030 20:02:29.470764    4892 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qxp7s\" (UniqueName: \"kubernetes.io/projected/10a5c7c8-6146-47b0-9643-bb137cf10bf5-kube-api-access-qxp7s\") on node \"ip-10-0-132-82.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:02:30.080535 ip-10-0-132-82 kubenswrapper[4892]: I1030 20:02:30.080505    4892 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w3-ada8d7/simulator-w3-ada8d7-kserve-7d575cfcb5-5txgn"
Oct 30 20:02:30.080721 ip-10-0-132-82 kubenswrapper[4892]: I1030 20:02:30.080549    4892 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w3-ada8d7/simulator-w3-ada8d7-kserve-7d575cfcb5-5txgn" event={"ID":"10a5c7c8-6146-47b0-9643-bb137cf10bf5","Type":"ContainerDied","Data":"d9b691aaff95d751183bfca03c9a3e6374fee23cd47652d8fd1d36b85c1b22d2"}
Oct 30 20:02:30.080721 ip-10-0-132-82 kubenswrapper[4892]: I1030 20:02:30.080601    4892 scope.go:122] "RemoveContainer" containerID="b16fc553522b87ac281529364f1d4a580ead182ca2d88e0462cce03fa304a0b9"
Oct 30 20:02:30.250508 ip-10-0-132-82 kubenswrapper[4892]: I1030 20:02:30.250464    4892 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w3-ada8d7/simulator-w3-ada8d7-kserve-7d575cfcb5-5txgn"]
Oct 30 20:02:30.297776 ip-10-0-132-82 kubenswrapper[4892]: I1030 20:02:30.297742    4892 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w3-ada8d7/simulator-w3-ada8d7-kserve-7d575cfcb5-5txgn"]
Oct 30 20:02:30.309599 ip-10-0-132-82 kubenswrapper[4892]: I1030 20:02:30.309572    4892 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6db586dd6c-k85mx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.1:15021/healthz/ready\": dial tcp 10.132.1.1:15021: connect: connection refused" start-of-body=
Oct 30 20:02:30.310057 ip-10-0-132-82 kubenswrapper[4892]: I1030 20:02:30.309624    4892 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-k85mx" podUID="6f05c89a-d631-4c95-aad6-d08d6c87cffd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.1:15021/healthz/ready\": dial tcp 10.132.1.1:15021: connect: connection refused"
Oct 30 20:02:30.575952 ip-10-0-132-82 kubenswrapper[4892]: I1030 20:02:30.575917    4892 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="10a5c7c8-6146-47b0-9643-bb137cf10bf5" path="/var/lib/kubelet/pods/10a5c7c8-6146-47b0-9643-bb137cf10bf5/volumes"
Oct 30 20:02:31.309296 ip-10-0-132-82 kubenswrapper[4892]: I1030 20:02:31.309266    4892 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6db586dd6c-k85mx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.1:15021/healthz/ready\": dial tcp 10.132.1.1:15021: connect: connection refused" start-of-body=
Oct 30 20:02:31.309565 ip-10-0-132-82 kubenswrapper[4892]: I1030 20:02:31.309536    4892 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-k85mx" podUID="6f05c89a-d631-4c95-aad6-d08d6c87cffd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.1:15021/healthz/ready\": dial tcp 10.132.1.1:15021: connect: connection refused"
Oct 30 20:02:32.309298 ip-10-0-132-82 kubenswrapper[4892]: I1030 20:02:32.309261    4892 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6db586dd6c-k85mx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.1:15021/healthz/ready\": dial tcp 10.132.1.1:15021: connect: connection refused" start-of-body=
Oct 30 20:02:32.309709 ip-10-0-132-82 kubenswrapper[4892]: I1030 20:02:32.309316    4892 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-k85mx" podUID="6f05c89a-d631-4c95-aad6-d08d6c87cffd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.1:15021/healthz/ready\": dial tcp 10.132.1.1:15021: connect: connection refused"
Oct 30 20:02:33.308978 ip-10-0-132-82 kubenswrapper[4892]: I1030 20:02:33.308937    4892 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6db586dd6c-k85mx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.1:15021/healthz/ready\": dial tcp 10.132.1.1:15021: connect: connection refused" start-of-body=
Oct 30 20:02:33.309174 ip-10-0-132-82 kubenswrapper[4892]: I1030 20:02:33.309015    4892 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-k85mx" podUID="6f05c89a-d631-4c95-aad6-d08d6c87cffd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.1:15021/healthz/ready\": dial tcp 10.132.1.1:15021: connect: connection refused"
Oct 30 20:02:35.070929 ip-10-0-132-82 kubenswrapper[4892]: I1030 20:02:35.070886    4892 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-k85mx"
Oct 30 20:02:35.071627 ip-10-0-132-82 kubenswrapper[4892]: I1030 20:02:35.071598    4892 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-k85mx"
Oct 30 20:02:35.073285 ip-10-0-132-82 kubenswrapper[4892]: I1030 20:02:35.073264    4892 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-k85mx"
Oct 30 20:02:44.483454 ip-10-0-132-82 kubenswrapper[4892]: I1030 20:02:44.483422    4892 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-fsfvk"]
Oct 30 20:02:46.608102 ip-10-0-132-82 kubenswrapper[4892]: I1030 20:02:46.608061    4892 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-ada8d7-ss6z5"]
Oct 30 20:02:46.657221 ip-10-0-132-82 kubenswrapper[4892]: I1030 20:02:46.657188    4892 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-ada8d7-ss6z5"]
Oct 30 20:02:47.801934 ip-10-0-132-82 kubenswrapper[4892]: I1030 20:02:47.801896    4892 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-a3f9b7-gfgvh"]
Oct 30 20:02:47.850928 ip-10-0-132-82 kubenswrapper[4892]: I1030 20:02:47.850891    4892 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-a3f9b7-gfgvh"]
Oct 30 20:02:48.569266 ip-10-0-132-82 kubenswrapper[4892]: I1030 20:02:48.569228    4892 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="497e96db-0aa8-4f9f-be89-4797e1fc6d0a" path="/var/lib/kubelet/pods/497e96db-0aa8-4f9f-be89-4797e1fc6d0a/volumes"
Oct 30 20:02:48.569655 ip-10-0-132-82 kubenswrapper[4892]: I1030 20:02:48.569639    4892 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="73e40af0-f49a-41bf-9dbd-e52de5027c21" path="/var/lib/kubelet/pods/73e40af0-f49a-41bf-9dbd-e52de5027c21/volumes"
Oct 30 20:02:53.777139 ip-10-0-132-82 kubenswrapper[4892]: I1030 20:02:53.777099    4892 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-ada8d7-openshift-default-5f8b49bdfc-khngd"]
Oct 30 20:02:53.777630 ip-10-0-132-82 kubenswrapper[4892]: I1030 20:02:53.777311    4892 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w3-ada8d7-openshift-default-5f8b49bdfc-khngd" podUID="eb58d944-e2ad-4505-a5df-d5c617c1dbf2" containerName="istio-proxy" containerID="cri-o://c20bba13fe09724c6650e876534a4db34e2ba5a5a45fe865f2d69660b9d213d6" gracePeriod=30
Oct 30 20:02:54.843759 ip-10-0-132-82 kubenswrapper[4892]: E1030 20:02:54.843716    4892 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-jrm87" containerName="extractor"
Oct 30 20:02:54.884836 ip-10-0-132-82 kubenswrapper[4892]: I1030 20:02:54.884804    4892 patch_prober.go:29] interesting pod/e2e-worker-w3-ada8d7-openshift-default-5f8b49bdfc-khngd container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Oct 30 20:02:54.885030 ip-10-0-132-82 kubenswrapper[4892]: I1030 20:02:54.884866    4892 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w3-ada8d7-openshift-default-5f8b49bdfc-khngd" podUID="eb58d944-e2ad-4505-a5df-d5c617c1dbf2" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Oct 30 20:02:58.193531 ip-10-0-132-82 kubenswrapper[4892]: I1030 20:02:58.193492    4892 scope.go:122] "RemoveContainer" containerID="97964c9a51be733508b19228fd127ff01a9cba57de9b3c6f5e185401b0b47134"
Oct 30 20:02:58.204270 ip-10-0-132-82 kubenswrapper[4892]: I1030 20:02:58.204251    4892 scope.go:122] "RemoveContainer" containerID="27391a1d7e8075a1bfc8293fd2e9cbdce81f08b5796699bb4c9a224239de129f"
Oct 30 20:02:58.213197 ip-10-0-132-82 kubenswrapper[4892]: I1030 20:02:58.213177    4892 scope.go:122] "RemoveContainer" containerID="556837ed75294bf62881db9353fc7bf97427b9f70c8f0fd3bc03cb435dc0dcfd"
Oct 30 20:02:58.222331 ip-10-0-132-82 kubenswrapper[4892]: I1030 20:02:58.222312    4892 scope.go:122] "RemoveContainer" containerID="a0e456c551f27194822d5c66f632d8254e7236024a5ec99882aa7e25b893009a"
Oct 30 20:02:58.230947 ip-10-0-132-82 kubenswrapper[4892]: I1030 20:02:58.230927    4892 scope.go:122] "RemoveContainer" containerID="4bec4ad58bd687ed8ebc8b6e4853203a4ed0ceb1dbc2377d38f44440e95b626b"
Oct 30 20:02:58.239772 ip-10-0-132-82 kubenswrapper[4892]: I1030 20:02:58.239751    4892 scope.go:122] "RemoveContainer" containerID="72f5e3a23d03763ac8f5839a202969d29f16f47a12ff92506f63ea6d30fe414f"
Oct 30 20:02:59.036564 ip-10-0-132-82 kubenswrapper[4892]: I1030 20:02:59.036543    4892 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-ada8d7-openshift-default-5f8b49bdfc-khngd"
Oct 30 20:02:59.183536 ip-10-0-132-82 kubenswrapper[4892]: I1030 20:02:59.183503    4892 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/eb58d944-e2ad-4505-a5df-d5c617c1dbf2-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/eb58d944-e2ad-4505-a5df-d5c617c1dbf2-istiod-ca-cert\") pod \"eb58d944-e2ad-4505-a5df-d5c617c1dbf2\" (UID: \"eb58d944-e2ad-4505-a5df-d5c617c1dbf2\") "
Oct 30 20:02:59.183713 ip-10-0-132-82 kubenswrapper[4892]: I1030 20:02:59.183575    4892 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/eb58d944-e2ad-4505-a5df-d5c617c1dbf2-istio-token\" (UniqueName: \"kubernetes.io/projected/eb58d944-e2ad-4505-a5df-d5c617c1dbf2-istio-token\") pod \"eb58d944-e2ad-4505-a5df-d5c617c1dbf2\" (UID: \"eb58d944-e2ad-4505-a5df-d5c617c1dbf2\") "
Oct 30 20:02:59.183713 ip-10-0-132-82 kubenswrapper[4892]: I1030 20:02:59.183678    4892 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/eb58d944-e2ad-4505-a5df-d5c617c1dbf2-kube-api-access-7qxtg\" (UniqueName: \"kubernetes.io/projected/eb58d944-e2ad-4505-a5df-d5c617c1dbf2-kube-api-access-7qxtg\") pod \"eb58d944-e2ad-4505-a5df-d5c617c1dbf2\" (UID: \"eb58d944-e2ad-4505-a5df-d5c617c1dbf2\") "
Oct 30 20:02:59.183859 ip-10-0-132-82 kubenswrapper[4892]: I1030 20:02:59.183729    4892 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/eb58d944-e2ad-4505-a5df-d5c617c1dbf2-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/eb58d944-e2ad-4505-a5df-d5c617c1dbf2-istio-data\") pod \"eb58d944-e2ad-4505-a5df-d5c617c1dbf2\" (UID: \"eb58d944-e2ad-4505-a5df-d5c617c1dbf2\") "
Oct 30 20:02:59.183859 ip-10-0-132-82 kubenswrapper[4892]: I1030 20:02:59.183847    4892 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/eb58d944-e2ad-4505-a5df-d5c617c1dbf2-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/eb58d944-e2ad-4505-a5df-d5c617c1dbf2-istio-envoy\") pod \"eb58d944-e2ad-4505-a5df-d5c617c1dbf2\" (UID: \"eb58d944-e2ad-4505-a5df-d5c617c1dbf2\") "
Oct 30 20:02:59.184012 ip-10-0-132-82 kubenswrapper[4892]: I1030 20:02:59.183873    4892 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/eb58d944-e2ad-4505-a5df-d5c617c1dbf2-istiod-ca-cert" pod "eb58d944-e2ad-4505-a5df-d5c617c1dbf2" (UID: "eb58d944-e2ad-4505-a5df-d5c617c1dbf2"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 20:02:59.184012 ip-10-0-132-82 kubenswrapper[4892]: I1030 20:02:59.183887    4892 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/eb58d944-e2ad-4505-a5df-d5c617c1dbf2-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/eb58d944-e2ad-4505-a5df-d5c617c1dbf2-workload-certs\") pod \"eb58d944-e2ad-4505-a5df-d5c617c1dbf2\" (UID: \"eb58d944-e2ad-4505-a5df-d5c617c1dbf2\") "
Oct 30 20:02:59.184012 ip-10-0-132-82 kubenswrapper[4892]: I1030 20:02:59.183930    4892 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/eb58d944-e2ad-4505-a5df-d5c617c1dbf2-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/eb58d944-e2ad-4505-a5df-d5c617c1dbf2-credential-socket\") pod \"eb58d944-e2ad-4505-a5df-d5c617c1dbf2\" (UID: \"eb58d944-e2ad-4505-a5df-d5c617c1dbf2\") "
Oct 30 20:02:59.184012 ip-10-0-132-82 kubenswrapper[4892]: I1030 20:02:59.183994    4892 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/eb58d944-e2ad-4505-a5df-d5c617c1dbf2-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/eb58d944-e2ad-4505-a5df-d5c617c1dbf2-workload-socket\") pod \"eb58d944-e2ad-4505-a5df-d5c617c1dbf2\" (UID: \"eb58d944-e2ad-4505-a5df-d5c617c1dbf2\") "
Oct 30 20:02:59.184231 ip-10-0-132-82 kubenswrapper[4892]: I1030 20:02:59.184036    4892 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/eb58d944-e2ad-4505-a5df-d5c617c1dbf2-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/eb58d944-e2ad-4505-a5df-d5c617c1dbf2-istio-podinfo\") pod \"eb58d944-e2ad-4505-a5df-d5c617c1dbf2\" (UID: \"eb58d944-e2ad-4505-a5df-d5c617c1dbf2\") "
Oct 30 20:02:59.184231 ip-10-0-132-82 kubenswrapper[4892]: I1030 20:02:59.184070    4892 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/eb58d944-e2ad-4505-a5df-d5c617c1dbf2-istio-data" pod "eb58d944-e2ad-4505-a5df-d5c617c1dbf2" (UID: "eb58d944-e2ad-4505-a5df-d5c617c1dbf2"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 20:02:59.184347 ip-10-0-132-82 kubenswrapper[4892]: I1030 20:02:59.184298    4892 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/eb58d944-e2ad-4505-a5df-d5c617c1dbf2-credential-socket" pod "eb58d944-e2ad-4505-a5df-d5c617c1dbf2" (UID: "eb58d944-e2ad-4505-a5df-d5c617c1dbf2"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 20:02:59.184411 ip-10-0-132-82 kubenswrapper[4892]: I1030 20:02:59.184359    4892 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/eb58d944-e2ad-4505-a5df-d5c617c1dbf2-workload-certs" pod "eb58d944-e2ad-4505-a5df-d5c617c1dbf2" (UID: "eb58d944-e2ad-4505-a5df-d5c617c1dbf2"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 20:02:59.184474 ip-10-0-132-82 kubenswrapper[4892]: I1030 20:02:59.184413    4892 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/eb58d944-e2ad-4505-a5df-d5c617c1dbf2-workload-socket" pod "eb58d944-e2ad-4505-a5df-d5c617c1dbf2" (UID: "eb58d944-e2ad-4505-a5df-d5c617c1dbf2"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 20:02:59.184620 ip-10-0-132-82 kubenswrapper[4892]: I1030 20:02:59.184595    4892 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/eb58d944-e2ad-4505-a5df-d5c617c1dbf2-istiod-ca-cert\") on node \"ip-10-0-132-82.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:02:59.184620 ip-10-0-132-82 kubenswrapper[4892]: I1030 20:02:59.184622    4892 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/eb58d944-e2ad-4505-a5df-d5c617c1dbf2-istio-data\") on node \"ip-10-0-132-82.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:02:59.184844 ip-10-0-132-82 kubenswrapper[4892]: I1030 20:02:59.184636    4892 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/eb58d944-e2ad-4505-a5df-d5c617c1dbf2-credential-socket\") on node \"ip-10-0-132-82.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:02:59.185957 ip-10-0-132-82 kubenswrapper[4892]: I1030 20:02:59.185937    4892 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/eb58d944-e2ad-4505-a5df-d5c617c1dbf2-istio-token" pod "eb58d944-e2ad-4505-a5df-d5c617c1dbf2" (UID: "eb58d944-e2ad-4505-a5df-d5c617c1dbf2"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 20:02:59.186089 ip-10-0-132-82 kubenswrapper[4892]: I1030 20:02:59.186004    4892 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/eb58d944-e2ad-4505-a5df-d5c617c1dbf2-kube-api-access-7qxtg" pod "eb58d944-e2ad-4505-a5df-d5c617c1dbf2" (UID: "eb58d944-e2ad-4505-a5df-d5c617c1dbf2"). InnerVolumeSpecName "kube-api-access-7qxtg". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 20:02:59.186089 ip-10-0-132-82 kubenswrapper[4892]: I1030 20:02:59.186080    4892 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/eb58d944-e2ad-4505-a5df-d5c617c1dbf2-istio-envoy" pod "eb58d944-e2ad-4505-a5df-d5c617c1dbf2" (UID: "eb58d944-e2ad-4505-a5df-d5c617c1dbf2"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 20:02:59.186433 ip-10-0-132-82 kubenswrapper[4892]: I1030 20:02:59.186412    4892 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/eb58d944-e2ad-4505-a5df-d5c617c1dbf2-istio-podinfo" pod "eb58d944-e2ad-4505-a5df-d5c617c1dbf2" (UID: "eb58d944-e2ad-4505-a5df-d5c617c1dbf2"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 20:02:59.285798 ip-10-0-132-82 kubenswrapper[4892]: I1030 20:02:59.285772    4892 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/eb58d944-e2ad-4505-a5df-d5c617c1dbf2-istio-token\") on node \"ip-10-0-132-82.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:02:59.285798 ip-10-0-132-82 kubenswrapper[4892]: I1030 20:02:59.285797    4892 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7qxtg\" (UniqueName: \"kubernetes.io/projected/eb58d944-e2ad-4505-a5df-d5c617c1dbf2-kube-api-access-7qxtg\") on node \"ip-10-0-132-82.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:02:59.286185 ip-10-0-132-82 kubenswrapper[4892]: I1030 20:02:59.285807    4892 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/eb58d944-e2ad-4505-a5df-d5c617c1dbf2-istio-envoy\") on node \"ip-10-0-132-82.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:02:59.286185 ip-10-0-132-82 kubenswrapper[4892]: I1030 20:02:59.285815    4892 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/eb58d944-e2ad-4505-a5df-d5c617c1dbf2-workload-certs\") on node \"ip-10-0-132-82.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:02:59.286185 ip-10-0-132-82 kubenswrapper[4892]: I1030 20:02:59.285825    4892 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/eb58d944-e2ad-4505-a5df-d5c617c1dbf2-workload-socket\") on node \"ip-10-0-132-82.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:02:59.286185 ip-10-0-132-82 kubenswrapper[4892]: I1030 20:02:59.285833    4892 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/eb58d944-e2ad-4505-a5df-d5c617c1dbf2-istio-podinfo\") on node \"ip-10-0-132-82.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:02:59.302313 ip-10-0-132-82 kubenswrapper[4892]: I1030 20:02:59.302286    4892 generic.go:356] "Generic (PLEG): container finished" podID="eb58d944-e2ad-4505-a5df-d5c617c1dbf2" containerID="c20bba13fe09724c6650e876534a4db34e2ba5a5a45fe865f2d69660b9d213d6" exitCode=0
Oct 30 20:02:59.302415 ip-10-0-132-82 kubenswrapper[4892]: I1030 20:02:59.302331    4892 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-ada8d7-openshift-default-5f8b49bdfc-khngd" event={"ID":"eb58d944-e2ad-4505-a5df-d5c617c1dbf2","Type":"ContainerDied","Data":"c20bba13fe09724c6650e876534a4db34e2ba5a5a45fe865f2d69660b9d213d6"}
Oct 30 20:02:59.302415 ip-10-0-132-82 kubenswrapper[4892]: I1030 20:02:59.302352    4892 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-ada8d7-openshift-default-5f8b49bdfc-khngd"
Oct 30 20:02:59.302415 ip-10-0-132-82 kubenswrapper[4892]: I1030 20:02:59.302393    4892 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-ada8d7-openshift-default-5f8b49bdfc-khngd" event={"ID":"eb58d944-e2ad-4505-a5df-d5c617c1dbf2","Type":"ContainerDied","Data":"e0903356a050cf11a595cf8ccfcc67e01f97f287dadb23554b4d22e1c2777400"}
Oct 30 20:02:59.302529 ip-10-0-132-82 kubenswrapper[4892]: I1030 20:02:59.302422    4892 scope.go:122] "RemoveContainer" containerID="c20bba13fe09724c6650e876534a4db34e2ba5a5a45fe865f2d69660b9d213d6"
Oct 30 20:02:59.312747 ip-10-0-132-82 kubenswrapper[4892]: I1030 20:02:59.312727    4892 scope.go:122] "RemoveContainer" containerID="c20bba13fe09724c6650e876534a4db34e2ba5a5a45fe865f2d69660b9d213d6"
Oct 30 20:02:59.313047 ip-10-0-132-82 kubenswrapper[4892]: E1030 20:02:59.313023    4892 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c20bba13fe09724c6650e876534a4db34e2ba5a5a45fe865f2d69660b9d213d6\": container with ID starting with c20bba13fe09724c6650e876534a4db34e2ba5a5a45fe865f2d69660b9d213d6 not found: ID does not exist" containerID="c20bba13fe09724c6650e876534a4db34e2ba5a5a45fe865f2d69660b9d213d6"
Oct 30 20:02:59.313138 ip-10-0-132-82 kubenswrapper[4892]: I1030 20:02:59.313056    4892 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c20bba13fe09724c6650e876534a4db34e2ba5a5a45fe865f2d69660b9d213d6"} err="failed to get container status \"c20bba13fe09724c6650e876534a4db34e2ba5a5a45fe865f2d69660b9d213d6\": rpc error: code = NotFound desc = could not find container \"c20bba13fe09724c6650e876534a4db34e2ba5a5a45fe865f2d69660b9d213d6\": container with ID starting with c20bba13fe09724c6650e876534a4db34e2ba5a5a45fe865f2d69660b9d213d6 not found: ID does not exist"
Oct 30 20:02:59.465866 ip-10-0-132-82 kubenswrapper[4892]: I1030 20:02:59.465791    4892 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-ada8d7-openshift-default-5f8b49bdfc-khngd"]
Oct 30 20:02:59.470212 ip-10-0-132-82 kubenswrapper[4892]: I1030 20:02:59.470188    4892 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w3-ada8d7-openshift-default-5f8b49bdfc-khngd"]
Oct 30 20:03:00.569413 ip-10-0-132-82 kubenswrapper[4892]: I1030 20:03:00.569381    4892 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="eb58d944-e2ad-4505-a5df-d5c617c1dbf2" path="/var/lib/kubelet/pods/eb58d944-e2ad-4505-a5df-d5c617c1dbf2/volumes"
Oct 30 20:03:10.770808 ip-10-0-132-82 kubenswrapper[4892]: I1030 20:03:10.770722    4892 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-fsfvk"]
Oct 30 20:03:24.842818 ip-10-0-132-82 kubenswrapper[4892]: E1030 20:03:24.842783    4892 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-jrm87" containerName="extractor"
Oct 30 20:03:30.580175 ip-10-0-132-82 kubenswrapper[4892]: I1030 20:03:30.580140    4892 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-fsfvk"]
Oct 30 20:03:45.048081 ip-10-0-132-82 kubenswrapper[4892]: I1030 20:03:45.048039    4892 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-aa5947e6-mzxvf"]
Oct 30 20:03:45.097244 ip-10-0-132-82 kubenswrapper[4892]: I1030 20:03:45.097213    4892 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-aa5947e6-mzxvf"]
Oct 30 20:03:46.046002 ip-10-0-132-82 kubenswrapper[4892]: I1030 20:03:46.045954    4892 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-4ad5eb9b-czdvj"]
Oct 30 20:03:46.095497 ip-10-0-132-82 kubenswrapper[4892]: I1030 20:03:46.095419    4892 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-4ad5eb9b-czdvj"]
Oct 30 20:03:46.569020 ip-10-0-132-82 kubenswrapper[4892]: I1030 20:03:46.568989    4892 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="9cea7b8f-5b17-4130-b392-61640b58fa52" path="/var/lib/kubelet/pods/9cea7b8f-5b17-4130-b392-61640b58fa52/volumes"
Oct 30 20:03:46.569469 ip-10-0-132-82 kubenswrapper[4892]: I1030 20:03:46.569449    4892 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a6645cd6-a8c2-42ea-9284-aa7bc3dcac48" path="/var/lib/kubelet/pods/a6645cd6-a8c2-42ea-9284-aa7bc3dcac48/volumes"
Oct 30 20:03:54.843489 ip-10-0-132-82 kubenswrapper[4892]: E1030 20:03:54.843451    4892 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-jrm87" containerName="extractor"
Oct 30 20:03:58.300888 ip-10-0-132-82 kubenswrapper[4892]: I1030 20:03:58.300854    4892 scope.go:122] "RemoveContainer" containerID="71c3d80f18f2a473bf806b9f0168ec3acfc61f9cd43fc4a62570438a1e115425"
Oct 30 20:04:08.711150 ip-10-0-132-82 kubenswrapper[4892]: I1030 20:04:08.711122    4892 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-bb1331d0-pqd49"]
Oct 30 20:04:08.711656 ip-10-0-132-82 kubenswrapper[4892]: I1030 20:04:08.711515    4892 cpu_manager.go:414] "containerMap: removing container" podUID="eb58d944-e2ad-4505-a5df-d5c617c1dbf2" containerName="istio-proxy"
Oct 30 20:04:08.711656 ip-10-0-132-82 kubenswrapper[4892]: I1030 20:04:08.711526    4892 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="eb58d944-e2ad-4505-a5df-d5c617c1dbf2" containerName="istio-proxy"
Oct 30 20:04:08.711656 ip-10-0-132-82 kubenswrapper[4892]: I1030 20:04:08.711549    4892 cpu_manager.go:414] "containerMap: removing container" podUID="10a5c7c8-6146-47b0-9643-bb137cf10bf5" containerName="main"
Oct 30 20:04:08.711656 ip-10-0-132-82 kubenswrapper[4892]: I1030 20:04:08.711555    4892 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="10a5c7c8-6146-47b0-9643-bb137cf10bf5" containerName="main"
Oct 30 20:04:08.711656 ip-10-0-132-82 kubenswrapper[4892]: I1030 20:04:08.711608    4892 memory_manager.go:365] "RemoveStaleState removing state" podUID="eb58d944-e2ad-4505-a5df-d5c617c1dbf2" containerName="istio-proxy"
Oct 30 20:04:08.711656 ip-10-0-132-82 kubenswrapper[4892]: I1030 20:04:08.711618    4892 memory_manager.go:365] "RemoveStaleState removing state" podUID="10a5c7c8-6146-47b0-9643-bb137cf10bf5" containerName="main"
Oct 30 20:04:08.716077 ip-10-0-132-82 kubenswrapper[4892]: I1030 20:04:08.716059    4892 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-bb1331d0-pqd49"
Oct 30 20:04:08.810663 ip-10-0-132-82 kubenswrapper[4892]: I1030 20:04:08.810635    4892 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 30 20:04:08.817409 ip-10-0-132-82 kubenswrapper[4892]: I1030 20:04:08.817382    4892 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 20:04:08.817536 ip-10-0-132-82 kubenswrapper[4892]: I1030 20:04:08.817463    4892 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-nlxp4\""
Oct 30 20:04:08.817536 ip-10-0-132-82 kubenswrapper[4892]: I1030 20:04:08.817498    4892 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 30 20:04:08.820263 ip-10-0-132-82 kubenswrapper[4892]: I1030 20:04:08.820243    4892 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-bb1331d0-pqd49"]
Oct 30 20:04:08.907259 ip-10-0-132-82 kubenswrapper[4892]: I1030 20:04:08.907215    4892 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-w66t6\" (UniqueName: \"kubernetes.io/projected/c849e37b-a605-43c3-a628-5fa823c378dd-kube-api-access-w66t6\") pod \"maas-api-revoke-sub-bb1331d0-pqd49\" (UID: \"c849e37b-a605-43c3-a628-5fa823c378dd\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-bb1331d0-pqd49"
Oct 30 20:04:08.907401 ip-10-0-132-82 kubenswrapper[4892]: I1030 20:04:08.907288    4892 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c849e37b-a605-43c3-a628-5fa823c378dd-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-bb1331d0-pqd49\" (UID: \"c849e37b-a605-43c3-a628-5fa823c378dd\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-bb1331d0-pqd49"
Oct 30 20:04:08.907401 ip-10-0-132-82 kubenswrapper[4892]: I1030 20:04:08.907339    4892 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c849e37b-a605-43c3-a628-5fa823c378dd-maas-api-xks-ca\") pod \"maas-api-revoke-sub-bb1331d0-pqd49\" (UID: \"c849e37b-a605-43c3-a628-5fa823c378dd\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-bb1331d0-pqd49"
Oct 30 20:04:09.008816 ip-10-0-132-82 kubenswrapper[4892]: I1030 20:04:09.008751    4892 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-w66t6\" (UniqueName: \"kubernetes.io/projected/c849e37b-a605-43c3-a628-5fa823c378dd-kube-api-access-w66t6\") pod \"maas-api-revoke-sub-bb1331d0-pqd49\" (UID: \"c849e37b-a605-43c3-a628-5fa823c378dd\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-bb1331d0-pqd49"
Oct 30 20:04:09.008816 ip-10-0-132-82 kubenswrapper[4892]: I1030 20:04:09.008786    4892 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c849e37b-a605-43c3-a628-5fa823c378dd-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-bb1331d0-pqd49\" (UID: \"c849e37b-a605-43c3-a628-5fa823c378dd\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-bb1331d0-pqd49"
Oct 30 20:04:09.008816 ip-10-0-132-82 kubenswrapper[4892]: I1030 20:04:09.008813    4892 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c849e37b-a605-43c3-a628-5fa823c378dd-maas-api-xks-ca\") pod \"maas-api-revoke-sub-bb1331d0-pqd49\" (UID: \"c849e37b-a605-43c3-a628-5fa823c378dd\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-bb1331d0-pqd49"
Oct 30 20:04:09.009417 ip-10-0-132-82 kubenswrapper[4892]: I1030 20:04:09.009395    4892 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c849e37b-a605-43c3-a628-5fa823c378dd-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-bb1331d0-pqd49\" (UID: \"c849e37b-a605-43c3-a628-5fa823c378dd\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-bb1331d0-pqd49"
Oct 30 20:04:09.011290 ip-10-0-132-82 kubenswrapper[4892]: I1030 20:04:09.011274    4892 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c849e37b-a605-43c3-a628-5fa823c378dd-maas-api-xks-ca\") pod \"maas-api-revoke-sub-bb1331d0-pqd49\" (UID: \"c849e37b-a605-43c3-a628-5fa823c378dd\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-bb1331d0-pqd49"
Oct 30 20:04:09.065345 ip-10-0-132-82 kubenswrapper[4892]: I1030 20:04:09.065311    4892 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-w66t6\" (UniqueName: \"kubernetes.io/projected/c849e37b-a605-43c3-a628-5fa823c378dd-kube-api-access-w66t6\") pod \"maas-api-revoke-sub-bb1331d0-pqd49\" (UID: \"c849e37b-a605-43c3-a628-5fa823c378dd\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-bb1331d0-pqd49"
Oct 30 20:04:09.328981 ip-10-0-132-82 kubenswrapper[4892]: I1030 20:04:09.328888    4892 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-bb1331d0-pqd49"
Oct 30 20:04:09.704411 ip-10-0-132-82 kubenswrapper[4892]: I1030 20:04:09.704380    4892 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-bb1331d0-pqd49"]
Oct 30 20:04:09.705218 ip-10-0-132-82 kubenswrapper[4892]: W1030 20:04:09.705180    4892 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc849e37b_a605_43c3_a628_5fa823c378dd.slice/crio-66efe679484085d5581064075f36a2bee606cb1447ec8438bdfb859a6f2d0506 WatchSource:0}: Error finding container 66efe679484085d5581064075f36a2bee606cb1447ec8438bdfb859a6f2d0506: Status 404 returned error can't find the container with id 66efe679484085d5581064075f36a2bee606cb1447ec8438bdfb859a6f2d0506
Oct 30 20:04:09.753631 ip-10-0-132-82 kubenswrapper[4892]: I1030 20:04:09.753608    4892 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-bb1331d0-pqd49" event={"ID":"c849e37b-a605-43c3-a628-5fa823c378dd","Type":"ContainerStarted","Data":"66efe679484085d5581064075f36a2bee606cb1447ec8438bdfb859a6f2d0506"}
Oct 30 20:04:10.761610 ip-10-0-132-82 kubenswrapper[4892]: I1030 20:04:10.761571    4892 generic.go:356] "Generic (PLEG): container finished" podID="c849e37b-a605-43c3-a628-5fa823c378dd" containerID="42d989106da7bedd9a59ac5f5ca2d0dd2b2e56ec32d2d94cdb635446434e4541" exitCode=0
Oct 30 20:04:10.761992 ip-10-0-132-82 kubenswrapper[4892]: I1030 20:04:10.761654    4892 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-bb1331d0-pqd49" event={"ID":"c849e37b-a605-43c3-a628-5fa823c378dd","Type":"ContainerDied","Data":"42d989106da7bedd9a59ac5f5ca2d0dd2b2e56ec32d2d94cdb635446434e4541"}
Oct 30 20:04:11.904734 ip-10-0-132-82 kubenswrapper[4892]: I1030 20:04:11.904712    4892 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-bb1331d0-pqd49"
Oct 30 20:04:12.037418 ip-10-0-132-82 kubenswrapper[4892]: I1030 20:04:12.037322    4892 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c849e37b-a605-43c3-a628-5fa823c378dd-kube-api-access-w66t6\" (UniqueName: \"kubernetes.io/projected/c849e37b-a605-43c3-a628-5fa823c378dd-kube-api-access-w66t6\") pod \"c849e37b-a605-43c3-a628-5fa823c378dd\" (UID: \"c849e37b-a605-43c3-a628-5fa823c378dd\") "
Oct 30 20:04:12.037418 ip-10-0-132-82 kubenswrapper[4892]: I1030 20:04:12.037373    4892 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c849e37b-a605-43c3-a628-5fa823c378dd-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c849e37b-a605-43c3-a628-5fa823c378dd-maas-api-openshift-ca\") pod \"c849e37b-a605-43c3-a628-5fa823c378dd\" (UID: \"c849e37b-a605-43c3-a628-5fa823c378dd\") "
Oct 30 20:04:12.037766 ip-10-0-132-82 kubenswrapper[4892]: I1030 20:04:12.037737    4892 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c849e37b-a605-43c3-a628-5fa823c378dd-maas-api-openshift-ca" pod "c849e37b-a605-43c3-a628-5fa823c378dd" (UID: "c849e37b-a605-43c3-a628-5fa823c378dd"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 20:04:12.037766 ip-10-0-132-82 kubenswrapper[4892]: I1030 20:04:12.037752    4892 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/c849e37b-a605-43c3-a628-5fa823c378dd-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c849e37b-a605-43c3-a628-5fa823c378dd-maas-api-xks-ca\") pod \"c849e37b-a605-43c3-a628-5fa823c378dd\" (UID: \"c849e37b-a605-43c3-a628-5fa823c378dd\") "
Oct 30 20:04:12.038332 ip-10-0-132-82 kubenswrapper[4892]: I1030 20:04:12.038307    4892 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c849e37b-a605-43c3-a628-5fa823c378dd-maas-api-openshift-ca\") on node \"ip-10-0-132-82.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:04:12.040268 ip-10-0-132-82 kubenswrapper[4892]: I1030 20:04:12.040238    4892 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c849e37b-a605-43c3-a628-5fa823c378dd-maas-api-xks-ca" pod "c849e37b-a605-43c3-a628-5fa823c378dd" (UID: "c849e37b-a605-43c3-a628-5fa823c378dd"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 20:04:12.040423 ip-10-0-132-82 kubenswrapper[4892]: I1030 20:04:12.040320    4892 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c849e37b-a605-43c3-a628-5fa823c378dd-kube-api-access-w66t6" pod "c849e37b-a605-43c3-a628-5fa823c378dd" (UID: "c849e37b-a605-43c3-a628-5fa823c378dd"). InnerVolumeSpecName "kube-api-access-w66t6". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 20:04:12.139222 ip-10-0-132-82 kubenswrapper[4892]: I1030 20:04:12.139196    4892 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-w66t6\" (UniqueName: \"kubernetes.io/projected/c849e37b-a605-43c3-a628-5fa823c378dd-kube-api-access-w66t6\") on node \"ip-10-0-132-82.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:04:12.139222 ip-10-0-132-82 kubenswrapper[4892]: I1030 20:04:12.139219    4892 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c849e37b-a605-43c3-a628-5fa823c378dd-maas-api-xks-ca\") on node \"ip-10-0-132-82.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:04:12.774775 ip-10-0-132-82 kubenswrapper[4892]: I1030 20:04:12.774747    4892 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-bb1331d0-pqd49"
Oct 30 20:04:12.774932 ip-10-0-132-82 kubenswrapper[4892]: I1030 20:04:12.774782    4892 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-bb1331d0-pqd49" event={"ID":"c849e37b-a605-43c3-a628-5fa823c378dd","Type":"ContainerDied","Data":"66efe679484085d5581064075f36a2bee606cb1447ec8438bdfb859a6f2d0506"}
Oct 30 20:04:12.774932 ip-10-0-132-82 kubenswrapper[4892]: I1030 20:04:12.774824    4892 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="66efe679484085d5581064075f36a2bee606cb1447ec8438bdfb859a6f2d0506"
Oct 30 20:04:21.622804 ip-10-0-132-82 kubenswrapper[4892]: I1030 20:04:21.622767    4892 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-4fa2da17-4vlhs"]
Oct 30 20:04:21.623331 ip-10-0-132-82 kubenswrapper[4892]: I1030 20:04:21.623187    4892 cpu_manager.go:414] "containerMap: removing container" podUID="c849e37b-a605-43c3-a628-5fa823c378dd" containerName="revoke-keys"
Oct 30 20:04:21.623331 ip-10-0-132-82 kubenswrapper[4892]: I1030 20:04:21.623198    4892 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c849e37b-a605-43c3-a628-5fa823c378dd" containerName="revoke-keys"
Oct 30 20:04:21.623331 ip-10-0-132-82 kubenswrapper[4892]: I1030 20:04:21.623291    4892 memory_manager.go:365] "RemoveStaleState removing state" podUID="c849e37b-a605-43c3-a628-5fa823c378dd" containerName="revoke-keys"
Oct 30 20:04:21.626466 ip-10-0-132-82 kubenswrapper[4892]: I1030 20:04:21.626450    4892 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4fa2da17-4vlhs"
Oct 30 20:04:21.723753 ip-10-0-132-82 kubenswrapper[4892]: I1030 20:04:21.723716    4892 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/69927d30-c42f-4f56-8257-10b2e8f0dba0-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-4fa2da17-4vlhs\" (UID: \"69927d30-c42f-4f56-8257-10b2e8f0dba0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4fa2da17-4vlhs"
Oct 30 20:04:21.723753 ip-10-0-132-82 kubenswrapper[4892]: I1030 20:04:21.723769    4892 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/69927d30-c42f-4f56-8257-10b2e8f0dba0-maas-api-xks-ca\") pod \"maas-api-revoke-sub-4fa2da17-4vlhs\" (UID: \"69927d30-c42f-4f56-8257-10b2e8f0dba0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4fa2da17-4vlhs"
Oct 30 20:04:21.724070 ip-10-0-132-82 kubenswrapper[4892]: I1030 20:04:21.723867    4892 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lgthm\" (UniqueName: \"kubernetes.io/projected/69927d30-c42f-4f56-8257-10b2e8f0dba0-kube-api-access-lgthm\") pod \"maas-api-revoke-sub-4fa2da17-4vlhs\" (UID: \"69927d30-c42f-4f56-8257-10b2e8f0dba0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4fa2da17-4vlhs"
Oct 30 20:04:21.729324 ip-10-0-132-82 kubenswrapper[4892]: I1030 20:04:21.729291    4892 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 30 20:04:21.729493 ip-10-0-132-82 kubenswrapper[4892]: I1030 20:04:21.729346    4892 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 20:04:21.729493 ip-10-0-132-82 kubenswrapper[4892]: I1030 20:04:21.729386    4892 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 30 20:04:21.729493 ip-10-0-132-82 kubenswrapper[4892]: I1030 20:04:21.729388    4892 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-nlxp4\""
Oct 30 20:04:21.736123 ip-10-0-132-82 kubenswrapper[4892]: I1030 20:04:21.736094    4892 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-4fa2da17-4vlhs"]
Oct 30 20:04:21.824715 ip-10-0-132-82 kubenswrapper[4892]: I1030 20:04:21.824678    4892 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/69927d30-c42f-4f56-8257-10b2e8f0dba0-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-4fa2da17-4vlhs\" (UID: \"69927d30-c42f-4f56-8257-10b2e8f0dba0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4fa2da17-4vlhs"
Oct 30 20:04:21.824715 ip-10-0-132-82 kubenswrapper[4892]: I1030 20:04:21.824730    4892 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/69927d30-c42f-4f56-8257-10b2e8f0dba0-maas-api-xks-ca\") pod \"maas-api-revoke-sub-4fa2da17-4vlhs\" (UID: \"69927d30-c42f-4f56-8257-10b2e8f0dba0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4fa2da17-4vlhs"
Oct 30 20:04:21.824920 ip-10-0-132-82 kubenswrapper[4892]: I1030 20:04:21.824848    4892 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lgthm\" (UniqueName: \"kubernetes.io/projected/69927d30-c42f-4f56-8257-10b2e8f0dba0-kube-api-access-lgthm\") pod \"maas-api-revoke-sub-4fa2da17-4vlhs\" (UID: \"69927d30-c42f-4f56-8257-10b2e8f0dba0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4fa2da17-4vlhs"
Oct 30 20:04:21.825354 ip-10-0-132-82 kubenswrapper[4892]: I1030 20:04:21.825329    4892 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/69927d30-c42f-4f56-8257-10b2e8f0dba0-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-4fa2da17-4vlhs\" (UID: \"69927d30-c42f-4f56-8257-10b2e8f0dba0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4fa2da17-4vlhs"
Oct 30 20:04:21.827228 ip-10-0-132-82 kubenswrapper[4892]: I1030 20:04:21.827207    4892 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/69927d30-c42f-4f56-8257-10b2e8f0dba0-maas-api-xks-ca\") pod \"maas-api-revoke-sub-4fa2da17-4vlhs\" (UID: \"69927d30-c42f-4f56-8257-10b2e8f0dba0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4fa2da17-4vlhs"
Oct 30 20:04:21.881562 ip-10-0-132-82 kubenswrapper[4892]: I1030 20:04:21.881496    4892 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lgthm\" (UniqueName: \"kubernetes.io/projected/69927d30-c42f-4f56-8257-10b2e8f0dba0-kube-api-access-lgthm\") pod \"maas-api-revoke-sub-4fa2da17-4vlhs\" (UID: \"69927d30-c42f-4f56-8257-10b2e8f0dba0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4fa2da17-4vlhs"
Oct 30 20:04:21.939204 ip-10-0-132-82 kubenswrapper[4892]: I1030 20:04:21.939178    4892 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4fa2da17-4vlhs"
Oct 30 20:04:22.313306 ip-10-0-132-82 kubenswrapper[4892]: W1030 20:04:22.313278    4892 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod69927d30_c42f_4f56_8257_10b2e8f0dba0.slice/crio-e0c3233d44f3a92fded9a923bed76ec9831bf329bc9d893aff835017081d504d WatchSource:0}: Error finding container e0c3233d44f3a92fded9a923bed76ec9831bf329bc9d893aff835017081d504d: Status 404 returned error can't find the container with id e0c3233d44f3a92fded9a923bed76ec9831bf329bc9d893aff835017081d504d
Oct 30 20:04:22.313998 ip-10-0-132-82 kubenswrapper[4892]: I1030 20:04:22.313950    4892 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-4fa2da17-4vlhs"]
Oct 30 20:04:22.845595 ip-10-0-132-82 kubenswrapper[4892]: I1030 20:04:22.845560    4892 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4fa2da17-4vlhs" event={"ID":"69927d30-c42f-4f56-8257-10b2e8f0dba0","Type":"ContainerStarted","Data":"c684557a881fec919fd89c08eed24ae4dcc12f02ebc4a6e338bc233833c27e56"}
Oct 30 20:04:22.846045 ip-10-0-132-82 kubenswrapper[4892]: I1030 20:04:22.845615    4892 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4fa2da17-4vlhs" event={"ID":"69927d30-c42f-4f56-8257-10b2e8f0dba0","Type":"ContainerStarted","Data":"e0c3233d44f3a92fded9a923bed76ec9831bf329bc9d893aff835017081d504d"}
Oct 30 20:04:22.962375 ip-10-0-132-82 kubenswrapper[4892]: I1030 20:04:22.962290    4892 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4fa2da17-4vlhs" podStartSLOduration=1.962276108 podStartE2EDuration="1.962276108s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 20:04:21 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 20:04:22.958934445 +0000 UTC m=+2910.529400043" watchObservedRunningTime="2026-10-30 20:04:22.962276108 +0000 UTC m=+2910.532741706"
Oct 30 20:04:23.855433 ip-10-0-132-82 kubenswrapper[4892]: I1030 20:04:23.855394    4892 generic.go:356] "Generic (PLEG): container finished" podID="69927d30-c42f-4f56-8257-10b2e8f0dba0" containerID="c684557a881fec919fd89c08eed24ae4dcc12f02ebc4a6e338bc233833c27e56" exitCode=0
Oct 30 20:04:23.855871 ip-10-0-132-82 kubenswrapper[4892]: I1030 20:04:23.855480    4892 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4fa2da17-4vlhs" event={"ID":"69927d30-c42f-4f56-8257-10b2e8f0dba0","Type":"ContainerDied","Data":"c684557a881fec919fd89c08eed24ae4dcc12f02ebc4a6e338bc233833c27e56"}
Oct 30 20:04:24.477259 ip-10-0-132-82 kubenswrapper[4892]: I1030 20:04:24.477228    4892 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-fsfvk"]
Oct 30 20:04:24.843737 ip-10-0-132-82 kubenswrapper[4892]: E1030 20:04:24.843651    4892 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-jrm87" containerName="extractor"
Oct 30 20:04:25.001795 ip-10-0-132-82 kubenswrapper[4892]: I1030 20:04:25.001773    4892 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4fa2da17-4vlhs"
Oct 30 20:04:25.054475 ip-10-0-132-82 kubenswrapper[4892]: I1030 20:04:25.054448    4892 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/69927d30-c42f-4f56-8257-10b2e8f0dba0-kube-api-access-lgthm\" (UniqueName: \"kubernetes.io/projected/69927d30-c42f-4f56-8257-10b2e8f0dba0-kube-api-access-lgthm\") pod \"69927d30-c42f-4f56-8257-10b2e8f0dba0\" (UID: \"69927d30-c42f-4f56-8257-10b2e8f0dba0\") "
Oct 30 20:04:25.054622 ip-10-0-132-82 kubenswrapper[4892]: I1030 20:04:25.054488    4892 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/69927d30-c42f-4f56-8257-10b2e8f0dba0-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/69927d30-c42f-4f56-8257-10b2e8f0dba0-maas-api-xks-ca\") pod \"69927d30-c42f-4f56-8257-10b2e8f0dba0\" (UID: \"69927d30-c42f-4f56-8257-10b2e8f0dba0\") "
Oct 30 20:04:25.054622 ip-10-0-132-82 kubenswrapper[4892]: I1030 20:04:25.054587    4892 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/69927d30-c42f-4f56-8257-10b2e8f0dba0-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/69927d30-c42f-4f56-8257-10b2e8f0dba0-maas-api-openshift-ca\") pod \"69927d30-c42f-4f56-8257-10b2e8f0dba0\" (UID: \"69927d30-c42f-4f56-8257-10b2e8f0dba0\") "
Oct 30 20:04:25.054898 ip-10-0-132-82 kubenswrapper[4892]: I1030 20:04:25.054875    4892 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/69927d30-c42f-4f56-8257-10b2e8f0dba0-maas-api-openshift-ca" pod "69927d30-c42f-4f56-8257-10b2e8f0dba0" (UID: "69927d30-c42f-4f56-8257-10b2e8f0dba0"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 20:04:25.056819 ip-10-0-132-82 kubenswrapper[4892]: I1030 20:04:25.056798    4892 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/69927d30-c42f-4f56-8257-10b2e8f0dba0-maas-api-xks-ca" pod "69927d30-c42f-4f56-8257-10b2e8f0dba0" (UID: "69927d30-c42f-4f56-8257-10b2e8f0dba0"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 20:04:25.056910 ip-10-0-132-82 kubenswrapper[4892]: I1030 20:04:25.056894    4892 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/69927d30-c42f-4f56-8257-10b2e8f0dba0-kube-api-access-lgthm" pod "69927d30-c42f-4f56-8257-10b2e8f0dba0" (UID: "69927d30-c42f-4f56-8257-10b2e8f0dba0"). InnerVolumeSpecName "kube-api-access-lgthm". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 20:04:25.155908 ip-10-0-132-82 kubenswrapper[4892]: I1030 20:04:25.155875    4892 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/69927d30-c42f-4f56-8257-10b2e8f0dba0-maas-api-openshift-ca\") on node \"ip-10-0-132-82.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:04:25.155908 ip-10-0-132-82 kubenswrapper[4892]: I1030 20:04:25.155905    4892 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lgthm\" (UniqueName: \"kubernetes.io/projected/69927d30-c42f-4f56-8257-10b2e8f0dba0-kube-api-access-lgthm\") on node \"ip-10-0-132-82.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:04:25.156097 ip-10-0-132-82 kubenswrapper[4892]: I1030 20:04:25.155920    4892 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/69927d30-c42f-4f56-8257-10b2e8f0dba0-maas-api-xks-ca\") on node \"ip-10-0-132-82.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:04:25.869860 ip-10-0-132-82 kubenswrapper[4892]: I1030 20:04:25.869825    4892 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4fa2da17-4vlhs"
Oct 30 20:04:25.869860 ip-10-0-132-82 kubenswrapper[4892]: I1030 20:04:25.869837    4892 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4fa2da17-4vlhs" event={"ID":"69927d30-c42f-4f56-8257-10b2e8f0dba0","Type":"ContainerDied","Data":"e0c3233d44f3a92fded9a923bed76ec9831bf329bc9d893aff835017081d504d"}
Oct 30 20:04:25.870165 ip-10-0-132-82 kubenswrapper[4892]: I1030 20:04:25.869888    4892 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="e0c3233d44f3a92fded9a923bed76ec9831bf329bc9d893aff835017081d504d"
Oct 30 20:04:43.968853 ip-10-0-132-82 kubenswrapper[4892]: I1030 20:04:43.968771    4892 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-fsfvk"]
Oct 30 20:04:46.988350 ip-10-0-132-82 kubenswrapper[4892]: I1030 20:04:46.988311    4892 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-132119-hcv7q"]
Oct 30 20:04:46.988954 ip-10-0-132-82 kubenswrapper[4892]: I1030 20:04:46.988933    4892 cpu_manager.go:414] "containerMap: removing container" podUID="69927d30-c42f-4f56-8257-10b2e8f0dba0" containerName="revoke-keys"
Oct 30 20:04:46.989096 ip-10-0-132-82 kubenswrapper[4892]: I1030 20:04:46.988956    4892 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="69927d30-c42f-4f56-8257-10b2e8f0dba0" containerName="revoke-keys"
Oct 30 20:04:46.989096 ip-10-0-132-82 kubenswrapper[4892]: I1030 20:04:46.989083    4892 memory_manager.go:365] "RemoveStaleState removing state" podUID="69927d30-c42f-4f56-8257-10b2e8f0dba0" containerName="revoke-keys"
Oct 30 20:04:46.992506 ip-10-0-132-82 kubenswrapper[4892]: I1030 20:04:46.992485    4892 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-132119-hcv7q"
Oct 30 20:04:47.045250 ip-10-0-132-82 kubenswrapper[4892]: I1030 20:04:47.045224    4892 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 20:04:47.054836 ip-10-0-132-82 kubenswrapper[4892]: I1030 20:04:47.054813    4892 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/194a7217-d395-482b-a09d-b5d0b502284a-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-132119-hcv7q\" (UID: \"194a7217-d395-482b-a09d-b5d0b502284a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-132119-hcv7q"
Oct 30 20:04:47.054997 ip-10-0-132-82 kubenswrapper[4892]: I1030 20:04:47.054855    4892 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-n2nkz\" (UniqueName: \"kubernetes.io/projected/194a7217-d395-482b-a09d-b5d0b502284a-kube-api-access-n2nkz\") pod \"maas-api-revoke-keys-e2e-inf-a-132119-hcv7q\" (UID: \"194a7217-d395-482b-a09d-b5d0b502284a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-132119-hcv7q"
Oct 30 20:04:47.054997 ip-10-0-132-82 kubenswrapper[4892]: I1030 20:04:47.054879    4892 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/194a7217-d395-482b-a09d-b5d0b502284a-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-132119-hcv7q\" (UID: \"194a7217-d395-482b-a09d-b5d0b502284a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-132119-hcv7q"
Oct 30 20:04:47.094921 ip-10-0-132-82 kubenswrapper[4892]: I1030 20:04:47.094892    4892 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-nlxp4\""
Oct 30 20:04:47.097865 ip-10-0-132-82 kubenswrapper[4892]: I1030 20:04:47.097842    4892 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-132119-hcv7q"]
Oct 30 20:04:47.102954 ip-10-0-132-82 kubenswrapper[4892]: I1030 20:04:47.102933    4892 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 30 20:04:47.103064 ip-10-0-132-82 kubenswrapper[4892]: I1030 20:04:47.103006    4892 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 30 20:04:47.156482 ip-10-0-132-82 kubenswrapper[4892]: I1030 20:04:47.156453    4892 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/194a7217-d395-482b-a09d-b5d0b502284a-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-132119-hcv7q\" (UID: \"194a7217-d395-482b-a09d-b5d0b502284a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-132119-hcv7q"
Oct 30 20:04:47.156635 ip-10-0-132-82 kubenswrapper[4892]: I1030 20:04:47.156518    4892 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-n2nkz\" (UniqueName: \"kubernetes.io/projected/194a7217-d395-482b-a09d-b5d0b502284a-kube-api-access-n2nkz\") pod \"maas-api-revoke-keys-e2e-inf-a-132119-hcv7q\" (UID: \"194a7217-d395-482b-a09d-b5d0b502284a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-132119-hcv7q"
Oct 30 20:04:47.156848 ip-10-0-132-82 kubenswrapper[4892]: I1030 20:04:47.156813    4892 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/194a7217-d395-482b-a09d-b5d0b502284a-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-132119-hcv7q\" (UID: \"194a7217-d395-482b-a09d-b5d0b502284a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-132119-hcv7q"
Oct 30 20:04:47.157383 ip-10-0-132-82 kubenswrapper[4892]: I1030 20:04:47.157358    4892 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/194a7217-d395-482b-a09d-b5d0b502284a-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-132119-hcv7q\" (UID: \"194a7217-d395-482b-a09d-b5d0b502284a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-132119-hcv7q"
Oct 30 20:04:47.160759 ip-10-0-132-82 kubenswrapper[4892]: I1030 20:04:47.160732    4892 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/194a7217-d395-482b-a09d-b5d0b502284a-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-132119-hcv7q\" (UID: \"194a7217-d395-482b-a09d-b5d0b502284a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-132119-hcv7q"
Oct 30 20:04:47.214923 ip-10-0-132-82 kubenswrapper[4892]: I1030 20:04:47.214887    4892 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-n2nkz\" (UniqueName: \"kubernetes.io/projected/194a7217-d395-482b-a09d-b5d0b502284a-kube-api-access-n2nkz\") pod \"maas-api-revoke-keys-e2e-inf-a-132119-hcv7q\" (UID: \"194a7217-d395-482b-a09d-b5d0b502284a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-132119-hcv7q"
Oct 30 20:04:47.305572 ip-10-0-132-82 kubenswrapper[4892]: I1030 20:04:47.305509    4892 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-132119-hcv7q"
Oct 30 20:04:47.683986 ip-10-0-132-82 kubenswrapper[4892]: I1030 20:04:47.683944    4892 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-132119-hcv7q"]
Oct 30 20:04:47.685097 ip-10-0-132-82 kubenswrapper[4892]: W1030 20:04:47.685075    4892 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod194a7217_d395_482b_a09d_b5d0b502284a.slice/crio-93e6e4ff203bd4bff2211fac947fa64fa58531d1604fc5fd7e2b6c4979687f38 WatchSource:0}: Error finding container 93e6e4ff203bd4bff2211fac947fa64fa58531d1604fc5fd7e2b6c4979687f38: Status 404 returned error can't find the container with id 93e6e4ff203bd4bff2211fac947fa64fa58531d1604fc5fd7e2b6c4979687f38
Oct 30 20:04:48.017395 ip-10-0-132-82 kubenswrapper[4892]: I1030 20:04:48.017361    4892 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-132119-hcv7q" event={"ID":"194a7217-d395-482b-a09d-b5d0b502284a","Type":"ContainerStarted","Data":"bbd2134cee542d7349c51b912a853d84e7a9b31dc6f4c0b12c2d7824045695c4"}
Oct 30 20:04:48.017755 ip-10-0-132-82 kubenswrapper[4892]: I1030 20:04:48.017417    4892 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-132119-hcv7q" event={"ID":"194a7217-d395-482b-a09d-b5d0b502284a","Type":"ContainerStarted","Data":"93e6e4ff203bd4bff2211fac947fa64fa58531d1604fc5fd7e2b6c4979687f38"}
Oct 30 20:04:48.132252 ip-10-0-132-82 kubenswrapper[4892]: I1030 20:04:48.132193    4892 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-132119-hcv7q" podStartSLOduration=2.132176614 podStartE2EDuration="2.132176614s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 20:04:46 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 20:04:48.131715539 +0000 UTC m=+2935.702181151" watchObservedRunningTime="2026-10-30 20:04:48.132176614 +0000 UTC m=+2935.702642214"
Oct 30 20:04:49.025695 ip-10-0-132-82 kubenswrapper[4892]: I1030 20:04:49.025645    4892 generic.go:356] "Generic (PLEG): container finished" podID="194a7217-d395-482b-a09d-b5d0b502284a" containerID="bbd2134cee542d7349c51b912a853d84e7a9b31dc6f4c0b12c2d7824045695c4" exitCode=0
Oct 30 20:04:49.026122 ip-10-0-132-82 kubenswrapper[4892]: I1030 20:04:49.025713    4892 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-132119-hcv7q" event={"ID":"194a7217-d395-482b-a09d-b5d0b502284a","Type":"ContainerDied","Data":"bbd2134cee542d7349c51b912a853d84e7a9b31dc6f4c0b12c2d7824045695c4"}
Oct 30 20:04:50.168847 ip-10-0-132-82 kubenswrapper[4892]: I1030 20:04:50.168824    4892 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-132119-hcv7q"
Oct 30 20:04:50.294232 ip-10-0-132-82 kubenswrapper[4892]: I1030 20:04:50.294143    4892 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/194a7217-d395-482b-a09d-b5d0b502284a-kube-api-access-n2nkz\" (UniqueName: \"kubernetes.io/projected/194a7217-d395-482b-a09d-b5d0b502284a-kube-api-access-n2nkz\") pod \"194a7217-d395-482b-a09d-b5d0b502284a\" (UID: \"194a7217-d395-482b-a09d-b5d0b502284a\") "
Oct 30 20:04:50.294232 ip-10-0-132-82 kubenswrapper[4892]: I1030 20:04:50.294199    4892 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/194a7217-d395-482b-a09d-b5d0b502284a-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/194a7217-d395-482b-a09d-b5d0b502284a-maas-api-xks-ca\") pod \"194a7217-d395-482b-a09d-b5d0b502284a\" (UID: \"194a7217-d395-482b-a09d-b5d0b502284a\") "
Oct 30 20:04:50.294481 ip-10-0-132-82 kubenswrapper[4892]: I1030 20:04:50.294325    4892 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/194a7217-d395-482b-a09d-b5d0b502284a-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/194a7217-d395-482b-a09d-b5d0b502284a-maas-api-openshift-ca\") pod \"194a7217-d395-482b-a09d-b5d0b502284a\" (UID: \"194a7217-d395-482b-a09d-b5d0b502284a\") "
Oct 30 20:04:50.294751 ip-10-0-132-82 kubenswrapper[4892]: I1030 20:04:50.294719    4892 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/194a7217-d395-482b-a09d-b5d0b502284a-maas-api-openshift-ca" pod "194a7217-d395-482b-a09d-b5d0b502284a" (UID: "194a7217-d395-482b-a09d-b5d0b502284a"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 20:04:50.297112 ip-10-0-132-82 kubenswrapper[4892]: I1030 20:04:50.297078    4892 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/194a7217-d395-482b-a09d-b5d0b502284a-kube-api-access-n2nkz" pod "194a7217-d395-482b-a09d-b5d0b502284a" (UID: "194a7217-d395-482b-a09d-b5d0b502284a"). InnerVolumeSpecName "kube-api-access-n2nkz". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 20:04:50.297366 ip-10-0-132-82 kubenswrapper[4892]: I1030 20:04:50.297337    4892 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/194a7217-d395-482b-a09d-b5d0b502284a-maas-api-xks-ca" pod "194a7217-d395-482b-a09d-b5d0b502284a" (UID: "194a7217-d395-482b-a09d-b5d0b502284a"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 20:04:50.395649 ip-10-0-132-82 kubenswrapper[4892]: I1030 20:04:50.395619    4892 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-n2nkz\" (UniqueName: \"kubernetes.io/projected/194a7217-d395-482b-a09d-b5d0b502284a-kube-api-access-n2nkz\") on node \"ip-10-0-132-82.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:04:50.395649 ip-10-0-132-82 kubenswrapper[4892]: I1030 20:04:50.395648    4892 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/194a7217-d395-482b-a09d-b5d0b502284a-maas-api-xks-ca\") on node \"ip-10-0-132-82.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:04:50.395839 ip-10-0-132-82 kubenswrapper[4892]: I1030 20:04:50.395658    4892 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/194a7217-d395-482b-a09d-b5d0b502284a-maas-api-openshift-ca\") on node \"ip-10-0-132-82.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:04:51.041211 ip-10-0-132-82 kubenswrapper[4892]: I1030 20:04:51.041177    4892 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-132119-hcv7q"
Oct 30 20:04:51.041211 ip-10-0-132-82 kubenswrapper[4892]: I1030 20:04:51.041207    4892 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-132119-hcv7q" event={"ID":"194a7217-d395-482b-a09d-b5d0b502284a","Type":"ContainerDied","Data":"93e6e4ff203bd4bff2211fac947fa64fa58531d1604fc5fd7e2b6c4979687f38"}
Oct 30 20:04:51.041496 ip-10-0-132-82 kubenswrapper[4892]: I1030 20:04:51.041251    4892 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="93e6e4ff203bd4bff2211fac947fa64fa58531d1604fc5fd7e2b6c4979687f38"
Oct 30 20:04:54.846045 ip-10-0-132-82 kubenswrapper[4892]: E1030 20:04:54.845990    4892 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-jrm87" containerName="extractor"
Oct 30 20:04:57.673252 ip-10-0-132-82 kubenswrapper[4892]: I1030 20:04:57.673220    4892 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-fsfvk"]
Oct 30 20:04:58.331823 ip-10-0-132-82 kubenswrapper[4892]: I1030 20:04:58.331792    4892 scope.go:122] "RemoveContainer" containerID="55a5d749961896f71efa97ff668654dd9fc0cbd86d1d05b392cdeaa11b803014"
Oct 30 20:04:58.341847 ip-10-0-132-82 kubenswrapper[4892]: I1030 20:04:58.341823    4892 scope.go:122] "RemoveContainer" containerID="24f73394c85811dcf684ed6d069342beafb02e29b236513cd228bac26990d25c"
Oct 30 20:04:58.353680 ip-10-0-132-82 kubenswrapper[4892]: I1030 20:04:58.352123    4892 scope.go:122] "RemoveContainer" containerID="463a0af2ea6209b33957bc2bd5d1479e0d6e5e2bb760c91a99e705037c7e615e"
Oct 30 20:04:58.362192 ip-10-0-132-82 kubenswrapper[4892]: I1030 20:04:58.362173    4892 scope.go:122] "RemoveContainer" containerID="671d0bb085a20c8e3af6c8ccaedfe450644082e5b869d96f66122f9a135e06ff"
Oct 30 20:04:58.370700 ip-10-0-132-82 kubenswrapper[4892]: I1030 20:04:58.370679    4892 scope.go:122] "RemoveContainer" containerID="5ceb7015507274cf3f097cdae638b98af2af71ed4bf059c264bdffa1e54c706a"
Oct 30 20:04:58.378851 ip-10-0-132-82 kubenswrapper[4892]: I1030 20:04:58.378834    4892 scope.go:122] "RemoveContainer" containerID="4c714a6cddb76596b4c14a02ef169db62458392f8d937ca7941d018953404f73"
Oct 30 20:04:58.677819 ip-10-0-132-82 kubenswrapper[4892]: I1030 20:04:58.677791    4892 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-132119-79464cf78b-w2p5b"]
Oct 30 20:04:58.785239 ip-10-0-132-82 kubenswrapper[4892]: I1030 20:04:58.785202    4892 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-132119-6b76cf689f-tk59n"]
Oct 30 20:05:03.678928 ip-10-0-132-82 kubenswrapper[4892]: I1030 20:05:03.678889    4892 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-inf-a-132119-79464cf78b-w2p5b" podUID="c63fdd71-1e7d-4c3d-8946-a20291b4f6b8" containerName="payload-processing" containerID="cri-o://ddc641c3b05923c27aa337967673403bad9627b19af544efb0d32025429bbb67" gracePeriod=25
Oct 30 20:05:03.785937 ip-10-0-132-82 kubenswrapper[4892]: I1030 20:05:03.785890    4892 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-132119-6b76cf689f-tk59n" podUID="60856d7a-fd76-42e9-ac9f-c28861d49701" containerName="payload-pre-processing" containerID="cri-o://5814939db6bc91961288ac2cef5bf2d36b4045fd5b06af2fccc45d3bb6032ac6" gracePeriod=25
Oct 30 20:05:04.118508 ip-10-0-132-82 kubenswrapper[4892]: I1030 20:05:04.118414    4892 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-132119-6b76cf689f-tk59n" podUID="60856d7a-fd76-42e9-ac9f-c28861d49701" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 30 20:05:04.118508 ip-10-0-132-82 kubenswrapper[4892]: I1030 20:05:04.118427    4892 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-inf-a-132119-79464cf78b-w2p5b" podUID="c63fdd71-1e7d-4c3d-8946-a20291b4f6b8" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 30 20:05:14.118516 ip-10-0-132-82 kubenswrapper[4892]: I1030 20:05:14.118474    4892 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-132119-6b76cf689f-tk59n" podUID="60856d7a-fd76-42e9-ac9f-c28861d49701" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 30 20:05:14.118954 ip-10-0-132-82 kubenswrapper[4892]: I1030 20:05:14.118475    4892 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-inf-a-132119-79464cf78b-w2p5b" podUID="c63fdd71-1e7d-4c3d-8946-a20291b4f6b8" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 30 20:05:14.724249 ip-10-0-132-82 kubenswrapper[4892]: I1030 20:05:14.724205    4892 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-132119-hcv7q"]
Oct 30 20:05:14.773258 ip-10-0-132-82 kubenswrapper[4892]: I1030 20:05:14.773232    4892 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-132119-hcv7q"]
Oct 30 20:05:16.568639 ip-10-0-132-82 kubenswrapper[4892]: I1030 20:05:16.568604    4892 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="194a7217-d395-482b-a09d-b5d0b502284a" path="/var/lib/kubelet/pods/194a7217-d395-482b-a09d-b5d0b502284a/volumes"
Oct 30 20:05:17.966817 ip-10-0-132-82 kubenswrapper[4892]: I1030 20:05:17.966783    4892 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-fsfvk"]
Oct 30 20:05:23.935875 ip-10-0-132-82 kubenswrapper[4892]: I1030 20:05:23.935812    4892 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-a-132119-79464cf78b-w2p5b"
Oct 30 20:05:24.010585 ip-10-0-132-82 kubenswrapper[4892]: I1030 20:05:24.010550    4892 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c63fdd71-1e7d-4c3d-8946-a20291b4f6b8-kube-api-access-v9ssq\" (UniqueName: \"kubernetes.io/projected/c63fdd71-1e7d-4c3d-8946-a20291b4f6b8-kube-api-access-v9ssq\") pod \"c63fdd71-1e7d-4c3d-8946-a20291b4f6b8\" (UID: \"c63fdd71-1e7d-4c3d-8946-a20291b4f6b8\") "
Oct 30 20:05:24.010778 ip-10-0-132-82 kubenswrapper[4892]: I1030 20:05:24.010674    4892 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c63fdd71-1e7d-4c3d-8946-a20291b4f6b8-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c63fdd71-1e7d-4c3d-8946-a20291b4f6b8-plugins-config-volume\") pod \"c63fdd71-1e7d-4c3d-8946-a20291b4f6b8\" (UID: \"c63fdd71-1e7d-4c3d-8946-a20291b4f6b8\") "
Oct 30 20:05:24.011077 ip-10-0-132-82 kubenswrapper[4892]: I1030 20:05:24.011056    4892 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c63fdd71-1e7d-4c3d-8946-a20291b4f6b8-plugins-config-volume" pod "c63fdd71-1e7d-4c3d-8946-a20291b4f6b8" (UID: "c63fdd71-1e7d-4c3d-8946-a20291b4f6b8"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 20:05:24.012735 ip-10-0-132-82 kubenswrapper[4892]: I1030 20:05:24.012702    4892 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c63fdd71-1e7d-4c3d-8946-a20291b4f6b8-kube-api-access-v9ssq" pod "c63fdd71-1e7d-4c3d-8946-a20291b4f6b8" (UID: "c63fdd71-1e7d-4c3d-8946-a20291b4f6b8"). InnerVolumeSpecName "kube-api-access-v9ssq". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 20:05:24.027675 ip-10-0-132-82 kubenswrapper[4892]: I1030 20:05:24.027655    4892 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-132119-6b76cf689f-tk59n"
Oct 30 20:05:24.112065 ip-10-0-132-82 kubenswrapper[4892]: I1030 20:05:24.112034    4892 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/60856d7a-fd76-42e9-ac9f-c28861d49701-kube-api-access-s6kk6\" (UniqueName: \"kubernetes.io/projected/60856d7a-fd76-42e9-ac9f-c28861d49701-kube-api-access-s6kk6\") pod \"60856d7a-fd76-42e9-ac9f-c28861d49701\" (UID: \"60856d7a-fd76-42e9-ac9f-c28861d49701\") "
Oct 30 20:05:24.112253 ip-10-0-132-82 kubenswrapper[4892]: I1030 20:05:24.112106    4892 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/60856d7a-fd76-42e9-ac9f-c28861d49701-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/60856d7a-fd76-42e9-ac9f-c28861d49701-plugins-config-volume\") pod \"60856d7a-fd76-42e9-ac9f-c28861d49701\" (UID: \"60856d7a-fd76-42e9-ac9f-c28861d49701\") "
Oct 30 20:05:24.112297 ip-10-0-132-82 kubenswrapper[4892]: I1030 20:05:24.112284    4892 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-v9ssq\" (UniqueName: \"kubernetes.io/projected/c63fdd71-1e7d-4c3d-8946-a20291b4f6b8-kube-api-access-v9ssq\") on node \"ip-10-0-132-82.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:05:24.112338 ip-10-0-132-82 kubenswrapper[4892]: I1030 20:05:24.112296    4892 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c63fdd71-1e7d-4c3d-8946-a20291b4f6b8-plugins-config-volume\") on node \"ip-10-0-132-82.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:05:24.112475 ip-10-0-132-82 kubenswrapper[4892]: I1030 20:05:24.112453    4892 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/60856d7a-fd76-42e9-ac9f-c28861d49701-plugins-config-volume" pod "60856d7a-fd76-42e9-ac9f-c28861d49701" (UID: "60856d7a-fd76-42e9-ac9f-c28861d49701"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 20:05:24.114348 ip-10-0-132-82 kubenswrapper[4892]: I1030 20:05:24.114324    4892 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/60856d7a-fd76-42e9-ac9f-c28861d49701-kube-api-access-s6kk6" pod "60856d7a-fd76-42e9-ac9f-c28861d49701" (UID: "60856d7a-fd76-42e9-ac9f-c28861d49701"). InnerVolumeSpecName "kube-api-access-s6kk6". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 20:05:24.213428 ip-10-0-132-82 kubenswrapper[4892]: I1030 20:05:24.213358    4892 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-s6kk6\" (UniqueName: \"kubernetes.io/projected/60856d7a-fd76-42e9-ac9f-c28861d49701-kube-api-access-s6kk6\") on node \"ip-10-0-132-82.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:05:24.213428 ip-10-0-132-82 kubenswrapper[4892]: I1030 20:05:24.213387    4892 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/60856d7a-fd76-42e9-ac9f-c28861d49701-plugins-config-volume\") on node \"ip-10-0-132-82.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:05:24.266487 ip-10-0-132-82 kubenswrapper[4892]: I1030 20:05:24.266453    4892 generic.go:356] "Generic (PLEG): container finished" podID="c63fdd71-1e7d-4c3d-8946-a20291b4f6b8" containerID="ddc641c3b05923c27aa337967673403bad9627b19af544efb0d32025429bbb67" exitCode=0
Oct 30 20:05:24.266633 ip-10-0-132-82 kubenswrapper[4892]: I1030 20:05:24.266500    4892 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-a-132119-79464cf78b-w2p5b"
Oct 30 20:05:24.266633 ip-10-0-132-82 kubenswrapper[4892]: I1030 20:05:24.266537    4892 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-a-132119-79464cf78b-w2p5b" event={"ID":"c63fdd71-1e7d-4c3d-8946-a20291b4f6b8","Type":"ContainerDied","Data":"ddc641c3b05923c27aa337967673403bad9627b19af544efb0d32025429bbb67"}
Oct 30 20:05:24.266738 ip-10-0-132-82 kubenswrapper[4892]: I1030 20:05:24.266636    4892 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-a-132119-79464cf78b-w2p5b" event={"ID":"c63fdd71-1e7d-4c3d-8946-a20291b4f6b8","Type":"ContainerDied","Data":"392c8e9ebba7fae354ffcd589b4a87ddc9c3b79f4b9ea946a2ded0402c8df98a"}
Oct 30 20:05:24.266738 ip-10-0-132-82 kubenswrapper[4892]: I1030 20:05:24.266654    4892 scope.go:122] "RemoveContainer" containerID="ddc641c3b05923c27aa337967673403bad9627b19af544efb0d32025429bbb67"
Oct 30 20:05:24.268026 ip-10-0-132-82 kubenswrapper[4892]: I1030 20:05:24.268006    4892 generic.go:356] "Generic (PLEG): container finished" podID="60856d7a-fd76-42e9-ac9f-c28861d49701" containerID="5814939db6bc91961288ac2cef5bf2d36b4045fd5b06af2fccc45d3bb6032ac6" exitCode=0
Oct 30 20:05:24.268132 ip-10-0-132-82 kubenswrapper[4892]: I1030 20:05:24.268073    4892 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-132119-6b76cf689f-tk59n" event={"ID":"60856d7a-fd76-42e9-ac9f-c28861d49701","Type":"ContainerDied","Data":"5814939db6bc91961288ac2cef5bf2d36b4045fd5b06af2fccc45d3bb6032ac6"}
Oct 30 20:05:24.268132 ip-10-0-132-82 kubenswrapper[4892]: I1030 20:05:24.268096    4892 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-132119-6b76cf689f-tk59n"
Oct 30 20:05:24.268132 ip-10-0-132-82 kubenswrapper[4892]: I1030 20:05:24.268108    4892 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-132119-6b76cf689f-tk59n" event={"ID":"60856d7a-fd76-42e9-ac9f-c28861d49701","Type":"ContainerDied","Data":"1975dc48a88d0e50c89aab6af7c856ef96b1c0b055beb84348d5ddcdd1f77146"}
Oct 30 20:05:24.276901 ip-10-0-132-82 kubenswrapper[4892]: I1030 20:05:24.276778    4892 scope.go:122] "RemoveContainer" containerID="ddc641c3b05923c27aa337967673403bad9627b19af544efb0d32025429bbb67"
Oct 30 20:05:24.277251 ip-10-0-132-82 kubenswrapper[4892]: E1030 20:05:24.277228    4892 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ddc641c3b05923c27aa337967673403bad9627b19af544efb0d32025429bbb67\": container with ID starting with ddc641c3b05923c27aa337967673403bad9627b19af544efb0d32025429bbb67 not found: ID does not exist" containerID="ddc641c3b05923c27aa337967673403bad9627b19af544efb0d32025429bbb67"
Oct 30 20:05:24.277351 ip-10-0-132-82 kubenswrapper[4892]: I1030 20:05:24.277261    4892 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ddc641c3b05923c27aa337967673403bad9627b19af544efb0d32025429bbb67"} err="failed to get container status \"ddc641c3b05923c27aa337967673403bad9627b19af544efb0d32025429bbb67\": rpc error: code = NotFound desc = could not find container \"ddc641c3b05923c27aa337967673403bad9627b19af544efb0d32025429bbb67\": container with ID starting with ddc641c3b05923c27aa337967673403bad9627b19af544efb0d32025429bbb67 not found: ID does not exist"
Oct 30 20:05:24.277351 ip-10-0-132-82 kubenswrapper[4892]: I1030 20:05:24.277285    4892 scope.go:122] "RemoveContainer" containerID="5814939db6bc91961288ac2cef5bf2d36b4045fd5b06af2fccc45d3bb6032ac6"
Oct 30 20:05:24.286375 ip-10-0-132-82 kubenswrapper[4892]: I1030 20:05:24.286359    4892 scope.go:122] "RemoveContainer" containerID="5814939db6bc91961288ac2cef5bf2d36b4045fd5b06af2fccc45d3bb6032ac6"
Oct 30 20:05:24.286637 ip-10-0-132-82 kubenswrapper[4892]: E1030 20:05:24.286617    4892 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5814939db6bc91961288ac2cef5bf2d36b4045fd5b06af2fccc45d3bb6032ac6\": container with ID starting with 5814939db6bc91961288ac2cef5bf2d36b4045fd5b06af2fccc45d3bb6032ac6 not found: ID does not exist" containerID="5814939db6bc91961288ac2cef5bf2d36b4045fd5b06af2fccc45d3bb6032ac6"
Oct 30 20:05:24.286721 ip-10-0-132-82 kubenswrapper[4892]: I1030 20:05:24.286643    4892 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5814939db6bc91961288ac2cef5bf2d36b4045fd5b06af2fccc45d3bb6032ac6"} err="failed to get container status \"5814939db6bc91961288ac2cef5bf2d36b4045fd5b06af2fccc45d3bb6032ac6\": rpc error: code = NotFound desc = could not find container \"5814939db6bc91961288ac2cef5bf2d36b4045fd5b06af2fccc45d3bb6032ac6\": container with ID starting with 5814939db6bc91961288ac2cef5bf2d36b4045fd5b06af2fccc45d3bb6032ac6 not found: ID does not exist"
Oct 30 20:05:24.432068 ip-10-0-132-82 kubenswrapper[4892]: I1030 20:05:24.431910    4892 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-132119-6b76cf689f-tk59n"]
Oct 30 20:05:24.435906 ip-10-0-132-82 kubenswrapper[4892]: I1030 20:05:24.435877    4892 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-132119-6b76cf689f-tk59n"]
Oct 30 20:05:24.546492 ip-10-0-132-82 kubenswrapper[4892]: I1030 20:05:24.546419    4892 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-132119-79464cf78b-w2p5b"]
Oct 30 20:05:24.550473 ip-10-0-132-82 kubenswrapper[4892]: I1030 20:05:24.550456    4892 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-132119-79464cf78b-w2p5b"]
Oct 30 20:05:24.570913 ip-10-0-132-82 kubenswrapper[4892]: I1030 20:05:24.570882    4892 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="60856d7a-fd76-42e9-ac9f-c28861d49701" path="/var/lib/kubelet/pods/60856d7a-fd76-42e9-ac9f-c28861d49701/volumes"
Oct 30 20:05:24.571442 ip-10-0-132-82 kubenswrapper[4892]: I1030 20:05:24.571424    4892 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c63fdd71-1e7d-4c3d-8946-a20291b4f6b8" path="/var/lib/kubelet/pods/c63fdd71-1e7d-4c3d-8946-a20291b4f6b8/volumes"
Oct 30 20:05:24.848453 ip-10-0-132-82 kubenswrapper[4892]: E1030 20:05:24.848368    4892 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-jrm87" containerName="extractor"
Oct 30 20:05:31.802048 ip-10-0-132-82 kubenswrapper[4892]: I1030 20:05:31.802015    4892 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-a-132119-openshift-default-849b46588f-29f2x"]
Oct 30 20:05:31.802574 ip-10-0-132-82 kubenswrapper[4892]: I1030 20:05:31.802207    4892 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-inf-a-132119-openshift-default-849b46588f-29f2x" podUID="c76fc24b-0fe2-490e-8967-9a5eb4e9ce12" containerName="istio-proxy" containerID="cri-o://25d1a04e483f98b8c00b859aaec861ddeae787e748fe9b510c5cbfc99be258c3" gracePeriod=30
Oct 30 20:05:32.447880 ip-10-0-132-82 kubenswrapper[4892]: I1030 20:05:32.447843    4892 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-f8a9d8-kj6zz"]
Oct 30 20:05:32.448297 ip-10-0-132-82 kubenswrapper[4892]: I1030 20:05:32.448283    4892 cpu_manager.go:414] "containerMap: removing container" podUID="60856d7a-fd76-42e9-ac9f-c28861d49701" containerName="payload-pre-processing"
Oct 30 20:05:32.448349 ip-10-0-132-82 kubenswrapper[4892]: I1030 20:05:32.448298    4892 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="60856d7a-fd76-42e9-ac9f-c28861d49701" containerName="payload-pre-processing"
Oct 30 20:05:32.448349 ip-10-0-132-82 kubenswrapper[4892]: I1030 20:05:32.448310    4892 cpu_manager.go:414] "containerMap: removing container" podUID="194a7217-d395-482b-a09d-b5d0b502284a" containerName="revoke-keys"
Oct 30 20:05:32.448349 ip-10-0-132-82 kubenswrapper[4892]: I1030 20:05:32.448315    4892 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="194a7217-d395-482b-a09d-b5d0b502284a" containerName="revoke-keys"
Oct 30 20:05:32.448454 ip-10-0-132-82 kubenswrapper[4892]: I1030 20:05:32.448358    4892 cpu_manager.go:414] "containerMap: removing container" podUID="c63fdd71-1e7d-4c3d-8946-a20291b4f6b8" containerName="payload-processing"
Oct 30 20:05:32.448454 ip-10-0-132-82 kubenswrapper[4892]: I1030 20:05:32.448364    4892 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c63fdd71-1e7d-4c3d-8946-a20291b4f6b8" containerName="payload-processing"
Oct 30 20:05:32.448454 ip-10-0-132-82 kubenswrapper[4892]: I1030 20:05:32.448433    4892 memory_manager.go:365] "RemoveStaleState removing state" podUID="c63fdd71-1e7d-4c3d-8946-a20291b4f6b8" containerName="payload-processing"
Oct 30 20:05:32.448454 ip-10-0-132-82 kubenswrapper[4892]: I1030 20:05:32.448443    4892 memory_manager.go:365] "RemoveStaleState removing state" podUID="60856d7a-fd76-42e9-ac9f-c28861d49701" containerName="payload-pre-processing"
Oct 30 20:05:32.448454 ip-10-0-132-82 kubenswrapper[4892]: I1030 20:05:32.448449    4892 memory_manager.go:365] "RemoveStaleState removing state" podUID="194a7217-d395-482b-a09d-b5d0b502284a" containerName="revoke-keys"
Oct 30 20:05:32.453081 ip-10-0-132-82 kubenswrapper[4892]: I1030 20:05:32.453065    4892 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-f8a9d8-kj6zz"
Oct 30 20:05:32.509191 ip-10-0-132-82 kubenswrapper[4892]: I1030 20:05:32.509163    4892 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-f8a9d8-kj6zz"]
Oct 30 20:05:32.554747 ip-10-0-132-82 kubenswrapper[4892]: I1030 20:05:32.554717    4892 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 20:05:32.554893 ip-10-0-132-82 kubenswrapper[4892]: I1030 20:05:32.554790    4892 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 30 20:05:32.554893 ip-10-0-132-82 kubenswrapper[4892]: I1030 20:05:32.554814    4892 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 30 20:05:32.554893 ip-10-0-132-82 kubenswrapper[4892]: I1030 20:05:32.554877    4892 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-nlxp4\""
Oct 30 20:05:32.600576 ip-10-0-132-82 kubenswrapper[4892]: I1030 20:05:32.600551    4892 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9ef10533-c954-4f94-ab57-1e56838a418f-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-f8a9d8-kj6zz\" (UID: \"9ef10533-c954-4f94-ab57-1e56838a418f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-f8a9d8-kj6zz"
Oct 30 20:05:32.600707 ip-10-0-132-82 kubenswrapper[4892]: I1030 20:05:32.600691    4892 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9ef10533-c954-4f94-ab57-1e56838a418f-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-f8a9d8-kj6zz\" (UID: \"9ef10533-c954-4f94-ab57-1e56838a418f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-f8a9d8-kj6zz"
Oct 30 20:05:32.600754 ip-10-0-132-82 kubenswrapper[4892]: I1030 20:05:32.600726    4892 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vffh8\" (UniqueName: \"kubernetes.io/projected/9ef10533-c954-4f94-ab57-1e56838a418f-kube-api-access-vffh8\") pod \"maas-api-revoke-keys-e2e-inf-b-f8a9d8-kj6zz\" (UID: \"9ef10533-c954-4f94-ab57-1e56838a418f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-f8a9d8-kj6zz"
Oct 30 20:05:32.702249 ip-10-0-132-82 kubenswrapper[4892]: I1030 20:05:32.702185    4892 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9ef10533-c954-4f94-ab57-1e56838a418f-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-f8a9d8-kj6zz\" (UID: \"9ef10533-c954-4f94-ab57-1e56838a418f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-f8a9d8-kj6zz"
Oct 30 20:05:32.702249 ip-10-0-132-82 kubenswrapper[4892]: I1030 20:05:32.702217    4892 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vffh8\" (UniqueName: \"kubernetes.io/projected/9ef10533-c954-4f94-ab57-1e56838a418f-kube-api-access-vffh8\") pod \"maas-api-revoke-keys-e2e-inf-b-f8a9d8-kj6zz\" (UID: \"9ef10533-c954-4f94-ab57-1e56838a418f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-f8a9d8-kj6zz"
Oct 30 20:05:32.702458 ip-10-0-132-82 kubenswrapper[4892]: I1030 20:05:32.702264    4892 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9ef10533-c954-4f94-ab57-1e56838a418f-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-f8a9d8-kj6zz\" (UID: \"9ef10533-c954-4f94-ab57-1e56838a418f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-f8a9d8-kj6zz"
Oct 30 20:05:32.702937 ip-10-0-132-82 kubenswrapper[4892]: I1030 20:05:32.702919    4892 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9ef10533-c954-4f94-ab57-1e56838a418f-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-f8a9d8-kj6zz\" (UID: \"9ef10533-c954-4f94-ab57-1e56838a418f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-f8a9d8-kj6zz"
Oct 30 20:05:32.704621 ip-10-0-132-82 kubenswrapper[4892]: I1030 20:05:32.704597    4892 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9ef10533-c954-4f94-ab57-1e56838a418f-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-f8a9d8-kj6zz\" (UID: \"9ef10533-c954-4f94-ab57-1e56838a418f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-f8a9d8-kj6zz"
Oct 30 20:05:32.759577 ip-10-0-132-82 kubenswrapper[4892]: I1030 20:05:32.759547    4892 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vffh8\" (UniqueName: \"kubernetes.io/projected/9ef10533-c954-4f94-ab57-1e56838a418f-kube-api-access-vffh8\") pod \"maas-api-revoke-keys-e2e-inf-b-f8a9d8-kj6zz\" (UID: \"9ef10533-c954-4f94-ab57-1e56838a418f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-f8a9d8-kj6zz"
Oct 30 20:05:32.765316 ip-10-0-132-82 kubenswrapper[4892]: I1030 20:05:32.765295    4892 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-f8a9d8-kj6zz"
Oct 30 20:05:33.138446 ip-10-0-132-82 kubenswrapper[4892]: I1030 20:05:33.138417    4892 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-f8a9d8-kj6zz"]
Oct 30 20:05:33.140049 ip-10-0-132-82 kubenswrapper[4892]: W1030 20:05:33.140025    4892 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9ef10533_c954_4f94_ab57_1e56838a418f.slice/crio-ca28e58c2726eee10c9e9a04c76aabfc13387bff9b4a7b6796a0cdf13abb7c6f WatchSource:0}: Error finding container ca28e58c2726eee10c9e9a04c76aabfc13387bff9b4a7b6796a0cdf13abb7c6f: Status 404 returned error can't find the container with id ca28e58c2726eee10c9e9a04c76aabfc13387bff9b4a7b6796a0cdf13abb7c6f
Oct 30 20:05:33.329931 ip-10-0-132-82 kubenswrapper[4892]: I1030 20:05:33.329890    4892 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-f8a9d8-kj6zz" event={"ID":"9ef10533-c954-4f94-ab57-1e56838a418f","Type":"ContainerStarted","Data":"ca28e58c2726eee10c9e9a04c76aabfc13387bff9b4a7b6796a0cdf13abb7c6f"}
Oct 30 20:05:33.778743 ip-10-0-132-82 kubenswrapper[4892]: I1030 20:05:33.778663    4892 patch_prober.go:29] interesting pod/e2e-inf-a-132119-openshift-default-849b46588f-29f2x container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Oct 30 20:05:33.778743 ip-10-0-132-82 kubenswrapper[4892]: I1030 20:05:33.778719    4892 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-inf-a-132119-openshift-default-849b46588f-29f2x" podUID="c76fc24b-0fe2-490e-8967-9a5eb4e9ce12" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Oct 30 20:05:34.340409 ip-10-0-132-82 kubenswrapper[4892]: I1030 20:05:34.340368    4892 generic.go:356] "Generic (PLEG): container finished" podID="9ef10533-c954-4f94-ab57-1e56838a418f" containerID="2e3e6db3cefd15fc64a0e584121f1e6a4531b52dc855bf457abd6a680b2f09d4" exitCode=0
Oct 30 20:05:34.340815 ip-10-0-132-82 kubenswrapper[4892]: I1030 20:05:34.340453    4892 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-f8a9d8-kj6zz" event={"ID":"9ef10533-c954-4f94-ab57-1e56838a418f","Type":"ContainerDied","Data":"2e3e6db3cefd15fc64a0e584121f1e6a4531b52dc855bf457abd6a680b2f09d4"}
Oct 30 20:05:35.486206 ip-10-0-132-82 kubenswrapper[4892]: I1030 20:05:35.486182    4892 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-f8a9d8-kj6zz"
Oct 30 20:05:35.630625 ip-10-0-132-82 kubenswrapper[4892]: I1030 20:05:35.630550    4892 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/9ef10533-c954-4f94-ab57-1e56838a418f-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9ef10533-c954-4f94-ab57-1e56838a418f-maas-api-xks-ca\") pod \"9ef10533-c954-4f94-ab57-1e56838a418f\" (UID: \"9ef10533-c954-4f94-ab57-1e56838a418f\") "
Oct 30 20:05:35.630625 ip-10-0-132-82 kubenswrapper[4892]: I1030 20:05:35.630614    4892 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/9ef10533-c954-4f94-ab57-1e56838a418f-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9ef10533-c954-4f94-ab57-1e56838a418f-maas-api-openshift-ca\") pod \"9ef10533-c954-4f94-ab57-1e56838a418f\" (UID: \"9ef10533-c954-4f94-ab57-1e56838a418f\") "
Oct 30 20:05:35.630852 ip-10-0-132-82 kubenswrapper[4892]: I1030 20:05:35.630699    4892 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9ef10533-c954-4f94-ab57-1e56838a418f-kube-api-access-vffh8\" (UniqueName: \"kubernetes.io/projected/9ef10533-c954-4f94-ab57-1e56838a418f-kube-api-access-vffh8\") pod \"9ef10533-c954-4f94-ab57-1e56838a418f\" (UID: \"9ef10533-c954-4f94-ab57-1e56838a418f\") "
Oct 30 20:05:35.631072 ip-10-0-132-82 kubenswrapper[4892]: I1030 20:05:35.631035    4892 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9ef10533-c954-4f94-ab57-1e56838a418f-maas-api-openshift-ca" pod "9ef10533-c954-4f94-ab57-1e56838a418f" (UID: "9ef10533-c954-4f94-ab57-1e56838a418f"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 20:05:35.632931 ip-10-0-132-82 kubenswrapper[4892]: I1030 20:05:35.632905    4892 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9ef10533-c954-4f94-ab57-1e56838a418f-maas-api-xks-ca" pod "9ef10533-c954-4f94-ab57-1e56838a418f" (UID: "9ef10533-c954-4f94-ab57-1e56838a418f"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 20:05:35.633091 ip-10-0-132-82 kubenswrapper[4892]: I1030 20:05:35.632982    4892 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9ef10533-c954-4f94-ab57-1e56838a418f-kube-api-access-vffh8" pod "9ef10533-c954-4f94-ab57-1e56838a418f" (UID: "9ef10533-c954-4f94-ab57-1e56838a418f"). InnerVolumeSpecName "kube-api-access-vffh8". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 20:05:35.731544 ip-10-0-132-82 kubenswrapper[4892]: I1030 20:05:35.731518    4892 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vffh8\" (UniqueName: \"kubernetes.io/projected/9ef10533-c954-4f94-ab57-1e56838a418f-kube-api-access-vffh8\") on node \"ip-10-0-132-82.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:05:35.731650 ip-10-0-132-82 kubenswrapper[4892]: I1030 20:05:35.731547    4892 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9ef10533-c954-4f94-ab57-1e56838a418f-maas-api-xks-ca\") on node \"ip-10-0-132-82.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:05:35.731650 ip-10-0-132-82 kubenswrapper[4892]: I1030 20:05:35.731562    4892 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9ef10533-c954-4f94-ab57-1e56838a418f-maas-api-openshift-ca\") on node \"ip-10-0-132-82.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:05:36.355493 ip-10-0-132-82 kubenswrapper[4892]: I1030 20:05:36.355463    4892 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-f8a9d8-kj6zz"
Oct 30 20:05:36.355681 ip-10-0-132-82 kubenswrapper[4892]: I1030 20:05:36.355505    4892 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-f8a9d8-kj6zz" event={"ID":"9ef10533-c954-4f94-ab57-1e56838a418f","Type":"ContainerDied","Data":"ca28e58c2726eee10c9e9a04c76aabfc13387bff9b4a7b6796a0cdf13abb7c6f"}
Oct 30 20:05:36.355681 ip-10-0-132-82 kubenswrapper[4892]: I1030 20:05:36.355560    4892 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="ca28e58c2726eee10c9e9a04c76aabfc13387bff9b4a7b6796a0cdf13abb7c6f"
Oct 30 20:05:37.055457 ip-10-0-132-82 kubenswrapper[4892]: I1030 20:05:37.055434    4892 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-132119-openshift-default-849b46588f-29f2x"
Oct 30 20:05:37.143159 ip-10-0-132-82 kubenswrapper[4892]: I1030 20:05:37.143132    4892 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/c76fc24b-0fe2-490e-8967-9a5eb4e9ce12-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c76fc24b-0fe2-490e-8967-9a5eb4e9ce12-istio-podinfo\") pod \"c76fc24b-0fe2-490e-8967-9a5eb4e9ce12\" (UID: \"c76fc24b-0fe2-490e-8967-9a5eb4e9ce12\") "
Oct 30 20:05:37.143300 ip-10-0-132-82 kubenswrapper[4892]: I1030 20:05:37.143167    4892 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c76fc24b-0fe2-490e-8967-9a5eb4e9ce12-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c76fc24b-0fe2-490e-8967-9a5eb4e9ce12-credential-socket\") pod \"c76fc24b-0fe2-490e-8967-9a5eb4e9ce12\" (UID: \"c76fc24b-0fe2-490e-8967-9a5eb4e9ce12\") "
Oct 30 20:05:37.143300 ip-10-0-132-82 kubenswrapper[4892]: I1030 20:05:37.143186    4892 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c76fc24b-0fe2-490e-8967-9a5eb4e9ce12-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c76fc24b-0fe2-490e-8967-9a5eb4e9ce12-istio-envoy\") pod \"c76fc24b-0fe2-490e-8967-9a5eb4e9ce12\" (UID: \"c76fc24b-0fe2-490e-8967-9a5eb4e9ce12\") "
Oct 30 20:05:37.143300 ip-10-0-132-82 kubenswrapper[4892]: I1030 20:05:37.143204    4892 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c76fc24b-0fe2-490e-8967-9a5eb4e9ce12-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c76fc24b-0fe2-490e-8967-9a5eb4e9ce12-workload-certs\") pod \"c76fc24b-0fe2-490e-8967-9a5eb4e9ce12\" (UID: \"c76fc24b-0fe2-490e-8967-9a5eb4e9ce12\") "
Oct 30 20:05:37.143300 ip-10-0-132-82 kubenswrapper[4892]: I1030 20:05:37.143266    4892 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c76fc24b-0fe2-490e-8967-9a5eb4e9ce12-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c76fc24b-0fe2-490e-8967-9a5eb4e9ce12-istio-data\") pod \"c76fc24b-0fe2-490e-8967-9a5eb4e9ce12\" (UID: \"c76fc24b-0fe2-490e-8967-9a5eb4e9ce12\") "
Oct 30 20:05:37.143300 ip-10-0-132-82 kubenswrapper[4892]: I1030 20:05:37.143288    4892 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c76fc24b-0fe2-490e-8967-9a5eb4e9ce12-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c76fc24b-0fe2-490e-8967-9a5eb4e9ce12-istiod-ca-cert\") pod \"c76fc24b-0fe2-490e-8967-9a5eb4e9ce12\" (UID: \"c76fc24b-0fe2-490e-8967-9a5eb4e9ce12\") "
Oct 30 20:05:37.143585 ip-10-0-132-82 kubenswrapper[4892]: I1030 20:05:37.143325    4892 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c76fc24b-0fe2-490e-8967-9a5eb4e9ce12-kube-api-access-8snqf\" (UniqueName: \"kubernetes.io/projected/c76fc24b-0fe2-490e-8967-9a5eb4e9ce12-kube-api-access-8snqf\") pod \"c76fc24b-0fe2-490e-8967-9a5eb4e9ce12\" (UID: \"c76fc24b-0fe2-490e-8967-9a5eb4e9ce12\") "
Oct 30 20:05:37.143585 ip-10-0-132-82 kubenswrapper[4892]: I1030 20:05:37.143371    4892 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c76fc24b-0fe2-490e-8967-9a5eb4e9ce12-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c76fc24b-0fe2-490e-8967-9a5eb4e9ce12-workload-socket\") pod \"c76fc24b-0fe2-490e-8967-9a5eb4e9ce12\" (UID: \"c76fc24b-0fe2-490e-8967-9a5eb4e9ce12\") "
Oct 30 20:05:37.143585 ip-10-0-132-82 kubenswrapper[4892]: I1030 20:05:37.143421    4892 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c76fc24b-0fe2-490e-8967-9a5eb4e9ce12-istio-token\" (UniqueName: \"kubernetes.io/projected/c76fc24b-0fe2-490e-8967-9a5eb4e9ce12-istio-token\") pod \"c76fc24b-0fe2-490e-8967-9a5eb4e9ce12\" (UID: \"c76fc24b-0fe2-490e-8967-9a5eb4e9ce12\") "
Oct 30 20:05:37.143585 ip-10-0-132-82 kubenswrapper[4892]: I1030 20:05:37.143436    4892 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c76fc24b-0fe2-490e-8967-9a5eb4e9ce12-credential-socket" pod "c76fc24b-0fe2-490e-8967-9a5eb4e9ce12" (UID: "c76fc24b-0fe2-490e-8967-9a5eb4e9ce12"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 20:05:37.143827 ip-10-0-132-82 kubenswrapper[4892]: I1030 20:05:37.143614    4892 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c76fc24b-0fe2-490e-8967-9a5eb4e9ce12-workload-certs" pod "c76fc24b-0fe2-490e-8967-9a5eb4e9ce12" (UID: "c76fc24b-0fe2-490e-8967-9a5eb4e9ce12"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 20:05:37.143827 ip-10-0-132-82 kubenswrapper[4892]: I1030 20:05:37.143695    4892 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c76fc24b-0fe2-490e-8967-9a5eb4e9ce12-credential-socket\") on node \"ip-10-0-132-82.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:05:37.143827 ip-10-0-132-82 kubenswrapper[4892]: I1030 20:05:37.143689    4892 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c76fc24b-0fe2-490e-8967-9a5eb4e9ce12-workload-socket" pod "c76fc24b-0fe2-490e-8967-9a5eb4e9ce12" (UID: "c76fc24b-0fe2-490e-8967-9a5eb4e9ce12"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 20:05:37.144199 ip-10-0-132-82 kubenswrapper[4892]: I1030 20:05:37.143853    4892 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c76fc24b-0fe2-490e-8967-9a5eb4e9ce12-istiod-ca-cert" pod "c76fc24b-0fe2-490e-8967-9a5eb4e9ce12" (UID: "c76fc24b-0fe2-490e-8967-9a5eb4e9ce12"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 20:05:37.144199 ip-10-0-132-82 kubenswrapper[4892]: I1030 20:05:37.143914    4892 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c76fc24b-0fe2-490e-8967-9a5eb4e9ce12-istio-data" pod "c76fc24b-0fe2-490e-8967-9a5eb4e9ce12" (UID: "c76fc24b-0fe2-490e-8967-9a5eb4e9ce12"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 20:05:37.145585 ip-10-0-132-82 kubenswrapper[4892]: I1030 20:05:37.145561    4892 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/c76fc24b-0fe2-490e-8967-9a5eb4e9ce12-istio-podinfo" pod "c76fc24b-0fe2-490e-8967-9a5eb4e9ce12" (UID: "c76fc24b-0fe2-490e-8967-9a5eb4e9ce12"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 20:05:37.145676 ip-10-0-132-82 kubenswrapper[4892]: I1030 20:05:37.145653    4892 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c76fc24b-0fe2-490e-8967-9a5eb4e9ce12-istio-envoy" pod "c76fc24b-0fe2-490e-8967-9a5eb4e9ce12" (UID: "c76fc24b-0fe2-490e-8967-9a5eb4e9ce12"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 20:05:37.146059 ip-10-0-132-82 kubenswrapper[4892]: I1030 20:05:37.146038    4892 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c76fc24b-0fe2-490e-8967-9a5eb4e9ce12-kube-api-access-8snqf" pod "c76fc24b-0fe2-490e-8967-9a5eb4e9ce12" (UID: "c76fc24b-0fe2-490e-8967-9a5eb4e9ce12"). InnerVolumeSpecName "kube-api-access-8snqf". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 20:05:37.146138 ip-10-0-132-82 kubenswrapper[4892]: I1030 20:05:37.146062    4892 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c76fc24b-0fe2-490e-8967-9a5eb4e9ce12-istio-token" pod "c76fc24b-0fe2-490e-8967-9a5eb4e9ce12" (UID: "c76fc24b-0fe2-490e-8967-9a5eb4e9ce12"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 20:05:37.244737 ip-10-0-132-82 kubenswrapper[4892]: I1030 20:05:37.244705    4892 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c76fc24b-0fe2-490e-8967-9a5eb4e9ce12-workload-socket\") on node \"ip-10-0-132-82.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:05:37.244737 ip-10-0-132-82 kubenswrapper[4892]: I1030 20:05:37.244727    4892 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c76fc24b-0fe2-490e-8967-9a5eb4e9ce12-istio-token\") on node \"ip-10-0-132-82.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:05:37.244737 ip-10-0-132-82 kubenswrapper[4892]: I1030 20:05:37.244738    4892 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c76fc24b-0fe2-490e-8967-9a5eb4e9ce12-istio-podinfo\") on node \"ip-10-0-132-82.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:05:37.245043 ip-10-0-132-82 kubenswrapper[4892]: I1030 20:05:37.244748    4892 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c76fc24b-0fe2-490e-8967-9a5eb4e9ce12-istio-envoy\") on node \"ip-10-0-132-82.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:05:37.245043 ip-10-0-132-82 kubenswrapper[4892]: I1030 20:05:37.244759    4892 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c76fc24b-0fe2-490e-8967-9a5eb4e9ce12-workload-certs\") on node \"ip-10-0-132-82.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:05:37.245043 ip-10-0-132-82 kubenswrapper[4892]: I1030 20:05:37.244767    4892 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c76fc24b-0fe2-490e-8967-9a5eb4e9ce12-istio-data\") on node \"ip-10-0-132-82.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:05:37.245043 ip-10-0-132-82 kubenswrapper[4892]: I1030 20:05:37.244775    4892 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c76fc24b-0fe2-490e-8967-9a5eb4e9ce12-istiod-ca-cert\") on node \"ip-10-0-132-82.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:05:37.245043 ip-10-0-132-82 kubenswrapper[4892]: I1030 20:05:37.244783    4892 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8snqf\" (UniqueName: \"kubernetes.io/projected/c76fc24b-0fe2-490e-8967-9a5eb4e9ce12-kube-api-access-8snqf\") on node \"ip-10-0-132-82.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:05:37.363804 ip-10-0-132-82 kubenswrapper[4892]: I1030 20:05:37.363772    4892 generic.go:356] "Generic (PLEG): container finished" podID="c76fc24b-0fe2-490e-8967-9a5eb4e9ce12" containerID="25d1a04e483f98b8c00b859aaec861ddeae787e748fe9b510c5cbfc99be258c3" exitCode=0
Oct 30 20:05:37.363934 ip-10-0-132-82 kubenswrapper[4892]: I1030 20:05:37.363824    4892 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-132119-openshift-default-849b46588f-29f2x"
Oct 30 20:05:37.363934 ip-10-0-132-82 kubenswrapper[4892]: I1030 20:05:37.363856    4892 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-132119-openshift-default-849b46588f-29f2x" event={"ID":"c76fc24b-0fe2-490e-8967-9a5eb4e9ce12","Type":"ContainerDied","Data":"25d1a04e483f98b8c00b859aaec861ddeae787e748fe9b510c5cbfc99be258c3"}
Oct 30 20:05:37.363934 ip-10-0-132-82 kubenswrapper[4892]: I1030 20:05:37.363912    4892 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-132119-openshift-default-849b46588f-29f2x" event={"ID":"c76fc24b-0fe2-490e-8967-9a5eb4e9ce12","Type":"ContainerDied","Data":"218e97154c0674a68bc13085ac7061bbe47bb5693484891d67a20ed4762b356e"}
Oct 30 20:05:37.363934 ip-10-0-132-82 kubenswrapper[4892]: I1030 20:05:37.363934    4892 scope.go:122] "RemoveContainer" containerID="25d1a04e483f98b8c00b859aaec861ddeae787e748fe9b510c5cbfc99be258c3"
Oct 30 20:05:37.374378 ip-10-0-132-82 kubenswrapper[4892]: I1030 20:05:37.374361    4892 scope.go:122] "RemoveContainer" containerID="8104d1feef8b7a9b4f6b6cf7646ea8bfc8897bc7178a480aa99cb6ce22992041"
Oct 30 20:05:37.383280 ip-10-0-132-82 kubenswrapper[4892]: I1030 20:05:37.383264    4892 scope.go:122] "RemoveContainer" containerID="25d1a04e483f98b8c00b859aaec861ddeae787e748fe9b510c5cbfc99be258c3"
Oct 30 20:05:37.383525 ip-10-0-132-82 kubenswrapper[4892]: E1030 20:05:37.383508    4892 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"25d1a04e483f98b8c00b859aaec861ddeae787e748fe9b510c5cbfc99be258c3\": container with ID starting with 25d1a04e483f98b8c00b859aaec861ddeae787e748fe9b510c5cbfc99be258c3 not found: ID does not exist" containerID="25d1a04e483f98b8c00b859aaec861ddeae787e748fe9b510c5cbfc99be258c3"
Oct 30 20:05:37.383587 ip-10-0-132-82 kubenswrapper[4892]: I1030 20:05:37.383532    4892 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"25d1a04e483f98b8c00b859aaec861ddeae787e748fe9b510c5cbfc99be258c3"} err="failed to get container status \"25d1a04e483f98b8c00b859aaec861ddeae787e748fe9b510c5cbfc99be258c3\": rpc error: code = NotFound desc = could not find container \"25d1a04e483f98b8c00b859aaec861ddeae787e748fe9b510c5cbfc99be258c3\": container with ID starting with 25d1a04e483f98b8c00b859aaec861ddeae787e748fe9b510c5cbfc99be258c3 not found: ID does not exist"
Oct 30 20:05:37.383587 ip-10-0-132-82 kubenswrapper[4892]: I1030 20:05:37.383548    4892 scope.go:122] "RemoveContainer" containerID="8104d1feef8b7a9b4f6b6cf7646ea8bfc8897bc7178a480aa99cb6ce22992041"
Oct 30 20:05:37.383813 ip-10-0-132-82 kubenswrapper[4892]: E1030 20:05:37.383793    4892 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8104d1feef8b7a9b4f6b6cf7646ea8bfc8897bc7178a480aa99cb6ce22992041\": container with ID starting with 8104d1feef8b7a9b4f6b6cf7646ea8bfc8897bc7178a480aa99cb6ce22992041 not found: ID does not exist" containerID="8104d1feef8b7a9b4f6b6cf7646ea8bfc8897bc7178a480aa99cb6ce22992041"
Oct 30 20:05:37.383868 ip-10-0-132-82 kubenswrapper[4892]: I1030 20:05:37.383820    4892 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8104d1feef8b7a9b4f6b6cf7646ea8bfc8897bc7178a480aa99cb6ce22992041"} err="failed to get container status \"8104d1feef8b7a9b4f6b6cf7646ea8bfc8897bc7178a480aa99cb6ce22992041\": rpc error: code = NotFound desc = could not find container \"8104d1feef8b7a9b4f6b6cf7646ea8bfc8897bc7178a480aa99cb6ce22992041\": container with ID starting with 8104d1feef8b7a9b4f6b6cf7646ea8bfc8897bc7178a480aa99cb6ce22992041 not found: ID does not exist"
Oct 30 20:05:37.530340 ip-10-0-132-82 kubenswrapper[4892]: I1030 20:05:37.530261    4892 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-a-132119-openshift-default-849b46588f-29f2x"]
Oct 30 20:05:37.533380 ip-10-0-132-82 kubenswrapper[4892]: I1030 20:05:37.533359    4892 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-inf-a-132119-openshift-default-849b46588f-29f2x"]
Oct 30 20:05:38.568495 ip-10-0-132-82 kubenswrapper[4892]: I1030 20:05:38.568462    4892 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c76fc24b-0fe2-490e-8967-9a5eb4e9ce12" path="/var/lib/kubelet/pods/c76fc24b-0fe2-490e-8967-9a5eb4e9ce12/volumes"
Oct 30 20:05:44.134386 ip-10-0-132-82 kubenswrapper[4892]: I1030 20:05:44.134349    4892 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-f8a9d8-d9dccd5b6-h6wsr"]
Oct 30 20:05:44.242208 ip-10-0-132-82 kubenswrapper[4892]: I1030 20:05:44.242175    4892 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-f8a9d8-676dfc58bc-5jxrg"]
Oct 30 20:05:49.085529 ip-10-0-132-82 kubenswrapper[4892]: I1030 20:05:49.085497    4892 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-f8a9d8-kj6zz"]
Oct 30 20:05:49.134800 ip-10-0-132-82 kubenswrapper[4892]: I1030 20:05:49.134763    4892 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-inf-b-f8a9d8-d9dccd5b6-h6wsr" podUID="92e73440-f470-44d9-983e-2d38334d834c" containerName="payload-processing" containerID="cri-o://611900ee659f729ed841e28409759972fefe52ddbbad56aad81db74a646c56dd" gracePeriod=25
Oct 30 20:05:49.134996 ip-10-0-132-82 kubenswrapper[4892]: I1030 20:05:49.134975    4892 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-f8a9d8-kj6zz"]
Oct 30 20:05:49.242911 ip-10-0-132-82 kubenswrapper[4892]: I1030 20:05:49.242844    4892 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-f8a9d8-676dfc58bc-5jxrg" podUID="5f415c4a-4dda-40e3-bdae-8ad67ad15bdf" containerName="payload-pre-processing" containerID="cri-o://e42503fda96d54472d8152bbe0731f0a2c600231e7d4f52c4f092ad4f4aeaf00" gracePeriod=25
Oct 30 20:05:49.404317 ip-10-0-132-82 kubenswrapper[4892]: I1030 20:05:49.404293    4892 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-b-f8a9d8-d9dccd5b6-h6wsr"
Oct 30 20:05:49.448166 ip-10-0-132-82 kubenswrapper[4892]: I1030 20:05:49.448136    4892 generic.go:356] "Generic (PLEG): container finished" podID="92e73440-f470-44d9-983e-2d38334d834c" containerID="611900ee659f729ed841e28409759972fefe52ddbbad56aad81db74a646c56dd" exitCode=0
Oct 30 20:05:49.448335 ip-10-0-132-82 kubenswrapper[4892]: I1030 20:05:49.448182    4892 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-b-f8a9d8-d9dccd5b6-h6wsr"
Oct 30 20:05:49.448335 ip-10-0-132-82 kubenswrapper[4892]: I1030 20:05:49.448214    4892 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-b-f8a9d8-d9dccd5b6-h6wsr" event={"ID":"92e73440-f470-44d9-983e-2d38334d834c","Type":"ContainerDied","Data":"611900ee659f729ed841e28409759972fefe52ddbbad56aad81db74a646c56dd"}
Oct 30 20:05:49.448335 ip-10-0-132-82 kubenswrapper[4892]: I1030 20:05:49.448287    4892 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-b-f8a9d8-d9dccd5b6-h6wsr" event={"ID":"92e73440-f470-44d9-983e-2d38334d834c","Type":"ContainerDied","Data":"edf42999566353b1bbee45376492e373060a4e2db778c978b33af2c74757dbd7"}
Oct 30 20:05:49.448335 ip-10-0-132-82 kubenswrapper[4892]: I1030 20:05:49.448316    4892 scope.go:122] "RemoveContainer" containerID="611900ee659f729ed841e28409759972fefe52ddbbad56aad81db74a646c56dd"
Oct 30 20:05:49.450736 ip-10-0-132-82 kubenswrapper[4892]: I1030 20:05:49.450712    4892 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/92e73440-f470-44d9-983e-2d38334d834c-kube-api-access-4cnpw\" (UniqueName: \"kubernetes.io/projected/92e73440-f470-44d9-983e-2d38334d834c-kube-api-access-4cnpw\") pod \"92e73440-f470-44d9-983e-2d38334d834c\" (UID: \"92e73440-f470-44d9-983e-2d38334d834c\") "
Oct 30 20:05:49.450846 ip-10-0-132-82 kubenswrapper[4892]: I1030 20:05:49.450754    4892 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/92e73440-f470-44d9-983e-2d38334d834c-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/92e73440-f470-44d9-983e-2d38334d834c-plugins-config-volume\") pod \"92e73440-f470-44d9-983e-2d38334d834c\" (UID: \"92e73440-f470-44d9-983e-2d38334d834c\") "
Oct 30 20:05:49.451137 ip-10-0-132-82 kubenswrapper[4892]: I1030 20:05:49.451117    4892 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/92e73440-f470-44d9-983e-2d38334d834c-plugins-config-volume" pod "92e73440-f470-44d9-983e-2d38334d834c" (UID: "92e73440-f470-44d9-983e-2d38334d834c"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 20:05:49.452911 ip-10-0-132-82 kubenswrapper[4892]: I1030 20:05:49.452886    4892 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/92e73440-f470-44d9-983e-2d38334d834c-kube-api-access-4cnpw" pod "92e73440-f470-44d9-983e-2d38334d834c" (UID: "92e73440-f470-44d9-983e-2d38334d834c"). InnerVolumeSpecName "kube-api-access-4cnpw". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 20:05:49.465203 ip-10-0-132-82 kubenswrapper[4892]: I1030 20:05:49.465181    4892 scope.go:122] "RemoveContainer" containerID="611900ee659f729ed841e28409759972fefe52ddbbad56aad81db74a646c56dd"
Oct 30 20:05:49.465533 ip-10-0-132-82 kubenswrapper[4892]: E1030 20:05:49.465511    4892 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"611900ee659f729ed841e28409759972fefe52ddbbad56aad81db74a646c56dd\": container with ID starting with 611900ee659f729ed841e28409759972fefe52ddbbad56aad81db74a646c56dd not found: ID does not exist" containerID="611900ee659f729ed841e28409759972fefe52ddbbad56aad81db74a646c56dd"
Oct 30 20:05:49.465612 ip-10-0-132-82 kubenswrapper[4892]: I1030 20:05:49.465545    4892 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"611900ee659f729ed841e28409759972fefe52ddbbad56aad81db74a646c56dd"} err="failed to get container status \"611900ee659f729ed841e28409759972fefe52ddbbad56aad81db74a646c56dd\": rpc error: code = NotFound desc = could not find container \"611900ee659f729ed841e28409759972fefe52ddbbad56aad81db74a646c56dd\": container with ID starting with 611900ee659f729ed841e28409759972fefe52ddbbad56aad81db74a646c56dd not found: ID does not exist"
Oct 30 20:05:49.552726 ip-10-0-132-82 kubenswrapper[4892]: I1030 20:05:49.552690    4892 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4cnpw\" (UniqueName: \"kubernetes.io/projected/92e73440-f470-44d9-983e-2d38334d834c-kube-api-access-4cnpw\") on node \"ip-10-0-132-82.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:05:49.552726 ip-10-0-132-82 kubenswrapper[4892]: I1030 20:05:49.552725    4892 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/92e73440-f470-44d9-983e-2d38334d834c-plugins-config-volume\") on node \"ip-10-0-132-82.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:05:49.868020 ip-10-0-132-82 kubenswrapper[4892]: I1030 20:05:49.867959    4892 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-f8a9d8-d9dccd5b6-h6wsr"]
Oct 30 20:05:49.870653 ip-10-0-132-82 kubenswrapper[4892]: I1030 20:05:49.870630    4892 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-f8a9d8-d9dccd5b6-h6wsr"]
Oct 30 20:05:50.049849 ip-10-0-132-82 kubenswrapper[4892]: I1030 20:05:50.049767    4892 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-fa928509-4w4hc"]
Oct 30 20:05:50.052138 ip-10-0-132-82 kubenswrapper[4892]: I1030 20:05:50.052117    4892 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-fa928509-4w4hc"]
Oct 30 20:05:50.567691 ip-10-0-132-82 kubenswrapper[4892]: I1030 20:05:50.567656    4892 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="92e73440-f470-44d9-983e-2d38334d834c" path="/var/lib/kubelet/pods/92e73440-f470-44d9-983e-2d38334d834c/volumes"
Oct 30 20:05:50.568105 ip-10-0-132-82 kubenswrapper[4892]: I1030 20:05:50.568075    4892 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="9ef10533-c954-4f94-ab57-1e56838a418f" path="/var/lib/kubelet/pods/9ef10533-c954-4f94-ab57-1e56838a418f/volumes"
Oct 30 20:05:50.568438 ip-10-0-132-82 kubenswrapper[4892]: I1030 20:05:50.568424    4892 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a4f83474-cd19-4c19-b6d5-2914706168e8" path="/var/lib/kubelet/pods/a4f83474-cd19-4c19-b6d5-2914706168e8/volumes"
Oct 30 20:05:51.048903 ip-10-0-132-82 kubenswrapper[4892]: I1030 20:05:51.048861    4892 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b372b6b1-jhvr9"]
Oct 30 20:05:51.098696 ip-10-0-132-82 kubenswrapper[4892]: I1030 20:05:51.098660    4892 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-584d0bf7-wx8bn"]
Oct 30 20:05:51.101287 ip-10-0-132-82 kubenswrapper[4892]: I1030 20:05:51.101183    4892 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-de9ce181-rtrn7"]
Oct 30 20:05:51.103238 ip-10-0-132-82 kubenswrapper[4892]: I1030 20:05:51.103218    4892 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b372b6b1-jhvr9"]
Oct 30 20:05:51.105196 ip-10-0-132-82 kubenswrapper[4892]: I1030 20:05:51.105169    4892 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-584d0bf7-wx8bn"]
Oct 30 20:05:51.147444 ip-10-0-132-82 kubenswrapper[4892]: I1030 20:05:51.147421    4892 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-de9ce181-rtrn7"]
Oct 30 20:05:52.253140 ip-10-0-132-82 kubenswrapper[4892]: I1030 20:05:52.253104    4892 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-fsfvk"]
Oct 30 20:05:52.569047 ip-10-0-132-82 kubenswrapper[4892]: I1030 20:05:52.568951    4892 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="03e83bd1-55e2-4aa5-956b-7964b3529ad4" path="/var/lib/kubelet/pods/03e83bd1-55e2-4aa5-956b-7964b3529ad4/volumes"
Oct 30 20:05:52.569373 ip-10-0-132-82 kubenswrapper[4892]: I1030 20:05:52.569359    4892 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="05dbc393-f688-4634-ab42-8c00bf839520" path="/var/lib/kubelet/pods/05dbc393-f688-4634-ab42-8c00bf839520/volumes"
Oct 30 20:05:52.569716 ip-10-0-132-82 kubenswrapper[4892]: I1030 20:05:52.569704    4892 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="956121ad-97c4-4c74-8934-3367fbff6c24" path="/var/lib/kubelet/pods/956121ad-97c4-4c74-8934-3367fbff6c24/volumes"
Oct 30 20:05:53.387342 ip-10-0-132-82 kubenswrapper[4892]: I1030 20:05:53.387314    4892 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-tg5xj_11db9cec-653f-4aee-8937-0cf5b367efac/console-operator/3.log"
Oct 30 20:05:53.446693 ip-10-0-132-82 kubenswrapper[4892]: I1030 20:05:53.446670    4892 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-tg5xj_11db9cec-653f-4aee-8937-0cf5b367efac/console-operator/3.log"
Oct 30 20:05:54.636267 ip-10-0-132-82 kubenswrapper[4892]: I1030 20:05:54.636224    4892 prober.go:122] "Probe failed" probeType="Liveness" pod="kube-system/kube-apiserver-proxy-ip-10-0-132-82.us-west-1.compute.internal" podUID="f9af2f5b3d8deb871a93a1e3f0011dc6" containerName="haproxy" probeResult="failure" output="Get \"https://172.20.0.1:6443/version\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 20:05:54.793785 ip-10-0-132-82 kubenswrapper[4892]: I1030 20:05:54.793751    4892 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-f8a9d8-676dfc58bc-5jxrg" podUID="5f415c4a-4dda-40e3-bdae-8ad67ad15bdf" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 30 20:05:54.845126 ip-10-0-132-82 kubenswrapper[4892]: E1030 20:05:54.845088    4892 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-jrm87" containerName="extractor"
Oct 30 20:05:58.422958 ip-10-0-132-82 kubenswrapper[4892]: I1030 20:05:58.422924    4892 scope.go:122] "RemoveContainer" containerID="1e8d5e7453ec2ed5c8cb8c32fc0dd1685d911fa58453abb0baa70ee7fb0dd2db"
Oct 30 20:05:58.437792 ip-10-0-132-82 kubenswrapper[4892]: I1030 20:05:58.437762    4892 scope.go:122] "RemoveContainer" containerID="e8c019a83307a40a60c50815101688a37f0a919049d2dee08ec8e5b291be8bd3"
Oct 30 20:05:58.448555 ip-10-0-132-82 kubenswrapper[4892]: I1030 20:05:58.448533    4892 scope.go:122] "RemoveContainer" containerID="db349bb08440b61c12da3989ce700c20f39a84d1a9ad1874f6abd0144a6f7827"
Oct 30 20:05:58.458660 ip-10-0-132-82 kubenswrapper[4892]: I1030 20:05:58.458641    4892 scope.go:122] "RemoveContainer" containerID="ab6ce328922b158ffae34102b4c2e083ef1f419a8e5fbe377997ffd3c6e9cb2a"
Oct 30 20:06:01.795379 ip-10-0-132-82 kubenswrapper[4892]: I1030 20:06:01.795303    4892 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w4-b5178f/distinct-2-w4-b5178f-kserve-69ddf8b665-9pdth"]
Oct 30 20:06:04.793779 ip-10-0-132-82 kubenswrapper[4892]: I1030 20:06:04.793739    4892 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-f8a9d8-676dfc58bc-5jxrg" podUID="5f415c4a-4dda-40e3-bdae-8ad67ad15bdf" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 30 20:06:09.489887 ip-10-0-132-82 kubenswrapper[4892]: I1030 20:06:09.489859    4892 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-f8a9d8-676dfc58bc-5jxrg"
Oct 30 20:06:09.584724 ip-10-0-132-82 kubenswrapper[4892]: I1030 20:06:09.584659    4892 generic.go:356] "Generic (PLEG): container finished" podID="5f415c4a-4dda-40e3-bdae-8ad67ad15bdf" containerID="e42503fda96d54472d8152bbe0731f0a2c600231e7d4f52c4f092ad4f4aeaf00" exitCode=0
Oct 30 20:06:09.584724 ip-10-0-132-82 kubenswrapper[4892]: I1030 20:06:09.584702    4892 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-f8a9d8-676dfc58bc-5jxrg"
Oct 30 20:06:09.584876 ip-10-0-132-82 kubenswrapper[4892]: I1030 20:06:09.584742    4892 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-f8a9d8-676dfc58bc-5jxrg" event={"ID":"5f415c4a-4dda-40e3-bdae-8ad67ad15bdf","Type":"ContainerDied","Data":"e42503fda96d54472d8152bbe0731f0a2c600231e7d4f52c4f092ad4f4aeaf00"}
Oct 30 20:06:09.584876 ip-10-0-132-82 kubenswrapper[4892]: I1030 20:06:09.584794    4892 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-f8a9d8-676dfc58bc-5jxrg" event={"ID":"5f415c4a-4dda-40e3-bdae-8ad67ad15bdf","Type":"ContainerDied","Data":"d61ca2f9b84c2a345f2a91fb0a090622cfa74a58a5427b23164c516321f03de8"}
Oct 30 20:06:09.584876 ip-10-0-132-82 kubenswrapper[4892]: I1030 20:06:09.584816    4892 scope.go:122] "RemoveContainer" containerID="e42503fda96d54472d8152bbe0731f0a2c600231e7d4f52c4f092ad4f4aeaf00"
Oct 30 20:06:09.596654 ip-10-0-132-82 kubenswrapper[4892]: I1030 20:06:09.596634    4892 scope.go:122] "RemoveContainer" containerID="e42503fda96d54472d8152bbe0731f0a2c600231e7d4f52c4f092ad4f4aeaf00"
Oct 30 20:06:09.597017 ip-10-0-132-82 kubenswrapper[4892]: E1030 20:06:09.596987    4892 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e42503fda96d54472d8152bbe0731f0a2c600231e7d4f52c4f092ad4f4aeaf00\": container with ID starting with e42503fda96d54472d8152bbe0731f0a2c600231e7d4f52c4f092ad4f4aeaf00 not found: ID does not exist" containerID="e42503fda96d54472d8152bbe0731f0a2c600231e7d4f52c4f092ad4f4aeaf00"
Oct 30 20:06:09.597093 ip-10-0-132-82 kubenswrapper[4892]: I1030 20:06:09.597031    4892 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e42503fda96d54472d8152bbe0731f0a2c600231e7d4f52c4f092ad4f4aeaf00"} err="failed to get container status \"e42503fda96d54472d8152bbe0731f0a2c600231e7d4f52c4f092ad4f4aeaf00\": rpc error: code = NotFound desc = could not find container \"e42503fda96d54472d8152bbe0731f0a2c600231e7d4f52c4f092ad4f4aeaf00\": container with ID starting with e42503fda96d54472d8152bbe0731f0a2c600231e7d4f52c4f092ad4f4aeaf00 not found: ID does not exist"
Oct 30 20:06:09.633109 ip-10-0-132-82 kubenswrapper[4892]: I1030 20:06:09.633081    4892 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/5f415c4a-4dda-40e3-bdae-8ad67ad15bdf-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5f415c4a-4dda-40e3-bdae-8ad67ad15bdf-plugins-config-volume\") pod \"5f415c4a-4dda-40e3-bdae-8ad67ad15bdf\" (UID: \"5f415c4a-4dda-40e3-bdae-8ad67ad15bdf\") "
Oct 30 20:06:09.633226 ip-10-0-132-82 kubenswrapper[4892]: I1030 20:06:09.633209    4892 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5f415c4a-4dda-40e3-bdae-8ad67ad15bdf-kube-api-access-vrdbm\" (UniqueName: \"kubernetes.io/projected/5f415c4a-4dda-40e3-bdae-8ad67ad15bdf-kube-api-access-vrdbm\") pod \"5f415c4a-4dda-40e3-bdae-8ad67ad15bdf\" (UID: \"5f415c4a-4dda-40e3-bdae-8ad67ad15bdf\") "
Oct 30 20:06:09.633489 ip-10-0-132-82 kubenswrapper[4892]: I1030 20:06:09.633464    4892 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5f415c4a-4dda-40e3-bdae-8ad67ad15bdf-plugins-config-volume" pod "5f415c4a-4dda-40e3-bdae-8ad67ad15bdf" (UID: "5f415c4a-4dda-40e3-bdae-8ad67ad15bdf"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 20:06:09.635421 ip-10-0-132-82 kubenswrapper[4892]: I1030 20:06:09.635373    4892 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5f415c4a-4dda-40e3-bdae-8ad67ad15bdf-kube-api-access-vrdbm" pod "5f415c4a-4dda-40e3-bdae-8ad67ad15bdf" (UID: "5f415c4a-4dda-40e3-bdae-8ad67ad15bdf"). InnerVolumeSpecName "kube-api-access-vrdbm". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 20:06:09.734343 ip-10-0-132-82 kubenswrapper[4892]: I1030 20:06:09.734323    4892 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vrdbm\" (UniqueName: \"kubernetes.io/projected/5f415c4a-4dda-40e3-bdae-8ad67ad15bdf-kube-api-access-vrdbm\") on node \"ip-10-0-132-82.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:06:09.734343 ip-10-0-132-82 kubenswrapper[4892]: I1030 20:06:09.734344    4892 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5f415c4a-4dda-40e3-bdae-8ad67ad15bdf-plugins-config-volume\") on node \"ip-10-0-132-82.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:06:10.003715 ip-10-0-132-82 kubenswrapper[4892]: I1030 20:06:10.003684    4892 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-f8a9d8-676dfc58bc-5jxrg"]
Oct 30 20:06:10.052847 ip-10-0-132-82 kubenswrapper[4892]: I1030 20:06:10.052811    4892 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-f8a9d8-676dfc58bc-5jxrg"]
Oct 30 20:06:10.569496 ip-10-0-132-82 kubenswrapper[4892]: I1030 20:06:10.569464    4892 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="5f415c4a-4dda-40e3-bdae-8ad67ad15bdf" path="/var/lib/kubelet/pods/5f415c4a-4dda-40e3-bdae-8ad67ad15bdf/volumes"
Oct 30 20:06:12.235293 ip-10-0-132-82 kubenswrapper[4892]: I1030 20:06:12.235258    4892 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-70e0339c-cg5ql"]
Oct 30 20:06:12.235678 ip-10-0-132-82 kubenswrapper[4892]: I1030 20:06:12.235630    4892 cpu_manager.go:414] "containerMap: removing container" podUID="92e73440-f470-44d9-983e-2d38334d834c" containerName="payload-processing"
Oct 30 20:06:12.235678 ip-10-0-132-82 kubenswrapper[4892]: I1030 20:06:12.235641    4892 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="92e73440-f470-44d9-983e-2d38334d834c" containerName="payload-processing"
Oct 30 20:06:12.235678 ip-10-0-132-82 kubenswrapper[4892]: I1030 20:06:12.235658    4892 cpu_manager.go:414] "containerMap: removing container" podUID="5f415c4a-4dda-40e3-bdae-8ad67ad15bdf" containerName="payload-pre-processing"
Oct 30 20:06:12.235678 ip-10-0-132-82 kubenswrapper[4892]: I1030 20:06:12.235663    4892 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5f415c4a-4dda-40e3-bdae-8ad67ad15bdf" containerName="payload-pre-processing"
Oct 30 20:06:12.235678 ip-10-0-132-82 kubenswrapper[4892]: I1030 20:06:12.235676    4892 cpu_manager.go:414] "containerMap: removing container" podUID="9ef10533-c954-4f94-ab57-1e56838a418f" containerName="revoke-keys"
Oct 30 20:06:12.235678 ip-10-0-132-82 kubenswrapper[4892]: I1030 20:06:12.235682    4892 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9ef10533-c954-4f94-ab57-1e56838a418f" containerName="revoke-keys"
Oct 30 20:06:12.235922 ip-10-0-132-82 kubenswrapper[4892]: I1030 20:06:12.235699    4892 cpu_manager.go:414] "containerMap: removing container" podUID="c76fc24b-0fe2-490e-8967-9a5eb4e9ce12" containerName="istio-proxy"
Oct 30 20:06:12.235922 ip-10-0-132-82 kubenswrapper[4892]: I1030 20:06:12.235704    4892 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c76fc24b-0fe2-490e-8967-9a5eb4e9ce12" containerName="istio-proxy"
Oct 30 20:06:12.235922 ip-10-0-132-82 kubenswrapper[4892]: I1030 20:06:12.235714    4892 cpu_manager.go:414] "containerMap: removing container" podUID="c76fc24b-0fe2-490e-8967-9a5eb4e9ce12" containerName="istio-proxy"
Oct 30 20:06:12.235922 ip-10-0-132-82 kubenswrapper[4892]: I1030 20:06:12.235720    4892 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c76fc24b-0fe2-490e-8967-9a5eb4e9ce12" containerName="istio-proxy"
Oct 30 20:06:12.235922 ip-10-0-132-82 kubenswrapper[4892]: I1030 20:06:12.235796    4892 memory_manager.go:365] "RemoveStaleState removing state" podUID="c76fc24b-0fe2-490e-8967-9a5eb4e9ce12" containerName="istio-proxy"
Oct 30 20:06:12.235922 ip-10-0-132-82 kubenswrapper[4892]: I1030 20:06:12.235806    4892 memory_manager.go:365] "RemoveStaleState removing state" podUID="c76fc24b-0fe2-490e-8967-9a5eb4e9ce12" containerName="istio-proxy"
Oct 30 20:06:12.235922 ip-10-0-132-82 kubenswrapper[4892]: I1030 20:06:12.235813    4892 memory_manager.go:365] "RemoveStaleState removing state" podUID="92e73440-f470-44d9-983e-2d38334d834c" containerName="payload-processing"
Oct 30 20:06:12.235922 ip-10-0-132-82 kubenswrapper[4892]: I1030 20:06:12.235819    4892 memory_manager.go:365] "RemoveStaleState removing state" podUID="5f415c4a-4dda-40e3-bdae-8ad67ad15bdf" containerName="payload-pre-processing"
Oct 30 20:06:12.235922 ip-10-0-132-82 kubenswrapper[4892]: I1030 20:06:12.235825    4892 memory_manager.go:365] "RemoveStaleState removing state" podUID="9ef10533-c954-4f94-ab57-1e56838a418f" containerName="revoke-keys"
Oct 30 20:06:12.240543 ip-10-0-132-82 kubenswrapper[4892]: I1030 20:06:12.240518    4892 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-70e0339c-cg5ql"
Oct 30 20:06:12.296083 ip-10-0-132-82 kubenswrapper[4892]: I1030 20:06:12.296053    4892 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-70e0339c-cg5ql"]
Oct 30 20:06:12.341597 ip-10-0-132-82 kubenswrapper[4892]: I1030 20:06:12.341568    4892 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 30 20:06:12.341745 ip-10-0-132-82 kubenswrapper[4892]: I1030 20:06:12.341659    4892 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 20:06:12.341745 ip-10-0-132-82 kubenswrapper[4892]: I1030 20:06:12.341670    4892 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 30 20:06:12.341745 ip-10-0-132-82 kubenswrapper[4892]: I1030 20:06:12.341713    4892 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-nlxp4\""
Oct 30 20:06:12.364039 ip-10-0-132-82 kubenswrapper[4892]: I1030 20:06:12.364011    4892 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b2c258e4-8081-4033-92cb-4f7262144057-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-70e0339c-cg5ql\" (UID: \"b2c258e4-8081-4033-92cb-4f7262144057\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-70e0339c-cg5ql"
Oct 30 20:06:12.364162 ip-10-0-132-82 kubenswrapper[4892]: I1030 20:06:12.364060    4892 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nqf48\" (UniqueName: \"kubernetes.io/projected/b2c258e4-8081-4033-92cb-4f7262144057-kube-api-access-nqf48\") pod \"maas-api-revoke-sub-70e0339c-cg5ql\" (UID: \"b2c258e4-8081-4033-92cb-4f7262144057\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-70e0339c-cg5ql"
Oct 30 20:06:12.364162 ip-10-0-132-82 kubenswrapper[4892]: I1030 20:06:12.364080    4892 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b2c258e4-8081-4033-92cb-4f7262144057-maas-api-xks-ca\") pod \"maas-api-revoke-sub-70e0339c-cg5ql\" (UID: \"b2c258e4-8081-4033-92cb-4f7262144057\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-70e0339c-cg5ql"
Oct 30 20:06:12.464672 ip-10-0-132-82 kubenswrapper[4892]: I1030 20:06:12.464639    4892 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b2c258e4-8081-4033-92cb-4f7262144057-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-70e0339c-cg5ql\" (UID: \"b2c258e4-8081-4033-92cb-4f7262144057\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-70e0339c-cg5ql"
Oct 30 20:06:12.464810 ip-10-0-132-82 kubenswrapper[4892]: I1030 20:06:12.464693    4892 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nqf48\" (UniqueName: \"kubernetes.io/projected/b2c258e4-8081-4033-92cb-4f7262144057-kube-api-access-nqf48\") pod \"maas-api-revoke-sub-70e0339c-cg5ql\" (UID: \"b2c258e4-8081-4033-92cb-4f7262144057\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-70e0339c-cg5ql"
Oct 30 20:06:12.464810 ip-10-0-132-82 kubenswrapper[4892]: I1030 20:06:12.464718    4892 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b2c258e4-8081-4033-92cb-4f7262144057-maas-api-xks-ca\") pod \"maas-api-revoke-sub-70e0339c-cg5ql\" (UID: \"b2c258e4-8081-4033-92cb-4f7262144057\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-70e0339c-cg5ql"
Oct 30 20:06:12.465286 ip-10-0-132-82 kubenswrapper[4892]: I1030 20:06:12.465264    4892 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b2c258e4-8081-4033-92cb-4f7262144057-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-70e0339c-cg5ql\" (UID: \"b2c258e4-8081-4033-92cb-4f7262144057\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-70e0339c-cg5ql"
Oct 30 20:06:12.467194 ip-10-0-132-82 kubenswrapper[4892]: I1030 20:06:12.467168    4892 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b2c258e4-8081-4033-92cb-4f7262144057-maas-api-xks-ca\") pod \"maas-api-revoke-sub-70e0339c-cg5ql\" (UID: \"b2c258e4-8081-4033-92cb-4f7262144057\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-70e0339c-cg5ql"
Oct 30 20:06:12.520946 ip-10-0-132-82 kubenswrapper[4892]: I1030 20:06:12.520899    4892 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nqf48\" (UniqueName: \"kubernetes.io/projected/b2c258e4-8081-4033-92cb-4f7262144057-kube-api-access-nqf48\") pod \"maas-api-revoke-sub-70e0339c-cg5ql\" (UID: \"b2c258e4-8081-4033-92cb-4f7262144057\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-70e0339c-cg5ql"
Oct 30 20:06:12.554007 ip-10-0-132-82 kubenswrapper[4892]: I1030 20:06:12.553986    4892 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-70e0339c-cg5ql"
Oct 30 20:06:12.930628 ip-10-0-132-82 kubenswrapper[4892]: I1030 20:06:12.930604    4892 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-70e0339c-cg5ql"]
Oct 30 20:06:12.932469 ip-10-0-132-82 kubenswrapper[4892]: W1030 20:06:12.932439    4892 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb2c258e4_8081_4033_92cb_4f7262144057.slice/crio-d5ee45dd62138494fd96ea59adffcd5c6237e37e08cab95719b9934077e5deee WatchSource:0}: Error finding container d5ee45dd62138494fd96ea59adffcd5c6237e37e08cab95719b9934077e5deee: Status 404 returned error can't find the container with id d5ee45dd62138494fd96ea59adffcd5c6237e37e08cab95719b9934077e5deee
Oct 30 20:06:13.610700 ip-10-0-132-82 kubenswrapper[4892]: I1030 20:06:13.610662    4892 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-70e0339c-cg5ql" event={"ID":"b2c258e4-8081-4033-92cb-4f7262144057","Type":"ContainerStarted","Data":"56cb7ec6c1e3ec8dac662aa905be04a1ba3902905d385627a51324a4c0473ee8"}
Oct 30 20:06:13.611134 ip-10-0-132-82 kubenswrapper[4892]: I1030 20:06:13.610714    4892 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-70e0339c-cg5ql" event={"ID":"b2c258e4-8081-4033-92cb-4f7262144057","Type":"ContainerStarted","Data":"d5ee45dd62138494fd96ea59adffcd5c6237e37e08cab95719b9934077e5deee"}
Oct 30 20:06:13.722714 ip-10-0-132-82 kubenswrapper[4892]: I1030 20:06:13.722659    4892 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-70e0339c-cg5ql" podStartSLOduration=1.722642312 podStartE2EDuration="1.722642312s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 20:06:12 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 20:06:13.722583795 +0000 UTC m=+3021.293049387" watchObservedRunningTime="2026-10-30 20:06:13.722642312 +0000 UTC m=+3021.293107914"
Oct 30 20:06:14.617934 ip-10-0-132-82 kubenswrapper[4892]: I1030 20:06:14.617902    4892 generic.go:356] "Generic (PLEG): container finished" podID="b2c258e4-8081-4033-92cb-4f7262144057" containerID="56cb7ec6c1e3ec8dac662aa905be04a1ba3902905d385627a51324a4c0473ee8" exitCode=0
Oct 30 20:06:14.618377 ip-10-0-132-82 kubenswrapper[4892]: I1030 20:06:14.617991    4892 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-70e0339c-cg5ql" event={"ID":"b2c258e4-8081-4033-92cb-4f7262144057","Type":"ContainerDied","Data":"56cb7ec6c1e3ec8dac662aa905be04a1ba3902905d385627a51324a4c0473ee8"}
Oct 30 20:06:15.765250 ip-10-0-132-82 kubenswrapper[4892]: I1030 20:06:15.765220    4892 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-70e0339c-cg5ql"
Oct 30 20:06:15.891074 ip-10-0-132-82 kubenswrapper[4892]: I1030 20:06:15.891048    4892 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b2c258e4-8081-4033-92cb-4f7262144057-kube-api-access-nqf48\" (UniqueName: \"kubernetes.io/projected/b2c258e4-8081-4033-92cb-4f7262144057-kube-api-access-nqf48\") pod \"b2c258e4-8081-4033-92cb-4f7262144057\" (UID: \"b2c258e4-8081-4033-92cb-4f7262144057\") "
Oct 30 20:06:15.891229 ip-10-0-132-82 kubenswrapper[4892]: I1030 20:06:15.891097    4892 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b2c258e4-8081-4033-92cb-4f7262144057-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b2c258e4-8081-4033-92cb-4f7262144057-maas-api-openshift-ca\") pod \"b2c258e4-8081-4033-92cb-4f7262144057\" (UID: \"b2c258e4-8081-4033-92cb-4f7262144057\") "
Oct 30 20:06:15.891229 ip-10-0-132-82 kubenswrapper[4892]: I1030 20:06:15.891168    4892 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/b2c258e4-8081-4033-92cb-4f7262144057-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b2c258e4-8081-4033-92cb-4f7262144057-maas-api-xks-ca\") pod \"b2c258e4-8081-4033-92cb-4f7262144057\" (UID: \"b2c258e4-8081-4033-92cb-4f7262144057\") "
Oct 30 20:06:15.891497 ip-10-0-132-82 kubenswrapper[4892]: I1030 20:06:15.891470    4892 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b2c258e4-8081-4033-92cb-4f7262144057-maas-api-openshift-ca" pod "b2c258e4-8081-4033-92cb-4f7262144057" (UID: "b2c258e4-8081-4033-92cb-4f7262144057"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 20:06:15.893321 ip-10-0-132-82 kubenswrapper[4892]: I1030 20:06:15.893293    4892 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b2c258e4-8081-4033-92cb-4f7262144057-kube-api-access-nqf48" pod "b2c258e4-8081-4033-92cb-4f7262144057" (UID: "b2c258e4-8081-4033-92cb-4f7262144057"). InnerVolumeSpecName "kube-api-access-nqf48". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 20:06:15.893403 ip-10-0-132-82 kubenswrapper[4892]: I1030 20:06:15.893338    4892 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b2c258e4-8081-4033-92cb-4f7262144057-maas-api-xks-ca" pod "b2c258e4-8081-4033-92cb-4f7262144057" (UID: "b2c258e4-8081-4033-92cb-4f7262144057"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 20:06:15.991722 ip-10-0-132-82 kubenswrapper[4892]: I1030 20:06:15.991698    4892 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b2c258e4-8081-4033-92cb-4f7262144057-maas-api-xks-ca\") on node \"ip-10-0-132-82.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:06:15.991722 ip-10-0-132-82 kubenswrapper[4892]: I1030 20:06:15.991721    4892 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nqf48\" (UniqueName: \"kubernetes.io/projected/b2c258e4-8081-4033-92cb-4f7262144057-kube-api-access-nqf48\") on node \"ip-10-0-132-82.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:06:15.991869 ip-10-0-132-82 kubenswrapper[4892]: I1030 20:06:15.991732    4892 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b2c258e4-8081-4033-92cb-4f7262144057-maas-api-openshift-ca\") on node \"ip-10-0-132-82.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:06:16.630850 ip-10-0-132-82 kubenswrapper[4892]: I1030 20:06:16.630827    4892 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-70e0339c-cg5ql"
Oct 30 20:06:16.631019 ip-10-0-132-82 kubenswrapper[4892]: I1030 20:06:16.630819    4892 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-70e0339c-cg5ql" event={"ID":"b2c258e4-8081-4033-92cb-4f7262144057","Type":"ContainerDied","Data":"d5ee45dd62138494fd96ea59adffcd5c6237e37e08cab95719b9934077e5deee"}
Oct 30 20:06:16.631019 ip-10-0-132-82 kubenswrapper[4892]: I1030 20:06:16.630947    4892 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="d5ee45dd62138494fd96ea59adffcd5c6237e37e08cab95719b9934077e5deee"
Oct 30 20:06:16.815976 ip-10-0-132-82 kubenswrapper[4892]: I1030 20:06:16.815923    4892 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w4-b5178f/distinct-2-w4-b5178f-kserve-69ddf8b665-9pdth" podUID="fb71d961-8500-4367-b48c-88d385962bd5" containerName="main" containerID="cri-o://ee63f2eb62b582c733c1bbfd46528dc7e66ac14c84946fe2fbe09a604ca1d05d" gracePeriod=45
Oct 30 20:06:17.058017 ip-10-0-132-82 kubenswrapper[4892]: I1030 20:06:17.057990    4892 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-b5178f/distinct-2-w4-b5178f-kserve-69ddf8b665-9pdth"
Oct 30 20:06:17.201526 ip-10-0-132-82 kubenswrapper[4892]: I1030 20:06:17.201496    4892 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/fb71d961-8500-4367-b48c-88d385962bd5-tls-certs\" (UniqueName: \"kubernetes.io/secret/fb71d961-8500-4367-b48c-88d385962bd5-tls-certs\") pod \"fb71d961-8500-4367-b48c-88d385962bd5\" (UID: \"fb71d961-8500-4367-b48c-88d385962bd5\") "
Oct 30 20:06:17.201749 ip-10-0-132-82 kubenswrapper[4892]: I1030 20:06:17.201609    4892 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/fb71d961-8500-4367-b48c-88d385962bd5-kube-api-access-92v49\" (UniqueName: \"kubernetes.io/projected/fb71d961-8500-4367-b48c-88d385962bd5-kube-api-access-92v49\") pod \"fb71d961-8500-4367-b48c-88d385962bd5\" (UID: \"fb71d961-8500-4367-b48c-88d385962bd5\") "
Oct 30 20:06:17.201749 ip-10-0-132-82 kubenswrapper[4892]: I1030 20:06:17.201652    4892 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/fb71d961-8500-4367-b48c-88d385962bd5-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/fb71d961-8500-4367-b48c-88d385962bd5-tmp-dir\") pod \"fb71d961-8500-4367-b48c-88d385962bd5\" (UID: \"fb71d961-8500-4367-b48c-88d385962bd5\") "
Oct 30 20:06:17.201749 ip-10-0-132-82 kubenswrapper[4892]: I1030 20:06:17.201686    4892 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/fb71d961-8500-4367-b48c-88d385962bd5-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/fb71d961-8500-4367-b48c-88d385962bd5-model-cache\") pod \"fb71d961-8500-4367-b48c-88d385962bd5\" (UID: \"fb71d961-8500-4367-b48c-88d385962bd5\") "
Oct 30 20:06:17.201749 ip-10-0-132-82 kubenswrapper[4892]: I1030 20:06:17.201733    4892 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/fb71d961-8500-4367-b48c-88d385962bd5-dshm\" (UniqueName: \"kubernetes.io/empty-dir/fb71d961-8500-4367-b48c-88d385962bd5-dshm\") pod \"fb71d961-8500-4367-b48c-88d385962bd5\" (UID: \"fb71d961-8500-4367-b48c-88d385962bd5\") "
Oct 30 20:06:17.202923 ip-10-0-132-82 kubenswrapper[4892]: I1030 20:06:17.201813    4892 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/fb71d961-8500-4367-b48c-88d385962bd5-home\" (UniqueName: \"kubernetes.io/empty-dir/fb71d961-8500-4367-b48c-88d385962bd5-home\") pod \"fb71d961-8500-4367-b48c-88d385962bd5\" (UID: \"fb71d961-8500-4367-b48c-88d385962bd5\") "
Oct 30 20:06:17.202923 ip-10-0-132-82 kubenswrapper[4892]: I1030 20:06:17.201949    4892 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fb71d961-8500-4367-b48c-88d385962bd5-tmp-dir" pod "fb71d961-8500-4367-b48c-88d385962bd5" (UID: "fb71d961-8500-4367-b48c-88d385962bd5"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 20:06:17.202923 ip-10-0-132-82 kubenswrapper[4892]: I1030 20:06:17.202164    4892 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/fb71d961-8500-4367-b48c-88d385962bd5-tmp-dir\") on node \"ip-10-0-132-82.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:06:17.202923 ip-10-0-132-82 kubenswrapper[4892]: I1030 20:06:17.202200    4892 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fb71d961-8500-4367-b48c-88d385962bd5-home" pod "fb71d961-8500-4367-b48c-88d385962bd5" (UID: "fb71d961-8500-4367-b48c-88d385962bd5"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 20:06:17.202923 ip-10-0-132-82 kubenswrapper[4892]: I1030 20:06:17.202304    4892 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fb71d961-8500-4367-b48c-88d385962bd5-model-cache" pod "fb71d961-8500-4367-b48c-88d385962bd5" (UID: "fb71d961-8500-4367-b48c-88d385962bd5"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 20:06:17.204502 ip-10-0-132-82 kubenswrapper[4892]: I1030 20:06:17.204465    4892 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fb71d961-8500-4367-b48c-88d385962bd5-dshm" pod "fb71d961-8500-4367-b48c-88d385962bd5" (UID: "fb71d961-8500-4367-b48c-88d385962bd5"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 20:06:17.204674 ip-10-0-132-82 kubenswrapper[4892]: I1030 20:06:17.204602    4892 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/fb71d961-8500-4367-b48c-88d385962bd5-tls-certs" pod "fb71d961-8500-4367-b48c-88d385962bd5" (UID: "fb71d961-8500-4367-b48c-88d385962bd5"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 20:06:17.204674 ip-10-0-132-82 kubenswrapper[4892]: I1030 20:06:17.204634    4892 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fb71d961-8500-4367-b48c-88d385962bd5-kube-api-access-92v49" pod "fb71d961-8500-4367-b48c-88d385962bd5" (UID: "fb71d961-8500-4367-b48c-88d385962bd5"). InnerVolumeSpecName "kube-api-access-92v49". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 20:06:17.303159 ip-10-0-132-82 kubenswrapper[4892]: I1030 20:06:17.303131    4892 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/fb71d961-8500-4367-b48c-88d385962bd5-home\") on node \"ip-10-0-132-82.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:06:17.303159 ip-10-0-132-82 kubenswrapper[4892]: I1030 20:06:17.303156    4892 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/fb71d961-8500-4367-b48c-88d385962bd5-tls-certs\") on node \"ip-10-0-132-82.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:06:17.303360 ip-10-0-132-82 kubenswrapper[4892]: I1030 20:06:17.303168    4892 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-92v49\" (UniqueName: \"kubernetes.io/projected/fb71d961-8500-4367-b48c-88d385962bd5-kube-api-access-92v49\") on node \"ip-10-0-132-82.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:06:17.303360 ip-10-0-132-82 kubenswrapper[4892]: I1030 20:06:17.303181    4892 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/fb71d961-8500-4367-b48c-88d385962bd5-model-cache\") on node \"ip-10-0-132-82.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:06:17.303360 ip-10-0-132-82 kubenswrapper[4892]: I1030 20:06:17.303190    4892 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/fb71d961-8500-4367-b48c-88d385962bd5-dshm\") on node \"ip-10-0-132-82.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:06:17.637598 ip-10-0-132-82 kubenswrapper[4892]: I1030 20:06:17.637561    4892 generic.go:356] "Generic (PLEG): container finished" podID="fb71d961-8500-4367-b48c-88d385962bd5" containerID="ee63f2eb62b582c733c1bbfd46528dc7e66ac14c84946fe2fbe09a604ca1d05d" exitCode=0
Oct 30 20:06:17.637768 ip-10-0-132-82 kubenswrapper[4892]: I1030 20:06:17.637606    4892 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-b5178f/distinct-2-w4-b5178f-kserve-69ddf8b665-9pdth"
Oct 30 20:06:17.637768 ip-10-0-132-82 kubenswrapper[4892]: I1030 20:06:17.637628    4892 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-b5178f/distinct-2-w4-b5178f-kserve-69ddf8b665-9pdth" event={"ID":"fb71d961-8500-4367-b48c-88d385962bd5","Type":"ContainerDied","Data":"ee63f2eb62b582c733c1bbfd46528dc7e66ac14c84946fe2fbe09a604ca1d05d"}
Oct 30 20:06:17.637768 ip-10-0-132-82 kubenswrapper[4892]: I1030 20:06:17.637659    4892 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-b5178f/distinct-2-w4-b5178f-kserve-69ddf8b665-9pdth" event={"ID":"fb71d961-8500-4367-b48c-88d385962bd5","Type":"ContainerDied","Data":"db43981c0f9cbbd30fb0e757523de51563b791d2c2aa9357f01b461d9acb174c"}
Oct 30 20:06:17.637768 ip-10-0-132-82 kubenswrapper[4892]: I1030 20:06:17.637674    4892 scope.go:122] "RemoveContainer" containerID="ee63f2eb62b582c733c1bbfd46528dc7e66ac14c84946fe2fbe09a604ca1d05d"
Oct 30 20:06:17.647843 ip-10-0-132-82 kubenswrapper[4892]: I1030 20:06:17.647821    4892 scope.go:122] "RemoveContainer" containerID="ee63f2eb62b582c733c1bbfd46528dc7e66ac14c84946fe2fbe09a604ca1d05d"
Oct 30 20:06:17.648142 ip-10-0-132-82 kubenswrapper[4892]: E1030 20:06:17.648120    4892 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ee63f2eb62b582c733c1bbfd46528dc7e66ac14c84946fe2fbe09a604ca1d05d\": container with ID starting with ee63f2eb62b582c733c1bbfd46528dc7e66ac14c84946fe2fbe09a604ca1d05d not found: ID does not exist" containerID="ee63f2eb62b582c733c1bbfd46528dc7e66ac14c84946fe2fbe09a604ca1d05d"
Oct 30 20:06:17.648221 ip-10-0-132-82 kubenswrapper[4892]: I1030 20:06:17.648152    4892 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ee63f2eb62b582c733c1bbfd46528dc7e66ac14c84946fe2fbe09a604ca1d05d"} err="failed to get container status \"ee63f2eb62b582c733c1bbfd46528dc7e66ac14c84946fe2fbe09a604ca1d05d\": rpc error: code = NotFound desc = could not find container \"ee63f2eb62b582c733c1bbfd46528dc7e66ac14c84946fe2fbe09a604ca1d05d\": container with ID starting with ee63f2eb62b582c733c1bbfd46528dc7e66ac14c84946fe2fbe09a604ca1d05d not found: ID does not exist"
Oct 30 20:06:17.800862 ip-10-0-132-82 kubenswrapper[4892]: I1030 20:06:17.800838    4892 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w4-b5178f/distinct-2-w4-b5178f-kserve-69ddf8b665-9pdth"]
Oct 30 20:06:17.803029 ip-10-0-132-82 kubenswrapper[4892]: I1030 20:06:17.803004    4892 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w4-b5178f/distinct-2-w4-b5178f-kserve-69ddf8b665-9pdth"]
Oct 30 20:06:18.568591 ip-10-0-132-82 kubenswrapper[4892]: I1030 20:06:18.568558    4892 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="fb71d961-8500-4367-b48c-88d385962bd5" path="/var/lib/kubelet/pods/fb71d961-8500-4367-b48c-88d385962bd5/volumes"
Oct 30 20:06:24.076045 ip-10-0-132-82 kubenswrapper[4892]: I1030 20:06:24.076004    4892 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-96f166af-wv982"]
Oct 30 20:06:24.126123 ip-10-0-132-82 kubenswrapper[4892]: I1030 20:06:24.126087    4892 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-cc81bdfd-pprtv"]
Oct 30 20:06:24.128212 ip-10-0-132-82 kubenswrapper[4892]: I1030 20:06:24.128199    4892 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-96f166af-wv982"]
Oct 30 20:06:24.130073 ip-10-0-132-82 kubenswrapper[4892]: I1030 20:06:24.130051    4892 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-cc81bdfd-pprtv"]
Oct 30 20:06:24.568126 ip-10-0-132-82 kubenswrapper[4892]: I1030 20:06:24.568098    4892 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="46408b41-3826-472e-ab55-22cf9c43ebb5" path="/var/lib/kubelet/pods/46408b41-3826-472e-ab55-22cf9c43ebb5/volumes"
Oct 30 20:06:24.568485 ip-10-0-132-82 kubenswrapper[4892]: I1030 20:06:24.568471    4892 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="714775e4-a883-4b7c-bea6-cdb90ae3bff4" path="/var/lib/kubelet/pods/714775e4-a883-4b7c-bea6-cdb90ae3bff4/volumes"
Oct 30 20:06:24.843866 ip-10-0-132-82 kubenswrapper[4892]: E1030 20:06:24.843784    4892 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-jrm87" containerName="extractor"
Oct 30 20:06:27.373526 ip-10-0-132-82 kubenswrapper[4892]: I1030 20:06:27.373494    4892 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-fsfvk"]
Oct 30 20:06:43.861776 ip-10-0-132-82 kubenswrapper[4892]: I1030 20:06:43.861743    4892 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-fsfvk"]
Oct 30 20:06:50.046980 ip-10-0-132-82 kubenswrapper[4892]: I1030 20:06:50.046937    4892 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-8bdbd0b9-qpqgz"]
Oct 30 20:06:50.096077 ip-10-0-132-82 kubenswrapper[4892]: I1030 20:06:50.096040    4892 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-8bdbd0b9-qpqgz"]
Oct 30 20:06:50.568918 ip-10-0-132-82 kubenswrapper[4892]: I1030 20:06:50.568872    4892 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="56bf0c05-25c3-4fc3-9299-2eaa33c76d07" path="/var/lib/kubelet/pods/56bf0c05-25c3-4fc3-9299-2eaa33c76d07/volumes"
Oct 30 20:06:54.843551 ip-10-0-132-82 kubenswrapper[4892]: E1030 20:06:54.843516    4892 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-jrm87" containerName="extractor"
Oct 30 20:06:58.517249 ip-10-0-132-82 kubenswrapper[4892]: I1030 20:06:58.517213    4892 scope.go:122] "RemoveContainer" containerID="47442b1f1e65b6d69a87050fedc25434e5fc3e9eaa41e51f8171a483d9ba9c08"
Oct 30 20:06:58.527475 ip-10-0-132-82 kubenswrapper[4892]: I1030 20:06:58.527452    4892 scope.go:122] "RemoveContainer" containerID="87784e77de2b9c57d01e135ffc4d04dd8cb3807a44e6c686fc17a1a3bcf72583"
Oct 30 20:06:58.536399 ip-10-0-132-82 kubenswrapper[4892]: I1030 20:06:58.536380    4892 scope.go:122] "RemoveContainer" containerID="258af30f9920c6703e06161d0a4019f9c452a0bdc78f4d27359a7a2444b2694d"
Oct 30 20:06:58.544503 ip-10-0-132-82 kubenswrapper[4892]: I1030 20:06:58.544486    4892 scope.go:122] "RemoveContainer" containerID="8aa9d2b80f9c0d20e6c7010ceb122d6fbf82d8b5b3b80b94d78a44f21316e326"
Oct 30 20:06:58.552344 ip-10-0-132-82 kubenswrapper[4892]: I1030 20:06:58.552325    4892 scope.go:122] "RemoveContainer" containerID="db0d3018ceb00f3dea046d1269b7ac0fa7ba6998159960f6b6b146602e027de0"
Oct 30 20:06:58.560163 ip-10-0-132-82 kubenswrapper[4892]: I1030 20:06:58.560146    4892 scope.go:122] "RemoveContainer" containerID="ddf751afe27a96f71b7695b975e5daf54bafd08d5680891f8a516c52dc068ed5"
Oct 30 20:06:58.570054 ip-10-0-132-82 kubenswrapper[4892]: I1030 20:06:58.570030    4892 scope.go:122] "RemoveContainer" containerID="51025d0f9bd9e146de4ea0bf0049f9b1569843e327823e5309f32b43e48842fb"
Oct 30 20:07:21.134474 ip-10-0-132-82 kubenswrapper[4892]: I1030 20:07:21.134086    4892 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6db586dd6c-k85mx container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.1.1:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 30 20:07:21.134474 ip-10-0-132-82 kubenswrapper[4892]: I1030 20:07:21.134151    4892 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-k85mx" podUID="6f05c89a-d631-4c95-aad6-d08d6c87cffd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.1:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 30 20:07:24.046680 ip-10-0-132-82 kubenswrapper[4892]: I1030 20:07:24.046634    4892 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3556bc06-5mltw"]
Oct 30 20:07:24.095820 ip-10-0-132-82 kubenswrapper[4892]: I1030 20:07:24.095791    4892 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3556bc06-5mltw"]
Oct 30 20:07:24.567454 ip-10-0-132-82 kubenswrapper[4892]: I1030 20:07:24.567420    4892 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="83cb2031-9ebe-487f-8053-361008386451" path="/var/lib/kubelet/pods/83cb2031-9ebe-487f-8053-361008386451/volumes"
Oct 30 20:07:24.844496 ip-10-0-132-82 kubenswrapper[4892]: E1030 20:07:24.844408    4892 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-jrm87" containerName="extractor"
Oct 30 20:07:25.046988 ip-10-0-132-82 kubenswrapper[4892]: I1030 20:07:25.046947    4892 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-7a1cd64a-9lx2c"]
Oct 30 20:07:25.097055 ip-10-0-132-82 kubenswrapper[4892]: I1030 20:07:25.096949    4892 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-fcb89cf2-pfwt8"]
Oct 30 20:07:25.098974 ip-10-0-132-82 kubenswrapper[4892]: I1030 20:07:25.098941    4892 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-7a1cd64a-9lx2c"]
Oct 30 20:07:25.100874 ip-10-0-132-82 kubenswrapper[4892]: I1030 20:07:25.100855    4892 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-fcb89cf2-pfwt8"]
Oct 30 20:07:26.568225 ip-10-0-132-82 kubenswrapper[4892]: I1030 20:07:26.568191    4892 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0fb0d83f-df47-49bc-aa9b-16b4eb5708dc" path="/var/lib/kubelet/pods/0fb0d83f-df47-49bc-aa9b-16b4eb5708dc/volumes"
Oct 30 20:07:26.568632 ip-10-0-132-82 kubenswrapper[4892]: I1030 20:07:26.568611    4892 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="5dec99a8-bafc-496d-baa2-8974fdcd53eb" path="/var/lib/kubelet/pods/5dec99a8-bafc-496d-baa2-8974fdcd53eb/volumes"
Oct 30 20:07:54.842928 ip-10-0-132-82 kubenswrapper[4892]: E1030 20:07:54.842877    4892 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-jrm87" containerName="extractor"
Oct 30 20:07:58.622026 ip-10-0-132-82 kubenswrapper[4892]: I1030 20:07:58.621992    4892 scope.go:122] "RemoveContainer" containerID="c2955dbce66a6d69474bfacfc27986e2c512185941ace175bdd1d87971953e46"
Oct 30 20:07:58.630562 ip-10-0-132-82 kubenswrapper[4892]: I1030 20:07:58.630545    4892 scope.go:122] "RemoveContainer" containerID="3eafeae8988c1c0b0eefcb83d62e59b90f941a740bdfa2169906b78c48cdb07f"
Oct 30 20:07:58.638639 ip-10-0-132-82 kubenswrapper[4892]: I1030 20:07:58.638622    4892 scope.go:122] "RemoveContainer" containerID="231fbf87cbd4f8237c06f7eedfb0b2994b2ba049d0332d1bc213ef4d8d7894c9"
Oct 30 20:07:58.646134 ip-10-0-132-82 kubenswrapper[4892]: I1030 20:07:58.646105    4892 scope.go:122] "RemoveContainer" containerID="c9f0039cbe33119de37b0bed2fbf5a0c8ee21d851638f12ead36ccd48eaf67f3"
Oct 30 20:08:24.843886 ip-10-0-132-82 kubenswrapper[4892]: E1030 20:08:24.843843    4892 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-jrm87" containerName="extractor"
Oct 30 20:08:26.882233 ip-10-0-132-82 kubenswrapper[4892]: I1030 20:08:26.882197    4892 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-fsfvk"]
Oct 30 20:08:54.843472 ip-10-0-132-82 kubenswrapper[4892]: E1030 20:08:54.843437    4892 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-jrm87" containerName="extractor"
Oct 30 20:08:58.680636 ip-10-0-132-82 kubenswrapper[4892]: I1030 20:08:58.680602    4892 scope.go:122] "RemoveContainer" containerID="616a2776cc0caf27afe59f3085659095da970931699b815c9ba30f32362ed90f"
Oct 30 20:08:58.689610 ip-10-0-132-82 kubenswrapper[4892]: I1030 20:08:58.689584    4892 scope.go:122] "RemoveContainer" containerID="530375a37374506ec1d13e2ffefc8cbb2c4c939ad609ca8a15d08fa6b116e2c0"
Oct 30 20:08:58.697740 ip-10-0-132-82 kubenswrapper[4892]: I1030 20:08:58.697722    4892 scope.go:122] "RemoveContainer" containerID="46ffd73a30a924f956acc4f0c93f6bfe18ad7428170697876eb41c7ae2197cca"
Oct 30 20:08:58.706113 ip-10-0-132-82 kubenswrapper[4892]: I1030 20:08:58.706092    4892 scope.go:122] "RemoveContainer" containerID="6e2f1bb171ba7408ba5268f7d248482cd9666a47606ddee2b30525ef31256a7b"
Oct 30 20:08:58.714705 ip-10-0-132-82 kubenswrapper[4892]: I1030 20:08:58.714687    4892 scope.go:122] "RemoveContainer" containerID="c31c586e2429efe19c8075f4d13e541005ae6f8e68432d3ebddc471b1c56e4ba"
Oct 30 20:09:00.269894 ip-10-0-132-82 kubenswrapper[4892]: I1030 20:09:00.269862    4892 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-fsfvk"]
Oct 30 20:09:13.057892 ip-10-0-132-82 kubenswrapper[4892]: I1030 20:09:13.057818    4892 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-bb1331d0-pqd49"]
Oct 30 20:09:13.107427 ip-10-0-132-82 kubenswrapper[4892]: I1030 20:09:13.107393    4892 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-bb1331d0-pqd49"]
Oct 30 20:09:14.568776 ip-10-0-132-82 kubenswrapper[4892]: I1030 20:09:14.568746    4892 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c849e37b-a605-43c3-a628-5fa823c378dd" path="/var/lib/kubelet/pods/c849e37b-a605-43c3-a628-5fa823c378dd/volumes"
Oct 30 20:09:14.789747 ip-10-0-132-82 kubenswrapper[4892]: I1030 20:09:14.789716    4892 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-fsfvk"]
Oct 30 20:09:24.842820 ip-10-0-132-82 kubenswrapper[4892]: E1030 20:09:24.842784    4892 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-jrm87" containerName="extractor"
Oct 30 20:09:25.047674 ip-10-0-132-82 kubenswrapper[4892]: I1030 20:09:25.047640    4892 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-4fa2da17-4vlhs"]
Oct 30 20:09:25.097020 ip-10-0-132-82 kubenswrapper[4892]: I1030 20:09:25.096933    4892 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-4fa2da17-4vlhs"]
Oct 30 20:09:26.568481 ip-10-0-132-82 kubenswrapper[4892]: I1030 20:09:26.568448    4892 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="69927d30-c42f-4f56-8257-10b2e8f0dba0" path="/var/lib/kubelet/pods/69927d30-c42f-4f56-8257-10b2e8f0dba0/volumes"
Oct 30 20:09:45.731819 ip-10-0-132-82 kubenswrapper[4892]: I1030 20:09:45.731784    4892 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-fsfvk"]
Oct 30 20:09:54.849957 ip-10-0-132-82 kubenswrapper[4892]: E1030 20:09:54.849913    4892 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-jrm87" containerName="extractor"
Oct 30 20:10:13.771096 ip-10-0-132-82 kubenswrapper[4892]: I1030 20:10:13.771064    4892 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-fsfvk"]
Oct 30 20:10:24.843131 ip-10-0-132-82 kubenswrapper[4892]: E1030 20:10:24.843092    4892 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-jrm87" containerName="extractor"
Oct 30 20:10:47.035023 ip-10-0-132-82 kubenswrapper[4892]: I1030 20:10:47.034935    4892 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-fsfvk"]
Oct 30 20:10:53.422251 ip-10-0-132-82 kubenswrapper[4892]: I1030 20:10:53.422221    4892 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-tg5xj_11db9cec-653f-4aee-8937-0cf5b367efac/console-operator/3.log"
Oct 30 20:10:53.485711 ip-10-0-132-82 kubenswrapper[4892]: I1030 20:10:53.485686    4892 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-tg5xj_11db9cec-653f-4aee-8937-0cf5b367efac/console-operator/3.log"
Oct 30 20:10:54.843914 ip-10-0-132-82 kubenswrapper[4892]: E1030 20:10:54.843874    4892 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-jrm87" containerName="extractor"
Oct 30 20:10:57.730678 ip-10-0-132-82 kubenswrapper[4892]: I1030 20:10:57.730644    4892 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-fsfvk"]
Oct 30 20:10:58.759439 ip-10-0-132-82 kubenswrapper[4892]: I1030 20:10:58.759406    4892 scope.go:122] "RemoveContainer" containerID="bbd2134cee542d7349c51b912a853d84e7a9b31dc6f4c0b12c2d7824045695c4"
Oct 30 20:10:58.768720 ip-10-0-132-82 kubenswrapper[4892]: I1030 20:10:58.768692    4892 scope.go:122] "RemoveContainer" containerID="42d989106da7bedd9a59ac5f5ca2d0dd2b2e56ec32d2d94cdb635446434e4541"
Oct 30 20:10:58.777009 ip-10-0-132-82 kubenswrapper[4892]: I1030 20:10:58.776957    4892 scope.go:122] "RemoveContainer" containerID="c684557a881fec919fd89c08eed24ae4dcc12f02ebc4a6e338bc233833c27e56"
Oct 30 20:11:06.120136 ip-10-0-132-82 kubenswrapper[4892]: I1030 20:11:06.120096    4892 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6db586dd6c-k85mx container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.1.1:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 30 20:11:06.120553 ip-10-0-132-82 kubenswrapper[4892]: I1030 20:11:06.120168    4892 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-k85mx" podUID="6f05c89a-d631-4c95-aad6-d08d6c87cffd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.1:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 30 20:11:06.559140 ip-10-0-132-82 kubenswrapper[4892]: I1030 20:11:06.559087    4892 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6db586dd6c-p7m7x container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.252:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 30 20:11:06.559507 ip-10-0-132-82 kubenswrapper[4892]: I1030 20:11:06.559474    4892 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-p7m7x" podUID="741c37d3-758a-4ced-ab0e-9038baea2d88" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.252:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 30 20:11:09.554494 ip-10-0-132-82 kubenswrapper[4892]: I1030 20:11:09.554458    4892 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-fsfvk"]
Oct 30 20:11:16.047086 ip-10-0-132-82 kubenswrapper[4892]: I1030 20:11:16.047044    4892 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-70e0339c-cg5ql"]
Oct 30 20:11:16.096638 ip-10-0-132-82 kubenswrapper[4892]: I1030 20:11:16.096606    4892 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-70e0339c-cg5ql"]
Oct 30 20:11:16.567388 ip-10-0-132-82 kubenswrapper[4892]: I1030 20:11:16.567353    4892 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b2c258e4-8081-4033-92cb-4f7262144057" path="/var/lib/kubelet/pods/b2c258e4-8081-4033-92cb-4f7262144057/volumes"
Oct 30 20:11:24.843272 ip-10-0-132-82 kubenswrapper[4892]: E1030 20:11:24.843233    4892 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-jrm87" containerName="extractor"
Oct 30 20:11:32.959837 ip-10-0-132-82 kubenswrapper[4892]: I1030 20:11:32.959804    4892 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-fsfvk"]
Oct 30 20:11:45.035174 ip-10-0-132-82 kubenswrapper[4892]: I1030 20:11:45.035136    4892 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-fsfvk"]
Oct 30 20:11:54.844133 ip-10-0-132-82 kubenswrapper[4892]: E1030 20:11:54.844096    4892 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-jrm87" containerName="extractor"
Oct 30 20:11:58.808444 ip-10-0-132-82 kubenswrapper[4892]: I1030 20:11:58.808411    4892 scope.go:122] "RemoveContainer" containerID="2e3e6db3cefd15fc64a0e584121f1e6a4531b52dc855bf457abd6a680b2f09d4"
Oct 30 20:12:08.534015 ip-10-0-132-82 kubenswrapper[4892]: I1030 20:12:08.533896    4892 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-fsfvk"]
Oct 30 20:12:23.634904 ip-10-0-132-82 kubenswrapper[4892]: I1030 20:12:23.634868    4892 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-fsfvk"]
Oct 30 20:12:24.845232 ip-10-0-132-82 kubenswrapper[4892]: E1030 20:12:24.845190    4892 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-jrm87" containerName="extractor"
Oct 30 20:12:54.331114 ip-10-0-132-82 kubenswrapper[4892]: I1030 20:12:54.331078    4892 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-fsfvk"]
Oct 30 20:12:54.851007 ip-10-0-132-82 kubenswrapper[4892]: E1030 20:12:54.850928    4892 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-jrm87" containerName="extractor"
Oct 30 20:12:58.828839 ip-10-0-132-82 kubenswrapper[4892]: I1030 20:12:58.828809    4892 scope.go:122] "RemoveContainer" containerID="56cb7ec6c1e3ec8dac662aa905be04a1ba3902905d385627a51324a4c0473ee8"
Oct 30 20:13:16.941449 ip-10-0-132-82 kubenswrapper[4892]: I1030 20:13:16.941408    4892 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-fsfvk"]
Oct 30 20:13:24.843490 ip-10-0-132-82 kubenswrapper[4892]: E1030 20:13:24.843455    4892 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-jrm87" containerName="extractor"
Oct 30 20:13:30.046398 ip-10-0-132-82 kubenswrapper[4892]: I1030 20:13:30.046365    4892 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-fsfvk"]
Oct 30 20:13:39.967608 ip-10-0-132-82 kubenswrapper[4892]: I1030 20:13:39.967526    4892 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-fsfvk"]
Oct 30 20:13:54.635901 ip-10-0-132-82 kubenswrapper[4892]: I1030 20:13:54.635858    4892 prober.go:122] "Probe failed" probeType="Liveness" pod="kube-system/kube-apiserver-proxy-ip-10-0-132-82.us-west-1.compute.internal" podUID="f9af2f5b3d8deb871a93a1e3f0011dc6" containerName="haproxy" probeResult="failure" output="Get \"https://172.20.0.1:6443/version\": context deadline exceeded"
Oct 30 20:13:54.843269 ip-10-0-132-82 kubenswrapper[4892]: E1030 20:13:54.843237    4892 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-jrm87" containerName="extractor"
Oct 30 20:14:00.568599 ip-10-0-132-82 kubenswrapper[4892]: I1030 20:14:00.568568    4892 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-fsfvk"]
Oct 30 20:14:09.952871 ip-10-0-132-82 kubenswrapper[4892]: I1030 20:14:09.952831    4892 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-fsfvk"]
Oct 30 20:14:24.844472 ip-10-0-132-82 kubenswrapper[4892]: E1030 20:14:24.844436    4892 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-jrm87" containerName="extractor"
Oct 30 20:14:45.261206 ip-10-0-132-82 kubenswrapper[4892]: I1030 20:14:45.261167    4892 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-fsfvk"]
Oct 30 20:14:54.843562 ip-10-0-132-82 kubenswrapper[4892]: E1030 20:14:54.843504    4892 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-jrm87" containerName="extractor"
Oct 30 20:14:54.930513 ip-10-0-132-82 kubenswrapper[4892]: I1030 20:14:54.930479    4892 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-fsfvk"]
Oct 30 20:15:16.348565 ip-10-0-132-82 kubenswrapper[4892]: I1030 20:15:16.348525    4892 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-fsfvk"]
Oct 30 20:15:24.843459 ip-10-0-132-82 kubenswrapper[4892]: E1030 20:15:24.843419    4892 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-jrm87" containerName="extractor"
Oct 30 20:15:47.587578 ip-10-0-132-82 kubenswrapper[4892]: I1030 20:15:47.587540    4892 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-fsfvk"]
Oct 30 20:15:53.457586 ip-10-0-132-82 kubenswrapper[4892]: I1030 20:15:53.457558    4892 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-tg5xj_11db9cec-653f-4aee-8937-0cf5b367efac/console-operator/3.log"
Oct 30 20:15:53.521955 ip-10-0-132-82 kubenswrapper[4892]: I1030 20:15:53.521927    4892 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-tg5xj_11db9cec-653f-4aee-8937-0cf5b367efac/console-operator/3.log"
Oct 30 20:15:54.843590 ip-10-0-132-82 kubenswrapper[4892]: E1030 20:15:54.843554    4892 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-jrm87" containerName="extractor"
Oct 30 20:16:12.155826 ip-10-0-132-82 kubenswrapper[4892]: I1030 20:16:12.155794    4892 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-fsfvk"]
Oct 30 20:16:23.249233 ip-10-0-132-82 kubenswrapper[4892]: I1030 20:16:23.249200    4892 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-8e78ffb3-qhtjz"]
Oct 30 20:16:23.249728 ip-10-0-132-82 kubenswrapper[4892]: I1030 20:16:23.249556    4892 cpu_manager.go:414] "containerMap: removing container" podUID="b2c258e4-8081-4033-92cb-4f7262144057" containerName="revoke-keys"
Oct 30 20:16:23.249728 ip-10-0-132-82 kubenswrapper[4892]: I1030 20:16:23.249567    4892 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b2c258e4-8081-4033-92cb-4f7262144057" containerName="revoke-keys"
Oct 30 20:16:23.249728 ip-10-0-132-82 kubenswrapper[4892]: I1030 20:16:23.249601    4892 cpu_manager.go:414] "containerMap: removing container" podUID="fb71d961-8500-4367-b48c-88d385962bd5" containerName="main"
Oct 30 20:16:23.249728 ip-10-0-132-82 kubenswrapper[4892]: I1030 20:16:23.249606    4892 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="fb71d961-8500-4367-b48c-88d385962bd5" containerName="main"
Oct 30 20:16:23.249728 ip-10-0-132-82 kubenswrapper[4892]: I1030 20:16:23.249671    4892 memory_manager.go:365] "RemoveStaleState removing state" podUID="fb71d961-8500-4367-b48c-88d385962bd5" containerName="main"
Oct 30 20:16:23.249728 ip-10-0-132-82 kubenswrapper[4892]: I1030 20:16:23.249687    4892 memory_manager.go:365] "RemoveStaleState removing state" podUID="b2c258e4-8081-4033-92cb-4f7262144057" containerName="revoke-keys"
Oct 30 20:16:23.252760 ip-10-0-132-82 kubenswrapper[4892]: I1030 20:16:23.252744    4892 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8e78ffb3-qhtjz"
Oct 30 20:16:23.269224 ip-10-0-132-82 kubenswrapper[4892]: I1030 20:16:23.269201    4892 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jvdzn\" (UniqueName: \"kubernetes.io/projected/bac05a0f-9ed2-41f0-9880-4f6d01ba29cf-kube-api-access-jvdzn\") pod \"maas-api-revoke-sub-8e78ffb3-qhtjz\" (UID: \"bac05a0f-9ed2-41f0-9880-4f6d01ba29cf\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8e78ffb3-qhtjz"
Oct 30 20:16:23.269349 ip-10-0-132-82 kubenswrapper[4892]: I1030 20:16:23.269304    4892 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/bac05a0f-9ed2-41f0-9880-4f6d01ba29cf-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-8e78ffb3-qhtjz\" (UID: \"bac05a0f-9ed2-41f0-9880-4f6d01ba29cf\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8e78ffb3-qhtjz"
Oct 30 20:16:23.269349 ip-10-0-132-82 kubenswrapper[4892]: I1030 20:16:23.269339    4892 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/bac05a0f-9ed2-41f0-9880-4f6d01ba29cf-maas-api-xks-ca\") pod \"maas-api-revoke-sub-8e78ffb3-qhtjz\" (UID: \"bac05a0f-9ed2-41f0-9880-4f6d01ba29cf\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8e78ffb3-qhtjz"
Oct 30 20:16:23.311269 ip-10-0-132-82 kubenswrapper[4892]: I1030 20:16:23.311238    4892 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-8e78ffb3-qhtjz"]
Oct 30 20:16:23.356343 ip-10-0-132-82 kubenswrapper[4892]: I1030 20:16:23.356317    4892 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 20:16:23.356343 ip-10-0-132-82 kubenswrapper[4892]: I1030 20:16:23.356320    4892 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 30 20:16:23.356547 ip-10-0-132-82 kubenswrapper[4892]: I1030 20:16:23.356348    4892 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 30 20:16:23.356547 ip-10-0-132-82 kubenswrapper[4892]: I1030 20:16:23.356394    4892 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-nlxp4\""
Oct 30 20:16:23.370150 ip-10-0-132-82 kubenswrapper[4892]: I1030 20:16:23.370119    4892 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/bac05a0f-9ed2-41f0-9880-4f6d01ba29cf-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-8e78ffb3-qhtjz\" (UID: \"bac05a0f-9ed2-41f0-9880-4f6d01ba29cf\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8e78ffb3-qhtjz"
Oct 30 20:16:23.370314 ip-10-0-132-82 kubenswrapper[4892]: I1030 20:16:23.370163    4892 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/bac05a0f-9ed2-41f0-9880-4f6d01ba29cf-maas-api-xks-ca\") pod \"maas-api-revoke-sub-8e78ffb3-qhtjz\" (UID: \"bac05a0f-9ed2-41f0-9880-4f6d01ba29cf\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8e78ffb3-qhtjz"
Oct 30 20:16:23.370314 ip-10-0-132-82 kubenswrapper[4892]: I1030 20:16:23.370217    4892 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jvdzn\" (UniqueName: \"kubernetes.io/projected/bac05a0f-9ed2-41f0-9880-4f6d01ba29cf-kube-api-access-jvdzn\") pod \"maas-api-revoke-sub-8e78ffb3-qhtjz\" (UID: \"bac05a0f-9ed2-41f0-9880-4f6d01ba29cf\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8e78ffb3-qhtjz"
Oct 30 20:16:23.370875 ip-10-0-132-82 kubenswrapper[4892]: I1030 20:16:23.370853    4892 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/bac05a0f-9ed2-41f0-9880-4f6d01ba29cf-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-8e78ffb3-qhtjz\" (UID: \"bac05a0f-9ed2-41f0-9880-4f6d01ba29cf\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8e78ffb3-qhtjz"
Oct 30 20:16:23.372690 ip-10-0-132-82 kubenswrapper[4892]: I1030 20:16:23.372669    4892 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/bac05a0f-9ed2-41f0-9880-4f6d01ba29cf-maas-api-xks-ca\") pod \"maas-api-revoke-sub-8e78ffb3-qhtjz\" (UID: \"bac05a0f-9ed2-41f0-9880-4f6d01ba29cf\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8e78ffb3-qhtjz"
Oct 30 20:16:23.426640 ip-10-0-132-82 kubenswrapper[4892]: I1030 20:16:23.426617    4892 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jvdzn\" (UniqueName: \"kubernetes.io/projected/bac05a0f-9ed2-41f0-9880-4f6d01ba29cf-kube-api-access-jvdzn\") pod \"maas-api-revoke-sub-8e78ffb3-qhtjz\" (UID: \"bac05a0f-9ed2-41f0-9880-4f6d01ba29cf\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8e78ffb3-qhtjz"
Oct 30 20:16:23.565033 ip-10-0-132-82 kubenswrapper[4892]: I1030 20:16:23.564957    4892 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8e78ffb3-qhtjz"
Oct 30 20:16:23.937344 ip-10-0-132-82 kubenswrapper[4892]: I1030 20:16:23.937321    4892 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-8e78ffb3-qhtjz"]
Oct 30 20:16:23.939713 ip-10-0-132-82 kubenswrapper[4892]: W1030 20:16:23.939681    4892 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbac05a0f_9ed2_41f0_9880_4f6d01ba29cf.slice/crio-a3d493103698c28ed3e8a6e0a892042282f2d37ed7fe60ee0cf0b5d156ac4021 WatchSource:0}: Error finding container a3d493103698c28ed3e8a6e0a892042282f2d37ed7fe60ee0cf0b5d156ac4021: Status 404 returned error can't find the container with id a3d493103698c28ed3e8a6e0a892042282f2d37ed7fe60ee0cf0b5d156ac4021
Oct 30 20:16:23.959261 ip-10-0-132-82 kubenswrapper[4892]: I1030 20:16:23.959232    4892 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8e78ffb3-qhtjz" event={"ID":"bac05a0f-9ed2-41f0-9880-4f6d01ba29cf","Type":"ContainerStarted","Data":"a3d493103698c28ed3e8a6e0a892042282f2d37ed7fe60ee0cf0b5d156ac4021"}
Oct 30 20:16:24.844438 ip-10-0-132-82 kubenswrapper[4892]: E1030 20:16:24.844403    4892 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-jrm87" containerName="extractor"
Oct 30 20:16:24.972374 ip-10-0-132-82 kubenswrapper[4892]: I1030 20:16:24.972339    4892 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8e78ffb3-qhtjz" event={"ID":"bac05a0f-9ed2-41f0-9880-4f6d01ba29cf","Type":"ContainerStarted","Data":"6f1f402ec204d7ee59040b68ce9ff5d654ba14c4821be41926d573b3fae13c09"}
Oct 30 20:16:25.087386 ip-10-0-132-82 kubenswrapper[4892]: I1030 20:16:25.087226    4892 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8e78ffb3-qhtjz" podStartSLOduration=2.087208851 podStartE2EDuration="2.087208851s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 20:16:23 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 20:16:25.085135442 +0000 UTC m=+3632.655601038" watchObservedRunningTime="2026-10-30 20:16:25.087208851 +0000 UTC m=+3632.657674453"
Oct 30 20:16:25.978939 ip-10-0-132-82 kubenswrapper[4892]: I1030 20:16:25.978899    4892 generic.go:356] "Generic (PLEG): container finished" podID="bac05a0f-9ed2-41f0-9880-4f6d01ba29cf" containerID="6f1f402ec204d7ee59040b68ce9ff5d654ba14c4821be41926d573b3fae13c09" exitCode=0
Oct 30 20:16:25.979366 ip-10-0-132-82 kubenswrapper[4892]: I1030 20:16:25.978999    4892 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8e78ffb3-qhtjz" event={"ID":"bac05a0f-9ed2-41f0-9880-4f6d01ba29cf","Type":"ContainerDied","Data":"6f1f402ec204d7ee59040b68ce9ff5d654ba14c4821be41926d573b3fae13c09"}
Oct 30 20:16:27.117748 ip-10-0-132-82 kubenswrapper[4892]: I1030 20:16:27.117725    4892 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8e78ffb3-qhtjz"
Oct 30 20:16:27.202349 ip-10-0-132-82 kubenswrapper[4892]: I1030 20:16:27.202324    4892 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/bac05a0f-9ed2-41f0-9880-4f6d01ba29cf-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/bac05a0f-9ed2-41f0-9880-4f6d01ba29cf-maas-api-openshift-ca\") pod \"bac05a0f-9ed2-41f0-9880-4f6d01ba29cf\" (UID: \"bac05a0f-9ed2-41f0-9880-4f6d01ba29cf\") "
Oct 30 20:16:27.202442 ip-10-0-132-82 kubenswrapper[4892]: I1030 20:16:27.202383    4892 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/bac05a0f-9ed2-41f0-9880-4f6d01ba29cf-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/bac05a0f-9ed2-41f0-9880-4f6d01ba29cf-maas-api-xks-ca\") pod \"bac05a0f-9ed2-41f0-9880-4f6d01ba29cf\" (UID: \"bac05a0f-9ed2-41f0-9880-4f6d01ba29cf\") "
Oct 30 20:16:27.202514 ip-10-0-132-82 kubenswrapper[4892]: I1030 20:16:27.202464    4892 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/bac05a0f-9ed2-41f0-9880-4f6d01ba29cf-kube-api-access-jvdzn\" (UniqueName: \"kubernetes.io/projected/bac05a0f-9ed2-41f0-9880-4f6d01ba29cf-kube-api-access-jvdzn\") pod \"bac05a0f-9ed2-41f0-9880-4f6d01ba29cf\" (UID: \"bac05a0f-9ed2-41f0-9880-4f6d01ba29cf\") "
Oct 30 20:16:27.202683 ip-10-0-132-82 kubenswrapper[4892]: I1030 20:16:27.202661    4892 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/bac05a0f-9ed2-41f0-9880-4f6d01ba29cf-maas-api-openshift-ca" pod "bac05a0f-9ed2-41f0-9880-4f6d01ba29cf" (UID: "bac05a0f-9ed2-41f0-9880-4f6d01ba29cf"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 20:16:27.204503 ip-10-0-132-82 kubenswrapper[4892]: I1030 20:16:27.204478    4892 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/bac05a0f-9ed2-41f0-9880-4f6d01ba29cf-maas-api-xks-ca" pod "bac05a0f-9ed2-41f0-9880-4f6d01ba29cf" (UID: "bac05a0f-9ed2-41f0-9880-4f6d01ba29cf"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 20:16:27.204592 ip-10-0-132-82 kubenswrapper[4892]: I1030 20:16:27.204536    4892 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bac05a0f-9ed2-41f0-9880-4f6d01ba29cf-kube-api-access-jvdzn" pod "bac05a0f-9ed2-41f0-9880-4f6d01ba29cf" (UID: "bac05a0f-9ed2-41f0-9880-4f6d01ba29cf"). InnerVolumeSpecName "kube-api-access-jvdzn". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 20:16:27.303546 ip-10-0-132-82 kubenswrapper[4892]: I1030 20:16:27.303491    4892 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/bac05a0f-9ed2-41f0-9880-4f6d01ba29cf-maas-api-xks-ca\") on node \"ip-10-0-132-82.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:16:27.303546 ip-10-0-132-82 kubenswrapper[4892]: I1030 20:16:27.303513    4892 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jvdzn\" (UniqueName: \"kubernetes.io/projected/bac05a0f-9ed2-41f0-9880-4f6d01ba29cf-kube-api-access-jvdzn\") on node \"ip-10-0-132-82.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:16:27.303546 ip-10-0-132-82 kubenswrapper[4892]: I1030 20:16:27.303524    4892 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/bac05a0f-9ed2-41f0-9880-4f6d01ba29cf-maas-api-openshift-ca\") on node \"ip-10-0-132-82.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:16:27.989713 ip-10-0-132-82 kubenswrapper[4892]: I1030 20:16:27.989683    4892 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8e78ffb3-qhtjz"
Oct 30 20:16:27.989924 ip-10-0-132-82 kubenswrapper[4892]: I1030 20:16:27.989681    4892 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8e78ffb3-qhtjz" event={"ID":"bac05a0f-9ed2-41f0-9880-4f6d01ba29cf","Type":"ContainerDied","Data":"a3d493103698c28ed3e8a6e0a892042282f2d37ed7fe60ee0cf0b5d156ac4021"}
Oct 30 20:16:27.989924 ip-10-0-132-82 kubenswrapper[4892]: I1030 20:16:27.989810    4892 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="a3d493103698c28ed3e8a6e0a892042282f2d37ed7fe60ee0cf0b5d156ac4021"
Oct 30 20:16:35.313207 ip-10-0-132-82 kubenswrapper[4892]: I1030 20:16:35.313129    4892 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-f1aa42-cn68q"]
Oct 30 20:16:35.313580 ip-10-0-132-82 kubenswrapper[4892]: I1030 20:16:35.313507    4892 cpu_manager.go:414] "containerMap: removing container" podUID="bac05a0f-9ed2-41f0-9880-4f6d01ba29cf" containerName="revoke-keys"
Oct 30 20:16:35.313580 ip-10-0-132-82 kubenswrapper[4892]: I1030 20:16:35.313517    4892 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="bac05a0f-9ed2-41f0-9880-4f6d01ba29cf" containerName="revoke-keys"
Oct 30 20:16:35.313658 ip-10-0-132-82 kubenswrapper[4892]: I1030 20:16:35.313591    4892 memory_manager.go:365] "RemoveStaleState removing state" podUID="bac05a0f-9ed2-41f0-9880-4f6d01ba29cf" containerName="revoke-keys"
Oct 30 20:16:35.316758 ip-10-0-132-82 kubenswrapper[4892]: I1030 20:16:35.316739    4892 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-f1aa42-cn68q"
Oct 30 20:16:35.367159 ip-10-0-132-82 kubenswrapper[4892]: I1030 20:16:35.367136    4892 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qtcsz\" (UniqueName: \"kubernetes.io/projected/4d9fe6dc-c2c4-4c83-ac34-e187378a3c2e-kube-api-access-qtcsz\") pod \"maas-api-revoke-keys-e2e-worker-single-main-f1aa42-cn68q\" (UID: \"4d9fe6dc-c2c4-4c83-ac34-e187378a3c2e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-f1aa42-cn68q"
Oct 30 20:16:35.367267 ip-10-0-132-82 kubenswrapper[4892]: I1030 20:16:35.367168    4892 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4d9fe6dc-c2c4-4c83-ac34-e187378a3c2e-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-single-main-f1aa42-cn68q\" (UID: \"4d9fe6dc-c2c4-4c83-ac34-e187378a3c2e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-f1aa42-cn68q"
Oct 30 20:16:35.367310 ip-10-0-132-82 kubenswrapper[4892]: I1030 20:16:35.367296    4892 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4d9fe6dc-c2c4-4c83-ac34-e187378a3c2e-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-single-main-f1aa42-cn68q\" (UID: \"4d9fe6dc-c2c4-4c83-ac34-e187378a3c2e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-f1aa42-cn68q"
Oct 30 20:16:35.420763 ip-10-0-132-82 kubenswrapper[4892]: I1030 20:16:35.420730    4892 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 20:16:35.420905 ip-10-0-132-82 kubenswrapper[4892]: I1030 20:16:35.420797    4892 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-nlxp4\""
Oct 30 20:16:35.423002 ip-10-0-132-82 kubenswrapper[4892]: I1030 20:16:35.422957    4892 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-f1aa42-cn68q"]
Oct 30 20:16:35.458037 ip-10-0-132-82 kubenswrapper[4892]: I1030 20:16:35.458011    4892 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 30 20:16:35.458169 ip-10-0-132-82 kubenswrapper[4892]: I1030 20:16:35.458074    4892 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 30 20:16:35.468516 ip-10-0-132-82 kubenswrapper[4892]: I1030 20:16:35.468491    4892 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4d9fe6dc-c2c4-4c83-ac34-e187378a3c2e-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-single-main-f1aa42-cn68q\" (UID: \"4d9fe6dc-c2c4-4c83-ac34-e187378a3c2e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-f1aa42-cn68q"
Oct 30 20:16:35.468612 ip-10-0-132-82 kubenswrapper[4892]: I1030 20:16:35.468538    4892 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qtcsz\" (UniqueName: \"kubernetes.io/projected/4d9fe6dc-c2c4-4c83-ac34-e187378a3c2e-kube-api-access-qtcsz\") pod \"maas-api-revoke-keys-e2e-worker-single-main-f1aa42-cn68q\" (UID: \"4d9fe6dc-c2c4-4c83-ac34-e187378a3c2e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-f1aa42-cn68q"
Oct 30 20:16:35.468612 ip-10-0-132-82 kubenswrapper[4892]: I1030 20:16:35.468556    4892 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4d9fe6dc-c2c4-4c83-ac34-e187378a3c2e-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-single-main-f1aa42-cn68q\" (UID: \"4d9fe6dc-c2c4-4c83-ac34-e187378a3c2e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-f1aa42-cn68q"
Oct 30 20:16:35.469166 ip-10-0-132-82 kubenswrapper[4892]: I1030 20:16:35.469145    4892 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4d9fe6dc-c2c4-4c83-ac34-e187378a3c2e-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-single-main-f1aa42-cn68q\" (UID: \"4d9fe6dc-c2c4-4c83-ac34-e187378a3c2e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-f1aa42-cn68q"
Oct 30 20:16:35.471002 ip-10-0-132-82 kubenswrapper[4892]: I1030 20:16:35.470983    4892 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4d9fe6dc-c2c4-4c83-ac34-e187378a3c2e-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-single-main-f1aa42-cn68q\" (UID: \"4d9fe6dc-c2c4-4c83-ac34-e187378a3c2e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-f1aa42-cn68q"
Oct 30 20:16:35.524419 ip-10-0-132-82 kubenswrapper[4892]: I1030 20:16:35.524391    4892 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qtcsz\" (UniqueName: \"kubernetes.io/projected/4d9fe6dc-c2c4-4c83-ac34-e187378a3c2e-kube-api-access-qtcsz\") pod \"maas-api-revoke-keys-e2e-worker-single-main-f1aa42-cn68q\" (UID: \"4d9fe6dc-c2c4-4c83-ac34-e187378a3c2e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-f1aa42-cn68q"
Oct 30 20:16:35.629786 ip-10-0-132-82 kubenswrapper[4892]: I1030 20:16:35.629723    4892 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-f1aa42-cn68q"
Oct 30 20:16:36.002294 ip-10-0-132-82 kubenswrapper[4892]: I1030 20:16:36.002269    4892 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-f1aa42-cn68q"]
Oct 30 20:16:36.004366 ip-10-0-132-82 kubenswrapper[4892]: W1030 20:16:36.004337    4892 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4d9fe6dc_c2c4_4c83_ac34_e187378a3c2e.slice/crio-0dbb9e10a4e3adb062cf48168a9a3a110ab3450078c18e46dc0b4d6961947da0 WatchSource:0}: Error finding container 0dbb9e10a4e3adb062cf48168a9a3a110ab3450078c18e46dc0b4d6961947da0: Status 404 returned error can't find the container with id 0dbb9e10a4e3adb062cf48168a9a3a110ab3450078c18e46dc0b4d6961947da0
Oct 30 20:16:36.035895 ip-10-0-132-82 kubenswrapper[4892]: I1030 20:16:36.035863    4892 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-f1aa42-cn68q" event={"ID":"4d9fe6dc-c2c4-4c83-ac34-e187378a3c2e","Type":"ContainerStarted","Data":"0dbb9e10a4e3adb062cf48168a9a3a110ab3450078c18e46dc0b4d6961947da0"}
Oct 30 20:16:36.159023 ip-10-0-132-82 kubenswrapper[4892]: I1030 20:16:36.158981    4892 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-fsfvk"]
Oct 30 20:16:37.043470 ip-10-0-132-82 kubenswrapper[4892]: I1030 20:16:37.043441    4892 generic.go:356] "Generic (PLEG): container finished" podID="4d9fe6dc-c2c4-4c83-ac34-e187378a3c2e" containerID="4cc121ff2ff525bf3f09bd637e69148b1e4347931aaa3f70a87a4e33cb8c5c28" exitCode=0
Oct 30 20:16:37.043896 ip-10-0-132-82 kubenswrapper[4892]: I1030 20:16:37.043487    4892 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-f1aa42-cn68q" event={"ID":"4d9fe6dc-c2c4-4c83-ac34-e187378a3c2e","Type":"ContainerDied","Data":"4cc121ff2ff525bf3f09bd637e69148b1e4347931aaa3f70a87a4e33cb8c5c28"}
Oct 30 20:16:38.182122 ip-10-0-132-82 kubenswrapper[4892]: I1030 20:16:38.182099    4892 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-f1aa42-cn68q"
Oct 30 20:16:38.296033 ip-10-0-132-82 kubenswrapper[4892]: I1030 20:16:38.295989    4892 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4d9fe6dc-c2c4-4c83-ac34-e187378a3c2e-kube-api-access-qtcsz\" (UniqueName: \"kubernetes.io/projected/4d9fe6dc-c2c4-4c83-ac34-e187378a3c2e-kube-api-access-qtcsz\") pod \"4d9fe6dc-c2c4-4c83-ac34-e187378a3c2e\" (UID: \"4d9fe6dc-c2c4-4c83-ac34-e187378a3c2e\") "
Oct 30 20:16:38.296209 ip-10-0-132-82 kubenswrapper[4892]: I1030 20:16:38.296087    4892 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/4d9fe6dc-c2c4-4c83-ac34-e187378a3c2e-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4d9fe6dc-c2c4-4c83-ac34-e187378a3c2e-maas-api-xks-ca\") pod \"4d9fe6dc-c2c4-4c83-ac34-e187378a3c2e\" (UID: \"4d9fe6dc-c2c4-4c83-ac34-e187378a3c2e\") "
Oct 30 20:16:38.296209 ip-10-0-132-82 kubenswrapper[4892]: I1030 20:16:38.296156    4892 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/4d9fe6dc-c2c4-4c83-ac34-e187378a3c2e-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4d9fe6dc-c2c4-4c83-ac34-e187378a3c2e-maas-api-openshift-ca\") pod \"4d9fe6dc-c2c4-4c83-ac34-e187378a3c2e\" (UID: \"4d9fe6dc-c2c4-4c83-ac34-e187378a3c2e\") "
Oct 30 20:16:38.296819 ip-10-0-132-82 kubenswrapper[4892]: I1030 20:16:38.296782    4892 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4d9fe6dc-c2c4-4c83-ac34-e187378a3c2e-maas-api-openshift-ca" pod "4d9fe6dc-c2c4-4c83-ac34-e187378a3c2e" (UID: "4d9fe6dc-c2c4-4c83-ac34-e187378a3c2e"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 20:16:38.299465 ip-10-0-132-82 kubenswrapper[4892]: I1030 20:16:38.299399    4892 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/4d9fe6dc-c2c4-4c83-ac34-e187378a3c2e-maas-api-xks-ca" pod "4d9fe6dc-c2c4-4c83-ac34-e187378a3c2e" (UID: "4d9fe6dc-c2c4-4c83-ac34-e187378a3c2e"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 20:16:38.300418 ip-10-0-132-82 kubenswrapper[4892]: I1030 20:16:38.300390    4892 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4d9fe6dc-c2c4-4c83-ac34-e187378a3c2e-kube-api-access-qtcsz" pod "4d9fe6dc-c2c4-4c83-ac34-e187378a3c2e" (UID: "4d9fe6dc-c2c4-4c83-ac34-e187378a3c2e"). InnerVolumeSpecName "kube-api-access-qtcsz". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 20:16:38.397504 ip-10-0-132-82 kubenswrapper[4892]: I1030 20:16:38.397472    4892 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4d9fe6dc-c2c4-4c83-ac34-e187378a3c2e-maas-api-openshift-ca\") on node \"ip-10-0-132-82.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:16:38.397504 ip-10-0-132-82 kubenswrapper[4892]: I1030 20:16:38.397500    4892 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qtcsz\" (UniqueName: \"kubernetes.io/projected/4d9fe6dc-c2c4-4c83-ac34-e187378a3c2e-kube-api-access-qtcsz\") on node \"ip-10-0-132-82.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:16:38.397703 ip-10-0-132-82 kubenswrapper[4892]: I1030 20:16:38.397514    4892 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4d9fe6dc-c2c4-4c83-ac34-e187378a3c2e-maas-api-xks-ca\") on node \"ip-10-0-132-82.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:16:39.054234 ip-10-0-132-82 kubenswrapper[4892]: I1030 20:16:39.054198    4892 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-f1aa42-cn68q"
Oct 30 20:16:39.054403 ip-10-0-132-82 kubenswrapper[4892]: I1030 20:16:39.054229    4892 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-f1aa42-cn68q" event={"ID":"4d9fe6dc-c2c4-4c83-ac34-e187378a3c2e","Type":"ContainerDied","Data":"0dbb9e10a4e3adb062cf48168a9a3a110ab3450078c18e46dc0b4d6961947da0"}
Oct 30 20:16:39.054403 ip-10-0-132-82 kubenswrapper[4892]: I1030 20:16:39.054276    4892 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="0dbb9e10a4e3adb062cf48168a9a3a110ab3450078c18e46dc0b4d6961947da0"
Oct 30 20:16:54.844118 ip-10-0-132-82 kubenswrapper[4892]: E1030 20:16:54.844081    4892 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-jrm87" containerName="extractor"
Oct 30 20:16:55.175270 ip-10-0-132-82 kubenswrapper[4892]: I1030 20:16:55.175235    4892 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-f1aa42-cn68q"]
Oct 30 20:16:55.224924 ip-10-0-132-82 kubenswrapper[4892]: I1030 20:16:55.224892    4892 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-f1aa42-cn68q"]
Oct 30 20:16:56.568095 ip-10-0-132-82 kubenswrapper[4892]: I1030 20:16:56.568065    4892 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="4d9fe6dc-c2c4-4c83-ac34-e187378a3c2e" path="/var/lib/kubelet/pods/4d9fe6dc-c2c4-4c83-ac34-e187378a3c2e/volumes"
Oct 30 20:16:57.546296 ip-10-0-132-82 kubenswrapper[4892]: I1030 20:16:57.546257    4892 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-fsfvk"]
Oct 30 20:17:20.158239 ip-10-0-132-82 kubenswrapper[4892]: I1030 20:17:20.158198    4892 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-fsfvk"]
Oct 30 20:17:24.843294 ip-10-0-132-82 kubenswrapper[4892]: E1030 20:17:24.843256    4892 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-jrm87" containerName="extractor"
Oct 30 20:17:30.082610 ip-10-0-132-82 kubenswrapper[4892]: I1030 20:17:30.082579    4892 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-fsfvk"]
Oct 30 20:17:39.073111 ip-10-0-132-82 kubenswrapper[4892]: I1030 20:17:39.073077    4892 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-57bb9bfff-qflpx"]
Oct 30 20:17:39.073683 ip-10-0-132-82 kubenswrapper[4892]: I1030 20:17:39.073333    4892 kuberuntime_container.go:898] "Killing container with a grace period" pod="kuadrant-system/kuadrant-operator-controller-manager-57bb9bfff-qflpx" podUID="55c71a1d-feca-47a1-b100-738a982b6d47" containerName="manager" containerID="cri-o://9ec83f08186618bf35d3b4f36f878125a5c7e34ed6992d613bd37006c90e0a55" gracePeriod=10
Oct 30 20:17:39.137502 ip-10-0-132-82 kubenswrapper[4892]: I1030 20:17:39.137464    4892 prober.go:122] "Probe failed" probeType="Readiness" pod="kuadrant-system/kuadrant-operator-controller-manager-57bb9bfff-qflpx" podUID="55c71a1d-feca-47a1-b100-738a982b6d47" containerName="manager" probeResult="failure" output="Get \"http://10.132.0.45:8081/readyz\": dial tcp 10.132.0.45:8081: connect: connection refused"
Oct 30 20:17:39.326571 ip-10-0-132-82 kubenswrapper[4892]: I1030 20:17:39.326515    4892 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-57bb9bfff-qflpx"
Oct 30 20:17:39.379318 ip-10-0-132-82 kubenswrapper[4892]: I1030 20:17:39.379286    4892 generic.go:356] "Generic (PLEG): container finished" podID="55c71a1d-feca-47a1-b100-738a982b6d47" containerID="9ec83f08186618bf35d3b4f36f878125a5c7e34ed6992d613bd37006c90e0a55" exitCode=0
Oct 30 20:17:39.379443 ip-10-0-132-82 kubenswrapper[4892]: I1030 20:17:39.379328    4892 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-57bb9bfff-qflpx"
Oct 30 20:17:39.379443 ip-10-0-132-82 kubenswrapper[4892]: I1030 20:17:39.379372    4892 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-57bb9bfff-qflpx" event={"ID":"55c71a1d-feca-47a1-b100-738a982b6d47","Type":"ContainerDied","Data":"9ec83f08186618bf35d3b4f36f878125a5c7e34ed6992d613bd37006c90e0a55"}
Oct 30 20:17:39.379443 ip-10-0-132-82 kubenswrapper[4892]: I1030 20:17:39.379423    4892 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-57bb9bfff-qflpx" event={"ID":"55c71a1d-feca-47a1-b100-738a982b6d47","Type":"ContainerDied","Data":"fc44631b285988f7f2f54a4da2d22d6fa9563574f02ae91669b5b027b0ea56f1"}
Oct 30 20:17:39.379570 ip-10-0-132-82 kubenswrapper[4892]: I1030 20:17:39.379445    4892 scope.go:122] "RemoveContainer" containerID="9ec83f08186618bf35d3b4f36f878125a5c7e34ed6992d613bd37006c90e0a55"
Oct 30 20:17:39.389753 ip-10-0-132-82 kubenswrapper[4892]: I1030 20:17:39.389738    4892 scope.go:122] "RemoveContainer" containerID="9ec83f08186618bf35d3b4f36f878125a5c7e34ed6992d613bd37006c90e0a55"
Oct 30 20:17:39.390062 ip-10-0-132-82 kubenswrapper[4892]: E1030 20:17:39.390041    4892 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9ec83f08186618bf35d3b4f36f878125a5c7e34ed6992d613bd37006c90e0a55\": container with ID starting with 9ec83f08186618bf35d3b4f36f878125a5c7e34ed6992d613bd37006c90e0a55 not found: ID does not exist" containerID="9ec83f08186618bf35d3b4f36f878125a5c7e34ed6992d613bd37006c90e0a55"
Oct 30 20:17:39.390128 ip-10-0-132-82 kubenswrapper[4892]: I1030 20:17:39.390072    4892 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9ec83f08186618bf35d3b4f36f878125a5c7e34ed6992d613bd37006c90e0a55"} err="failed to get container status \"9ec83f08186618bf35d3b4f36f878125a5c7e34ed6992d613bd37006c90e0a55\": rpc error: code = NotFound desc = could not find container \"9ec83f08186618bf35d3b4f36f878125a5c7e34ed6992d613bd37006c90e0a55\": container with ID starting with 9ec83f08186618bf35d3b4f36f878125a5c7e34ed6992d613bd37006c90e0a55 not found: ID does not exist"
Oct 30 20:17:39.468122 ip-10-0-132-82 kubenswrapper[4892]: I1030 20:17:39.468095    4892 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/55c71a1d-feca-47a1-b100-738a982b6d47-extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/55c71a1d-feca-47a1-b100-738a982b6d47-extensions-socket-volume\") pod \"55c71a1d-feca-47a1-b100-738a982b6d47\" (UID: \"55c71a1d-feca-47a1-b100-738a982b6d47\") "
Oct 30 20:17:39.468292 ip-10-0-132-82 kubenswrapper[4892]: I1030 20:17:39.468145    4892 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/55c71a1d-feca-47a1-b100-738a982b6d47-kube-api-access-7b4sb\" (UniqueName: \"kubernetes.io/projected/55c71a1d-feca-47a1-b100-738a982b6d47-kube-api-access-7b4sb\") pod \"55c71a1d-feca-47a1-b100-738a982b6d47\" (UID: \"55c71a1d-feca-47a1-b100-738a982b6d47\") "
Oct 30 20:17:39.468615 ip-10-0-132-82 kubenswrapper[4892]: I1030 20:17:39.468582    4892 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/55c71a1d-feca-47a1-b100-738a982b6d47-extensions-socket-volume" pod "55c71a1d-feca-47a1-b100-738a982b6d47" (UID: "55c71a1d-feca-47a1-b100-738a982b6d47"). InnerVolumeSpecName "extensions-socket-volume". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 20:17:39.470483 ip-10-0-132-82 kubenswrapper[4892]: I1030 20:17:39.470462    4892 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/55c71a1d-feca-47a1-b100-738a982b6d47-kube-api-access-7b4sb" pod "55c71a1d-feca-47a1-b100-738a982b6d47" (UID: "55c71a1d-feca-47a1-b100-738a982b6d47"). InnerVolumeSpecName "kube-api-access-7b4sb". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 20:17:39.569617 ip-10-0-132-82 kubenswrapper[4892]: I1030 20:17:39.569587    4892 reconciler_common.go:299] "Volume detached for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/55c71a1d-feca-47a1-b100-738a982b6d47-extensions-socket-volume\") on node \"ip-10-0-132-82.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:17:39.569617 ip-10-0-132-82 kubenswrapper[4892]: I1030 20:17:39.569614    4892 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7b4sb\" (UniqueName: \"kubernetes.io/projected/55c71a1d-feca-47a1-b100-738a982b6d47-kube-api-access-7b4sb\") on node \"ip-10-0-132-82.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:17:39.798478 ip-10-0-132-82 kubenswrapper[4892]: I1030 20:17:39.798446    4892 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-57bb9bfff-qflpx"]
Oct 30 20:17:39.802014 ip-10-0-132-82 kubenswrapper[4892]: I1030 20:17:39.801989    4892 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-57bb9bfff-qflpx"]
Oct 30 20:17:40.574312 ip-10-0-132-82 kubenswrapper[4892]: I1030 20:17:40.574268    4892 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="55c71a1d-feca-47a1-b100-738a982b6d47" path="/var/lib/kubelet/pods/55c71a1d-feca-47a1-b100-738a982b6d47/volumes"
Oct 30 20:17:54.635718 ip-10-0-132-82 kubenswrapper[4892]: I1030 20:17:54.635675    4892 prober.go:122] "Probe failed" probeType="Liveness" pod="kube-system/kube-apiserver-proxy-ip-10-0-132-82.us-west-1.compute.internal" podUID="f9af2f5b3d8deb871a93a1e3f0011dc6" containerName="haproxy" probeResult="failure" output="Get \"https://172.20.0.1:6443/version\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 20:17:54.849324 ip-10-0-132-82 kubenswrapper[4892]: E1030 20:17:54.849287    4892 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-jrm87" containerName="extractor"
Oct 30 20:18:24.842789 ip-10-0-132-82 kubenswrapper[4892]: E1030 20:18:24.842752    4892 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-jrm87" containerName="extractor"
Oct 30 20:18:52.658239 ip-10-0-132-82 kubenswrapper[4892]: I1030 20:18:52.658211    4892 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-57bb9bfff-ck96g"]
Oct 30 20:18:52.658626 ip-10-0-132-82 kubenswrapper[4892]: I1030 20:18:52.658534    4892 cpu_manager.go:414] "containerMap: removing container" podUID="55c71a1d-feca-47a1-b100-738a982b6d47" containerName="manager"
Oct 30 20:18:52.658626 ip-10-0-132-82 kubenswrapper[4892]: I1030 20:18:52.658544    4892 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="55c71a1d-feca-47a1-b100-738a982b6d47" containerName="manager"
Oct 30 20:18:52.658626 ip-10-0-132-82 kubenswrapper[4892]: I1030 20:18:52.658580    4892 cpu_manager.go:414] "containerMap: removing container" podUID="4d9fe6dc-c2c4-4c83-ac34-e187378a3c2e" containerName="revoke-keys"
Oct 30 20:18:52.658626 ip-10-0-132-82 kubenswrapper[4892]: I1030 20:18:52.658586    4892 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4d9fe6dc-c2c4-4c83-ac34-e187378a3c2e" containerName="revoke-keys"
Oct 30 20:18:52.658769 ip-10-0-132-82 kubenswrapper[4892]: I1030 20:18:52.658636    4892 memory_manager.go:365] "RemoveStaleState removing state" podUID="4d9fe6dc-c2c4-4c83-ac34-e187378a3c2e" containerName="revoke-keys"
Oct 30 20:18:52.658769 ip-10-0-132-82 kubenswrapper[4892]: I1030 20:18:52.658644    4892 memory_manager.go:365] "RemoveStaleState removing state" podUID="55c71a1d-feca-47a1-b100-738a982b6d47" containerName="manager"
Oct 30 20:18:52.661639 ip-10-0-132-82 kubenswrapper[4892]: I1030 20:18:52.661621    4892 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-57bb9bfff-ck96g"
Oct 30 20:18:52.713683 ip-10-0-132-82 kubenswrapper[4892]: I1030 20:18:52.713661    4892 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"kuadrant-operator-controller-manager-dockercfg-82bmx\""
Oct 30 20:18:52.748919 ip-10-0-132-82 kubenswrapper[4892]: I1030 20:18:52.748890    4892 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/0c40923a-2096-43f0-8d0f-315f2e4a2270-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-57bb9bfff-ck96g\" (UID: \"0c40923a-2096-43f0-8d0f-315f2e4a2270\") " pod="kuadrant-system/kuadrant-operator-controller-manager-57bb9bfff-ck96g"
Oct 30 20:18:52.749068 ip-10-0-132-82 kubenswrapper[4892]: I1030 20:18:52.749002    4892 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pfq6k\" (UniqueName: \"kubernetes.io/projected/0c40923a-2096-43f0-8d0f-315f2e4a2270-kube-api-access-pfq6k\") pod \"kuadrant-operator-controller-manager-57bb9bfff-ck96g\" (UID: \"0c40923a-2096-43f0-8d0f-315f2e4a2270\") " pod="kuadrant-system/kuadrant-operator-controller-manager-57bb9bfff-ck96g"
Oct 30 20:18:52.765099 ip-10-0-132-82 kubenswrapper[4892]: I1030 20:18:52.765076    4892 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-57bb9bfff-ck96g"]
Oct 30 20:18:52.849724 ip-10-0-132-82 kubenswrapper[4892]: I1030 20:18:52.849696    4892 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pfq6k\" (UniqueName: \"kubernetes.io/projected/0c40923a-2096-43f0-8d0f-315f2e4a2270-kube-api-access-pfq6k\") pod \"kuadrant-operator-controller-manager-57bb9bfff-ck96g\" (UID: \"0c40923a-2096-43f0-8d0f-315f2e4a2270\") " pod="kuadrant-system/kuadrant-operator-controller-manager-57bb9bfff-ck96g"
Oct 30 20:18:52.849839 ip-10-0-132-82 kubenswrapper[4892]: I1030 20:18:52.849772    4892 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/0c40923a-2096-43f0-8d0f-315f2e4a2270-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-57bb9bfff-ck96g\" (UID: \"0c40923a-2096-43f0-8d0f-315f2e4a2270\") " pod="kuadrant-system/kuadrant-operator-controller-manager-57bb9bfff-ck96g"
Oct 30 20:18:52.850192 ip-10-0-132-82 kubenswrapper[4892]: I1030 20:18:52.850175    4892 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/0c40923a-2096-43f0-8d0f-315f2e4a2270-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-57bb9bfff-ck96g\" (UID: \"0c40923a-2096-43f0-8d0f-315f2e4a2270\") " pod="kuadrant-system/kuadrant-operator-controller-manager-57bb9bfff-ck96g"
Oct 30 20:18:52.906331 ip-10-0-132-82 kubenswrapper[4892]: I1030 20:18:52.906308    4892 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-pfq6k\" (UniqueName: \"kubernetes.io/projected/0c40923a-2096-43f0-8d0f-315f2e4a2270-kube-api-access-pfq6k\") pod \"kuadrant-operator-controller-manager-57bb9bfff-ck96g\" (UID: \"0c40923a-2096-43f0-8d0f-315f2e4a2270\") " pod="kuadrant-system/kuadrant-operator-controller-manager-57bb9bfff-ck96g"
Oct 30 20:18:52.974378 ip-10-0-132-82 kubenswrapper[4892]: I1030 20:18:52.974307    4892 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-57bb9bfff-ck96g"
Oct 30 20:18:53.399236 ip-10-0-132-82 kubenswrapper[4892]: I1030 20:18:53.399213    4892 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-57bb9bfff-ck96g"]
Oct 30 20:18:53.401441 ip-10-0-132-82 kubenswrapper[4892]: W1030 20:18:53.401416    4892 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0c40923a_2096_43f0_8d0f_315f2e4a2270.slice/crio-c95657f341ea658e059c360d03c767ea66f001be0ed16567465d348ef219c8af WatchSource:0}: Error finding container c95657f341ea658e059c360d03c767ea66f001be0ed16567465d348ef219c8af: Status 404 returned error can't find the container with id c95657f341ea658e059c360d03c767ea66f001be0ed16567465d348ef219c8af
Oct 30 20:18:53.743403 ip-10-0-132-82 kubenswrapper[4892]: I1030 20:18:53.743317    4892 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-57bb9bfff-ck96g" event={"ID":"0c40923a-2096-43f0-8d0f-315f2e4a2270","Type":"ContainerStarted","Data":"27dad75e3fad5c6aff343622573af267d4f39961dbc3151d640b8edccf5e7ec5"}
Oct 30 20:18:53.743403 ip-10-0-132-82 kubenswrapper[4892]: I1030 20:18:53.743367    4892 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-57bb9bfff-ck96g" event={"ID":"0c40923a-2096-43f0-8d0f-315f2e4a2270","Type":"ContainerStarted","Data":"c95657f341ea658e059c360d03c767ea66f001be0ed16567465d348ef219c8af"}
Oct 30 20:18:53.743875 ip-10-0-132-82 kubenswrapper[4892]: I1030 20:18:53.743430    4892 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/kuadrant-operator-controller-manager-57bb9bfff-ck96g"
Oct 30 20:18:53.874662 ip-10-0-132-82 kubenswrapper[4892]: I1030 20:18:53.874614    4892 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kuadrant-system/kuadrant-operator-controller-manager-57bb9bfff-ck96g" podStartSLOduration=1.8746019440000001 podStartE2EDuration="1.874601944s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 20:18:52 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 20:18:53.874473269 +0000 UTC m=+3781.444938867" watchObservedRunningTime="2026-10-30 20:18:53.874601944 +0000 UTC m=+3781.445067542"
Oct 30 20:18:54.842933 ip-10-0-132-82 kubenswrapper[4892]: E1030 20:18:54.842896    4892 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-jrm87" containerName="extractor"
Oct 30 20:19:04.750312 ip-10-0-132-82 kubenswrapper[4892]: I1030 20:19:04.750284    4892 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/kuadrant-operator-controller-manager-57bb9bfff-ck96g"
Oct 30 20:19:13.263529 ip-10-0-132-82 kubenswrapper[4892]: I1030 20:19:13.263481    4892 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-fsfvk"]
Oct 30 20:19:22.053792 ip-10-0-132-82 kubenswrapper[4892]: I1030 20:19:22.053757    4892 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-fsfvk"]
Oct 30 20:19:24.844485 ip-10-0-132-82 kubenswrapper[4892]: E1030 20:19:24.844450    4892 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-jrm87" containerName="extractor"
Oct 30 20:19:54.843169 ip-10-0-132-82 kubenswrapper[4892]: E1030 20:19:54.843086    4892 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-jrm87" containerName="extractor"
Oct 30 20:20:24.843625 ip-10-0-132-82 kubenswrapper[4892]: E1030 20:20:24.843577    4892 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-jrm87" containerName="extractor"
Oct 30 20:20:27.635379 ip-10-0-132-82 kubenswrapper[4892]: I1030 20:20:27.635346    4892 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-fsfvk"]
Oct 30 20:20:47.176945 ip-10-0-132-82 kubenswrapper[4892]: I1030 20:20:47.176905    4892 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-fsfvk"]
Oct 30 20:20:53.492319 ip-10-0-132-82 kubenswrapper[4892]: I1030 20:20:53.492294    4892 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-tg5xj_11db9cec-653f-4aee-8937-0cf5b367efac/console-operator/3.log"
Oct 30 20:20:53.561518 ip-10-0-132-82 kubenswrapper[4892]: I1030 20:20:53.561496    4892 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-tg5xj_11db9cec-653f-4aee-8937-0cf5b367efac/console-operator/3.log"
Oct 30 20:20:54.844400 ip-10-0-132-82 kubenswrapper[4892]: E1030 20:20:54.844358    4892 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-jrm87" containerName="extractor"
Oct 30 20:21:11.158293 ip-10-0-132-82 kubenswrapper[4892]: I1030 20:21:11.158260    4892 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-fsfvk"]
Oct 30 20:21:12.960560 ip-10-0-132-82 kubenswrapper[4892]: I1030 20:21:12.960527    4892 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b4beeec2-s2mwf"]
Oct 30 20:21:12.964125 ip-10-0-132-82 kubenswrapper[4892]: I1030 20:21:12.964108    4892 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b4beeec2-s2mwf"
Oct 30 20:21:13.018030 ip-10-0-132-82 kubenswrapper[4892]: I1030 20:21:13.018002    4892 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 20:21:13.053282 ip-10-0-132-82 kubenswrapper[4892]: I1030 20:21:13.053254    4892 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/57233a47-7491-4605-ab04-b61557546a96-maas-api-xks-ca\") pod \"maas-api-revoke-sub-b4beeec2-s2mwf\" (UID: \"57233a47-7491-4605-ab04-b61557546a96\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b4beeec2-s2mwf"
Oct 30 20:21:13.053436 ip-10-0-132-82 kubenswrapper[4892]: I1030 20:21:13.053293    4892 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/57233a47-7491-4605-ab04-b61557546a96-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-b4beeec2-s2mwf\" (UID: \"57233a47-7491-4605-ab04-b61557546a96\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b4beeec2-s2mwf"
Oct 30 20:21:13.053436 ip-10-0-132-82 kubenswrapper[4892]: I1030 20:21:13.053317    4892 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kg45g\" (UniqueName: \"kubernetes.io/projected/57233a47-7491-4605-ab04-b61557546a96-kube-api-access-kg45g\") pod \"maas-api-revoke-sub-b4beeec2-s2mwf\" (UID: \"57233a47-7491-4605-ab04-b61557546a96\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b4beeec2-s2mwf"
Oct 30 20:21:13.067332 ip-10-0-132-82 kubenswrapper[4892]: I1030 20:21:13.067307    4892 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-nlxp4\""
Oct 30 20:21:13.067477 ip-10-0-132-82 kubenswrapper[4892]: I1030 20:21:13.067361    4892 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 30 20:21:13.067477 ip-10-0-132-82 kubenswrapper[4892]: I1030 20:21:13.067467    4892 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 30 20:21:13.070135 ip-10-0-132-82 kubenswrapper[4892]: I1030 20:21:13.070111    4892 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b4beeec2-s2mwf"]
Oct 30 20:21:13.154109 ip-10-0-132-82 kubenswrapper[4892]: I1030 20:21:13.154081    4892 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/57233a47-7491-4605-ab04-b61557546a96-maas-api-xks-ca\") pod \"maas-api-revoke-sub-b4beeec2-s2mwf\" (UID: \"57233a47-7491-4605-ab04-b61557546a96\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b4beeec2-s2mwf"
Oct 30 20:21:13.154233 ip-10-0-132-82 kubenswrapper[4892]: I1030 20:21:13.154119    4892 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/57233a47-7491-4605-ab04-b61557546a96-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-b4beeec2-s2mwf\" (UID: \"57233a47-7491-4605-ab04-b61557546a96\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b4beeec2-s2mwf"
Oct 30 20:21:13.154233 ip-10-0-132-82 kubenswrapper[4892]: I1030 20:21:13.154145    4892 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kg45g\" (UniqueName: \"kubernetes.io/projected/57233a47-7491-4605-ab04-b61557546a96-kube-api-access-kg45g\") pod \"maas-api-revoke-sub-b4beeec2-s2mwf\" (UID: \"57233a47-7491-4605-ab04-b61557546a96\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b4beeec2-s2mwf"
Oct 30 20:21:13.154737 ip-10-0-132-82 kubenswrapper[4892]: I1030 20:21:13.154718    4892 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/57233a47-7491-4605-ab04-b61557546a96-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-b4beeec2-s2mwf\" (UID: \"57233a47-7491-4605-ab04-b61557546a96\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b4beeec2-s2mwf"
Oct 30 20:21:13.156666 ip-10-0-132-82 kubenswrapper[4892]: I1030 20:21:13.156643    4892 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/57233a47-7491-4605-ab04-b61557546a96-maas-api-xks-ca\") pod \"maas-api-revoke-sub-b4beeec2-s2mwf\" (UID: \"57233a47-7491-4605-ab04-b61557546a96\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b4beeec2-s2mwf"
Oct 30 20:21:13.211283 ip-10-0-132-82 kubenswrapper[4892]: I1030 20:21:13.211228    4892 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-kg45g\" (UniqueName: \"kubernetes.io/projected/57233a47-7491-4605-ab04-b61557546a96-kube-api-access-kg45g\") pod \"maas-api-revoke-sub-b4beeec2-s2mwf\" (UID: \"57233a47-7491-4605-ab04-b61557546a96\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b4beeec2-s2mwf"
Oct 30 20:21:13.277203 ip-10-0-132-82 kubenswrapper[4892]: I1030 20:21:13.277178    4892 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b4beeec2-s2mwf"
Oct 30 20:21:13.647184 ip-10-0-132-82 kubenswrapper[4892]: I1030 20:21:13.647162    4892 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b4beeec2-s2mwf"]
Oct 30 20:21:13.649241 ip-10-0-132-82 kubenswrapper[4892]: W1030 20:21:13.649217    4892 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod57233a47_7491_4605_ab04_b61557546a96.slice/crio-11bf7ba0a2301e23490f710c31f26bab907a8d8f0db4f4cf8429dce45df620f6 WatchSource:0}: Error finding container 11bf7ba0a2301e23490f710c31f26bab907a8d8f0db4f4cf8429dce45df620f6: Status 404 returned error can't find the container with id 11bf7ba0a2301e23490f710c31f26bab907a8d8f0db4f4cf8429dce45df620f6
Oct 30 20:21:14.487524 ip-10-0-132-82 kubenswrapper[4892]: I1030 20:21:14.487492    4892 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b4beeec2-s2mwf" event={"ID":"57233a47-7491-4605-ab04-b61557546a96","Type":"ContainerStarted","Data":"bc2518691792913f55a9632d5acf4e6c7e12d23c6aa4fc74909d7b8958a8ec5a"}
Oct 30 20:21:14.487894 ip-10-0-132-82 kubenswrapper[4892]: I1030 20:21:14.487546    4892 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b4beeec2-s2mwf" event={"ID":"57233a47-7491-4605-ab04-b61557546a96","Type":"ContainerStarted","Data":"11bf7ba0a2301e23490f710c31f26bab907a8d8f0db4f4cf8429dce45df620f6"}
Oct 30 20:21:14.600879 ip-10-0-132-82 kubenswrapper[4892]: I1030 20:21:14.600834    4892 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b4beeec2-s2mwf" podStartSLOduration=2.6008154020000003 podStartE2EDuration="2.600815402s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 20:21:12 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 20:21:14.598518516 +0000 UTC m=+3922.168984118" watchObservedRunningTime="2026-10-30 20:21:14.600815402 +0000 UTC m=+3922.171281000"
Oct 30 20:21:15.494489 ip-10-0-132-82 kubenswrapper[4892]: I1030 20:21:15.494455    4892 generic.go:356] "Generic (PLEG): container finished" podID="57233a47-7491-4605-ab04-b61557546a96" containerID="bc2518691792913f55a9632d5acf4e6c7e12d23c6aa4fc74909d7b8958a8ec5a" exitCode=0
Oct 30 20:21:15.494888 ip-10-0-132-82 kubenswrapper[4892]: I1030 20:21:15.494522    4892 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b4beeec2-s2mwf" event={"ID":"57233a47-7491-4605-ab04-b61557546a96","Type":"ContainerDied","Data":"bc2518691792913f55a9632d5acf4e6c7e12d23c6aa4fc74909d7b8958a8ec5a"}
Oct 30 20:21:16.632635 ip-10-0-132-82 kubenswrapper[4892]: I1030 20:21:16.632613    4892 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b4beeec2-s2mwf"
Oct 30 20:21:16.786665 ip-10-0-132-82 kubenswrapper[4892]: I1030 20:21:16.786593    4892 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/57233a47-7491-4605-ab04-b61557546a96-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/57233a47-7491-4605-ab04-b61557546a96-maas-api-xks-ca\") pod \"57233a47-7491-4605-ab04-b61557546a96\" (UID: \"57233a47-7491-4605-ab04-b61557546a96\") "
Oct 30 20:21:16.786665 ip-10-0-132-82 kubenswrapper[4892]: I1030 20:21:16.786657    4892 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/57233a47-7491-4605-ab04-b61557546a96-kube-api-access-kg45g\" (UniqueName: \"kubernetes.io/projected/57233a47-7491-4605-ab04-b61557546a96-kube-api-access-kg45g\") pod \"57233a47-7491-4605-ab04-b61557546a96\" (UID: \"57233a47-7491-4605-ab04-b61557546a96\") "
Oct 30 20:21:16.786847 ip-10-0-132-82 kubenswrapper[4892]: I1030 20:21:16.786708    4892 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/57233a47-7491-4605-ab04-b61557546a96-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/57233a47-7491-4605-ab04-b61557546a96-maas-api-openshift-ca\") pod \"57233a47-7491-4605-ab04-b61557546a96\" (UID: \"57233a47-7491-4605-ab04-b61557546a96\") "
Oct 30 20:21:16.787120 ip-10-0-132-82 kubenswrapper[4892]: I1030 20:21:16.787093    4892 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/57233a47-7491-4605-ab04-b61557546a96-maas-api-openshift-ca" pod "57233a47-7491-4605-ab04-b61557546a96" (UID: "57233a47-7491-4605-ab04-b61557546a96"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 20:21:16.788787 ip-10-0-132-82 kubenswrapper[4892]: I1030 20:21:16.788761    4892 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/57233a47-7491-4605-ab04-b61557546a96-maas-api-xks-ca" pod "57233a47-7491-4605-ab04-b61557546a96" (UID: "57233a47-7491-4605-ab04-b61557546a96"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 20:21:16.788887 ip-10-0-132-82 kubenswrapper[4892]: I1030 20:21:16.788827    4892 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/57233a47-7491-4605-ab04-b61557546a96-kube-api-access-kg45g" pod "57233a47-7491-4605-ab04-b61557546a96" (UID: "57233a47-7491-4605-ab04-b61557546a96"). InnerVolumeSpecName "kube-api-access-kg45g". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 20:21:16.887499 ip-10-0-132-82 kubenswrapper[4892]: I1030 20:21:16.887468    4892 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/57233a47-7491-4605-ab04-b61557546a96-maas-api-openshift-ca\") on node \"ip-10-0-132-82.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:21:16.887499 ip-10-0-132-82 kubenswrapper[4892]: I1030 20:21:16.887496    4892 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/57233a47-7491-4605-ab04-b61557546a96-maas-api-xks-ca\") on node \"ip-10-0-132-82.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:21:16.887499 ip-10-0-132-82 kubenswrapper[4892]: I1030 20:21:16.887505    4892 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kg45g\" (UniqueName: \"kubernetes.io/projected/57233a47-7491-4605-ab04-b61557546a96-kube-api-access-kg45g\") on node \"ip-10-0-132-82.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:21:17.506568 ip-10-0-132-82 kubenswrapper[4892]: I1030 20:21:17.506527    4892 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b4beeec2-s2mwf" event={"ID":"57233a47-7491-4605-ab04-b61557546a96","Type":"ContainerDied","Data":"11bf7ba0a2301e23490f710c31f26bab907a8d8f0db4f4cf8429dce45df620f6"}
Oct 30 20:21:17.506568 ip-10-0-132-82 kubenswrapper[4892]: I1030 20:21:17.506562    4892 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b4beeec2-s2mwf"
Oct 30 20:21:17.506823 ip-10-0-132-82 kubenswrapper[4892]: I1030 20:21:17.506614    4892 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="11bf7ba0a2301e23490f710c31f26bab907a8d8f0db4f4cf8429dce45df620f6"
Oct 30 20:21:24.843677 ip-10-0-132-82 kubenswrapper[4892]: E1030 20:21:24.843639    4892 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-jrm87" containerName="extractor"
Oct 30 20:21:25.767663 ip-10-0-132-82 kubenswrapper[4892]: I1030 20:21:25.767631    4892 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-fsfvk"]
Oct 30 20:21:28.046758 ip-10-0-132-82 kubenswrapper[4892]: I1030 20:21:28.046723    4892 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-8e78ffb3-qhtjz"]
Oct 30 20:21:28.096163 ip-10-0-132-82 kubenswrapper[4892]: I1030 20:21:28.096130    4892 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-8e78ffb3-qhtjz"]
Oct 30 20:21:28.569240 ip-10-0-132-82 kubenswrapper[4892]: I1030 20:21:28.569200    4892 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="bac05a0f-9ed2-41f0-9880-4f6d01ba29cf" path="/var/lib/kubelet/pods/bac05a0f-9ed2-41f0-9880-4f6d01ba29cf/volumes"
Oct 30 20:21:34.323133 ip-10-0-132-82 kubenswrapper[4892]: I1030 20:21:34.323100    4892 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-14ab4d-ztbl4"]
Oct 30 20:21:34.323541 ip-10-0-132-82 kubenswrapper[4892]: I1030 20:21:34.323527    4892 cpu_manager.go:414] "containerMap: removing container" podUID="57233a47-7491-4605-ab04-b61557546a96" containerName="revoke-keys"
Oct 30 20:21:34.323592 ip-10-0-132-82 kubenswrapper[4892]: I1030 20:21:34.323542    4892 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="57233a47-7491-4605-ab04-b61557546a96" containerName="revoke-keys"
Oct 30 20:21:34.323664 ip-10-0-132-82 kubenswrapper[4892]: I1030 20:21:34.323654    4892 memory_manager.go:365] "RemoveStaleState removing state" podUID="57233a47-7491-4605-ab04-b61557546a96" containerName="revoke-keys"
Oct 30 20:21:34.326890 ip-10-0-132-82 kubenswrapper[4892]: I1030 20:21:34.326872    4892 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-14ab4d-ztbl4"
Oct 30 20:21:34.342036 ip-10-0-132-82 kubenswrapper[4892]: I1030 20:21:34.342013    4892 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-v6mhf\" (UniqueName: \"kubernetes.io/projected/809cb537-1472-41ad-800b-69ba73c28cd9-kube-api-access-v6mhf\") pod \"maas-api-revoke-keys-e2e-worker-single-main-14ab4d-ztbl4\" (UID: \"809cb537-1472-41ad-800b-69ba73c28cd9\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-14ab4d-ztbl4"
Oct 30 20:21:34.342156 ip-10-0-132-82 kubenswrapper[4892]: I1030 20:21:34.342090    4892 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/809cb537-1472-41ad-800b-69ba73c28cd9-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-single-main-14ab4d-ztbl4\" (UID: \"809cb537-1472-41ad-800b-69ba73c28cd9\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-14ab4d-ztbl4"
Oct 30 20:21:34.342202 ip-10-0-132-82 kubenswrapper[4892]: I1030 20:21:34.342152    4892 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/809cb537-1472-41ad-800b-69ba73c28cd9-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-single-main-14ab4d-ztbl4\" (UID: \"809cb537-1472-41ad-800b-69ba73c28cd9\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-14ab4d-ztbl4"
Oct 30 20:21:34.431085 ip-10-0-132-82 kubenswrapper[4892]: I1030 20:21:34.431053    4892 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 30 20:21:34.431248 ip-10-0-132-82 kubenswrapper[4892]: I1030 20:21:34.431215    4892 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 20:21:34.431331 ip-10-0-132-82 kubenswrapper[4892]: I1030 20:21:34.431282    4892 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 30 20:21:34.431331 ip-10-0-132-82 kubenswrapper[4892]: I1030 20:21:34.431283    4892 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-nlxp4\""
Oct 30 20:21:34.433404 ip-10-0-132-82 kubenswrapper[4892]: I1030 20:21:34.433380    4892 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-14ab4d-ztbl4"]
Oct 30 20:21:34.442715 ip-10-0-132-82 kubenswrapper[4892]: I1030 20:21:34.442695    4892 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/809cb537-1472-41ad-800b-69ba73c28cd9-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-single-main-14ab4d-ztbl4\" (UID: \"809cb537-1472-41ad-800b-69ba73c28cd9\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-14ab4d-ztbl4"
Oct 30 20:21:34.442802 ip-10-0-132-82 kubenswrapper[4892]: I1030 20:21:34.442725    4892 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-v6mhf\" (UniqueName: \"kubernetes.io/projected/809cb537-1472-41ad-800b-69ba73c28cd9-kube-api-access-v6mhf\") pod \"maas-api-revoke-keys-e2e-worker-single-main-14ab4d-ztbl4\" (UID: \"809cb537-1472-41ad-800b-69ba73c28cd9\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-14ab4d-ztbl4"
Oct 30 20:21:34.442802 ip-10-0-132-82 kubenswrapper[4892]: I1030 20:21:34.442798    4892 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/809cb537-1472-41ad-800b-69ba73c28cd9-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-single-main-14ab4d-ztbl4\" (UID: \"809cb537-1472-41ad-800b-69ba73c28cd9\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-14ab4d-ztbl4"
Oct 30 20:21:34.443379 ip-10-0-132-82 kubenswrapper[4892]: I1030 20:21:34.443362    4892 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/809cb537-1472-41ad-800b-69ba73c28cd9-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-single-main-14ab4d-ztbl4\" (UID: \"809cb537-1472-41ad-800b-69ba73c28cd9\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-14ab4d-ztbl4"
Oct 30 20:21:34.445318 ip-10-0-132-82 kubenswrapper[4892]: I1030 20:21:34.445299    4892 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/809cb537-1472-41ad-800b-69ba73c28cd9-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-single-main-14ab4d-ztbl4\" (UID: \"809cb537-1472-41ad-800b-69ba73c28cd9\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-14ab4d-ztbl4"
Oct 30 20:21:34.503976 ip-10-0-132-82 kubenswrapper[4892]: I1030 20:21:34.499708    4892 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-v6mhf\" (UniqueName: \"kubernetes.io/projected/809cb537-1472-41ad-800b-69ba73c28cd9-kube-api-access-v6mhf\") pod \"maas-api-revoke-keys-e2e-worker-single-main-14ab4d-ztbl4\" (UID: \"809cb537-1472-41ad-800b-69ba73c28cd9\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-14ab4d-ztbl4"
Oct 30 20:21:34.639889 ip-10-0-132-82 kubenswrapper[4892]: I1030 20:21:34.639868    4892 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-14ab4d-ztbl4"
Oct 30 20:21:35.013751 ip-10-0-132-82 kubenswrapper[4892]: I1030 20:21:35.013721    4892 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-14ab4d-ztbl4"]
Oct 30 20:21:35.016857 ip-10-0-132-82 kubenswrapper[4892]: W1030 20:21:35.016829    4892 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod809cb537_1472_41ad_800b_69ba73c28cd9.slice/crio-d1cfc0b104a48e4521c468b0be9085020d8904d45d33394e74fc127cd02900c5 WatchSource:0}: Error finding container d1cfc0b104a48e4521c468b0be9085020d8904d45d33394e74fc127cd02900c5: Status 404 returned error can't find the container with id d1cfc0b104a48e4521c468b0be9085020d8904d45d33394e74fc127cd02900c5
Oct 30 20:21:35.598488 ip-10-0-132-82 kubenswrapper[4892]: I1030 20:21:35.598404    4892 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-14ab4d-ztbl4" event={"ID":"809cb537-1472-41ad-800b-69ba73c28cd9","Type":"ContainerStarted","Data":"eeb48d01131b7509e114ca1a405287f656b9f9d60e51da8c01427442a26cd0c6"}
Oct 30 20:21:35.598488 ip-10-0-132-82 kubenswrapper[4892]: I1030 20:21:35.598454    4892 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-14ab4d-ztbl4" event={"ID":"809cb537-1472-41ad-800b-69ba73c28cd9","Type":"ContainerStarted","Data":"d1cfc0b104a48e4521c468b0be9085020d8904d45d33394e74fc127cd02900c5"}
Oct 30 20:21:35.711602 ip-10-0-132-82 kubenswrapper[4892]: I1030 20:21:35.711557    4892 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-14ab4d-ztbl4" podStartSLOduration=1.711542937 podStartE2EDuration="1.711542937s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 20:21:34 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 20:21:35.710194137 +0000 UTC m=+3943.280659736" watchObservedRunningTime="2026-10-30 20:21:35.711542937 +0000 UTC m=+3943.282008535"
Oct 30 20:21:36.604817 ip-10-0-132-82 kubenswrapper[4892]: I1030 20:21:36.604784    4892 generic.go:356] "Generic (PLEG): container finished" podID="809cb537-1472-41ad-800b-69ba73c28cd9" containerID="eeb48d01131b7509e114ca1a405287f656b9f9d60e51da8c01427442a26cd0c6" exitCode=0
Oct 30 20:21:36.605265 ip-10-0-132-82 kubenswrapper[4892]: I1030 20:21:36.604875    4892 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-14ab4d-ztbl4" event={"ID":"809cb537-1472-41ad-800b-69ba73c28cd9","Type":"ContainerDied","Data":"eeb48d01131b7509e114ca1a405287f656b9f9d60e51da8c01427442a26cd0c6"}
Oct 30 20:21:37.742615 ip-10-0-132-82 kubenswrapper[4892]: I1030 20:21:37.742590    4892 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-14ab4d-ztbl4"
Oct 30 20:21:37.773486 ip-10-0-132-82 kubenswrapper[4892]: I1030 20:21:37.773458    4892 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/809cb537-1472-41ad-800b-69ba73c28cd9-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/809cb537-1472-41ad-800b-69ba73c28cd9-maas-api-xks-ca\") pod \"809cb537-1472-41ad-800b-69ba73c28cd9\" (UID: \"809cb537-1472-41ad-800b-69ba73c28cd9\") "
Oct 30 20:21:37.773617 ip-10-0-132-82 kubenswrapper[4892]: I1030 20:21:37.773540    4892 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/809cb537-1472-41ad-800b-69ba73c28cd9-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/809cb537-1472-41ad-800b-69ba73c28cd9-maas-api-openshift-ca\") pod \"809cb537-1472-41ad-800b-69ba73c28cd9\" (UID: \"809cb537-1472-41ad-800b-69ba73c28cd9\") "
Oct 30 20:21:37.773724 ip-10-0-132-82 kubenswrapper[4892]: I1030 20:21:37.773612    4892 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/809cb537-1472-41ad-800b-69ba73c28cd9-kube-api-access-v6mhf\" (UniqueName: \"kubernetes.io/projected/809cb537-1472-41ad-800b-69ba73c28cd9-kube-api-access-v6mhf\") pod \"809cb537-1472-41ad-800b-69ba73c28cd9\" (UID: \"809cb537-1472-41ad-800b-69ba73c28cd9\") "
Oct 30 20:21:37.774053 ip-10-0-132-82 kubenswrapper[4892]: I1030 20:21:37.773884    4892 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/809cb537-1472-41ad-800b-69ba73c28cd9-maas-api-openshift-ca" pod "809cb537-1472-41ad-800b-69ba73c28cd9" (UID: "809cb537-1472-41ad-800b-69ba73c28cd9"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 20:21:37.774151 ip-10-0-132-82 kubenswrapper[4892]: I1030 20:21:37.774049    4892 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/809cb537-1472-41ad-800b-69ba73c28cd9-maas-api-openshift-ca\") on node \"ip-10-0-132-82.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:21:37.775692 ip-10-0-132-82 kubenswrapper[4892]: I1030 20:21:37.775667    4892 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/809cb537-1472-41ad-800b-69ba73c28cd9-maas-api-xks-ca" pod "809cb537-1472-41ad-800b-69ba73c28cd9" (UID: "809cb537-1472-41ad-800b-69ba73c28cd9"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 20:21:37.775801 ip-10-0-132-82 kubenswrapper[4892]: I1030 20:21:37.775767    4892 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/809cb537-1472-41ad-800b-69ba73c28cd9-kube-api-access-v6mhf" pod "809cb537-1472-41ad-800b-69ba73c28cd9" (UID: "809cb537-1472-41ad-800b-69ba73c28cd9"). InnerVolumeSpecName "kube-api-access-v6mhf". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 20:21:37.874776 ip-10-0-132-82 kubenswrapper[4892]: I1030 20:21:37.874699    4892 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-v6mhf\" (UniqueName: \"kubernetes.io/projected/809cb537-1472-41ad-800b-69ba73c28cd9-kube-api-access-v6mhf\") on node \"ip-10-0-132-82.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:21:37.874776 ip-10-0-132-82 kubenswrapper[4892]: I1030 20:21:37.874724    4892 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/809cb537-1472-41ad-800b-69ba73c28cd9-maas-api-xks-ca\") on node \"ip-10-0-132-82.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:21:38.616835 ip-10-0-132-82 kubenswrapper[4892]: I1030 20:21:38.616798    4892 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-14ab4d-ztbl4"
Oct 30 20:21:38.616835 ip-10-0-132-82 kubenswrapper[4892]: I1030 20:21:38.616836    4892 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-14ab4d-ztbl4" event={"ID":"809cb537-1472-41ad-800b-69ba73c28cd9","Type":"ContainerDied","Data":"d1cfc0b104a48e4521c468b0be9085020d8904d45d33394e74fc127cd02900c5"}
Oct 30 20:21:38.617092 ip-10-0-132-82 kubenswrapper[4892]: I1030 20:21:38.616882    4892 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="d1cfc0b104a48e4521c468b0be9085020d8904d45d33394e74fc127cd02900c5"
Oct 30 20:21:54.843631 ip-10-0-132-82 kubenswrapper[4892]: E1030 20:21:54.843593    4892 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-jrm87" containerName="extractor"
Oct 30 20:21:55.405703 ip-10-0-132-82 kubenswrapper[4892]: I1030 20:21:55.405669    4892 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-14ab4d-ztbl4"]
Oct 30 20:21:55.454375 ip-10-0-132-82 kubenswrapper[4892]: I1030 20:21:55.454344    4892 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-14ab4d-ztbl4"]
Oct 30 20:21:56.568116 ip-10-0-132-82 kubenswrapper[4892]: I1030 20:21:56.568081    4892 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="809cb537-1472-41ad-800b-69ba73c28cd9" path="/var/lib/kubelet/pods/809cb537-1472-41ad-800b-69ba73c28cd9/volumes"
Oct 30 20:21:57.750917 ip-10-0-132-82 kubenswrapper[4892]: I1030 20:21:57.750882    4892 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-fsfvk"]
Oct 30 20:22:24.842678 ip-10-0-132-82 kubenswrapper[4892]: E1030 20:22:24.842641    4892 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-jrm87" containerName="extractor"
Oct 30 20:22:39.225242 ip-10-0-132-82 kubenswrapper[4892]: I1030 20:22:39.225160    4892 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-fsfvk"]
Oct 30 20:22:54.843249 ip-10-0-132-82 kubenswrapper[4892]: E1030 20:22:54.843208    4892 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-jrm87" containerName="extractor"
Oct 30 20:22:58.923483 ip-10-0-132-82 kubenswrapper[4892]: I1030 20:22:58.923448    4892 scope.go:122] "RemoveContainer" containerID="4cc121ff2ff525bf3f09bd637e69148b1e4347931aaa3f70a87a4e33cb8c5c28"
Oct 30 20:22:58.934445 ip-10-0-132-82 kubenswrapper[4892]: I1030 20:22:58.934422    4892 scope.go:122] "RemoveContainer" containerID="6f1f402ec204d7ee59040b68ce9ff5d654ba14c4821be41926d573b3fae13c09"
Oct 30 20:23:02.838659 ip-10-0-132-82 kubenswrapper[4892]: I1030 20:23:02.838627    4892 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-fsfvk"]
Oct 30 20:23:06.118605 ip-10-0-132-82 kubenswrapper[4892]: I1030 20:23:06.118554    4892 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6db586dd6c-k85mx container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.1.1:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 30 20:23:06.119735 ip-10-0-132-82 kubenswrapper[4892]: I1030 20:23:06.119292    4892 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-k85mx" podUID="6f05c89a-d631-4c95-aad6-d08d6c87cffd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.1:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 30 20:23:06.552178 ip-10-0-132-82 kubenswrapper[4892]: I1030 20:23:06.552138    4892 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6db586dd6c-p7m7x container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.252:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 30 20:23:06.552532 ip-10-0-132-82 kubenswrapper[4892]: I1030 20:23:06.552477    4892 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-p7m7x" podUID="741c37d3-758a-4ced-ab0e-9038baea2d88" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.252:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 30 20:23:06.666103 ip-10-0-132-82 kubenswrapper[4892]: I1030 20:23:06.666072    4892 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6db586dd6c-mb59v container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.253:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 30 20:23:06.666305 ip-10-0-132-82 kubenswrapper[4892]: I1030 20:23:06.666123    4892 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-mb59v" podUID="72926e14-705b-493a-879f-f595c35a7490" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.253:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 30 20:23:24.842940 ip-10-0-132-82 kubenswrapper[4892]: E1030 20:23:24.842904    4892 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-jrm87" containerName="extractor"
Oct 30 20:23:32.033618 ip-10-0-132-82 kubenswrapper[4892]: I1030 20:23:32.033582    4892 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-fsfvk"]
Oct 30 20:23:47.478911 ip-10-0-132-82 kubenswrapper[4892]: I1030 20:23:47.478875    4892 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-fsfvk"]
Oct 30 20:23:54.842105 ip-10-0-132-82 kubenswrapper[4892]: E1030 20:23:54.842065    4892 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-jrm87" containerName="extractor"
Oct 30 20:24:03.345365 ip-10-0-132-82 kubenswrapper[4892]: I1030 20:24:03.345328    4892 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-fsfvk"]
Oct 30 20:24:06.120142 ip-10-0-132-82 kubenswrapper[4892]: I1030 20:24:06.120097    4892 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6db586dd6c-k85mx container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.1.1:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 30 20:24:06.120577 ip-10-0-132-82 kubenswrapper[4892]: I1030 20:24:06.120170    4892 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-k85mx" podUID="6f05c89a-d631-4c95-aad6-d08d6c87cffd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.1:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 30 20:24:16.033899 ip-10-0-132-82 kubenswrapper[4892]: I1030 20:24:16.033863    4892 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-fsfvk"]
Oct 30 20:24:24.844359 ip-10-0-132-82 kubenswrapper[4892]: E1030 20:24:24.844325    4892 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-jrm87" containerName="extractor"
Oct 30 20:24:29.956992 ip-10-0-132-82 kubenswrapper[4892]: I1030 20:24:29.956941    4892 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-fsfvk"]
Oct 30 20:24:50.356206 ip-10-0-132-82 kubenswrapper[4892]: I1030 20:24:50.356174    4892 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-fsfvk"]
Oct 30 20:24:54.845873 ip-10-0-132-82 kubenswrapper[4892]: E1030 20:24:54.845831    4892 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-jrm87" containerName="extractor"
Oct 30 20:25:01.458363 ip-10-0-132-82 kubenswrapper[4892]: I1030 20:25:01.458323    4892 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-fsfvk"]
Oct 30 20:25:12.686174 ip-10-0-132-82 kubenswrapper[4892]: I1030 20:25:12.686139    4892 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-fsfvk"]
Oct 30 20:25:18.025927 ip-10-0-132-82 kubenswrapper[4892]: I1030 20:25:18.025890    4892 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f03629db-xc594"]
Oct 30 20:25:18.026332 ip-10-0-132-82 kubenswrapper[4892]: I1030 20:25:18.026279    4892 cpu_manager.go:414] "containerMap: removing container" podUID="809cb537-1472-41ad-800b-69ba73c28cd9" containerName="revoke-keys"
Oct 30 20:25:18.026332 ip-10-0-132-82 kubenswrapper[4892]: I1030 20:25:18.026290    4892 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="809cb537-1472-41ad-800b-69ba73c28cd9" containerName="revoke-keys"
Oct 30 20:25:18.026415 ip-10-0-132-82 kubenswrapper[4892]: I1030 20:25:18.026359    4892 memory_manager.go:365] "RemoveStaleState removing state" podUID="809cb537-1472-41ad-800b-69ba73c28cd9" containerName="revoke-keys"
Oct 30 20:25:18.029472 ip-10-0-132-82 kubenswrapper[4892]: I1030 20:25:18.029452    4892 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f03629db-xc594"
Oct 30 20:25:18.090762 ip-10-0-132-82 kubenswrapper[4892]: I1030 20:25:18.090733    4892 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 30 20:25:18.133645 ip-10-0-132-82 kubenswrapper[4892]: I1030 20:25:18.133611    4892 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 30 20:25:18.133802 ip-10-0-132-82 kubenswrapper[4892]: I1030 20:25:18.133680    4892 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 20:25:18.133802 ip-10-0-132-82 kubenswrapper[4892]: I1030 20:25:18.133689    4892 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-nlxp4\""
Oct 30 20:25:18.135803 ip-10-0-132-82 kubenswrapper[4892]: I1030 20:25:18.135782    4892 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f03629db-xc594"]
Oct 30 20:25:18.143063 ip-10-0-132-82 kubenswrapper[4892]: I1030 20:25:18.143041    4892 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/856a6aa7-8f9f-44f0-830b-f381709b4651-maas-api-xks-ca\") pod \"maas-api-revoke-sub-f03629db-xc594\" (UID: \"856a6aa7-8f9f-44f0-830b-f381709b4651\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f03629db-xc594"
Oct 30 20:25:18.143139 ip-10-0-132-82 kubenswrapper[4892]: I1030 20:25:18.143105    4892 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/856a6aa7-8f9f-44f0-830b-f381709b4651-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-f03629db-xc594\" (UID: \"856a6aa7-8f9f-44f0-830b-f381709b4651\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f03629db-xc594"
Oct 30 20:25:18.143241 ip-10-0-132-82 kubenswrapper[4892]: I1030 20:25:18.143225    4892 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-58bz6\" (UniqueName: \"kubernetes.io/projected/856a6aa7-8f9f-44f0-830b-f381709b4651-kube-api-access-58bz6\") pod \"maas-api-revoke-sub-f03629db-xc594\" (UID: \"856a6aa7-8f9f-44f0-830b-f381709b4651\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f03629db-xc594"
Oct 30 20:25:18.244619 ip-10-0-132-82 kubenswrapper[4892]: I1030 20:25:18.244594    4892 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/856a6aa7-8f9f-44f0-830b-f381709b4651-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-f03629db-xc594\" (UID: \"856a6aa7-8f9f-44f0-830b-f381709b4651\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f03629db-xc594"
Oct 30 20:25:18.244746 ip-10-0-132-82 kubenswrapper[4892]: I1030 20:25:18.244646    4892 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-58bz6\" (UniqueName: \"kubernetes.io/projected/856a6aa7-8f9f-44f0-830b-f381709b4651-kube-api-access-58bz6\") pod \"maas-api-revoke-sub-f03629db-xc594\" (UID: \"856a6aa7-8f9f-44f0-830b-f381709b4651\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f03629db-xc594"
Oct 30 20:25:18.244746 ip-10-0-132-82 kubenswrapper[4892]: I1030 20:25:18.244678    4892 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/856a6aa7-8f9f-44f0-830b-f381709b4651-maas-api-xks-ca\") pod \"maas-api-revoke-sub-f03629db-xc594\" (UID: \"856a6aa7-8f9f-44f0-830b-f381709b4651\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f03629db-xc594"
Oct 30 20:25:18.245398 ip-10-0-132-82 kubenswrapper[4892]: I1030 20:25:18.245339    4892 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/856a6aa7-8f9f-44f0-830b-f381709b4651-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-f03629db-xc594\" (UID: \"856a6aa7-8f9f-44f0-830b-f381709b4651\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f03629db-xc594"
Oct 30 20:25:18.247164 ip-10-0-132-82 kubenswrapper[4892]: I1030 20:25:18.247142    4892 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/856a6aa7-8f9f-44f0-830b-f381709b4651-maas-api-xks-ca\") pod \"maas-api-revoke-sub-f03629db-xc594\" (UID: \"856a6aa7-8f9f-44f0-830b-f381709b4651\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f03629db-xc594"
Oct 30 20:25:18.300496 ip-10-0-132-82 kubenswrapper[4892]: I1030 20:25:18.300446    4892 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-58bz6\" (UniqueName: \"kubernetes.io/projected/856a6aa7-8f9f-44f0-830b-f381709b4651-kube-api-access-58bz6\") pod \"maas-api-revoke-sub-f03629db-xc594\" (UID: \"856a6aa7-8f9f-44f0-830b-f381709b4651\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f03629db-xc594"
Oct 30 20:25:18.342492 ip-10-0-132-82 kubenswrapper[4892]: I1030 20:25:18.342469    4892 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f03629db-xc594"
Oct 30 20:25:18.712048 ip-10-0-132-82 kubenswrapper[4892]: I1030 20:25:18.712023    4892 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f03629db-xc594"]
Oct 30 20:25:18.713833 ip-10-0-132-82 kubenswrapper[4892]: W1030 20:25:18.713805    4892 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod856a6aa7_8f9f_44f0_830b_f381709b4651.slice/crio-2621f95f9394d986a5ad26c5d9e769350a28fda28b570097574c5cfaf963e31e WatchSource:0}: Error finding container 2621f95f9394d986a5ad26c5d9e769350a28fda28b570097574c5cfaf963e31e: Status 404 returned error can't find the container with id 2621f95f9394d986a5ad26c5d9e769350a28fda28b570097574c5cfaf963e31e
Oct 30 20:25:19.008416 ip-10-0-132-82 kubenswrapper[4892]: I1030 20:25:19.008354    4892 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f03629db-xc594" event={"ID":"856a6aa7-8f9f-44f0-830b-f381709b4651","Type":"ContainerStarted","Data":"2621f95f9394d986a5ad26c5d9e769350a28fda28b570097574c5cfaf963e31e"}
Oct 30 20:25:20.015031 ip-10-0-132-82 kubenswrapper[4892]: I1030 20:25:20.014996    4892 generic.go:356] "Generic (PLEG): container finished" podID="856a6aa7-8f9f-44f0-830b-f381709b4651" containerID="f984832b32b90e92b2ab500bbb48dedad5c73086e7bbcf70e3236bee1bd98d41" exitCode=0
Oct 30 20:25:20.015442 ip-10-0-132-82 kubenswrapper[4892]: I1030 20:25:20.015077    4892 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f03629db-xc594" event={"ID":"856a6aa7-8f9f-44f0-830b-f381709b4651","Type":"ContainerDied","Data":"f984832b32b90e92b2ab500bbb48dedad5c73086e7bbcf70e3236bee1bd98d41"}
Oct 30 20:25:21.156786 ip-10-0-132-82 kubenswrapper[4892]: I1030 20:25:21.156761    4892 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f03629db-xc594"
Oct 30 20:25:21.271020 ip-10-0-132-82 kubenswrapper[4892]: I1030 20:25:21.270901    4892 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/856a6aa7-8f9f-44f0-830b-f381709b4651-kube-api-access-58bz6\" (UniqueName: \"kubernetes.io/projected/856a6aa7-8f9f-44f0-830b-f381709b4651-kube-api-access-58bz6\") pod \"856a6aa7-8f9f-44f0-830b-f381709b4651\" (UID: \"856a6aa7-8f9f-44f0-830b-f381709b4651\") "
Oct 30 20:25:21.271020 ip-10-0-132-82 kubenswrapper[4892]: I1030 20:25:21.271000    4892 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/856a6aa7-8f9f-44f0-830b-f381709b4651-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/856a6aa7-8f9f-44f0-830b-f381709b4651-maas-api-openshift-ca\") pod \"856a6aa7-8f9f-44f0-830b-f381709b4651\" (UID: \"856a6aa7-8f9f-44f0-830b-f381709b4651\") "
Oct 30 20:25:21.271237 ip-10-0-132-82 kubenswrapper[4892]: I1030 20:25:21.271030    4892 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/856a6aa7-8f9f-44f0-830b-f381709b4651-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/856a6aa7-8f9f-44f0-830b-f381709b4651-maas-api-xks-ca\") pod \"856a6aa7-8f9f-44f0-830b-f381709b4651\" (UID: \"856a6aa7-8f9f-44f0-830b-f381709b4651\") "
Oct 30 20:25:21.271810 ip-10-0-132-82 kubenswrapper[4892]: I1030 20:25:21.271767    4892 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/856a6aa7-8f9f-44f0-830b-f381709b4651-maas-api-openshift-ca" pod "856a6aa7-8f9f-44f0-830b-f381709b4651" (UID: "856a6aa7-8f9f-44f0-830b-f381709b4651"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 20:25:21.273695 ip-10-0-132-82 kubenswrapper[4892]: I1030 20:25:21.273664    4892 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/856a6aa7-8f9f-44f0-830b-f381709b4651-kube-api-access-58bz6" pod "856a6aa7-8f9f-44f0-830b-f381709b4651" (UID: "856a6aa7-8f9f-44f0-830b-f381709b4651"). InnerVolumeSpecName "kube-api-access-58bz6". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 20:25:21.274423 ip-10-0-132-82 kubenswrapper[4892]: I1030 20:25:21.274396    4892 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/856a6aa7-8f9f-44f0-830b-f381709b4651-maas-api-xks-ca" pod "856a6aa7-8f9f-44f0-830b-f381709b4651" (UID: "856a6aa7-8f9f-44f0-830b-f381709b4651"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 20:25:21.372256 ip-10-0-132-82 kubenswrapper[4892]: I1030 20:25:21.372222    4892 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-58bz6\" (UniqueName: \"kubernetes.io/projected/856a6aa7-8f9f-44f0-830b-f381709b4651-kube-api-access-58bz6\") on node \"ip-10-0-132-82.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:25:21.372256 ip-10-0-132-82 kubenswrapper[4892]: I1030 20:25:21.372255    4892 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/856a6aa7-8f9f-44f0-830b-f381709b4651-maas-api-openshift-ca\") on node \"ip-10-0-132-82.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:25:21.372256 ip-10-0-132-82 kubenswrapper[4892]: I1030 20:25:21.372266    4892 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/856a6aa7-8f9f-44f0-830b-f381709b4651-maas-api-xks-ca\") on node \"ip-10-0-132-82.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:25:22.025658 ip-10-0-132-82 kubenswrapper[4892]: I1030 20:25:22.025627    4892 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f03629db-xc594" event={"ID":"856a6aa7-8f9f-44f0-830b-f381709b4651","Type":"ContainerDied","Data":"2621f95f9394d986a5ad26c5d9e769350a28fda28b570097574c5cfaf963e31e"}
Oct 30 20:25:22.025958 ip-10-0-132-82 kubenswrapper[4892]: I1030 20:25:22.025676    4892 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="2621f95f9394d986a5ad26c5d9e769350a28fda28b570097574c5cfaf963e31e"
Oct 30 20:25:22.025958 ip-10-0-132-82 kubenswrapper[4892]: I1030 20:25:22.025632    4892 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f03629db-xc594"
Oct 30 20:25:24.848364 ip-10-0-132-82 kubenswrapper[4892]: E1030 20:25:24.848258    4892 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-jrm87" containerName="extractor"
Oct 30 20:25:30.649556 ip-10-0-132-82 kubenswrapper[4892]: I1030 20:25:30.649522    4892 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-fsfvk"]
Oct 30 20:25:53.527369 ip-10-0-132-82 kubenswrapper[4892]: I1030 20:25:53.527341    4892 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-tg5xj_11db9cec-653f-4aee-8937-0cf5b367efac/console-operator/3.log"
Oct 30 20:25:53.597326 ip-10-0-132-82 kubenswrapper[4892]: I1030 20:25:53.597296    4892 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-tg5xj_11db9cec-653f-4aee-8937-0cf5b367efac/console-operator/3.log"
Oct 30 20:25:54.636938 ip-10-0-132-82 kubenswrapper[4892]: I1030 20:25:54.636900    4892 prober.go:122] "Probe failed" probeType="Liveness" pod="kube-system/kube-apiserver-proxy-ip-10-0-132-82.us-west-1.compute.internal" podUID="f9af2f5b3d8deb871a93a1e3f0011dc6" containerName="haproxy" probeResult="failure" output="Get \"https://172.20.0.1:6443/version\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 20:25:54.843468 ip-10-0-132-82 kubenswrapper[4892]: E1030 20:25:54.843437    4892 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-jrm87" containerName="extractor"
Oct 30 20:25:56.888063 ip-10-0-132-82 kubenswrapper[4892]: I1030 20:25:56.888031    4892 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0165d7d2-sf2wp"]
Oct 30 20:25:56.888485 ip-10-0-132-82 kubenswrapper[4892]: I1030 20:25:56.888467    4892 cpu_manager.go:414] "containerMap: removing container" podUID="856a6aa7-8f9f-44f0-830b-f381709b4651" containerName="revoke-keys"
Oct 30 20:25:56.888532 ip-10-0-132-82 kubenswrapper[4892]: I1030 20:25:56.888487    4892 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="856a6aa7-8f9f-44f0-830b-f381709b4651" containerName="revoke-keys"
Oct 30 20:25:56.888578 ip-10-0-132-82 kubenswrapper[4892]: I1030 20:25:56.888567    4892 memory_manager.go:365] "RemoveStaleState removing state" podUID="856a6aa7-8f9f-44f0-830b-f381709b4651" containerName="revoke-keys"
Oct 30 20:25:56.891919 ip-10-0-132-82 kubenswrapper[4892]: I1030 20:25:56.891902    4892 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0165d7d2-sf2wp"
Oct 30 20:25:56.995640 ip-10-0-132-82 kubenswrapper[4892]: I1030 20:25:56.995608    4892 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5hhgf\" (UniqueName: \"kubernetes.io/projected/dedad02e-2673-4853-8f93-dcd631dc1ffa-kube-api-access-5hhgf\") pod \"maas-api-revoke-sub-0165d7d2-sf2wp\" (UID: \"dedad02e-2673-4853-8f93-dcd631dc1ffa\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0165d7d2-sf2wp"
Oct 30 20:25:56.995811 ip-10-0-132-82 kubenswrapper[4892]: I1030 20:25:56.995659    4892 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/dedad02e-2673-4853-8f93-dcd631dc1ffa-maas-api-xks-ca\") pod \"maas-api-revoke-sub-0165d7d2-sf2wp\" (UID: \"dedad02e-2673-4853-8f93-dcd631dc1ffa\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0165d7d2-sf2wp"
Oct 30 20:25:56.995811 ip-10-0-132-82 kubenswrapper[4892]: I1030 20:25:56.995763    4892 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/dedad02e-2673-4853-8f93-dcd631dc1ffa-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-0165d7d2-sf2wp\" (UID: \"dedad02e-2673-4853-8f93-dcd631dc1ffa\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0165d7d2-sf2wp"
Oct 30 20:25:56.998058 ip-10-0-132-82 kubenswrapper[4892]: I1030 20:25:56.998038    4892 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 20:25:56.998133 ip-10-0-132-82 kubenswrapper[4892]: I1030 20:25:56.998088    4892 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-nlxp4\""
Oct 30 20:25:56.998133 ip-10-0-132-82 kubenswrapper[4892]: I1030 20:25:56.998116    4892 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 30 20:25:56.998225 ip-10-0-132-82 kubenswrapper[4892]: I1030 20:25:56.998206    4892 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 30 20:25:57.000301 ip-10-0-132-82 kubenswrapper[4892]: I1030 20:25:57.000282    4892 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0165d7d2-sf2wp"]
Oct 30 20:25:57.096706 ip-10-0-132-82 kubenswrapper[4892]: I1030 20:25:57.096677    4892 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5hhgf\" (UniqueName: \"kubernetes.io/projected/dedad02e-2673-4853-8f93-dcd631dc1ffa-kube-api-access-5hhgf\") pod \"maas-api-revoke-sub-0165d7d2-sf2wp\" (UID: \"dedad02e-2673-4853-8f93-dcd631dc1ffa\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0165d7d2-sf2wp"
Oct 30 20:25:57.096861 ip-10-0-132-82 kubenswrapper[4892]: I1030 20:25:57.096725    4892 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/dedad02e-2673-4853-8f93-dcd631dc1ffa-maas-api-xks-ca\") pod \"maas-api-revoke-sub-0165d7d2-sf2wp\" (UID: \"dedad02e-2673-4853-8f93-dcd631dc1ffa\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0165d7d2-sf2wp"
Oct 30 20:25:57.096861 ip-10-0-132-82 kubenswrapper[4892]: I1030 20:25:57.096757    4892 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/dedad02e-2673-4853-8f93-dcd631dc1ffa-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-0165d7d2-sf2wp\" (UID: \"dedad02e-2673-4853-8f93-dcd631dc1ffa\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0165d7d2-sf2wp"
Oct 30 20:25:57.097380 ip-10-0-132-82 kubenswrapper[4892]: I1030 20:25:57.097361    4892 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/dedad02e-2673-4853-8f93-dcd631dc1ffa-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-0165d7d2-sf2wp\" (UID: \"dedad02e-2673-4853-8f93-dcd631dc1ffa\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0165d7d2-sf2wp"
Oct 30 20:25:57.099312 ip-10-0-132-82 kubenswrapper[4892]: I1030 20:25:57.099290    4892 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/dedad02e-2673-4853-8f93-dcd631dc1ffa-maas-api-xks-ca\") pod \"maas-api-revoke-sub-0165d7d2-sf2wp\" (UID: \"dedad02e-2673-4853-8f93-dcd631dc1ffa\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0165d7d2-sf2wp"
Oct 30 20:25:57.157201 ip-10-0-132-82 kubenswrapper[4892]: I1030 20:25:57.157142    4892 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5hhgf\" (UniqueName: \"kubernetes.io/projected/dedad02e-2673-4853-8f93-dcd631dc1ffa-kube-api-access-5hhgf\") pod \"maas-api-revoke-sub-0165d7d2-sf2wp\" (UID: \"dedad02e-2673-4853-8f93-dcd631dc1ffa\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0165d7d2-sf2wp"
Oct 30 20:25:57.208776 ip-10-0-132-82 kubenswrapper[4892]: I1030 20:25:57.208750    4892 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0165d7d2-sf2wp"
Oct 30 20:25:57.584505 ip-10-0-132-82 kubenswrapper[4892]: I1030 20:25:57.584478    4892 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0165d7d2-sf2wp"]
Oct 30 20:25:57.586484 ip-10-0-132-82 kubenswrapper[4892]: W1030 20:25:57.586456    4892 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddedad02e_2673_4853_8f93_dcd631dc1ffa.slice/crio-afc38dd075ef9655ecacf7c5b0c0a534aad8bfc224b681ec13dcb82e355b2a99 WatchSource:0}: Error finding container afc38dd075ef9655ecacf7c5b0c0a534aad8bfc224b681ec13dcb82e355b2a99: Status 404 returned error can't find the container with id afc38dd075ef9655ecacf7c5b0c0a534aad8bfc224b681ec13dcb82e355b2a99
Oct 30 20:25:58.242538 ip-10-0-132-82 kubenswrapper[4892]: I1030 20:25:58.242505    4892 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0165d7d2-sf2wp" event={"ID":"dedad02e-2673-4853-8f93-dcd631dc1ffa","Type":"ContainerStarted","Data":"6ab44e696c56e7d061475e78b0dee0bb396ff7e93673b6c87dddbb77aae2bd2d"}
Oct 30 20:25:58.242906 ip-10-0-132-82 kubenswrapper[4892]: I1030 20:25:58.242559    4892 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0165d7d2-sf2wp" event={"ID":"dedad02e-2673-4853-8f93-dcd631dc1ffa","Type":"ContainerStarted","Data":"afc38dd075ef9655ecacf7c5b0c0a534aad8bfc224b681ec13dcb82e355b2a99"}
Oct 30 20:25:58.364671 ip-10-0-132-82 kubenswrapper[4892]: I1030 20:25:58.364623    4892 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0165d7d2-sf2wp" podStartSLOduration=2.364609525 podStartE2EDuration="2.364609525s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 20:25:56 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 20:25:58.362246245 +0000 UTC m=+4205.932711848" watchObservedRunningTime="2026-10-30 20:25:58.364609525 +0000 UTC m=+4205.935075149"
Oct 30 20:25:59.256342 ip-10-0-132-82 kubenswrapper[4892]: I1030 20:25:59.256299    4892 generic.go:356] "Generic (PLEG): container finished" podID="dedad02e-2673-4853-8f93-dcd631dc1ffa" containerID="6ab44e696c56e7d061475e78b0dee0bb396ff7e93673b6c87dddbb77aae2bd2d" exitCode=0
Oct 30 20:25:59.256766 ip-10-0-132-82 kubenswrapper[4892]: I1030 20:25:59.256378    4892 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0165d7d2-sf2wp" event={"ID":"dedad02e-2673-4853-8f93-dcd631dc1ffa","Type":"ContainerDied","Data":"6ab44e696c56e7d061475e78b0dee0bb396ff7e93673b6c87dddbb77aae2bd2d"}
Oct 30 20:26:00.398425 ip-10-0-132-82 kubenswrapper[4892]: I1030 20:26:00.398403    4892 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0165d7d2-sf2wp"
Oct 30 20:26:00.426488 ip-10-0-132-82 kubenswrapper[4892]: I1030 20:26:00.426465    4892 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/dedad02e-2673-4853-8f93-dcd631dc1ffa-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/dedad02e-2673-4853-8f93-dcd631dc1ffa-maas-api-xks-ca\") pod \"dedad02e-2673-4853-8f93-dcd631dc1ffa\" (UID: \"dedad02e-2673-4853-8f93-dcd631dc1ffa\") "
Oct 30 20:26:00.426608 ip-10-0-132-82 kubenswrapper[4892]: I1030 20:26:00.426525    4892 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/dedad02e-2673-4853-8f93-dcd631dc1ffa-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/dedad02e-2673-4853-8f93-dcd631dc1ffa-maas-api-openshift-ca\") pod \"dedad02e-2673-4853-8f93-dcd631dc1ffa\" (UID: \"dedad02e-2673-4853-8f93-dcd631dc1ffa\") "
Oct 30 20:26:00.426608 ip-10-0-132-82 kubenswrapper[4892]: I1030 20:26:00.426580    4892 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/dedad02e-2673-4853-8f93-dcd631dc1ffa-kube-api-access-5hhgf\" (UniqueName: \"kubernetes.io/projected/dedad02e-2673-4853-8f93-dcd631dc1ffa-kube-api-access-5hhgf\") pod \"dedad02e-2673-4853-8f93-dcd631dc1ffa\" (UID: \"dedad02e-2673-4853-8f93-dcd631dc1ffa\") "
Oct 30 20:26:00.426848 ip-10-0-132-82 kubenswrapper[4892]: I1030 20:26:00.426824    4892 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/dedad02e-2673-4853-8f93-dcd631dc1ffa-maas-api-openshift-ca" pod "dedad02e-2673-4853-8f93-dcd631dc1ffa" (UID: "dedad02e-2673-4853-8f93-dcd631dc1ffa"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 20:26:00.428652 ip-10-0-132-82 kubenswrapper[4892]: I1030 20:26:00.428630    4892 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/dedad02e-2673-4853-8f93-dcd631dc1ffa-maas-api-xks-ca" pod "dedad02e-2673-4853-8f93-dcd631dc1ffa" (UID: "dedad02e-2673-4853-8f93-dcd631dc1ffa"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 20:26:00.428652 ip-10-0-132-82 kubenswrapper[4892]: I1030 20:26:00.428638    4892 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/dedad02e-2673-4853-8f93-dcd631dc1ffa-kube-api-access-5hhgf" pod "dedad02e-2673-4853-8f93-dcd631dc1ffa" (UID: "dedad02e-2673-4853-8f93-dcd631dc1ffa"). InnerVolumeSpecName "kube-api-access-5hhgf". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 20:26:00.527839 ip-10-0-132-82 kubenswrapper[4892]: I1030 20:26:00.527782    4892 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/dedad02e-2673-4853-8f93-dcd631dc1ffa-maas-api-openshift-ca\") on node \"ip-10-0-132-82.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:26:00.527839 ip-10-0-132-82 kubenswrapper[4892]: I1030 20:26:00.527805    4892 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5hhgf\" (UniqueName: \"kubernetes.io/projected/dedad02e-2673-4853-8f93-dcd631dc1ffa-kube-api-access-5hhgf\") on node \"ip-10-0-132-82.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:26:00.527839 ip-10-0-132-82 kubenswrapper[4892]: I1030 20:26:00.527819    4892 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/dedad02e-2673-4853-8f93-dcd631dc1ffa-maas-api-xks-ca\") on node \"ip-10-0-132-82.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:26:01.267220 ip-10-0-132-82 kubenswrapper[4892]: I1030 20:26:01.267192    4892 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0165d7d2-sf2wp"
Oct 30 20:26:01.267392 ip-10-0-132-82 kubenswrapper[4892]: I1030 20:26:01.267192    4892 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0165d7d2-sf2wp" event={"ID":"dedad02e-2673-4853-8f93-dcd631dc1ffa","Type":"ContainerDied","Data":"afc38dd075ef9655ecacf7c5b0c0a534aad8bfc224b681ec13dcb82e355b2a99"}
Oct 30 20:26:01.267392 ip-10-0-132-82 kubenswrapper[4892]: I1030 20:26:01.267315    4892 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="afc38dd075ef9655ecacf7c5b0c0a534aad8bfc224b681ec13dcb82e355b2a99"
Oct 30 20:26:08.111859 ip-10-0-132-82 kubenswrapper[4892]: I1030 20:26:08.111826    4892 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-6b19338f-nt67d"]
Oct 30 20:26:08.112293 ip-10-0-132-82 kubenswrapper[4892]: I1030 20:26:08.112279    4892 cpu_manager.go:414] "containerMap: removing container" podUID="dedad02e-2673-4853-8f93-dcd631dc1ffa" containerName="revoke-keys"
Oct 30 20:26:08.112293 ip-10-0-132-82 kubenswrapper[4892]: I1030 20:26:08.112291    4892 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="dedad02e-2673-4853-8f93-dcd631dc1ffa" containerName="revoke-keys"
Oct 30 20:26:08.112377 ip-10-0-132-82 kubenswrapper[4892]: I1030 20:26:08.112366    4892 memory_manager.go:365] "RemoveStaleState removing state" podUID="dedad02e-2673-4853-8f93-dcd631dc1ffa" containerName="revoke-keys"
Oct 30 20:26:08.115613 ip-10-0-132-82 kubenswrapper[4892]: I1030 20:26:08.115594    4892 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6b19338f-nt67d"
Oct 30 20:26:08.169830 ip-10-0-132-82 kubenswrapper[4892]: I1030 20:26:08.169800    4892 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 30 20:26:08.190395 ip-10-0-132-82 kubenswrapper[4892]: I1030 20:26:08.190358    4892 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7f896b90-72e0-4a1a-8c31-5e945063348e-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-6b19338f-nt67d\" (UID: \"7f896b90-72e0-4a1a-8c31-5e945063348e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6b19338f-nt67d"
Oct 30 20:26:08.190541 ip-10-0-132-82 kubenswrapper[4892]: I1030 20:26:08.190410    4892 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ncptk\" (UniqueName: \"kubernetes.io/projected/7f896b90-72e0-4a1a-8c31-5e945063348e-kube-api-access-ncptk\") pod \"maas-api-revoke-sub-6b19338f-nt67d\" (UID: \"7f896b90-72e0-4a1a-8c31-5e945063348e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6b19338f-nt67d"
Oct 30 20:26:08.190541 ip-10-0-132-82 kubenswrapper[4892]: I1030 20:26:08.190474    4892 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7f896b90-72e0-4a1a-8c31-5e945063348e-maas-api-xks-ca\") pod \"maas-api-revoke-sub-6b19338f-nt67d\" (UID: \"7f896b90-72e0-4a1a-8c31-5e945063348e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6b19338f-nt67d"
Oct 30 20:26:08.219162 ip-10-0-132-82 kubenswrapper[4892]: I1030 20:26:08.219131    4892 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 20:26:08.219162 ip-10-0-132-82 kubenswrapper[4892]: I1030 20:26:08.219151    4892 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 30 20:26:08.219360 ip-10-0-132-82 kubenswrapper[4892]: I1030 20:26:08.219249    4892 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-nlxp4\""
Oct 30 20:26:08.221470 ip-10-0-132-82 kubenswrapper[4892]: I1030 20:26:08.221454    4892 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-6b19338f-nt67d"]
Oct 30 20:26:08.291414 ip-10-0-132-82 kubenswrapper[4892]: I1030 20:26:08.291387    4892 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7f896b90-72e0-4a1a-8c31-5e945063348e-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-6b19338f-nt67d\" (UID: \"7f896b90-72e0-4a1a-8c31-5e945063348e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6b19338f-nt67d"
Oct 30 20:26:08.291591 ip-10-0-132-82 kubenswrapper[4892]: I1030 20:26:08.291443    4892 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ncptk\" (UniqueName: \"kubernetes.io/projected/7f896b90-72e0-4a1a-8c31-5e945063348e-kube-api-access-ncptk\") pod \"maas-api-revoke-sub-6b19338f-nt67d\" (UID: \"7f896b90-72e0-4a1a-8c31-5e945063348e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6b19338f-nt67d"
Oct 30 20:26:08.291591 ip-10-0-132-82 kubenswrapper[4892]: I1030 20:26:08.291478    4892 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7f896b90-72e0-4a1a-8c31-5e945063348e-maas-api-xks-ca\") pod \"maas-api-revoke-sub-6b19338f-nt67d\" (UID: \"7f896b90-72e0-4a1a-8c31-5e945063348e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6b19338f-nt67d"
Oct 30 20:26:08.292080 ip-10-0-132-82 kubenswrapper[4892]: I1030 20:26:08.292060    4892 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7f896b90-72e0-4a1a-8c31-5e945063348e-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-6b19338f-nt67d\" (UID: \"7f896b90-72e0-4a1a-8c31-5e945063348e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6b19338f-nt67d"
Oct 30 20:26:08.293925 ip-10-0-132-82 kubenswrapper[4892]: I1030 20:26:08.293904    4892 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7f896b90-72e0-4a1a-8c31-5e945063348e-maas-api-xks-ca\") pod \"maas-api-revoke-sub-6b19338f-nt67d\" (UID: \"7f896b90-72e0-4a1a-8c31-5e945063348e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6b19338f-nt67d"
Oct 30 20:26:08.348268 ip-10-0-132-82 kubenswrapper[4892]: I1030 20:26:08.348239    4892 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ncptk\" (UniqueName: \"kubernetes.io/projected/7f896b90-72e0-4a1a-8c31-5e945063348e-kube-api-access-ncptk\") pod \"maas-api-revoke-sub-6b19338f-nt67d\" (UID: \"7f896b90-72e0-4a1a-8c31-5e945063348e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6b19338f-nt67d"
Oct 30 20:26:08.430161 ip-10-0-132-82 kubenswrapper[4892]: I1030 20:26:08.430135    4892 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6b19338f-nt67d"
Oct 30 20:26:08.805897 ip-10-0-132-82 kubenswrapper[4892]: I1030 20:26:08.805872    4892 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-6b19338f-nt67d"]
Oct 30 20:26:08.808948 ip-10-0-132-82 kubenswrapper[4892]: W1030 20:26:08.808917    4892 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7f896b90_72e0_4a1a_8c31_5e945063348e.slice/crio-948d77b6b0f0558996742c1697d5b2d8b275db5f69bfd0a22e49611977fa8ccd WatchSource:0}: Error finding container 948d77b6b0f0558996742c1697d5b2d8b275db5f69bfd0a22e49611977fa8ccd: Status 404 returned error can't find the container with id 948d77b6b0f0558996742c1697d5b2d8b275db5f69bfd0a22e49611977fa8ccd
Oct 30 20:26:09.310726 ip-10-0-132-82 kubenswrapper[4892]: I1030 20:26:09.310698    4892 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6b19338f-nt67d" event={"ID":"7f896b90-72e0-4a1a-8c31-5e945063348e","Type":"ContainerStarted","Data":"9da08c74c4297b721df4d048a3b006a49f21c0ba9e995e8b48efa27997933034"}
Oct 30 20:26:09.311140 ip-10-0-132-82 kubenswrapper[4892]: I1030 20:26:09.310742    4892 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6b19338f-nt67d" event={"ID":"7f896b90-72e0-4a1a-8c31-5e945063348e","Type":"ContainerStarted","Data":"948d77b6b0f0558996742c1697d5b2d8b275db5f69bfd0a22e49611977fa8ccd"}
Oct 30 20:26:09.426750 ip-10-0-132-82 kubenswrapper[4892]: I1030 20:26:09.426657    4892 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6b19338f-nt67d" podStartSLOduration=1.426644908 podStartE2EDuration="1.426644908s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 20:26:08 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 20:26:09.424014523 +0000 UTC m=+4216.994480125" watchObservedRunningTime="2026-10-30 20:26:09.426644908 +0000 UTC m=+4216.997110501"
Oct 30 20:26:10.317800 ip-10-0-132-82 kubenswrapper[4892]: I1030 20:26:10.317757    4892 generic.go:356] "Generic (PLEG): container finished" podID="7f896b90-72e0-4a1a-8c31-5e945063348e" containerID="9da08c74c4297b721df4d048a3b006a49f21c0ba9e995e8b48efa27997933034" exitCode=0
Oct 30 20:26:10.318337 ip-10-0-132-82 kubenswrapper[4892]: I1030 20:26:10.317821    4892 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6b19338f-nt67d" event={"ID":"7f896b90-72e0-4a1a-8c31-5e945063348e","Type":"ContainerDied","Data":"9da08c74c4297b721df4d048a3b006a49f21c0ba9e995e8b48efa27997933034"}
Oct 30 20:26:11.458680 ip-10-0-132-82 kubenswrapper[4892]: I1030 20:26:11.458651    4892 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6b19338f-nt67d"
Oct 30 20:26:11.522743 ip-10-0-132-82 kubenswrapper[4892]: I1030 20:26:11.522718    4892 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/7f896b90-72e0-4a1a-8c31-5e945063348e-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7f896b90-72e0-4a1a-8c31-5e945063348e-maas-api-xks-ca\") pod \"7f896b90-72e0-4a1a-8c31-5e945063348e\" (UID: \"7f896b90-72e0-4a1a-8c31-5e945063348e\") "
Oct 30 20:26:11.522881 ip-10-0-132-82 kubenswrapper[4892]: I1030 20:26:11.522866    4892 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7f896b90-72e0-4a1a-8c31-5e945063348e-kube-api-access-ncptk\" (UniqueName: \"kubernetes.io/projected/7f896b90-72e0-4a1a-8c31-5e945063348e-kube-api-access-ncptk\") pod \"7f896b90-72e0-4a1a-8c31-5e945063348e\" (UID: \"7f896b90-72e0-4a1a-8c31-5e945063348e\") "
Oct 30 20:26:11.522923 ip-10-0-132-82 kubenswrapper[4892]: I1030 20:26:11.522901    4892 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/7f896b90-72e0-4a1a-8c31-5e945063348e-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7f896b90-72e0-4a1a-8c31-5e945063348e-maas-api-openshift-ca\") pod \"7f896b90-72e0-4a1a-8c31-5e945063348e\" (UID: \"7f896b90-72e0-4a1a-8c31-5e945063348e\") "
Oct 30 20:26:11.523290 ip-10-0-132-82 kubenswrapper[4892]: I1030 20:26:11.523266    4892 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7f896b90-72e0-4a1a-8c31-5e945063348e-maas-api-openshift-ca" pod "7f896b90-72e0-4a1a-8c31-5e945063348e" (UID: "7f896b90-72e0-4a1a-8c31-5e945063348e"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 20:26:11.524878 ip-10-0-132-82 kubenswrapper[4892]: I1030 20:26:11.524850    4892 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7f896b90-72e0-4a1a-8c31-5e945063348e-maas-api-xks-ca" pod "7f896b90-72e0-4a1a-8c31-5e945063348e" (UID: "7f896b90-72e0-4a1a-8c31-5e945063348e"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 20:26:11.525053 ip-10-0-132-82 kubenswrapper[4892]: I1030 20:26:11.525020    4892 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7f896b90-72e0-4a1a-8c31-5e945063348e-kube-api-access-ncptk" pod "7f896b90-72e0-4a1a-8c31-5e945063348e" (UID: "7f896b90-72e0-4a1a-8c31-5e945063348e"). InnerVolumeSpecName "kube-api-access-ncptk". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 20:26:11.623809 ip-10-0-132-82 kubenswrapper[4892]: I1030 20:26:11.623733    4892 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7f896b90-72e0-4a1a-8c31-5e945063348e-maas-api-openshift-ca\") on node \"ip-10-0-132-82.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:26:11.623809 ip-10-0-132-82 kubenswrapper[4892]: I1030 20:26:11.623768    4892 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7f896b90-72e0-4a1a-8c31-5e945063348e-maas-api-xks-ca\") on node \"ip-10-0-132-82.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:26:11.623809 ip-10-0-132-82 kubenswrapper[4892]: I1030 20:26:11.623779    4892 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ncptk\" (UniqueName: \"kubernetes.io/projected/7f896b90-72e0-4a1a-8c31-5e945063348e-kube-api-access-ncptk\") on node \"ip-10-0-132-82.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:26:12.328883 ip-10-0-132-82 kubenswrapper[4892]: I1030 20:26:12.328845    4892 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6b19338f-nt67d" event={"ID":"7f896b90-72e0-4a1a-8c31-5e945063348e","Type":"ContainerDied","Data":"948d77b6b0f0558996742c1697d5b2d8b275db5f69bfd0a22e49611977fa8ccd"}
Oct 30 20:26:12.329079 ip-10-0-132-82 kubenswrapper[4892]: I1030 20:26:12.328859    4892 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6b19338f-nt67d"
Oct 30 20:26:12.329079 ip-10-0-132-82 kubenswrapper[4892]: I1030 20:26:12.328941    4892 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="948d77b6b0f0558996742c1697d5b2d8b275db5f69bfd0a22e49611977fa8ccd"
Oct 30 20:26:17.048203 ip-10-0-132-82 kubenswrapper[4892]: I1030 20:26:17.048159    4892 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b4beeec2-s2mwf"]
Oct 30 20:26:17.097570 ip-10-0-132-82 kubenswrapper[4892]: I1030 20:26:17.097534    4892 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b4beeec2-s2mwf"]
Oct 30 20:26:18.568337 ip-10-0-132-82 kubenswrapper[4892]: I1030 20:26:18.568296    4892 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="57233a47-7491-4605-ab04-b61557546a96" path="/var/lib/kubelet/pods/57233a47-7491-4605-ab04-b61557546a96/volumes"
Oct 30 20:26:20.832518 ip-10-0-132-82 kubenswrapper[4892]: I1030 20:26:20.832482    4892 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-fsfvk"]
Oct 30 20:26:24.842873 ip-10-0-132-82 kubenswrapper[4892]: E1030 20:26:24.842834    4892 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-jrm87" containerName="extractor"
Oct 30 20:26:52.276419 ip-10-0-132-82 kubenswrapper[4892]: I1030 20:26:52.276386    4892 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_llmisvc-controller-manager-6466c554d8-q6d8c_f826b6a0-802b-4275-9bad-f437ed79959a/manager/0.log"
Oct 30 20:26:53.588198 ip-10-0-132-82 kubenswrapper[4892]: I1030 20:26:53.588170    4892 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_model-serving-api-75bd89fdfc-d6qpr_8bb5f724-d122-454a-a0db-0f9f67ee8c6a/server/0.log"
Oct 30 20:26:53.828267 ip-10-0-132-82 kubenswrapper[4892]: I1030 20:26:53.828235    4892 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_odh-model-controller-f87fcf56c-x697s_65933d4a-6b0a-4280-8e3c-a16a0ce6c6c3/manager/0.log"
Oct 30 20:26:54.854402 ip-10-0-132-82 kubenswrapper[4892]: E1030 20:26:54.854359    4892 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-jrm87" containerName="extractor"
Oct 30 20:26:54.987036 ip-10-0-132-82 kubenswrapper[4892]: I1030 20:26:54.987006    4892 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-69bfcc8945-w6764_aaef15fe-c1fe-44ed-88d9-ec1ac79f4896/copy-manifests/0.log"
Oct 30 20:26:55.055579 ip-10-0-132-82 kubenswrapper[4892]: I1030 20:26:55.055550    4892 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-69bfcc8945-w6764_aaef15fe-c1fe-44ed-88d9-ec1ac79f4896/manager/0.log"
Oct 30 20:26:57.518461 ip-10-0-132-82 kubenswrapper[4892]: I1030 20:26:57.518432    4892 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-revoke-sub-0165d7d2-sf2wp_dedad02e-2673-4853-8f93-dcd631dc1ffa/revoke-keys/0.log"
Oct 30 20:26:58.389329 ip-10-0-132-82 kubenswrapper[4892]: I1030 20:26:58.389298    4892 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-revoke-sub-6b19338f-nt67d_7f896b90-72e0-4a1a-8c31-5e945063348e/revoke-keys/0.log"
Oct 30 20:26:58.617621 ip-10-0-132-82 kubenswrapper[4892]: I1030 20:26:58.617594    4892 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-revoke-sub-f03629db-xc594_856a6aa7-8f9f-44f0-830b-f381709b4651/revoke-keys/0.log"
Oct 30 20:27:01.705251 ip-10-0-132-82 kubenswrapper[4892]: I1030 20:27:01.705174    4892 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-operator-controller-manager-57bb9bfff-ck96g_0c40923a-2096-43f0-8d0f-315f2e4a2270/manager/0.log"
Oct 30 20:27:02.186553 ip-10-0-132-82 kubenswrapper[4892]: I1030 20:27:02.186528    4892 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-limitador-7c9b87d574-fsfvk_a2799daf-05de-4827-90df-db8ac57d645a/limitador/0.log"
Oct 30 20:27:13.649424 ip-10-0-132-82 kubenswrapper[4892]: I1030 20:27:13.649388    4892 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_istiod-openshift-gateway-7ddd9cfdc8-rqhzf_b1eed800-8bc3-4333-a929-dcb1477293fd/discovery/0.log"
Oct 30 20:27:14.246207 ip-10-0-132-82 kubenswrapper[4892]: I1030 20:27:14.246175    4892 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-7f6998b7b5-wthwp_6fb66959-7822-48c9-9f8b-8daf7fc7adde/kube-auth-proxy/0.log"
Oct 30 20:27:15.666381 ip-10-0-132-82 kubenswrapper[4892]: I1030 20:27:15.666353    4892 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-6db586dd6c-k85mx_6f05c89a-d631-4c95-aad6-d08d6c87cffd/istio-proxy/0.log"
Oct 30 20:27:16.272156 ip-10-0-132-82 kubenswrapper[4892]: I1030 20:27:16.272131    4892 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-6db586dd6c-mb59v_72926e14-705b-493a-879f-f595c35a7490/istio-proxy/0.log"
Oct 30 20:27:16.547029 ip-10-0-132-82 kubenswrapper[4892]: I1030 20:27:16.546937    4892 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-6db586dd6c-p7m7x_741c37d3-758a-4ced-ab0e-9038baea2d88/istio-proxy/0.log"
Oct 30 20:27:17.867423 ip-10-0-132-82 kubenswrapper[4892]: I1030 20:27:17.867389    4892 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_router-default-776df59447-8nrhh_87082646-75c3-43b5-864b-3478e9a7373d/router/0.log"
Oct 30 20:27:24.843442 ip-10-0-132-82 kubenswrapper[4892]: E1030 20:27:24.843399    4892 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-jrm87" containerName="extractor"
Oct 30 20:27:54.844164 ip-10-0-132-82 kubenswrapper[4892]: E1030 20:27:54.844123    4892 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-jrm87" containerName="extractor"
Oct 30 20:27:58.992224 ip-10-0-132-82 kubenswrapper[4892]: I1030 20:27:58.992191    4892 scope.go:122] "RemoveContainer" containerID="eeb48d01131b7509e114ca1a405287f656b9f9d60e51da8c01427442a26cd0c6"
Oct 30 20:27:59.001302 ip-10-0-132-82 kubenswrapper[4892]: I1030 20:27:59.001283    4892 scope.go:122] "RemoveContainer" containerID="bc2518691792913f55a9632d5acf4e6c7e12d23c6aa4fc74909d7b8958a8ec5a"
Oct 30 20:28:01.656569 ip-10-0-132-82 kubenswrapper[4892]: I1030 20:28:01.656539    4892 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_llmisvc-controller-manager-6466c554d8-q6d8c_f826b6a0-802b-4275-9bad-f437ed79959a/manager/0.log"
Oct 30 20:28:02.489477 ip-10-0-132-82 kubenswrapper[4892]: I1030 20:28:02.489445    4892 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_model-serving-api-75bd89fdfc-d6qpr_8bb5f724-d122-454a-a0db-0f9f67ee8c6a/server/0.log"
Oct 30 20:28:02.715729 ip-10-0-132-82 kubenswrapper[4892]: I1030 20:28:02.715703    4892 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_odh-model-controller-f87fcf56c-x697s_65933d4a-6b0a-4280-8e3c-a16a0ce6c6c3/manager/0.log"
Oct 30 20:28:03.767275 ip-10-0-132-82 kubenswrapper[4892]: I1030 20:28:03.767242    4892 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-69bfcc8945-w6764_aaef15fe-c1fe-44ed-88d9-ec1ac79f4896/copy-manifests/0.log"
Oct 30 20:28:03.835927 ip-10-0-132-82 kubenswrapper[4892]: I1030 20:28:03.835898    4892 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-69bfcc8945-w6764_aaef15fe-c1fe-44ed-88d9-ec1ac79f4896/manager/0.log"
Oct 30 20:28:15.844441 ip-10-0-132-82 kubenswrapper[4892]: I1030 20:28:15.844402    4892 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-revoke-sub-0165d7d2-sf2wp_dedad02e-2673-4853-8f93-dcd631dc1ffa/revoke-keys/0.log"
Oct 30 20:28:16.771181 ip-10-0-132-82 kubenswrapper[4892]: I1030 20:28:16.771156    4892 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-revoke-sub-6b19338f-nt67d_7f896b90-72e0-4a1a-8c31-5e945063348e/revoke-keys/0.log"
Oct 30 20:28:16.995794 ip-10-0-132-82 kubenswrapper[4892]: I1030 20:28:16.995768    4892 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-revoke-sub-f03629db-xc594_856a6aa7-8f9f-44f0-830b-f381709b4651/revoke-keys/0.log"
Oct 30 20:28:24.842957 ip-10-0-132-82 kubenswrapper[4892]: E1030 20:28:24.842920    4892 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-jrm87" containerName="extractor"
Oct 30 20:28:29.716825 ip-10-0-132-82 kubenswrapper[4892]: I1030 20:28:29.716796    4892 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-operator-controller-manager-57bb9bfff-ck96g_0c40923a-2096-43f0-8d0f-315f2e4a2270/manager/0.log"
Oct 30 20:28:30.177311 ip-10-0-132-82 kubenswrapper[4892]: I1030 20:28:30.177285    4892 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-limitador-7c9b87d574-fsfvk_a2799daf-05de-4827-90df-db8ac57d645a/limitador/0.log"
Oct 30 20:28:31.360879 ip-10-0-132-82 kubenswrapper[4892]: I1030 20:28:31.360791    4892 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_istiod-openshift-gateway-7ddd9cfdc8-rqhzf_b1eed800-8bc3-4333-a929-dcb1477293fd/discovery/0.log"
Oct 30 20:28:31.900371 ip-10-0-132-82 kubenswrapper[4892]: I1030 20:28:31.900337    4892 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-7f6998b7b5-wthwp_6fb66959-7822-48c9-9f8b-8daf7fc7adde/kube-auth-proxy/0.log"
Oct 30 20:28:33.263362 ip-10-0-132-82 kubenswrapper[4892]: I1030 20:28:33.263332    4892 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-6db586dd6c-k85mx_6f05c89a-d631-4c95-aad6-d08d6c87cffd/istio-proxy/0.log"
Oct 30 20:28:33.829472 ip-10-0-132-82 kubenswrapper[4892]: I1030 20:28:33.829448    4892 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-6db586dd6c-mb59v_72926e14-705b-493a-879f-f595c35a7490/istio-proxy/0.log"
Oct 30 20:28:34.115396 ip-10-0-132-82 kubenswrapper[4892]: I1030 20:28:34.115314    4892 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-6db586dd6c-p7m7x_741c37d3-758a-4ced-ab0e-9038baea2d88/istio-proxy/0.log"
Oct 30 20:28:35.345906 ip-10-0-132-82 kubenswrapper[4892]: I1030 20:28:35.345871    4892 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_router-default-776df59447-8nrhh_87082646-75c3-43b5-864b-3478e9a7373d/router/0.log"
Oct 30 20:28:54.842865 ip-10-0-132-82 kubenswrapper[4892]: E1030 20:28:54.842829    4892 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-jrm87" containerName="extractor"
Oct 30 20:29:24.843973 ip-10-0-132-82 kubenswrapper[4892]: E1030 20:29:24.843907    4892 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-jrm87" containerName="extractor"
Oct 30 20:29:54.843838 ip-10-0-132-82 kubenswrapper[4892]: E1030 20:29:54.843800    4892 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-jrm87" containerName="extractor"
Oct 30 20:30:00.178389 ip-10-0-132-82 kubenswrapper[4892]: I1030 20:30:00.178350    4892 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29889870-q9hmg"]
Oct 30 20:30:00.178812 ip-10-0-132-82 kubenswrapper[4892]: I1030 20:30:00.178736    4892 cpu_manager.go:414] "containerMap: removing container" podUID="7f896b90-72e0-4a1a-8c31-5e945063348e" containerName="revoke-keys"
Oct 30 20:30:00.178812 ip-10-0-132-82 kubenswrapper[4892]: I1030 20:30:00.178748    4892 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7f896b90-72e0-4a1a-8c31-5e945063348e" containerName="revoke-keys"
Oct 30 20:30:00.178893 ip-10-0-132-82 kubenswrapper[4892]: I1030 20:30:00.178853    4892 memory_manager.go:365] "RemoveStaleState removing state" podUID="7f896b90-72e0-4a1a-8c31-5e945063348e" containerName="revoke-keys"
Oct 30 20:30:00.181909 ip-10-0-132-82 kubenswrapper[4892]: I1030 20:30:00.181892    4892 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889870-q9hmg"
Oct 30 20:30:00.283059 ip-10-0-132-82 kubenswrapper[4892]: I1030 20:30:00.283031    4892 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-nlxp4\""
Oct 30 20:30:00.283059 ip-10-0-132-82 kubenswrapper[4892]: I1030 20:30:00.283048    4892 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 30 20:30:00.283250 ip-10-0-132-82 kubenswrapper[4892]: I1030 20:30:00.283226    4892 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 30 20:30:00.285445 ip-10-0-132-82 kubenswrapper[4892]: I1030 20:30:00.285424    4892 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29889870-q9hmg"]
Oct 30 20:30:00.311829 ip-10-0-132-82 kubenswrapper[4892]: I1030 20:30:00.311802    4892 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/00ac42cd-07e0-4e77-aa8d-b268b06fc1cf-maas-api-service-ca\") pod \"maas-api-key-cleanup-29889870-q9hmg\" (UID: \"00ac42cd-07e0-4e77-aa8d-b268b06fc1cf\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889870-q9hmg"
Oct 30 20:30:00.311995 ip-10-0-132-82 kubenswrapper[4892]: I1030 20:30:00.311847    4892 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7vt5v\" (UniqueName: \"kubernetes.io/projected/00ac42cd-07e0-4e77-aa8d-b268b06fc1cf-kube-api-access-7vt5v\") pod \"maas-api-key-cleanup-29889870-q9hmg\" (UID: \"00ac42cd-07e0-4e77-aa8d-b268b06fc1cf\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889870-q9hmg"
Oct 30 20:30:00.413076 ip-10-0-132-82 kubenswrapper[4892]: I1030 20:30:00.413040    4892 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7vt5v\" (UniqueName: \"kubernetes.io/projected/00ac42cd-07e0-4e77-aa8d-b268b06fc1cf-kube-api-access-7vt5v\") pod \"maas-api-key-cleanup-29889870-q9hmg\" (UID: \"00ac42cd-07e0-4e77-aa8d-b268b06fc1cf\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889870-q9hmg"
Oct 30 20:30:00.413220 ip-10-0-132-82 kubenswrapper[4892]: I1030 20:30:00.413149    4892 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/00ac42cd-07e0-4e77-aa8d-b268b06fc1cf-maas-api-service-ca\") pod \"maas-api-key-cleanup-29889870-q9hmg\" (UID: \"00ac42cd-07e0-4e77-aa8d-b268b06fc1cf\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889870-q9hmg"
Oct 30 20:30:00.413719 ip-10-0-132-82 kubenswrapper[4892]: I1030 20:30:00.413703    4892 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/00ac42cd-07e0-4e77-aa8d-b268b06fc1cf-maas-api-service-ca\") pod \"maas-api-key-cleanup-29889870-q9hmg\" (UID: \"00ac42cd-07e0-4e77-aa8d-b268b06fc1cf\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889870-q9hmg"
Oct 30 20:30:00.469379 ip-10-0-132-82 kubenswrapper[4892]: I1030 20:30:00.469322    4892 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7vt5v\" (UniqueName: \"kubernetes.io/projected/00ac42cd-07e0-4e77-aa8d-b268b06fc1cf-kube-api-access-7vt5v\") pod \"maas-api-key-cleanup-29889870-q9hmg\" (UID: \"00ac42cd-07e0-4e77-aa8d-b268b06fc1cf\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889870-q9hmg"
Oct 30 20:30:00.494123 ip-10-0-132-82 kubenswrapper[4892]: I1030 20:30:00.494102    4892 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889870-q9hmg"
Oct 30 20:30:00.867401 ip-10-0-132-82 kubenswrapper[4892]: I1030 20:30:00.867377    4892 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29889870-q9hmg"]
Oct 30 20:30:00.869716 ip-10-0-132-82 kubenswrapper[4892]: W1030 20:30:00.869688    4892 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod00ac42cd_07e0_4e77_aa8d_b268b06fc1cf.slice/crio-309eab41d0f2777bd875ad155b2ec6af28dc664e6ba13099183285246447f32c WatchSource:0}: Error finding container 309eab41d0f2777bd875ad155b2ec6af28dc664e6ba13099183285246447f32c: Status 404 returned error can't find the container with id 309eab41d0f2777bd875ad155b2ec6af28dc664e6ba13099183285246447f32c
Oct 30 20:30:01.530530 ip-10-0-132-82 kubenswrapper[4892]: I1030 20:30:01.530448    4892 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889870-q9hmg" event={"ID":"00ac42cd-07e0-4e77-aa8d-b268b06fc1cf","Type":"ContainerStarted","Data":"abaf1de1e15521dafd0189f24031a1e8e20324ad7352931cfbd0113532b55da3"}
Oct 30 20:30:01.530530 ip-10-0-132-82 kubenswrapper[4892]: I1030 20:30:01.530497    4892 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889870-q9hmg" event={"ID":"00ac42cd-07e0-4e77-aa8d-b268b06fc1cf","Type":"ContainerStarted","Data":"309eab41d0f2777bd875ad155b2ec6af28dc664e6ba13099183285246447f32c"}
Oct 30 20:30:01.643848 ip-10-0-132-82 kubenswrapper[4892]: I1030 20:30:01.643799    4892 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889870-q9hmg" podStartSLOduration=1.643785878 podStartE2EDuration="1.643785878s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 20:30:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 20:30:01.642301439 +0000 UTC m=+4449.212767060" watchObservedRunningTime="2026-10-30 20:30:01.643785878 +0000 UTC m=+4449.214251476"
Oct 30 20:30:02.537765 ip-10-0-132-82 kubenswrapper[4892]: I1030 20:30:02.537727    4892 generic.go:356] "Generic (PLEG): container finished" podID="00ac42cd-07e0-4e77-aa8d-b268b06fc1cf" containerID="abaf1de1e15521dafd0189f24031a1e8e20324ad7352931cfbd0113532b55da3" exitCode=60
Oct 30 20:30:02.538197 ip-10-0-132-82 kubenswrapper[4892]: I1030 20:30:02.537858    4892 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889870-q9hmg" event={"ID":"00ac42cd-07e0-4e77-aa8d-b268b06fc1cf","Type":"ContainerDied","Data":"abaf1de1e15521dafd0189f24031a1e8e20324ad7352931cfbd0113532b55da3"}
Oct 30 20:30:02.538197 ip-10-0-132-82 kubenswrapper[4892]: I1030 20:30:02.538028    4892 scope.go:122] "RemoveContainer" containerID="abaf1de1e15521dafd0189f24031a1e8e20324ad7352931cfbd0113532b55da3"
Oct 30 20:30:03.545214 ip-10-0-132-82 kubenswrapper[4892]: I1030 20:30:03.545120    4892 generic.go:356] "Generic (PLEG): container finished" podID="00ac42cd-07e0-4e77-aa8d-b268b06fc1cf" containerID="8c7e03dee7cf1f79870bd9604d3b076d710ae7b7b7633a7cf315f3cf6ced4e2b" exitCode=60
Oct 30 20:30:03.545214 ip-10-0-132-82 kubenswrapper[4892]: I1030 20:30:03.545172    4892 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889870-q9hmg" event={"ID":"00ac42cd-07e0-4e77-aa8d-b268b06fc1cf","Type":"ContainerDied","Data":"8c7e03dee7cf1f79870bd9604d3b076d710ae7b7b7633a7cf315f3cf6ced4e2b"}
Oct 30 20:30:03.545745 ip-10-0-132-82 kubenswrapper[4892]: I1030 20:30:03.545220    4892 scope.go:122] "RemoveContainer" containerID="abaf1de1e15521dafd0189f24031a1e8e20324ad7352931cfbd0113532b55da3"
Oct 30 20:30:03.545745 ip-10-0-132-82 kubenswrapper[4892]: I1030 20:30:03.545452    4892 scope.go:122] "RemoveContainer" containerID="8c7e03dee7cf1f79870bd9604d3b076d710ae7b7b7633a7cf315f3cf6ced4e2b"
Oct 30 20:30:03.545745 ip-10-0-132-82 kubenswrapper[4892]: E1030 20:30:03.545700    4892 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-29889870-q9hmg_odh-ai-gateway-infra(00ac42cd-07e0-4e77-aa8d-b268b06fc1cf)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889870-q9hmg" podUID="00ac42cd-07e0-4e77-aa8d-b268b06fc1cf"
Oct 30 20:30:04.553019 ip-10-0-132-82 kubenswrapper[4892]: I1030 20:30:04.552992    4892 scope.go:122] "RemoveContainer" containerID="8c7e03dee7cf1f79870bd9604d3b076d710ae7b7b7633a7cf315f3cf6ced4e2b"
Oct 30 20:30:04.553440 ip-10-0-132-82 kubenswrapper[4892]: E1030 20:30:04.553203    4892 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-29889870-q9hmg_odh-ai-gateway-infra(00ac42cd-07e0-4e77-aa8d-b268b06fc1cf)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889870-q9hmg" podUID="00ac42cd-07e0-4e77-aa8d-b268b06fc1cf"
Oct 30 20:30:17.562280 ip-10-0-132-82 kubenswrapper[4892]: I1030 20:30:17.562245    4892 scope.go:122] "RemoveContainer" containerID="8c7e03dee7cf1f79870bd9604d3b076d710ae7b7b7633a7cf315f3cf6ced4e2b"
Oct 30 20:30:18.623411 ip-10-0-132-82 kubenswrapper[4892]: I1030 20:30:18.623342    4892 generic.go:356] "Generic (PLEG): container finished" podID="00ac42cd-07e0-4e77-aa8d-b268b06fc1cf" containerID="05e9c813be9e8bf2bce1c9819589af1348b6795ca8de839b9b18f9d240be0bbe" exitCode=60
Oct 30 20:30:18.623857 ip-10-0-132-82 kubenswrapper[4892]: I1030 20:30:18.623416    4892 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889870-q9hmg" event={"ID":"00ac42cd-07e0-4e77-aa8d-b268b06fc1cf","Type":"ContainerDied","Data":"05e9c813be9e8bf2bce1c9819589af1348b6795ca8de839b9b18f9d240be0bbe"}
Oct 30 20:30:18.623857 ip-10-0-132-82 kubenswrapper[4892]: I1030 20:30:18.623491    4892 scope.go:122] "RemoveContainer" containerID="8c7e03dee7cf1f79870bd9604d3b076d710ae7b7b7633a7cf315f3cf6ced4e2b"
Oct 30 20:30:18.623857 ip-10-0-132-82 kubenswrapper[4892]: I1030 20:30:18.623809    4892 scope.go:122] "RemoveContainer" containerID="05e9c813be9e8bf2bce1c9819589af1348b6795ca8de839b9b18f9d240be0bbe"
Oct 30 20:30:18.624095 ip-10-0-132-82 kubenswrapper[4892]: E1030 20:30:18.624074    4892 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-29889870-q9hmg_odh-ai-gateway-infra(00ac42cd-07e0-4e77-aa8d-b268b06fc1cf)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889870-q9hmg" podUID="00ac42cd-07e0-4e77-aa8d-b268b06fc1cf"
Oct 30 20:30:19.746651 ip-10-0-132-82 kubenswrapper[4892]: I1030 20:30:19.746618    4892 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29889870-q9hmg"]
Oct 30 20:30:19.887711 ip-10-0-132-82 kubenswrapper[4892]: I1030 20:30:19.887685    4892 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889870-q9hmg"
Oct 30 20:30:19.977908 ip-10-0-132-82 kubenswrapper[4892]: I1030 20:30:19.977877    4892 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/00ac42cd-07e0-4e77-aa8d-b268b06fc1cf-kube-api-access-7vt5v\" (UniqueName: \"kubernetes.io/projected/00ac42cd-07e0-4e77-aa8d-b268b06fc1cf-kube-api-access-7vt5v\") pod \"00ac42cd-07e0-4e77-aa8d-b268b06fc1cf\" (UID: \"00ac42cd-07e0-4e77-aa8d-b268b06fc1cf\") "
Oct 30 20:30:19.978086 ip-10-0-132-82 kubenswrapper[4892]: I1030 20:30:19.977988    4892 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/00ac42cd-07e0-4e77-aa8d-b268b06fc1cf-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/00ac42cd-07e0-4e77-aa8d-b268b06fc1cf-maas-api-service-ca\") pod \"00ac42cd-07e0-4e77-aa8d-b268b06fc1cf\" (UID: \"00ac42cd-07e0-4e77-aa8d-b268b06fc1cf\") "
Oct 30 20:30:19.978294 ip-10-0-132-82 kubenswrapper[4892]: I1030 20:30:19.978273    4892 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/00ac42cd-07e0-4e77-aa8d-b268b06fc1cf-maas-api-service-ca" pod "00ac42cd-07e0-4e77-aa8d-b268b06fc1cf" (UID: "00ac42cd-07e0-4e77-aa8d-b268b06fc1cf"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 20:30:19.980100 ip-10-0-132-82 kubenswrapper[4892]: I1030 20:30:19.980078    4892 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/00ac42cd-07e0-4e77-aa8d-b268b06fc1cf-kube-api-access-7vt5v" pod "00ac42cd-07e0-4e77-aa8d-b268b06fc1cf" (UID: "00ac42cd-07e0-4e77-aa8d-b268b06fc1cf"). InnerVolumeSpecName "kube-api-access-7vt5v". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 20:30:20.078983 ip-10-0-132-82 kubenswrapper[4892]: I1030 20:30:20.078870    4892 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/00ac42cd-07e0-4e77-aa8d-b268b06fc1cf-maas-api-service-ca\") on node \"ip-10-0-132-82.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:30:20.078983 ip-10-0-132-82 kubenswrapper[4892]: I1030 20:30:20.078897    4892 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7vt5v\" (UniqueName: \"kubernetes.io/projected/00ac42cd-07e0-4e77-aa8d-b268b06fc1cf-kube-api-access-7vt5v\") on node \"ip-10-0-132-82.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:30:20.635399 ip-10-0-132-82 kubenswrapper[4892]: I1030 20:30:20.635376    4892 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889870-q9hmg"
Oct 30 20:30:20.635399 ip-10-0-132-82 kubenswrapper[4892]: I1030 20:30:20.635383    4892 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889870-q9hmg" event={"ID":"00ac42cd-07e0-4e77-aa8d-b268b06fc1cf","Type":"ContainerDied","Data":"309eab41d0f2777bd875ad155b2ec6af28dc664e6ba13099183285246447f32c"}
Oct 30 20:30:20.635611 ip-10-0-132-82 kubenswrapper[4892]: I1030 20:30:20.635432    4892 scope.go:122] "RemoveContainer" containerID="05e9c813be9e8bf2bce1c9819589af1348b6795ca8de839b9b18f9d240be0bbe"
Oct 30 20:30:20.797891 ip-10-0-132-82 kubenswrapper[4892]: I1030 20:30:20.797866    4892 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29889870-q9hmg"]
Oct 30 20:30:20.846816 ip-10-0-132-82 kubenswrapper[4892]: I1030 20:30:20.846759    4892 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29889870-q9hmg"]
Oct 30 20:30:22.047150 ip-10-0-132-82 kubenswrapper[4892]: I1030 20:30:22.047119    4892 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f03629db-xc594"]
Oct 30 20:30:22.096399 ip-10-0-132-82 kubenswrapper[4892]: I1030 20:30:22.096375    4892 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f03629db-xc594"]
Oct 30 20:30:22.569217 ip-10-0-132-82 kubenswrapper[4892]: I1030 20:30:22.569188    4892 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="00ac42cd-07e0-4e77-aa8d-b268b06fc1cf" path="/var/lib/kubelet/pods/00ac42cd-07e0-4e77-aa8d-b268b06fc1cf/volumes"
Oct 30 20:30:22.569588 ip-10-0-132-82 kubenswrapper[4892]: I1030 20:30:22.569575    4892 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="856a6aa7-8f9f-44f0-830b-f381709b4651" path="/var/lib/kubelet/pods/856a6aa7-8f9f-44f0-830b-f381709b4651/volumes"
Oct 30 20:30:24.843414 ip-10-0-132-82 kubenswrapper[4892]: E1030 20:30:24.843379    4892 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-jrm87" containerName="extractor"
Oct 30 20:30:53.560641 ip-10-0-132-82 kubenswrapper[4892]: I1030 20:30:53.560613    4892 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-tg5xj_11db9cec-653f-4aee-8937-0cf5b367efac/console-operator/3.log"
Oct 30 20:30:53.636223 ip-10-0-132-82 kubenswrapper[4892]: I1030 20:30:53.636192    4892 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-tg5xj_11db9cec-653f-4aee-8937-0cf5b367efac/console-operator/3.log"
Oct 30 20:30:54.843620 ip-10-0-132-82 kubenswrapper[4892]: E1030 20:30:54.843581    4892 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-jrm87" containerName="extractor"
Oct 30 20:31:01.047420 ip-10-0-132-82 kubenswrapper[4892]: I1030 20:31:01.047385    4892 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0165d7d2-sf2wp"]
Oct 30 20:31:01.096824 ip-10-0-132-82 kubenswrapper[4892]: I1030 20:31:01.096786    4892 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0165d7d2-sf2wp"]
Oct 30 20:31:02.568252 ip-10-0-132-82 kubenswrapper[4892]: I1030 20:31:02.568216    4892 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="dedad02e-2673-4853-8f93-dcd631dc1ffa" path="/var/lib/kubelet/pods/dedad02e-2673-4853-8f93-dcd631dc1ffa/volumes"
Oct 30 20:31:08.269838 ip-10-0-132-82 kubenswrapper[4892]: I1030 20:31:08.269810    4892 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_global-pull-secret-syncer-76rht_c16c0a80-c5ce-4f5d-827a-2e3eb2c3cfb8/global-pull-secret-syncer/0.log"
Oct 30 20:31:09.416888 ip-10-0-132-82 kubenswrapper[4892]: I1030 20:31:09.416861    4892 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_kas-connection-checker-685895c7c9-9h85f_ba2c5571-23a3-46b6-a36d-7e8810a66261/connection-checker/0.log"
Oct 30 20:31:09.831987 ip-10-0-132-82 kubenswrapper[4892]: I1030 20:31:09.831884    4892 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_kas-connection-checker-685895c7c9-p8kjs_87da348d-b1ee-4cd3-806d-0560bff3c6f2/connection-checker/0.log"
Oct 30 20:31:10.251886 ip-10-0-132-82 kubenswrapper[4892]: I1030 20:31:10.251862    4892 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_kas-connection-checker-685895c7c9-tvmks_d260d4d7-b4aa-4f37-92b5-e75b22e13a3b/connection-checker/0.log"
Oct 30 20:31:11.259754 ip-10-0-132-82 kubenswrapper[4892]: I1030 20:31:11.259721    4892 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_konnectivity-agent-qnm56_af36f4d4-f985-4937-a9f7-9884936f3690/konnectivity-agent/0.log"
Oct 30 20:31:12.056605 ip-10-0-132-82 kubenswrapper[4892]: I1030 20:31:12.056569    4892 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-6b19338f-nt67d"]
Oct 30 20:31:12.069674 ip-10-0-132-82 kubenswrapper[4892]: I1030 20:31:12.069648    4892 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_kube-apiserver-proxy-ip-10-0-132-82.us-west-1.compute.internal_f9af2f5b3d8deb871a93a1e3f0011dc6/haproxy/0.log"
Oct 30 20:31:12.105877 ip-10-0-132-82 kubenswrapper[4892]: I1030 20:31:12.105853    4892 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-6b19338f-nt67d"]
Oct 30 20:31:12.568170 ip-10-0-132-82 kubenswrapper[4892]: I1030 20:31:12.568140    4892 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="7f896b90-72e0-4a1a-8c31-5e945063348e" path="/var/lib/kubelet/pods/7f896b90-72e0-4a1a-8c31-5e945063348e/volumes"
Oct 30 20:31:24.844410 ip-10-0-132-82 kubenswrapper[4892]: E1030 20:31:24.844373    4892 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-jrm87" containerName="extractor"
Oct 30 20:31:45.777879 ip-10-0-132-82 kubenswrapper[4892]: I1030 20:31:45.777799    4892 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-must-gather-n79m4/perf-node-gather-daemonset-xhjrt"]
Oct 30 20:31:45.778294 ip-10-0-132-82 kubenswrapper[4892]: I1030 20:31:45.778164    4892 cpu_manager.go:414] "containerMap: removing container" podUID="00ac42cd-07e0-4e77-aa8d-b268b06fc1cf" containerName="cleanup"
Oct 30 20:31:45.778294 ip-10-0-132-82 kubenswrapper[4892]: I1030 20:31:45.778176    4892 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="00ac42cd-07e0-4e77-aa8d-b268b06fc1cf" containerName="cleanup"
Oct 30 20:31:45.778294 ip-10-0-132-82 kubenswrapper[4892]: I1030 20:31:45.778200    4892 cpu_manager.go:414] "containerMap: removing container" podUID="00ac42cd-07e0-4e77-aa8d-b268b06fc1cf" containerName="cleanup"
Oct 30 20:31:45.778294 ip-10-0-132-82 kubenswrapper[4892]: I1030 20:31:45.778205    4892 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="00ac42cd-07e0-4e77-aa8d-b268b06fc1cf" containerName="cleanup"
Oct 30 20:31:45.778294 ip-10-0-132-82 kubenswrapper[4892]: I1030 20:31:45.778265    4892 memory_manager.go:365] "RemoveStaleState removing state" podUID="00ac42cd-07e0-4e77-aa8d-b268b06fc1cf" containerName="cleanup"
Oct 30 20:31:45.778294 ip-10-0-132-82 kubenswrapper[4892]: I1030 20:31:45.778272    4892 memory_manager.go:365] "RemoveStaleState removing state" podUID="00ac42cd-07e0-4e77-aa8d-b268b06fc1cf" containerName="cleanup"
Oct 30 20:31:45.781346 ip-10-0-132-82 kubenswrapper[4892]: I1030 20:31:45.781329    4892 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-n79m4/perf-node-gather-daemonset-xhjrt"
Oct 30 20:31:45.882676 ip-10-0-132-82 kubenswrapper[4892]: I1030 20:31:45.882632    4892 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-n79m4\"/\"openshift-service-ca.crt\""
Oct 30 20:31:45.882676 ip-10-0-132-82 kubenswrapper[4892]: I1030 20:31:45.882666    4892 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-n79m4\"/\"kube-root-ca.crt\""
Oct 30 20:31:45.882947 ip-10-0-132-82 kubenswrapper[4892]: I1030 20:31:45.882682    4892 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-must-gather-n79m4\"/\"default-dockercfg-jfgg8\""
Oct 30 20:31:45.885300 ip-10-0-132-82 kubenswrapper[4892]: I1030 20:31:45.885278    4892 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-n79m4/perf-node-gather-daemonset-xhjrt"]
Oct 30 20:31:45.895504 ip-10-0-132-82 kubenswrapper[4892]: I1030 20:31:45.895483    4892 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/07676190-4784-489e-a009-379a517fa9c5-proc\") pod \"perf-node-gather-daemonset-xhjrt\" (UID: \"07676190-4784-489e-a009-379a517fa9c5\") " pod="openshift-must-gather-n79m4/perf-node-gather-daemonset-xhjrt"
Oct 30 20:31:45.895613 ip-10-0-132-82 kubenswrapper[4892]: I1030 20:31:45.895530    4892 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-l24rk\" (UniqueName: \"kubernetes.io/projected/07676190-4784-489e-a009-379a517fa9c5-kube-api-access-l24rk\") pod \"perf-node-gather-daemonset-xhjrt\" (UID: \"07676190-4784-489e-a009-379a517fa9c5\") " pod="openshift-must-gather-n79m4/perf-node-gather-daemonset-xhjrt"
Oct 30 20:31:45.895669 ip-10-0-132-82 kubenswrapper[4892]: I1030 20:31:45.895613    4892 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/07676190-4784-489e-a009-379a517fa9c5-podres\") pod \"perf-node-gather-daemonset-xhjrt\" (UID: \"07676190-4784-489e-a009-379a517fa9c5\") " pod="openshift-must-gather-n79m4/perf-node-gather-daemonset-xhjrt"
Oct 30 20:31:45.895712 ip-10-0-132-82 kubenswrapper[4892]: I1030 20:31:45.895695    4892 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/07676190-4784-489e-a009-379a517fa9c5-sys\") pod \"perf-node-gather-daemonset-xhjrt\" (UID: \"07676190-4784-489e-a009-379a517fa9c5\") " pod="openshift-must-gather-n79m4/perf-node-gather-daemonset-xhjrt"
Oct 30 20:31:45.895748 ip-10-0-132-82 kubenswrapper[4892]: I1030 20:31:45.895727    4892 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/07676190-4784-489e-a009-379a517fa9c5-lib-modules\") pod \"perf-node-gather-daemonset-xhjrt\" (UID: \"07676190-4784-489e-a009-379a517fa9c5\") " pod="openshift-must-gather-n79m4/perf-node-gather-daemonset-xhjrt"
Oct 30 20:31:45.996434 ip-10-0-132-82 kubenswrapper[4892]: I1030 20:31:45.996407    4892 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/07676190-4784-489e-a009-379a517fa9c5-podres\") pod \"perf-node-gather-daemonset-xhjrt\" (UID: \"07676190-4784-489e-a009-379a517fa9c5\") " pod="openshift-must-gather-n79m4/perf-node-gather-daemonset-xhjrt"
Oct 30 20:31:45.996575 ip-10-0-132-82 kubenswrapper[4892]: I1030 20:31:45.996459    4892 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/07676190-4784-489e-a009-379a517fa9c5-sys\") pod \"perf-node-gather-daemonset-xhjrt\" (UID: \"07676190-4784-489e-a009-379a517fa9c5\") " pod="openshift-must-gather-n79m4/perf-node-gather-daemonset-xhjrt"
Oct 30 20:31:45.996575 ip-10-0-132-82 kubenswrapper[4892]: I1030 20:31:45.996495    4892 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/07676190-4784-489e-a009-379a517fa9c5-lib-modules\") pod \"perf-node-gather-daemonset-xhjrt\" (UID: \"07676190-4784-489e-a009-379a517fa9c5\") " pod="openshift-must-gather-n79m4/perf-node-gather-daemonset-xhjrt"
Oct 30 20:31:45.996575 ip-10-0-132-82 kubenswrapper[4892]: I1030 20:31:45.996532    4892 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/07676190-4784-489e-a009-379a517fa9c5-podres\") pod \"perf-node-gather-daemonset-xhjrt\" (UID: \"07676190-4784-489e-a009-379a517fa9c5\") " pod="openshift-must-gather-n79m4/perf-node-gather-daemonset-xhjrt"
Oct 30 20:31:45.996575 ip-10-0-132-82 kubenswrapper[4892]: I1030 20:31:45.996549    4892 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/07676190-4784-489e-a009-379a517fa9c5-proc\") pod \"perf-node-gather-daemonset-xhjrt\" (UID: \"07676190-4784-489e-a009-379a517fa9c5\") " pod="openshift-must-gather-n79m4/perf-node-gather-daemonset-xhjrt"
Oct 30 20:31:45.996812 ip-10-0-132-82 kubenswrapper[4892]: I1030 20:31:45.996591    4892 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/07676190-4784-489e-a009-379a517fa9c5-sys\") pod \"perf-node-gather-daemonset-xhjrt\" (UID: \"07676190-4784-489e-a009-379a517fa9c5\") " pod="openshift-must-gather-n79m4/perf-node-gather-daemonset-xhjrt"
Oct 30 20:31:45.996812 ip-10-0-132-82 kubenswrapper[4892]: I1030 20:31:45.996597    4892 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-l24rk\" (UniqueName: \"kubernetes.io/projected/07676190-4784-489e-a009-379a517fa9c5-kube-api-access-l24rk\") pod \"perf-node-gather-daemonset-xhjrt\" (UID: \"07676190-4784-489e-a009-379a517fa9c5\") " pod="openshift-must-gather-n79m4/perf-node-gather-daemonset-xhjrt"
Oct 30 20:31:45.996812 ip-10-0-132-82 kubenswrapper[4892]: I1030 20:31:45.996595    4892 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/07676190-4784-489e-a009-379a517fa9c5-proc\") pod \"perf-node-gather-daemonset-xhjrt\" (UID: \"07676190-4784-489e-a009-379a517fa9c5\") " pod="openshift-must-gather-n79m4/perf-node-gather-daemonset-xhjrt"
Oct 30 20:31:45.996812 ip-10-0-132-82 kubenswrapper[4892]: I1030 20:31:45.996630    4892 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/07676190-4784-489e-a009-379a517fa9c5-lib-modules\") pod \"perf-node-gather-daemonset-xhjrt\" (UID: \"07676190-4784-489e-a009-379a517fa9c5\") " pod="openshift-must-gather-n79m4/perf-node-gather-daemonset-xhjrt"
Oct 30 20:31:46.052982 ip-10-0-132-82 kubenswrapper[4892]: I1030 20:31:46.052903    4892 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-l24rk\" (UniqueName: \"kubernetes.io/projected/07676190-4784-489e-a009-379a517fa9c5-kube-api-access-l24rk\") pod \"perf-node-gather-daemonset-xhjrt\" (UID: \"07676190-4784-489e-a009-379a517fa9c5\") " pod="openshift-must-gather-n79m4/perf-node-gather-daemonset-xhjrt"
Oct 30 20:31:46.093718 ip-10-0-132-82 kubenswrapper[4892]: I1030 20:31:46.093697    4892 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-n79m4/perf-node-gather-daemonset-xhjrt"
Oct 30 20:31:46.463193 ip-10-0-132-82 kubenswrapper[4892]: I1030 20:31:46.463163    4892 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-n79m4/perf-node-gather-daemonset-xhjrt"]
Oct 30 20:31:46.465346 ip-10-0-132-82 kubenswrapper[4892]: W1030 20:31:46.465321    4892 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-pod07676190_4784_489e_a009_379a517fa9c5.slice/crio-efdea4e38c5ce4346b0c02a405a9f271a8d137cd57a3ba5c7904cd7953fb5fba WatchSource:0}: Error finding container efdea4e38c5ce4346b0c02a405a9f271a8d137cd57a3ba5c7904cd7953fb5fba: Status 404 returned error can't find the container with id efdea4e38c5ce4346b0c02a405a9f271a8d137cd57a3ba5c7904cd7953fb5fba
Oct 30 20:31:47.066850 ip-10-0-132-82 kubenswrapper[4892]: I1030 20:31:47.066805    4892 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-n79m4/perf-node-gather-daemonset-xhjrt" event={"ID":"07676190-4784-489e-a009-379a517fa9c5","Type":"ContainerStarted","Data":"13ad096ae7e9c9fe5061b8384a111315b4b88630dbd041a73f1702b34c5503b8"}
Oct 30 20:31:47.067280 ip-10-0-132-82 kubenswrapper[4892]: I1030 20:31:47.066881    4892 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-must-gather-n79m4/perf-node-gather-daemonset-xhjrt"
Oct 30 20:31:47.067280 ip-10-0-132-82 kubenswrapper[4892]: I1030 20:31:47.066910    4892 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-n79m4/perf-node-gather-daemonset-xhjrt" event={"ID":"07676190-4784-489e-a009-379a517fa9c5","Type":"ContainerStarted","Data":"efdea4e38c5ce4346b0c02a405a9f271a8d137cd57a3ba5c7904cd7953fb5fba"}
Oct 30 20:31:47.179817 ip-10-0-132-82 kubenswrapper[4892]: I1030 20:31:47.179758    4892 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-must-gather-n79m4/perf-node-gather-daemonset-xhjrt" podStartSLOduration=2.179744615 podStartE2EDuration="2.179744615s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-30 20:31:45 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 20:31:47.179019794 +0000 UTC m=+4554.749485395" watchObservedRunningTime="2026-10-30 20:31:47.179744615 +0000 UTC m=+4554.750210205"
Oct 30 20:31:52.417751 ip-10-0-132-82 kubenswrapper[4892]: I1030 20:31:52.417722    4892 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-mb59v"]
Oct 30 20:31:52.418247 ip-10-0-132-82 kubenswrapper[4892]: I1030 20:31:52.417946    4892 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-mb59v" podUID="72926e14-705b-493a-879f-f595c35a7490" containerName="istio-proxy" containerID="cri-o://daa545f6b5ed58fff46391eee8e6f47d0db43d87a6fee8b2c07925792dfcda7c" gracePeriod=30
Oct 30 20:31:52.467228 ip-10-0-132-82 kubenswrapper[4892]: I1030 20:31:52.467193    4892 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-k85mx"]
Oct 30 20:31:52.467454 ip-10-0-132-82 kubenswrapper[4892]: I1030 20:31:52.467420    4892 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-k85mx" podUID="6f05c89a-d631-4c95-aad6-d08d6c87cffd" containerName="istio-proxy" containerID="cri-o://d87139aa86bf1b1aacecc2c7327610a42fda8698ccf202823208d646da9c409d" gracePeriod=30
Oct 30 20:31:52.469198 ip-10-0-132-82 kubenswrapper[4892]: I1030 20:31:52.469176    4892 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-p7m7x"]
Oct 30 20:31:52.469372 ip-10-0-132-82 kubenswrapper[4892]: I1030 20:31:52.469338    4892 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-p7m7x" podUID="741c37d3-758a-4ced-ab0e-9038baea2d88" containerName="istio-proxy" containerID="cri-o://9989808b24a5895221146c0e35de04e3a5077e014a7900af7241e8077e77f2cf" gracePeriod=30
Oct 30 20:31:53.083157 ip-10-0-132-82 kubenswrapper[4892]: I1030 20:31:53.083130    4892 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-must-gather-n79m4/perf-node-gather-daemonset-xhjrt"
Oct 30 20:31:54.844846 ip-10-0-132-82 kubenswrapper[4892]: E1030 20:31:54.844804    4892 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-jrm87" containerName="extractor"
Oct 30 20:31:57.730450 ip-10-0-132-82 kubenswrapper[4892]: I1030 20:31:57.730416    4892 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-mb59v"
Oct 30 20:31:57.768569 ip-10-0-132-82 kubenswrapper[4892]: I1030 20:31:57.768545    4892 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-k85mx"
Oct 30 20:31:57.771562 ip-10-0-132-82 kubenswrapper[4892]: I1030 20:31:57.771541    4892 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-p7m7x"
Oct 30 20:31:57.800294 ip-10-0-132-82 kubenswrapper[4892]: I1030 20:31:57.800270    4892 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6f05c89a-d631-4c95-aad6-d08d6c87cffd-kube-api-access-sxn6r\" (UniqueName: \"kubernetes.io/projected/6f05c89a-d631-4c95-aad6-d08d6c87cffd-kube-api-access-sxn6r\") pod \"6f05c89a-d631-4c95-aad6-d08d6c87cffd\" (UID: \"6f05c89a-d631-4c95-aad6-d08d6c87cffd\") "
Oct 30 20:31:57.800407 ip-10-0-132-82 kubenswrapper[4892]: I1030 20:31:57.800343    4892 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/72926e14-705b-493a-879f-f595c35a7490-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/72926e14-705b-493a-879f-f595c35a7490-credential-socket\") pod \"72926e14-705b-493a-879f-f595c35a7490\" (UID: \"72926e14-705b-493a-879f-f595c35a7490\") "
Oct 30 20:31:57.800475 ip-10-0-132-82 kubenswrapper[4892]: I1030 20:31:57.800443    4892 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/741c37d3-758a-4ced-ab0e-9038baea2d88-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/741c37d3-758a-4ced-ab0e-9038baea2d88-istio-podinfo\") pod \"741c37d3-758a-4ced-ab0e-9038baea2d88\" (UID: \"741c37d3-758a-4ced-ab0e-9038baea2d88\") "
Oct 30 20:31:57.800475 ip-10-0-132-82 kubenswrapper[4892]: I1030 20:31:57.800470    4892 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6f05c89a-d631-4c95-aad6-d08d6c87cffd-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6f05c89a-d631-4c95-aad6-d08d6c87cffd-istio-envoy\") pod \"6f05c89a-d631-4c95-aad6-d08d6c87cffd\" (UID: \"6f05c89a-d631-4c95-aad6-d08d6c87cffd\") "
Oct 30 20:31:57.800598 ip-10-0-132-82 kubenswrapper[4892]: I1030 20:31:57.800491    4892 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/741c37d3-758a-4ced-ab0e-9038baea2d88-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/741c37d3-758a-4ced-ab0e-9038baea2d88-credential-socket\") pod \"741c37d3-758a-4ced-ab0e-9038baea2d88\" (UID: \"741c37d3-758a-4ced-ab0e-9038baea2d88\") "
Oct 30 20:31:57.800598 ip-10-0-132-82 kubenswrapper[4892]: I1030 20:31:57.800511    4892 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6f05c89a-d631-4c95-aad6-d08d6c87cffd-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6f05c89a-d631-4c95-aad6-d08d6c87cffd-workload-socket\") pod \"6f05c89a-d631-4c95-aad6-d08d6c87cffd\" (UID: \"6f05c89a-d631-4c95-aad6-d08d6c87cffd\") "
Oct 30 20:31:57.800598 ip-10-0-132-82 kubenswrapper[4892]: I1030 20:31:57.800553    4892 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/72926e14-705b-493a-879f-f595c35a7490-credential-socket" pod "72926e14-705b-493a-879f-f595c35a7490" (UID: "72926e14-705b-493a-879f-f595c35a7490"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 20:31:57.800768 ip-10-0-132-82 kubenswrapper[4892]: I1030 20:31:57.800622    4892 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/72926e14-705b-493a-879f-f595c35a7490-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/72926e14-705b-493a-879f-f595c35a7490-istio-data\") pod \"72926e14-705b-493a-879f-f595c35a7490\" (UID: \"72926e14-705b-493a-879f-f595c35a7490\") "
Oct 30 20:31:57.800768 ip-10-0-132-82 kubenswrapper[4892]: I1030 20:31:57.800703    4892 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/72926e14-705b-493a-879f-f595c35a7490-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/72926e14-705b-493a-879f-f595c35a7490-istiod-ca-cert\") pod \"72926e14-705b-493a-879f-f595c35a7490\" (UID: \"72926e14-705b-493a-879f-f595c35a7490\") "
Oct 30 20:31:57.800768 ip-10-0-132-82 kubenswrapper[4892]: I1030 20:31:57.800755    4892 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6f05c89a-d631-4c95-aad6-d08d6c87cffd-workload-socket" pod "6f05c89a-d631-4c95-aad6-d08d6c87cffd" (UID: "6f05c89a-d631-4c95-aad6-d08d6c87cffd"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 20:31:57.800932 ip-10-0-132-82 kubenswrapper[4892]: I1030 20:31:57.800754    4892 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/741c37d3-758a-4ced-ab0e-9038baea2d88-credential-socket" pod "741c37d3-758a-4ced-ab0e-9038baea2d88" (UID: "741c37d3-758a-4ced-ab0e-9038baea2d88"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 20:31:57.800932 ip-10-0-132-82 kubenswrapper[4892]: I1030 20:31:57.800759    4892 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6f05c89a-d631-4c95-aad6-d08d6c87cffd-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6f05c89a-d631-4c95-aad6-d08d6c87cffd-credential-socket\") pod \"6f05c89a-d631-4c95-aad6-d08d6c87cffd\" (UID: \"6f05c89a-d631-4c95-aad6-d08d6c87cffd\") "
Oct 30 20:31:57.800932 ip-10-0-132-82 kubenswrapper[4892]: I1030 20:31:57.800804    4892 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/741c37d3-758a-4ced-ab0e-9038baea2d88-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/741c37d3-758a-4ced-ab0e-9038baea2d88-workload-certs\") pod \"741c37d3-758a-4ced-ab0e-9038baea2d88\" (UID: \"741c37d3-758a-4ced-ab0e-9038baea2d88\") "
Oct 30 20:31:57.800932 ip-10-0-132-82 kubenswrapper[4892]: I1030 20:31:57.800835    4892 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/6f05c89a-d631-4c95-aad6-d08d6c87cffd-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6f05c89a-d631-4c95-aad6-d08d6c87cffd-istio-podinfo\") pod \"6f05c89a-d631-4c95-aad6-d08d6c87cffd\" (UID: \"6f05c89a-d631-4c95-aad6-d08d6c87cffd\") "
Oct 30 20:31:57.800932 ip-10-0-132-82 kubenswrapper[4892]: I1030 20:31:57.800873    4892 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/741c37d3-758a-4ced-ab0e-9038baea2d88-istio-token\" (UniqueName: \"kubernetes.io/projected/741c37d3-758a-4ced-ab0e-9038baea2d88-istio-token\") pod \"741c37d3-758a-4ced-ab0e-9038baea2d88\" (UID: \"741c37d3-758a-4ced-ab0e-9038baea2d88\") "
Oct 30 20:31:57.801248 ip-10-0-132-82 kubenswrapper[4892]: I1030 20:31:57.800931    4892 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/741c37d3-758a-4ced-ab0e-9038baea2d88-kube-api-access-cvv5r\" (UniqueName: \"kubernetes.io/projected/741c37d3-758a-4ced-ab0e-9038baea2d88-kube-api-access-cvv5r\") pod \"741c37d3-758a-4ced-ab0e-9038baea2d88\" (UID: \"741c37d3-758a-4ced-ab0e-9038baea2d88\") "
Oct 30 20:31:57.801248 ip-10-0-132-82 kubenswrapper[4892]: I1030 20:31:57.800959    4892 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6f05c89a-d631-4c95-aad6-d08d6c87cffd-credential-socket" pod "6f05c89a-d631-4c95-aad6-d08d6c87cffd" (UID: "6f05c89a-d631-4c95-aad6-d08d6c87cffd"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 20:31:57.801248 ip-10-0-132-82 kubenswrapper[4892]: I1030 20:31:57.800982    4892 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/72926e14-705b-493a-879f-f595c35a7490-istio-data" pod "72926e14-705b-493a-879f-f595c35a7490" (UID: "72926e14-705b-493a-879f-f595c35a7490"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 20:31:57.801248 ip-10-0-132-82 kubenswrapper[4892]: I1030 20:31:57.800996    4892 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/741c37d3-758a-4ced-ab0e-9038baea2d88-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/741c37d3-758a-4ced-ab0e-9038baea2d88-istio-data\") pod \"741c37d3-758a-4ced-ab0e-9038baea2d88\" (UID: \"741c37d3-758a-4ced-ab0e-9038baea2d88\") "
Oct 30 20:31:57.801248 ip-10-0-132-82 kubenswrapper[4892]: I1030 20:31:57.801052    4892 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/72926e14-705b-493a-879f-f595c35a7490-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/72926e14-705b-493a-879f-f595c35a7490-istio-envoy\") pod \"72926e14-705b-493a-879f-f595c35a7490\" (UID: \"72926e14-705b-493a-879f-f595c35a7490\") "
Oct 30 20:31:57.801248 ip-10-0-132-82 kubenswrapper[4892]: I1030 20:31:57.801080    4892 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/72926e14-705b-493a-879f-f595c35a7490-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/72926e14-705b-493a-879f-f595c35a7490-istio-podinfo\") pod \"72926e14-705b-493a-879f-f595c35a7490\" (UID: \"72926e14-705b-493a-879f-f595c35a7490\") "
Oct 30 20:31:57.801248 ip-10-0-132-82 kubenswrapper[4892]: I1030 20:31:57.801121    4892 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/72926e14-705b-493a-879f-f595c35a7490-istio-token\" (UniqueName: \"kubernetes.io/projected/72926e14-705b-493a-879f-f595c35a7490-istio-token\") pod \"72926e14-705b-493a-879f-f595c35a7490\" (UID: \"72926e14-705b-493a-879f-f595c35a7490\") "
Oct 30 20:31:57.801248 ip-10-0-132-82 kubenswrapper[4892]: I1030 20:31:57.801159    4892 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/741c37d3-758a-4ced-ab0e-9038baea2d88-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/741c37d3-758a-4ced-ab0e-9038baea2d88-istiod-ca-cert\") pod \"741c37d3-758a-4ced-ab0e-9038baea2d88\" (UID: \"741c37d3-758a-4ced-ab0e-9038baea2d88\") "
Oct 30 20:31:57.801248 ip-10-0-132-82 kubenswrapper[4892]: I1030 20:31:57.801194    4892 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/72926e14-705b-493a-879f-f595c35a7490-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/72926e14-705b-493a-879f-f595c35a7490-workload-socket\") pod \"72926e14-705b-493a-879f-f595c35a7490\" (UID: \"72926e14-705b-493a-879f-f595c35a7490\") "
Oct 30 20:31:57.801248 ip-10-0-132-82 kubenswrapper[4892]: I1030 20:31:57.801223    4892 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6f05c89a-d631-4c95-aad6-d08d6c87cffd-istio-token\" (UniqueName: \"kubernetes.io/projected/6f05c89a-d631-4c95-aad6-d08d6c87cffd-istio-token\") pod \"6f05c89a-d631-4c95-aad6-d08d6c87cffd\" (UID: \"6f05c89a-d631-4c95-aad6-d08d6c87cffd\") "
Oct 30 20:31:57.801780 ip-10-0-132-82 kubenswrapper[4892]: I1030 20:31:57.801262    4892 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/6f05c89a-d631-4c95-aad6-d08d6c87cffd-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6f05c89a-d631-4c95-aad6-d08d6c87cffd-istiod-ca-cert\") pod \"6f05c89a-d631-4c95-aad6-d08d6c87cffd\" (UID: \"6f05c89a-d631-4c95-aad6-d08d6c87cffd\") "
Oct 30 20:31:57.801780 ip-10-0-132-82 kubenswrapper[4892]: I1030 20:31:57.801293    4892 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/72926e14-705b-493a-879f-f595c35a7490-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/72926e14-705b-493a-879f-f595c35a7490-workload-certs\") pod \"72926e14-705b-493a-879f-f595c35a7490\" (UID: \"72926e14-705b-493a-879f-f595c35a7490\") "
Oct 30 20:31:57.801780 ip-10-0-132-82 kubenswrapper[4892]: I1030 20:31:57.801332    4892 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6f05c89a-d631-4c95-aad6-d08d6c87cffd-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6f05c89a-d631-4c95-aad6-d08d6c87cffd-workload-certs\") pod \"6f05c89a-d631-4c95-aad6-d08d6c87cffd\" (UID: \"6f05c89a-d631-4c95-aad6-d08d6c87cffd\") "
Oct 30 20:31:57.801780 ip-10-0-132-82 kubenswrapper[4892]: I1030 20:31:57.801358    4892 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/741c37d3-758a-4ced-ab0e-9038baea2d88-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/741c37d3-758a-4ced-ab0e-9038baea2d88-workload-socket\") pod \"741c37d3-758a-4ced-ab0e-9038baea2d88\" (UID: \"741c37d3-758a-4ced-ab0e-9038baea2d88\") "
Oct 30 20:31:57.801780 ip-10-0-132-82 kubenswrapper[4892]: I1030 20:31:57.801402    4892 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6f05c89a-d631-4c95-aad6-d08d6c87cffd-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6f05c89a-d631-4c95-aad6-d08d6c87cffd-istio-data\") pod \"6f05c89a-d631-4c95-aad6-d08d6c87cffd\" (UID: \"6f05c89a-d631-4c95-aad6-d08d6c87cffd\") "
Oct 30 20:31:57.801780 ip-10-0-132-82 kubenswrapper[4892]: I1030 20:31:57.801436    4892 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/72926e14-705b-493a-879f-f595c35a7490-kube-api-access-hgml7\" (UniqueName: \"kubernetes.io/projected/72926e14-705b-493a-879f-f595c35a7490-kube-api-access-hgml7\") pod \"72926e14-705b-493a-879f-f595c35a7490\" (UID: \"72926e14-705b-493a-879f-f595c35a7490\") "
Oct 30 20:31:57.801780 ip-10-0-132-82 kubenswrapper[4892]: I1030 20:31:57.801464    4892 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/741c37d3-758a-4ced-ab0e-9038baea2d88-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/741c37d3-758a-4ced-ab0e-9038baea2d88-istio-envoy\") pod \"741c37d3-758a-4ced-ab0e-9038baea2d88\" (UID: \"741c37d3-758a-4ced-ab0e-9038baea2d88\") "
Oct 30 20:31:57.802220 ip-10-0-132-82 kubenswrapper[4892]: I1030 20:31:57.801778    4892 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/741c37d3-758a-4ced-ab0e-9038baea2d88-istio-data" pod "741c37d3-758a-4ced-ab0e-9038baea2d88" (UID: "741c37d3-758a-4ced-ab0e-9038baea2d88"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 20:31:57.802220 ip-10-0-132-82 kubenswrapper[4892]: I1030 20:31:57.801787    4892 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/72926e14-705b-493a-879f-f595c35a7490-istiod-ca-cert" pod "72926e14-705b-493a-879f-f595c35a7490" (UID: "72926e14-705b-493a-879f-f595c35a7490"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 20:31:57.802220 ip-10-0-132-82 kubenswrapper[4892]: I1030 20:31:57.802011    4892 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/741c37d3-758a-4ced-ab0e-9038baea2d88-workload-certs" pod "741c37d3-758a-4ced-ab0e-9038baea2d88" (UID: "741c37d3-758a-4ced-ab0e-9038baea2d88"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 20:31:57.802404 ip-10-0-132-82 kubenswrapper[4892]: I1030 20:31:57.802329    4892 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/72926e14-705b-493a-879f-f595c35a7490-workload-certs" pod "72926e14-705b-493a-879f-f595c35a7490" (UID: "72926e14-705b-493a-879f-f595c35a7490"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 20:31:57.802471 ip-10-0-132-82 kubenswrapper[4892]: I1030 20:31:57.802394    4892 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6f05c89a-d631-4c95-aad6-d08d6c87cffd-credential-socket\") on node \"ip-10-0-132-82.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:31:57.802533 ip-10-0-132-82 kubenswrapper[4892]: I1030 20:31:57.802467    4892 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/741c37d3-758a-4ced-ab0e-9038baea2d88-workload-certs\") on node \"ip-10-0-132-82.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:31:57.802533 ip-10-0-132-82 kubenswrapper[4892]: I1030 20:31:57.802496    4892 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/741c37d3-758a-4ced-ab0e-9038baea2d88-istio-data\") on node \"ip-10-0-132-82.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:31:57.802533 ip-10-0-132-82 kubenswrapper[4892]: I1030 20:31:57.802511    4892 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/72926e14-705b-493a-879f-f595c35a7490-credential-socket\") on node \"ip-10-0-132-82.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:31:57.802533 ip-10-0-132-82 kubenswrapper[4892]: I1030 20:31:57.802525    4892 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/741c37d3-758a-4ced-ab0e-9038baea2d88-credential-socket\") on node \"ip-10-0-132-82.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:31:57.802762 ip-10-0-132-82 kubenswrapper[4892]: I1030 20:31:57.802539    4892 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6f05c89a-d631-4c95-aad6-d08d6c87cffd-workload-socket\") on node \"ip-10-0-132-82.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:31:57.802762 ip-10-0-132-82 kubenswrapper[4892]: I1030 20:31:57.802555    4892 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/72926e14-705b-493a-879f-f595c35a7490-istio-data\") on node \"ip-10-0-132-82.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:31:57.802762 ip-10-0-132-82 kubenswrapper[4892]: I1030 20:31:57.802568    4892 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/72926e14-705b-493a-879f-f595c35a7490-istiod-ca-cert\") on node \"ip-10-0-132-82.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:31:57.802762 ip-10-0-132-82 kubenswrapper[4892]: I1030 20:31:57.802571    4892 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/741c37d3-758a-4ced-ab0e-9038baea2d88-istiod-ca-cert" pod "741c37d3-758a-4ced-ab0e-9038baea2d88" (UID: "741c37d3-758a-4ced-ab0e-9038baea2d88"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 20:31:57.802998 ip-10-0-132-82 kubenswrapper[4892]: I1030 20:31:57.802808    4892 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/72926e14-705b-493a-879f-f595c35a7490-workload-socket" pod "72926e14-705b-493a-879f-f595c35a7490" (UID: "72926e14-705b-493a-879f-f595c35a7490"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 20:31:57.802998 ip-10-0-132-82 kubenswrapper[4892]: I1030 20:31:57.802921    4892 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6f05c89a-d631-4c95-aad6-d08d6c87cffd-workload-certs" pod "6f05c89a-d631-4c95-aad6-d08d6c87cffd" (UID: "6f05c89a-d631-4c95-aad6-d08d6c87cffd"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 20:31:57.803184 ip-10-0-132-82 kubenswrapper[4892]: I1030 20:31:57.803163    4892 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/741c37d3-758a-4ced-ab0e-9038baea2d88-workload-socket" pod "741c37d3-758a-4ced-ab0e-9038baea2d88" (UID: "741c37d3-758a-4ced-ab0e-9038baea2d88"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 20:31:57.803274 ip-10-0-132-82 kubenswrapper[4892]: I1030 20:31:57.803188    4892 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6f05c89a-d631-4c95-aad6-d08d6c87cffd-istiod-ca-cert" pod "6f05c89a-d631-4c95-aad6-d08d6c87cffd" (UID: "6f05c89a-d631-4c95-aad6-d08d6c87cffd"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 20:31:57.803401 ip-10-0-132-82 kubenswrapper[4892]: I1030 20:31:57.803381    4892 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6f05c89a-d631-4c95-aad6-d08d6c87cffd-istio-data" pod "6f05c89a-d631-4c95-aad6-d08d6c87cffd" (UID: "6f05c89a-d631-4c95-aad6-d08d6c87cffd"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 20:31:57.803591 ip-10-0-132-82 kubenswrapper[4892]: I1030 20:31:57.803562    4892 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6f05c89a-d631-4c95-aad6-d08d6c87cffd-istio-envoy" pod "6f05c89a-d631-4c95-aad6-d08d6c87cffd" (UID: "6f05c89a-d631-4c95-aad6-d08d6c87cffd"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 20:31:57.803708 ip-10-0-132-82 kubenswrapper[4892]: I1030 20:31:57.803607    4892 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/6f05c89a-d631-4c95-aad6-d08d6c87cffd-istio-podinfo" pod "6f05c89a-d631-4c95-aad6-d08d6c87cffd" (UID: "6f05c89a-d631-4c95-aad6-d08d6c87cffd"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 20:31:57.804627 ip-10-0-132-82 kubenswrapper[4892]: I1030 20:31:57.804586    4892 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6f05c89a-d631-4c95-aad6-d08d6c87cffd-kube-api-access-sxn6r" pod "6f05c89a-d631-4c95-aad6-d08d6c87cffd" (UID: "6f05c89a-d631-4c95-aad6-d08d6c87cffd"). InnerVolumeSpecName "kube-api-access-sxn6r". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 20:31:57.805660 ip-10-0-132-82 kubenswrapper[4892]: I1030 20:31:57.805617    4892 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/741c37d3-758a-4ced-ab0e-9038baea2d88-istio-envoy" pod "741c37d3-758a-4ced-ab0e-9038baea2d88" (UID: "741c37d3-758a-4ced-ab0e-9038baea2d88"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 20:31:57.806019 ip-10-0-132-82 kubenswrapper[4892]: I1030 20:31:57.805992    4892 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/72926e14-705b-493a-879f-f595c35a7490-istio-podinfo" pod "72926e14-705b-493a-879f-f595c35a7490" (UID: "72926e14-705b-493a-879f-f595c35a7490"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 20:31:57.806112 ip-10-0-132-82 kubenswrapper[4892]: I1030 20:31:57.806061    4892 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/741c37d3-758a-4ced-ab0e-9038baea2d88-istio-podinfo" pod "741c37d3-758a-4ced-ab0e-9038baea2d88" (UID: "741c37d3-758a-4ced-ab0e-9038baea2d88"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 20:31:57.806335 ip-10-0-132-82 kubenswrapper[4892]: I1030 20:31:57.806313    4892 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/741c37d3-758a-4ced-ab0e-9038baea2d88-istio-token" pod "741c37d3-758a-4ced-ab0e-9038baea2d88" (UID: "741c37d3-758a-4ced-ab0e-9038baea2d88"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 20:31:57.806479 ip-10-0-132-82 kubenswrapper[4892]: I1030 20:31:57.806382    4892 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/72926e14-705b-493a-879f-f595c35a7490-istio-token" pod "72926e14-705b-493a-879f-f595c35a7490" (UID: "72926e14-705b-493a-879f-f595c35a7490"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 20:31:57.806479 ip-10-0-132-82 kubenswrapper[4892]: I1030 20:31:57.806439    4892 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/72926e14-705b-493a-879f-f595c35a7490-kube-api-access-hgml7" pod "72926e14-705b-493a-879f-f595c35a7490" (UID: "72926e14-705b-493a-879f-f595c35a7490"). InnerVolumeSpecName "kube-api-access-hgml7". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 20:31:57.806620 ip-10-0-132-82 kubenswrapper[4892]: I1030 20:31:57.806488    4892 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6f05c89a-d631-4c95-aad6-d08d6c87cffd-istio-token" pod "6f05c89a-d631-4c95-aad6-d08d6c87cffd" (UID: "6f05c89a-d631-4c95-aad6-d08d6c87cffd"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 20:31:57.807159 ip-10-0-132-82 kubenswrapper[4892]: I1030 20:31:57.807143    4892 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/72926e14-705b-493a-879f-f595c35a7490-istio-envoy" pod "72926e14-705b-493a-879f-f595c35a7490" (UID: "72926e14-705b-493a-879f-f595c35a7490"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 20:31:57.807619 ip-10-0-132-82 kubenswrapper[4892]: I1030 20:31:57.807606    4892 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/741c37d3-758a-4ced-ab0e-9038baea2d88-kube-api-access-cvv5r" pod "741c37d3-758a-4ced-ab0e-9038baea2d88" (UID: "741c37d3-758a-4ced-ab0e-9038baea2d88"). InnerVolumeSpecName "kube-api-access-cvv5r". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 20:31:57.903104 ip-10-0-132-82 kubenswrapper[4892]: I1030 20:31:57.903079    4892 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6f05c89a-d631-4c95-aad6-d08d6c87cffd-istio-podinfo\") on node \"ip-10-0-132-82.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:31:57.903104 ip-10-0-132-82 kubenswrapper[4892]: I1030 20:31:57.903103    4892 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/741c37d3-758a-4ced-ab0e-9038baea2d88-istio-token\") on node \"ip-10-0-132-82.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:31:57.903281 ip-10-0-132-82 kubenswrapper[4892]: I1030 20:31:57.903113    4892 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-cvv5r\" (UniqueName: \"kubernetes.io/projected/741c37d3-758a-4ced-ab0e-9038baea2d88-kube-api-access-cvv5r\") on node \"ip-10-0-132-82.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:31:57.903281 ip-10-0-132-82 kubenswrapper[4892]: I1030 20:31:57.903124    4892 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/72926e14-705b-493a-879f-f595c35a7490-istio-envoy\") on node \"ip-10-0-132-82.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:31:57.903281 ip-10-0-132-82 kubenswrapper[4892]: I1030 20:31:57.903132    4892 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/72926e14-705b-493a-879f-f595c35a7490-istio-podinfo\") on node \"ip-10-0-132-82.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:31:57.903281 ip-10-0-132-82 kubenswrapper[4892]: I1030 20:31:57.903140    4892 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/72926e14-705b-493a-879f-f595c35a7490-istio-token\") on node \"ip-10-0-132-82.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:31:57.903281 ip-10-0-132-82 kubenswrapper[4892]: I1030 20:31:57.903148    4892 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/741c37d3-758a-4ced-ab0e-9038baea2d88-istiod-ca-cert\") on node \"ip-10-0-132-82.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:31:57.903281 ip-10-0-132-82 kubenswrapper[4892]: I1030 20:31:57.903156    4892 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/72926e14-705b-493a-879f-f595c35a7490-workload-socket\") on node \"ip-10-0-132-82.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:31:57.903281 ip-10-0-132-82 kubenswrapper[4892]: I1030 20:31:57.903164    4892 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6f05c89a-d631-4c95-aad6-d08d6c87cffd-istio-token\") on node \"ip-10-0-132-82.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:31:57.903281 ip-10-0-132-82 kubenswrapper[4892]: I1030 20:31:57.903171    4892 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6f05c89a-d631-4c95-aad6-d08d6c87cffd-istiod-ca-cert\") on node \"ip-10-0-132-82.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:31:57.903281 ip-10-0-132-82 kubenswrapper[4892]: I1030 20:31:57.903179    4892 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/72926e14-705b-493a-879f-f595c35a7490-workload-certs\") on node \"ip-10-0-132-82.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:31:57.903281 ip-10-0-132-82 kubenswrapper[4892]: I1030 20:31:57.903187    4892 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6f05c89a-d631-4c95-aad6-d08d6c87cffd-workload-certs\") on node \"ip-10-0-132-82.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:31:57.903281 ip-10-0-132-82 kubenswrapper[4892]: I1030 20:31:57.903195    4892 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/741c37d3-758a-4ced-ab0e-9038baea2d88-workload-socket\") on node \"ip-10-0-132-82.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:31:57.903281 ip-10-0-132-82 kubenswrapper[4892]: I1030 20:31:57.903202    4892 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6f05c89a-d631-4c95-aad6-d08d6c87cffd-istio-data\") on node \"ip-10-0-132-82.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:31:57.903281 ip-10-0-132-82 kubenswrapper[4892]: I1030 20:31:57.903212    4892 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hgml7\" (UniqueName: \"kubernetes.io/projected/72926e14-705b-493a-879f-f595c35a7490-kube-api-access-hgml7\") on node \"ip-10-0-132-82.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:31:57.903281 ip-10-0-132-82 kubenswrapper[4892]: I1030 20:31:57.903219    4892 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/741c37d3-758a-4ced-ab0e-9038baea2d88-istio-envoy\") on node \"ip-10-0-132-82.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:31:57.903281 ip-10-0-132-82 kubenswrapper[4892]: I1030 20:31:57.903228    4892 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-sxn6r\" (UniqueName: \"kubernetes.io/projected/6f05c89a-d631-4c95-aad6-d08d6c87cffd-kube-api-access-sxn6r\") on node \"ip-10-0-132-82.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:31:57.903281 ip-10-0-132-82 kubenswrapper[4892]: I1030 20:31:57.903237    4892 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/741c37d3-758a-4ced-ab0e-9038baea2d88-istio-podinfo\") on node \"ip-10-0-132-82.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:31:57.903281 ip-10-0-132-82 kubenswrapper[4892]: I1030 20:31:57.903244    4892 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6f05c89a-d631-4c95-aad6-d08d6c87cffd-istio-envoy\") on node \"ip-10-0-132-82.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:31:58.126483 ip-10-0-132-82 kubenswrapper[4892]: I1030 20:31:58.126398    4892 generic.go:356] "Generic (PLEG): container finished" podID="741c37d3-758a-4ced-ab0e-9038baea2d88" containerID="9989808b24a5895221146c0e35de04e3a5077e014a7900af7241e8077e77f2cf" exitCode=0
Oct 30 20:31:58.126483 ip-10-0-132-82 kubenswrapper[4892]: I1030 20:31:58.126449    4892 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-p7m7x"
Oct 30 20:31:58.126733 ip-10-0-132-82 kubenswrapper[4892]: I1030 20:31:58.126474    4892 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-p7m7x" event={"ID":"741c37d3-758a-4ced-ab0e-9038baea2d88","Type":"ContainerDied","Data":"9989808b24a5895221146c0e35de04e3a5077e014a7900af7241e8077e77f2cf"}
Oct 30 20:31:58.126733 ip-10-0-132-82 kubenswrapper[4892]: I1030 20:31:58.126574    4892 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-p7m7x" event={"ID":"741c37d3-758a-4ced-ab0e-9038baea2d88","Type":"ContainerDied","Data":"ad74710bb76347152ab4105a1ef41992b8d1ce7f66b43f49739ce2f674cec83e"}
Oct 30 20:31:58.126733 ip-10-0-132-82 kubenswrapper[4892]: I1030 20:31:58.126608    4892 scope.go:122] "RemoveContainer" containerID="9989808b24a5895221146c0e35de04e3a5077e014a7900af7241e8077e77f2cf"
Oct 30 20:31:58.128012 ip-10-0-132-82 kubenswrapper[4892]: I1030 20:31:58.127956    4892 generic.go:356] "Generic (PLEG): container finished" podID="72926e14-705b-493a-879f-f595c35a7490" containerID="daa545f6b5ed58fff46391eee8e6f47d0db43d87a6fee8b2c07925792dfcda7c" exitCode=0
Oct 30 20:31:58.128145 ip-10-0-132-82 kubenswrapper[4892]: I1030 20:31:58.128027    4892 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-mb59v"
Oct 30 20:31:58.128145 ip-10-0-132-82 kubenswrapper[4892]: I1030 20:31:58.128083    4892 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-mb59v" event={"ID":"72926e14-705b-493a-879f-f595c35a7490","Type":"ContainerDied","Data":"daa545f6b5ed58fff46391eee8e6f47d0db43d87a6fee8b2c07925792dfcda7c"}
Oct 30 20:31:58.128340 ip-10-0-132-82 kubenswrapper[4892]: I1030 20:31:58.128160    4892 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-mb59v" event={"ID":"72926e14-705b-493a-879f-f595c35a7490","Type":"ContainerDied","Data":"b197489a8cf534162a60956e538a6087a8c508c0ebde88c49cb86f69daab9e2a"}
Oct 30 20:31:58.136495 ip-10-0-132-82 kubenswrapper[4892]: I1030 20:31:58.129597    4892 generic.go:356] "Generic (PLEG): container finished" podID="6f05c89a-d631-4c95-aad6-d08d6c87cffd" containerID="d87139aa86bf1b1aacecc2c7327610a42fda8698ccf202823208d646da9c409d" exitCode=0
Oct 30 20:31:58.136495 ip-10-0-132-82 kubenswrapper[4892]: I1030 20:31:58.129637    4892 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-k85mx"
Oct 30 20:31:58.136495 ip-10-0-132-82 kubenswrapper[4892]: I1030 20:31:58.129653    4892 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-k85mx" event={"ID":"6f05c89a-d631-4c95-aad6-d08d6c87cffd","Type":"ContainerDied","Data":"d87139aa86bf1b1aacecc2c7327610a42fda8698ccf202823208d646da9c409d"}
Oct 30 20:31:58.136495 ip-10-0-132-82 kubenswrapper[4892]: I1030 20:31:58.129690    4892 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-k85mx" event={"ID":"6f05c89a-d631-4c95-aad6-d08d6c87cffd","Type":"ContainerDied","Data":"984a32728c751b3ca9653a8908c58a423c7a71015b2573fd91b0cedf79eba725"}
Oct 30 20:31:58.143940 ip-10-0-132-82 kubenswrapper[4892]: I1030 20:31:58.143919    4892 scope.go:122] "RemoveContainer" containerID="9989808b24a5895221146c0e35de04e3a5077e014a7900af7241e8077e77f2cf"
Oct 30 20:31:58.144907 ip-10-0-132-82 kubenswrapper[4892]: E1030 20:31:58.144886    4892 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9989808b24a5895221146c0e35de04e3a5077e014a7900af7241e8077e77f2cf\": container with ID starting with 9989808b24a5895221146c0e35de04e3a5077e014a7900af7241e8077e77f2cf not found: ID does not exist" containerID="9989808b24a5895221146c0e35de04e3a5077e014a7900af7241e8077e77f2cf"
Oct 30 20:31:58.145046 ip-10-0-132-82 kubenswrapper[4892]: I1030 20:31:58.144916    4892 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9989808b24a5895221146c0e35de04e3a5077e014a7900af7241e8077e77f2cf"} err="failed to get container status \"9989808b24a5895221146c0e35de04e3a5077e014a7900af7241e8077e77f2cf\": rpc error: code = NotFound desc = could not find container \"9989808b24a5895221146c0e35de04e3a5077e014a7900af7241e8077e77f2cf\": container with ID starting with 9989808b24a5895221146c0e35de04e3a5077e014a7900af7241e8077e77f2cf not found: ID does not exist"
Oct 30 20:31:58.145046 ip-10-0-132-82 kubenswrapper[4892]: I1030 20:31:58.144941    4892 scope.go:122] "RemoveContainer" containerID="daa545f6b5ed58fff46391eee8e6f47d0db43d87a6fee8b2c07925792dfcda7c"
Oct 30 20:31:58.153452 ip-10-0-132-82 kubenswrapper[4892]: I1030 20:31:58.153423    4892 scope.go:122] "RemoveContainer" containerID="daa545f6b5ed58fff46391eee8e6f47d0db43d87a6fee8b2c07925792dfcda7c"
Oct 30 20:31:58.153713 ip-10-0-132-82 kubenswrapper[4892]: E1030 20:31:58.153693    4892 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"daa545f6b5ed58fff46391eee8e6f47d0db43d87a6fee8b2c07925792dfcda7c\": container with ID starting with daa545f6b5ed58fff46391eee8e6f47d0db43d87a6fee8b2c07925792dfcda7c not found: ID does not exist" containerID="daa545f6b5ed58fff46391eee8e6f47d0db43d87a6fee8b2c07925792dfcda7c"
Oct 30 20:31:58.153799 ip-10-0-132-82 kubenswrapper[4892]: I1030 20:31:58.153724    4892 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"daa545f6b5ed58fff46391eee8e6f47d0db43d87a6fee8b2c07925792dfcda7c"} err="failed to get container status \"daa545f6b5ed58fff46391eee8e6f47d0db43d87a6fee8b2c07925792dfcda7c\": rpc error: code = NotFound desc = could not find container \"daa545f6b5ed58fff46391eee8e6f47d0db43d87a6fee8b2c07925792dfcda7c\": container with ID starting with daa545f6b5ed58fff46391eee8e6f47d0db43d87a6fee8b2c07925792dfcda7c not found: ID does not exist"
Oct 30 20:31:58.153799 ip-10-0-132-82 kubenswrapper[4892]: I1030 20:31:58.153748    4892 scope.go:122] "RemoveContainer" containerID="d87139aa86bf1b1aacecc2c7327610a42fda8698ccf202823208d646da9c409d"
Oct 30 20:31:58.162078 ip-10-0-132-82 kubenswrapper[4892]: I1030 20:31:58.162058    4892 scope.go:122] "RemoveContainer" containerID="d87139aa86bf1b1aacecc2c7327610a42fda8698ccf202823208d646da9c409d"
Oct 30 20:31:58.162350 ip-10-0-132-82 kubenswrapper[4892]: E1030 20:31:58.162332    4892 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d87139aa86bf1b1aacecc2c7327610a42fda8698ccf202823208d646da9c409d\": container with ID starting with d87139aa86bf1b1aacecc2c7327610a42fda8698ccf202823208d646da9c409d not found: ID does not exist" containerID="d87139aa86bf1b1aacecc2c7327610a42fda8698ccf202823208d646da9c409d"
Oct 30 20:31:58.162417 ip-10-0-132-82 kubenswrapper[4892]: I1030 20:31:58.162357    4892 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d87139aa86bf1b1aacecc2c7327610a42fda8698ccf202823208d646da9c409d"} err="failed to get container status \"d87139aa86bf1b1aacecc2c7327610a42fda8698ccf202823208d646da9c409d\": rpc error: code = NotFound desc = could not find container \"d87139aa86bf1b1aacecc2c7327610a42fda8698ccf202823208d646da9c409d\": container with ID starting with d87139aa86bf1b1aacecc2c7327610a42fda8698ccf202823208d646da9c409d not found: ID does not exist"
Oct 30 20:31:58.399648 ip-10-0-132-82 kubenswrapper[4892]: I1030 20:31:58.399620    4892 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-p7m7x"]
Oct 30 20:31:58.403345 ip-10-0-132-82 kubenswrapper[4892]: I1030 20:31:58.403319    4892 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-p7m7x"]
Oct 30 20:31:58.512463 ip-10-0-132-82 kubenswrapper[4892]: I1030 20:31:58.512435    4892 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-mb59v"]
Oct 30 20:31:58.517679 ip-10-0-132-82 kubenswrapper[4892]: I1030 20:31:58.517652    4892 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-mb59v"]
Oct 30 20:31:58.568330 ip-10-0-132-82 kubenswrapper[4892]: I1030 20:31:58.568302    4892 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="72926e14-705b-493a-879f-f595c35a7490" path="/var/lib/kubelet/pods/72926e14-705b-493a-879f-f595c35a7490/volumes"
Oct 30 20:31:58.568901 ip-10-0-132-82 kubenswrapper[4892]: I1030 20:31:58.568881    4892 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="741c37d3-758a-4ced-ab0e-9038baea2d88" path="/var/lib/kubelet/pods/741c37d3-758a-4ced-ab0e-9038baea2d88/volumes"
Oct 30 20:31:58.625319 ip-10-0-132-82 kubenswrapper[4892]: I1030 20:31:58.625294    4892 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-k85mx"]
Oct 30 20:31:58.628934 ip-10-0-132-82 kubenswrapper[4892]: I1030 20:31:58.628912    4892 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-k85mx"]
Oct 30 20:31:59.052055 ip-10-0-132-82 kubenswrapper[4892]: I1030 20:31:59.052027    4892 scope.go:122] "RemoveContainer" containerID="f984832b32b90e92b2ab500bbb48dedad5c73086e7bbcf70e3236bee1bd98d41"
Oct 30 20:31:59.063274 ip-10-0-132-82 kubenswrapper[4892]: I1030 20:31:59.063255    4892 scope.go:122] "RemoveContainer" containerID="6ab44e696c56e7d061475e78b0dee0bb396ff7e93673b6c87dddbb77aae2bd2d"
Oct 30 20:32:00.568568 ip-10-0-132-82 kubenswrapper[4892]: I1030 20:32:00.568533    4892 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="6f05c89a-d631-4c95-aad6-d08d6c87cffd" path="/var/lib/kubelet/pods/6f05c89a-d631-4c95-aad6-d08d6c87cffd/volumes"
Oct 30 20:32:16.813328 ip-10-0-132-82 kubenswrapper[4892]: I1030 20:32:16.813296    4892 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_istiod-openshift-gateway-7ddd9cfdc8-rqhzf_b1eed800-8bc3-4333-a929-dcb1477293fd/discovery/0.log"
Oct 30 20:32:17.184906 ip-10-0-132-82 kubenswrapper[4892]: I1030 20:32:17.184882    4892 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-7f6998b7b5-wthwp_6fb66959-7822-48c9-9f8b-8daf7fc7adde/kube-auth-proxy/0.log"
Oct 30 20:32:18.038592 ip-10-0-132-82 kubenswrapper[4892]: I1030 20:32:18.038558    4892 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_router-default-776df59447-8nrhh_87082646-75c3-43b5-864b-3478e9a7373d/router/0.log"
Oct 30 20:32:24.843690 ip-10-0-132-82 kubenswrapper[4892]: E1030 20:32:24.843653    4892 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-jrm87" containerName="extractor"